A Journey from Concept to Recognition Our journey with this research began with early discussions and progress shared here. The idea was simple yet profound: exploring the potential of Gabor filters, known for their exceptional feature extraction capabilities, in enhancing the performance of CNNs for object recognition tasks. This exploration led to a comprehensive study comparing the performance of Gabor-initialized CNNs against traditional CNNs with random initialization across six object recognition datasets.
Key Findings and Contributions The results were fascinating to us. The Gabor-initialized CNNs consistently outperformed traditional models in accuracy, area under the curve, minimum loss, and convergence speed. These findings provide robust evidence in favor of using Gabor-based methods for initializing the receptive fields of CNN architectures, a technique that was explored before with little success because researchers had been constraining Gabor filters during training, precluding gradient descent to optimize the filters as needed for general purpose object recognition, until now.
Our research contributes significantly to the field by demonstrating:
Improved performance in object classification tasks with Gabor-initialized CNNs.
Superior performance of random configurations of Gabor filters in the receptive layer, especially with complex datasets.
Enhanced performance of CNNs in a shorter time frame when incorporating Gabor filters.
Implications and Future Directions This study reaffirms the historical success of Gabor filters in image processing and opens new avenues for their application in modern CNN architectures. The impact of this research is vast, suggesting potential enhancements in various applications of CNNs, from medical imaging to autonomous vehicles.
As we celebrate this achievement, we also look forward to further research. Future studies could explore initializing other vision architectures, such as Vision Transformers (ViTs), with Gabor filters.
It’s a proud moment for us at the lab to see our research recognized on a global platform like ICML 2023 and published in a journal. This accomplishment is a testament to our commitment to pushing the boundaries of AI and ML research. We congratulate Mehang Rai for this remarkable achievement and thank the AI community for their continued support and recognition.
Regulatory Framework: The order emphasizes establishing a regulatory framework that ensures the safe and responsible development of AI. Companies must adhere to specific standards and best practices when developing and deploying AI technologies.
Transparency and Accountability: The industry is encouraged to adopt transparent methodologies and ensure that AI systems are explainable. This will likely lead to a surge in demand for tools and solutions that offer transparency in AI operations.
Collaboration with Federal Agencies: The order promotes cooperation between the private sector and federal agencies. This collaboration fosters innovation while ensuring AI technologies align with national interests and security.
Risk Management: Companies are urged to adopt risk management practices that identify and mitigate potential threats AI systems pose. This includes addressing biases, ensuring data privacy, and safeguarding against malicious uses of AI.
At the CRAIG/CSEAI, we’re committed to assisting industry and government partners in navigating this intricate AI regulatory terrain through our research, assessments, and training. Contact us to know more.
2. What does the EO mean for academics doing AI research?
Research Funding: The order highlights the importance of federal funding for AI research. Academics can expect increased support and resources for projects that align with the order’s objectives, especially those focusing on safety, security, and trustworthiness.
Ethical Considerations: Given the emphasis on trustworthy AI, researchers will be encouraged to incorporate ethical considerations into their work. This aligns with the growing movement towards AI ethics in the academic community.
Collaboration Opportunities: The directive promotes collaboration between academia and federal agencies. This could lead to new research opportunities, partnerships, and access to resources that were previously unavailable.
Publication and Transparency: The order underscores the importance of transparency in AI research. Academics will be encouraged to publish their findings, methodologies, and datasets to promote openness and reproducibility in the field.
K. Sooksatra, G. Bejarano, and P. Rivas, “Evaluating Robustness of Reconstruction Models with Adversarial Networks,” Procedia Computer Science, vol. 222, pp. 353-366, 2023. https://doi.org/10.1016/j.procs.2023.08.174.
In the ever-evolving landscape of artificial intelligence, our lab has made a significant breakthrough with our latest publication featured in Procedia Computer Science. This research, spearheaded by Korn Sooksatra, delves into the critical domain of adversarial robustness, mainly focusing on reconstruction models, which, until now, have been a less explored facet of adversarial research. This paper was accepted initially into IJCNN and chosen to be added to the INNS workshop and published as a journal article.
Innovative Frameworks: The team introduced two novel frameworks for assessing adversarial robustness: the standard framework, which perturbs input images to deceive reconstruction models, and the universal-attack framework, which generates adversarial perturbations from a dataset’s distribution.
Outperforming Benchmarks: Through rigorous testing on MNIST and Cropped Yale Face datasets, these frameworks demonstrated superior performance in altering image reconstruction and classification, surpassing existing state-of-the-art adversarial attacks.
Enhancing Model Resilience: A pivotal aspect of the study was using these frameworks to retrain reconstruction models, significantly improving their defense against adversarial perturbations and showcasing an ethical application of adversarial networks.
Latent Space Analysis: The research also included a thorough examination of the latent space, ensuring that adversarial attacks do not compromise the underlying features that are crucial for reconstruction integrity.
The implications of this research are profound for the AI community. It not only presents a method to evaluate and enhance the robustness of reconstruction models but also opens avenues for applying these frameworks to other image-to-image applications. The lab’s work is a call to the AI research community to prioritize the development of robust AI systems that can withstand adversarial threats, ensuring the security and reliability of AI applications across various domains.
While the frameworks developed are groundbreaking, the team acknowledges the need for reduced preprocessing time to enhance practicality. Future work aims to refine these frameworks and extend their application to other domains, such as video keypoint interpretation, anomaly detection, and graph prediction.
In today’s digital age, trust has become a precious commodity. It’s the invisible currency that fuels our interactions with technology and brands. Building trust, especially in technology, is a costly and time-consuming process. However, the payoff is immense. When users trust a system or a brand, they are more likely to engage with it, advocate for it, and remain loyal even when faced with alternatives.
One of the most effective ways to build trust in technology is to ensure it aligns with societal goals and values. When a system or technology operates in a way that benefits society and adheres to its values, it is more likely to be trusted and accepted.
However, artificial intelligence (AI) has faced significant challenges. Despite its immense potential and numerous benefits, trust in AI has suffered. This is due to various factors, including concerns about privacy, transparency, potential biases, and the lack of a clear ethical framework guiding its use.
This is where the concept of AI Orthopraxy comes in. AI Orthopraxy is all about the correct practice of AI. It’s about ensuring that AI is developed and used in a way that is ethical, responsible, and aligned with societal values. It’s about walking the talk of trustworthy AI.
In this talk, I will discuss the concept of AI Orthopraxy, the recent developments in AI, the associated risks, and the tools and strategies we can use to ensure the responsible use of AI. The goal is not just to highlight the challenges but also to provide a roadmap for moving forward in a way that is beneficial for all stakeholders.
Large Language Models (LLMs) and Large Multimodal Models: The Ethical Implications
The journey of Large Language Models (LLMs) has been remarkable. From the early successes of models like GPT and BERT, we have seen a rapid evolution in the capabilities of these models. The most recent iterations, such as ChatGPT, have demonstrated an impressive ability to generate human-like text, opening up many applications in areas like customer service, content creation, and more.
Parallel to this, the field of vision models has also seen significant advancements. Introducing models like Vision Transformer (ViT) has revolutionized how we process and understand visual data, leading to breakthroughs in medical imaging, autonomous driving, and more.
However, as with any powerful technology, these models come with their own challenges. One of the most concerning is their fragility, especially when faced with adversarial attacks. These attacks, which involve subtly modifying input data to mislead the model, have exposed the vulnerabilities of these models and raised questions about their reliability.
As someone deeply involved in this space, I see both the immense potential of these models and the serious risks they pose. But I firmly believe these risks can be mitigated with careful engineering and regulation.
Careful engineering involves developing robust models resistant to adversarial attacks and biases. It involves ensuring transparency in how these models work and making them interpretable so that their decisions can be understood and scrutinized.
On the other hand, regulation involves setting up rules and standards that guide the development and use of these models. It involves ensuring that these models are used responsibly and ethically and that there are mechanisms in place to hold those who misuse them accountable.
AI Ethics Standards: The Need for a Common Framework
Standards play a crucial role in ensuring technology’s responsible and ethical use. In the context of AI, they can help make systems fair, accountable, and transparent. They provide a common framework that guides the development and use of AI, ensuring that it aligns with societal values and goals.
One of the key initiatives in this space is the P70XX series of standards developed by the IEEE. These standards address various ethical considerations in system and software engineering and provide guidelines for embedding ethics into the design process.
Similarly, the International Organization for Standardization (ISO) has been working on standards related to AI. These standards cover various aspects of AI, including its terminology, trustworthiness, and use in specific sectors like healthcare and transportation.
The National Institute of Standards and Technology (NIST) has led efforts to develop a framework for AI standards in the United States. This framework aims to support the development and use of trustworthy AI systems and to promote innovation and public confidence in these systems.
The potential of these standards goes beyond just guiding the development and use of AI. There is a growing discussion about the possibility of these standards becoming recommended legal practice. This would mean that adherence to these standards would not just be a matter of ethical responsibility but also a legal requirement.
This possibility underscores the importance of these standards and their role in ensuring the responsible and ethical use of AI. However, standards alone are not enough. They need to be complemented by best practices in AI.
AI Best Practices: From Theory to Practice
As we navigate the complex landscape of AI ethics, best practices serve as our compass. They provide practical guidance on how to implement the principles of ethical AI in real-world systems.
One such best practice is the use of model cards for AI models. Model cards are like nutrition labels for AI models. They provide essential information about a model, including its purpose, performance, and potential biases. By providing this information, model cards help users understand what a model does, how well it does, and any limitations it might have.
Similarly, data sheets for datasets provide essential information about the datasets used to train AI models. They include details about the data collection process, the characteristics of the data, and any potential biases in the data. This helps users understand the strengths and weaknesses of the dataset and the models trained on it.
A newer practice is the use of Data Statements for Natural Language Processing, proposed to mitigate system bias and enable better science in NLP technologies. Data Statements are intended to address scientific and ethical issues arising from using data from specific populations in developing technology for other populations. They are designed to help alleviate exclusion and bias in language technology, lead to better precision in claims about how NLP research can generalize, and ultimately lead to language technology that respects its users’ preferred linguistic style and does not misrepresent them to others.
However, these best practices are only effective if a trained workforce understands them and can implement them in their work. This underscores the importance of education and training in AI ethics. It’s not enough to develop ethical AI systems; we must cultivate a workforce that can uphold these ethical standards in their work. Initiatives like the CSEAI promote responsible AI and develop a workforce equipped to navigate AI’s ethical challenges.
The Role of the CSEAI in Promoting Responsible AI
The Center for Standards and Ethics in AI (CSEAI) is pivotal in promoting responsible AI. Our mission at CSEAI is to provide applicable, actionable standard practices in trustworthy AI. We believe the path to responsible AI lies in the intersection of robust technical standards and ethical solid guidelines.
One of the critical areas of our work is developing these standards. We work closely with researchers, practitioners, and policymakers to develop standards that are technically sound and ethically grounded. These standards provide a common framework that guides the development and use of AI, ensuring that it aligns with societal values and goals.
In addition to developing standards, we also focus on state-of-the-art collaborative AI research and workforce development. We believe that responsible AI requires a workforce that is not just technically competent but also ethically aware. To this end, we offer training programs and resources that help individuals understand the ethical implications of AI, upcoming regulations, and the importance of bare minimum practices like Model Cards, Datasheets for Datasets, and Data Statements.
As the field of AI continues to evolve, so does the landscape of regulation, standardization, and best practices. At CSEAI, we are committed to staying ahead of these changes. We continuously update our value propositions and training programs to reflect the latest developments in the field and to ensure our standards and practices align with emerging regulations.
As the CSEAI initiative moves forward, we aim to ensure that AI is developed and used in a way that is beneficial for all stakeholders. We believe that with the right standards and practices, we can harness the power of AI in a way that is responsible, ethical, and aligned with societal values in a manner that is profitable for our industry partners and safe, robust, and trustworthy for all users.
Conclusion: The Future of Trustworthy AI
As we look toward the future of AI, we find ourselves amidst a cacophony of voices. As my colleagues put it, on one hand, we have the “AI Safety” group, which often stokes fear by highlighting existential risks from AI, potentially distracting from immediate concerns while simultaneously pushing for rapid AI development. On the other hand, we have the “AI Ethics” group, which tends to focus on the faults and dangers of AI at every turn, creating a brand of criticism hype and advocating for extreme caution in AI use.
However, most of us in the AI community operate in the quiet middle ground. We recognize the immense benefits that AI can bring to sectors like healthcare, education, and vision, among others. At the same time, we are acutely aware of the severe risks and harms that AI can pose. But we firmly believe that, like with electricity, cars, planes, and other transformative technologies, these risks can be minimized with careful engineering and regulation.
Consider the analogy of seatbelts in cars. Initially, many people resisted their use. We felt safe enough, with our mothers instinctively extending an arm in front of us during sudden stops. But when a serious accident occurred, the importance of seatbelts became painfully clear. AI regulation can be seen in a similar light. There may be resistance initially, but with proper safeguards in place, we can ensure that when something goes wrong—and it inevitably will—we will all be better prepared to handle it. More importantly, these safeguards will be able to protect those who are most vulnerable and unable to protect themselves.
As we continue to navigate the complex landscape of AI, let’s remember to stay grounded, to focus on the tangible and immediate impacts of our work, and to always strive for the responsible and ethical use of AI. Thank you.
This is a ChatGPT-generated summary of a noisy transcript of a keynote presented at Marist College on Tuesday, June 13, 2023, at 9 am as part of the Enterprise Computing Conference in Poughkeepsie, New York.
Today, we are on the brink of a new era where Artificial Intelligence (AI) promises to transform every aspect of our lives. However, with great power comes great responsibility. As AI permeates our societies and economies, ensuring its responsible use becomes more critical.
At the heart of the White House’s strategy is the principle that companies have a fundamental responsibility to ensure their products are safe before deployment. This safety-first approach means preventing harm and actively promoting the public good.
Meeting with CEOs of leading AI innovators, the Administration has underscored the importance of ethical and trustworthy AI systems, emphasizing safeguards to mitigate risks and potential harms.
Investing in Responsible AI
The Biden-Harris Administration is backing up its words with actions, committing $140 million to launch seven new National AI Research Institutes. These will bring the total number of institutes to 25 nationwide, and each is committed to ethical, responsible AI that serves the public good.
The Role of the NSF IUCRC, Center for Standards and Ethics in AI (CSEAI)
This is where our initiative, the NSF IUCRC, Center for Standards and Ethics in AI (CSEAI), comes into play. We align perfectly with the Administration’s vision for responsible AI, focusing on establishing standards and ethics that serve as the bedrock for AI development and application.
By joining and funding the CSEAI, industry members will directly collaborate with other industry members and with academia, contributing to responsible AI research and advancement. This aligns with the White House’s call for ethical and safe AI and benefits companies in the long run, ensuring their products meet the highest ethical and safety standards.
What this Means for AI Developers and Users
The White House’s recent announcements signal a shift towards more stringent AI development and use standards. This means industries must prioritize building and deploying AI systems that are ethical, trustworthy, and serve the public good.
In a world where AI is becoming ubiquitous, failing to meet these standards can lead to reputational damage, regulatory penalties, and even legal liability. Conversely, those who embrace these principles stand to gain significant competitive advantage, building trust with users and staying ahead of the regulatory curve.
The White House’s commitment to responsible AI is not just good news for Americans—it’s a call to action for industry members who develop or use AI. By aligning with the principles of responsible AI and supporting initiatives like the NSF IUCRC, Center for Standards and Ethics in AI (CSEAI), industry players can meet their ethical obligations and secure their future place in AI.
Two undergraduate students from the Computer Science Department at Baylor University, Misty and Andrew, have demonstrated the impactful role of undergraduate research in advancing technological frontiers. Their recent work on automatic information retrieval, conducted under the Baylor.AI lab, has been a testament to their dedication and intellectual curiosity.
Misty and Andrew’s journey in the AI lab involved engaging in research discussions that transcended typical undergraduate experiences. Their focus was on developing software to automate data collection, a crucial component in achieving the NSF grant objectives related to studying the illegal trafficking of stolen car parts on C2C marketplaces. This work is a critical piece in the larger puzzle of combating online criminal activities.
Their success in this endeavor was supported by the Research Experiences for Undergraduates (REU) program, highlighting the program’s role in fostering early research experiences. The REU program’s funding not only enabled these students to delve into real-world problems but also allowed them to contribute meaningfully to a project with significant societal implications.
This follow-up story is a continuation of our ongoing efforts, detailed in our previous post, to tackle online criminal activity under NSF’s REU program. The achievements of Misty and Andrew help in the grand scheme of AI research as we strive to have a safe and secure cyberspace; it’s essential to acknowledge and celebrate these steps in their academic journey. Their work exemplifies the potential of undergraduate research in contributing to complex and socially relevant projects.
As we continue to support and mentor our students in the Baylor.AI lab, we look forward to more such stories of perseverance, learning, and meaningful contributions to the field of AI and machine learning.
In the dynamic marketing world, the advent of AI-powered chatbot platforms like ChatGPT is a game-changer. This technology, leveraging natural language processing and machine learning, transforms how we interact with AI, offering significant advantages over previous tools. The open-access paper delves into the potential of ChatGPT to revolutionize marketing, provided ethical considerations are taken into account.
The paper argues that ChatGPT can accelerate content creation, enhance market research efficiency, and improve customer service automation. However, it also brings to light the ethical implications and potential risks for marketers, consumers, and other stakeholders. The ethical use of ChatGPT in marketing hinges on transparency, bias mitigation, privacy protection, and continuous monitoring to avoid potential harm while capitalizing on its benefits.
For marketing professionals, this paper is a crucial read to understand how to responsibly integrate ChatGPT into their strategies and ensure that its deployment enhances customer value without compromising ethical standards. Read the paper for free here.
I was honored to recently receive the ACM Senior Member designation from the Association for Computing Machinery (ACM). For my students who asked and anyone else interested, I would like to share with you what this honor is and why I received it.
First, let me tell you a little bit about the ACM. The ACM is the world’s largest educational and scientific computing society, with a mission to advance computing as a science and profession. The ACM Senior Member designation is a distinction awarded to members who have demonstrated significant accomplishments and impact in the computing field. To be considered for this honor, a candidate must have at least 10 years of professional experience in computing and have made significant contributions to the field through research, industry, or education. Being elevated to senior member status in ACM signifies that you are an established leader in the computing field, recognized by your peers for your expertise and contributions. It also comes with certain benefits, such as access to special resources and opportunities for professional development and networking.
Overall, being a senior member of ACM is a great honor and a recognition of your significant contributions to the computing field.
I was thrilled to learn that by recommendation of my mentor in the computer science department, Dr. Hamerly, the dean of the school of engineering and computer science, Dr. Baker, and of my peers, I am now an ACM Senior Member, and I believe that my contributions to the computing field over the past decade played a significant role in this recognition. Some of my most notable achievements include the following:
Technical leadership: leading industry-university collaborative projects, securing funding for students’ research, directing numerous theses and independent studies, developing graduate courses on data mining and machine learning, updating and developing courses, and participating in the education committees at Marist College and Baylor University.
Technical contributions: over 90 publications, research in machine learning and numerical optimization, contribution to SVM theory, recent research in efficient representation learning, adversarial learning, and ethical implications of biased and unfair models, and active involvement in developing AI ethics standards through work with IEEE Standards Association.
Professional contributions: participation in professional events, including serving as Sponsorship & Budget chair of ACM NYC of Women in Computing and as a Program Committee Chair for NAACL 2022 LXNLP workshop, active membership in professional organizations, and full-time industry experience designing end-to-end systems to support manufacturing and supply management.
Recognition: elevation to IEEE Senior Member, sought-after expertise and leadership in deep learning and ethics, involvement in developing AI ethics standards, and commitment to promoting diversity and inclusion in computing through work with ACM NYC of Women in Computing and participation in the AAAI Undergraduate Consortium.
My peers in the computing community have recognized these accomplishments and contributed to advancing the field. In addition to my technical contributions, I have been actively mentoring and teaching the next generation of computing professionals.
I am incredibly grateful to the ACM for this honor, and I hope it inspires some of my students to pursue academic excellence. I believe that we can all make a massive difference in the world through our work in computing, and I look forward to continuing to make meaningful contributions to the exciting and rapidly evolving field of machine learning and responsible AI.
Thank you for taking the time to read about my journey to becoming an ACM Senior Member. If you have any questions or would like to learn more about my work, please don’t hesitate to contact me.
In the rapidly evolving world of artificial intelligence (AI), the IEEE Transactions on Technology and Society recently published a special issue that delves into the heart of AI’s most pressing challenges and opportunities. This editorial piece has garnered widespread attention. Read the full editorial here.
J. R. Schoenherr, R. Abbas, K. Michael, P. Rivas and T. D. Anderson, “Designing AI Using a Human-Centered Approach: Explainability and Accuracy Toward Trustworthiness,” in IEEE Transactions on Technology and Society, vol. 4, no. 1, pp. 9-23, March 2023, doi: 10.1109/TTS.2023.3257627.
The Essence of the Special Issue
This special issue comprises eight thought-provoking papers that collectively address the multifaceted nature of AI. The journey begins with a reconceptualization of AI, leading to discussions on the pivotal role of explainability and accuracy in AI systems. The papers emphasize that designing AI with a human-centered approach while recognizing the importance of ethics is not a zero-sum game.
Reconceptualizing AI: Clarke, a Fellow of the Australian Computer Society, revisits the original conception of AI and proposes a fresh perspective, emphasizing the synergy between human and artifact capabilities.
The Challenge of Explainability: Adamson, a Past President of the IEEE’s Society on the Social Implications, delves into the complexities of AI systems, highlighting the concealed nature of many AI algorithms and the need for post-hoc reasoning.
Trustworthy AI: Petkovic underscores that trustworthy AI requires both accuracy and explainability. He emphasizes the importance of explainable AI (XAI) in ensuring user trust, especially in high-stakes applications.
Bias in AI: A team of researchers, including Nagpal, Singh, Singh, Vatsa, and Ratha, evaluates the behavior of face recognition models, shedding light on potential biases related to age and ethnicity.
AI in Healthcare: Dhar, Siuly, Borra, and Sherratt discuss the challenges and opportunities of deep learning in the healthcare domain, emphasizing the ethical considerations surrounding medical data.
AI in Education: Tham and Verhulsdonck introduce the “stack” analogy for designing ubiquitous learning, emphasizing the importance of a human-centered approach in smart city contexts.
Ethics in Computer Science Education: Peterson, Ferreira, and Vardi discuss the role of ethics in computer science education, emphasizing the need for emotional engagement to understand the potential impacts of technology.
A Call to Action
As guest editors deeply engaged in human-centric approaches to AI, we challenge all stakeholders in the AI design process to consider the multidimensionality of AI. It’s crucial to move beyond the trade-offs mindset and prioritize accuracy and explainability. If a decision made by an AI system cannot be explained, especially in critical sectors like finance and healthcare, should it even be proposed?
Do you need help determining which machine learning model is superior? This post presents a step-by-step guide using basic statistical techniques and a real case study! 🤖📈 #AIOrthoPraxy #MachineLearning #Statistics #DataScience
When employing Machine Learning to address problems, our choice of a model plays a crucial role. Evaluating models can be straightforward when performance disparities are substantial, for example, when comparing two large-language models (LLMS) on a masked language modeling (MLM) task with 71.01 and 28.56 perplexity, respectively. However, if differences among models are minute, making a solid analysis to discern if one model is genuinely superior to others can prove challenging.
This tutorial aims to present a step-by-step guide to determine if one model is superior to another. Our approach relies on basic statistical techniques and real datasets. Our study compares four models on six datasets using one metric, standard accuracy. Alternatively, other contexts may use different numbers of models, metrics, or datasets. We will work with the tables below that show the properties of the datasets and the performance of two baseline models and two of our proposed models, for which we hope to show that they are better, which would be our hypothesis to be tested.
One of the primary purposes of statistics is hypothesis testing. Statistical inference involves taking a sample from a population and determining how well the sample represents the population. In hypothesis testing, we formulate a null hypothesis, , and an alternative hypothesis, , based on the problem (comparing models). Both hypotheses must be concise, mutually exclusive, and exhaustive. For example, we could say that our null hypothesis is that the models perform equally, and the alternative could mean that the models perform differently.
Why is the ANOVA test not a good alternative?
The ANOVA (Analysis of Variance) test is a parametric test that compares the means of multiple groups. In our case, we have four models to compare with six datasets. The null hypothesis for ANOVA is that all the means are equal, and the alternative hypothesis is that at least one of the means is different. If the value of the ANOVA test is less than the significance level (usually 0.05), we reject the null hypothesis and conclude that at least one of the means is different, i.e., at least one model performs differently than the others. However, ANOVA may not always be the best choice for comparing the performance of different models.
One reason for this is that ANOVA assumes that the data follows a normal distribution, which may not always be the case for real-world data. Additionally, ANOVA does not take into account the difficulty of classifying certain data points. For example, in a dataset with a single numerical feature and binary labels, all models may achieve 100% accuracy on the training data. However, if the test set contains some mislabeled points, the models may perform differently. In this scenario, ANOVA would not be appropriate because it does not account for the difficulty of classifying certain data points.
Another issue with ANOVA is that it assumes that the variances of the groups being compared are equal. This assumption may not hold for datasets with different levels of noise or variability. In such cases, alternative statistical tests like the Friedman test or the Nemenyi test may be more appropriate.
The Friedman test is a non-parametric test that compares multiple models. In our example, we want to compare the performance of different models, i.e., two baseline models, Gabor randomized, and Gabor repeated, on datasets. First, the test calculates the average rank of each model’s performance on each dataset, with the best-performing model receiving a rank of 1. The Friedman test then tests the null hypothesis, , that all models are equally effective and their average ranks should be equal. The test statistic is calculated as follows:
where is the average ranking of each model.
The test result can be used to determine whether there is a statistically significant difference between the performance of the models by making sure that is not less than the critical value for the distribution for a particular confidence value . However, since could be too conservative, we also calculate the statistic as follows:
Based on the critical value, , and , we evaluate ;
once the null hypothesis is rejected, we apply a posthoc test. For this, we use the Nemenyi test to establish whether models differ significantly in their performance.
We will start the process of getting this test done by ranking the data. First, we can load the data and verify it with respect to the table shown earlier.
import pandas as pd
import numpy as np
data = [[0.8937, 0.8839, 0.9072, 0.9102],
[0.8023, 0.8024, 0.8229, 0.8238],
[0.7130, 0.7132, 0.7198, 0.7206],
[0.5084, 0.5085, 0.5232, 0.5273],
[0.2331, 0.2326, 0.3620, 0.3952],
[0.5174, 0.5175, 0.5307, 0.5178]]
model_names = ['Glorot N.', 'Glorot U.', 'Random G.', 'Repeated G.']
df = pd.DataFrame(data, columns=model_names)
print(df.describe()) #<- use averages to verify if matches table
Glorot N. Glorot U. Random G. Repeated G.
count 6.000000 6.000000 6.000000 6.000000
mean 0.611317 0.609683 0.644300 0.649150
std 0.240422 0.238318 0.206871 0.200173
min 0.233100 0.232600 0.362000 0.395200
25% 0.510650 0.510750 0.525075 0.520175
50% 0.615200 0.615350 0.625250 0.623950
75% 0.779975 0.780100 0.797125 0.798000
max 0.893700 0.883900 0.907200 0.910200
Next, we rank the models and get their averages like so:
data = df.rank(1, method='average', ascending=False)
Glorot N. Glorot U. Random G. Repeated G.
0 3.0 4.0 2.0 1.0
1 4.0 3.0 2.0 1.0
2 4.0 3.0 2.0 1.0
3 4.0 3.0 2.0 1.0
4 3.0 4.0 2.0 1.0
5 4.0 3.0 1.0 2.0
Glorot N. Glorot U. Random G. Repeated G.
count 6.000000 6.000000 6.000000 6.000000
mean 3.666667 3.333333 1.833333 1.166667
std 0.516398 0.516398 0.408248 0.408248
min 3.000000 3.000000 1.000000 1.000000
25% 3.250000 3.000000 2.000000 1.000000
50% 4.000000 3.000000 2.000000 1.000000
75% 4.000000 3.750000 2.000000 1.000000
max 4.000000 4.000000 2.000000 2.000000
With this information, we can expand our initial results table to show the rankings by dataset and the average rankings across all datasets for each model.
Now that we have the rankings, we can proceed with the statistical analysis and do the following:
The critical value at is 5.417. Thus, because the critical value is below our statistics obtained, we reject with 99% confidence.
The critical value can be obtained from any table that has the F distribution. In the table the degrees of freedom across columns (denoted as ) is , that is the number of models minus one; the degrees of freedom across rows (denoted as ) is , that is, the number of models minus one, times the number of datasets minus one. In our case this is and .
The Nemenyi test is a post-hoc test that compares multiple models after a significant result from Friedman’s test. The null hypothesis for Nemenyi is that there is no difference between any two models, and the alternative hypothesis is that at least one pair of models is different.
The formula for Nemenyi is as follows:
where is the critical difference of the Studentized range distribution at the chosen significance level and is the number of groups. The value can be obtained from the following table:
Thus, for our particular case study, the critical differences are:
Since the difference in rank between the randomized Gabor and baseline Glorot normal is 1.83 and is less than the , we conclude Gabor is better. Similarly, since the difference in rank between the fixed Gabor and baseline Glorot uniform is 2.17 and is less than the , we conclude that Gabor is better. Yes, there is sufficient statistical evidence to show that our model is better with high confidence.
Things we would like to see in papers
First of all, it would be nice to have a complete table that includes the results of the statistical tests as part of the caption or as a footnote, like this:
Second of all, graphics always help! A simple and visually appealing diagram is a powerful way to represent post hoc test results when comparing multiple classifiers. The figure below, which illustrates the data analysis from the table above, displays the average ranks of methods along the top line of the diagram. To facilitate interpretation, the axis is oriented so that the best ranks appear on the right side, which enables us to perceive the methods on the right as superior.
When comparing all the algorithms against each other, the groups of algorithms that are not significantly different are connected with a bold solid line. Such an approach clearly highlights the most effective models while also providing a robust analysis of the differences between models. Additionally, the critical difference is shown above the graph, further enhancing the visualization of the analysis results. Overall, this simple yet powerful diagrammatic approach provides a clear and concise representation of the performance of multiple classifiers, enabling more informed decision-making in selecting the best-performing model.
The statistical tests are based on this paper:
Demšar, Janez. “Statistical comparisons of classifiers over multiple data sets.” The Journal of Machine learning research 7 (2006): 1-30.
The case study is based on the following research:
Rai, Mehang. “On the Performance of Convolutional Neural Networks Initialized with Gabor Filters.” Thesis, Baylor University, 2021.