Smol but Mighty: Can Small Models Reason Well?

By /

Listen to this Post

2025-02-04

The rise of smaller, open-source models has sparked debates about their ability to compete with large, closed models in terms of reasoning capabilities and cultural sensitivity. Recent releases like DeepSeek-R1 and Qwen 2.5 have shown impressive performance, challenging the dominance of established names like OpenAI’s GPT models. These smaller models, despite their size, have demonstrated the potential to outperform larger counterparts in specific benchmarks. But how well do these models handle reasoning and the inherent biases within them?

In this analysis, I examined four different models—Llama 3.2, SmolLM, Qwen 2.5, and DeepSeek-R1—and evaluated their performance in reasoning tasks and bias detection. The results offer new insights into how small models can still pack a punch, but also reveal the limitations and nuances of their reasoning processes.

Key Takeaways:

  1. Smaller Models Show Big Potential: Despite being smaller in size, models like SmolLM prove that open-source and smaller models can hold their ground against larger, closed systems. SmolLM stands out not only for its performance but also for being fully open-source, with accessible code, training data, and weights. This transparency has become a major advantage for the open-source community.

  2. The “Thinking Trap” of DeepSeek-R1: DeepSeek-R1, while highly accurate, frequently hit the 512-token limit during its reasoning process, leading to incomplete answers. This issue arises from the high number of “thinking” tokens generated before an answer is provided. Interestingly, longer thinking sequences beyond 300 words did not lead to better accuracy and sometimes even resulted in worse performance.

  3. Bias Detection and Cultural Sensitivity: The models were tested on the Bias Benchmark for QA (BBQA), focusing on their ability to reason accurately while avoiding bias. The results indicated significant variation across models, with certain models (like DeepSeek) showing better performance in handling bias, especially when context was disambiguated. However, all models struggled with handling sensitive social categories such as religion, race, and sexual orientation in ambiguous contexts.

  4. Model-Specific Insights: DeepSeek generally performed the best in terms of accuracy across categories, particularly in disambiguated contexts. Qwen and SmolLM also performed well when explicit instructions or additional context were provided. However, Llama 3.2 struggled the most, particularly in ambiguous contexts, showing the highest susceptibility to biases.

  5. Chinese vs. American Bias: A closer look at how the models handled stereotypes related to Chinese and American contexts revealed interesting trends. Models like Qwen and SmolLM were more effective in handling disambiguated contexts, while Llama showed more erratic behavior. DeepSeek exhibited some positive bias in response to Chinese stereotypes, despite its higher accuracy.

What Undercode Says:

In today’s rapidly advancing world of AI models, the power of smaller and open-source models cannot be underestimated. The recent advancements in models like SmolLM, DeepSeek-R1, and Qwen 2.5 challenge the prevailing notion that only large-scale, proprietary models can produce top-tier results. These models not only showcase the potential for local, efficient computing but also point to the fact that open-source efforts can rival commercial giants.

However, the real question is how well these models truly reason and handle cultural complexities, especially biases embedded in their training. The experiment with the BBQA dataset highlights that while some models can excel in accuracy when given explicit context (disambiguated context), they often struggle in more ambiguous scenarios, where implicit biases can surface. This is particularly crucial when considering the implications of AI in sensitive areas like hiring, healthcare, and legal sectors.

The most compelling insight from this experiment is the realization that longer reasoning doesn’t always equate to better performance. In fact, DeepSeek-R1, despite its impressive capabilities, often finds itself “lost in thought,” hitting the token limit too early. This poses an important challenge for model developers: how do we balance thorough reasoning with computational efficiency? The sweet spot may lie somewhere between rapid responses and well-thought-out conclusions, without overstretching resources and time.

The fact that these smaller models are capable of handling complex tasks with reasonable efficiency—especially when the data is open and accessible—shows how powerful community-driven efforts can be. SmolLM’s fully open-source nature serves as a case study for the benefits of transparency and accessibility in the AI field. It’s not just about what the models can do, but also about how they can grow through community involvement. Smaller models like these show that you don’t need vast amounts of resources to build something impactful.

However, as the analysis of cultural biases demonstrated, the path to more accurate and ethical AI is far from straightforward. All models in the experiment struggled with certain categories like race, religion, and physical appearance, especially when context was ambiguous. The tendency of models to lean into stereotypes without clear contextual guidance exposes a critical flaw in how AI systems are trained and deployed.

Moreover, the intriguing differences between how the models processed Chinese versus American stereotypes reflect the challenges of building culturally sensitive AI systems. While all models showed improvement when provided with disambiguating context, it is evident that biases—whether related to nationality, race, or gender—are deeply ingrained in the training data. These biases can shape model outputs in subtle yet impactful ways.

This raises another crucial point for developers: AI models need to be continuously refined to account for the evolving landscape of social norms and cultural contexts. This can only be achieved through an ongoing effort to improve the data they are trained on and by incorporating more diverse perspectives in the development process.

Finally, the implications of these findings for the deployment of open-source AI systems are significant. Models like DeepSeek-R1 and SmolLM show that smaller, open models can achieve impressive results, but they also highlight the importance of transparent testing and evaluation. Without rigorous scrutiny, it’s difficult to understand where these models may fall short or perpetuate harmful biases.

In conclusion, the journey to improving small models is a balancing act. While they demonstrate the power of open-source and smaller-scale AI, their ability to reason effectively and handle cultural biases remains a work in progress. As the technology matures, we must continue to address these challenges to ensure that smaller models can live up to their full potential—delivering not only efficiency and power but also fairness and cultural sensitivity.

References:

Reported By: https://huggingface.co/blog/evijit/smollm-deepseek-bias-eval
https://stackoverflow.com
Wikipedia: https://www.wikipedia.org
Undercode AI: https://ai.undercodetesting.com

Image Source:

OpenAI: https://craiyon.com
Undercode AI DI v2: https://ai.undercode.helpFeatured Image

Explore More: