MamayLM: The New Benchmark for Ukrainian Language Models

MamayLM, the latest innovation in large language models (LLMs), has just been launched, bringing cutting-edge performance to the Ukrainian language space. This new model, developed through a collaboration between INSAIT and ETH Zurich, combines efficiency with remarkable multilingual capabilities, significantly surpassing other similarly-sized models in both Ukrainian and English. With 9 billion parameters, MamayLM not only demonstrates impressive versatility but also proves highly cost-effective, operating efficiently on a single GPU.

MamayLM, while smaller in size compared to its larger counterparts, delivers extraordinary results in various benchmarks, particularly excelling in Ukrainian-specific tasks. From government use to everyday applications, MamayLM holds great potential for practical deployment, offering a lightweight solution to AI-driven tasks that require understanding and generating Ukrainian text.

Overview of

MamayLM is based on Google’s Gemma 2 9B model, which INSAIT previously adapted for Bulgarian. Leveraging the robust multilingual foundation of Gemma 2, the team undertook substantial improvements, including advanced techniques in continual training, model merging, and the use of synthetic data. The result is a model that is exceptionally fine-tuned to handle both Ukrainian and English, boasting cultural and linguistic nuances vital for high-quality AI interaction in Ukrainian contexts.

To ensure data quality, the pretraining phase utilized a range of publicly available resources, including Ukrainian Wikipedia, FineWeb2, Malyuk, and CulturaX, all of which were filtered to ensure minimal noise and duplication. A unique approach to data packing, as well as a mix of Ukrainian and English content, allowed MamayLM to not only perform well but also avoid the common issue of catastrophic forgetting—a phenomenon where neural networks lose previously learned information when trained on new data.

In addition, MamayLM benefits from advanced instruction-tuning using datasets such as Nemotron SFT, OpenCoder, and Aya Collection, with a special emphasis on incorporating contributions from the Ukrainian open-source community. This inclusivity has helped the model become an expert in handling Ukrainian-specific tasks and applications.

What Undercode Says: A Deeper Dive into MamayLM’s Performance

MamayLM’s success lies in its unique approach to language transfer, combining state-of-the-art methods with a deep understanding of the Ukrainian language. The fine-tuning of the Gemma 2 model to the specifics of Ukrainian is what sets it apart from many other models. While the translation challenges posed by Ukrainian-specific benchmarks are significant, MamayLM addresses them by utilizing a custom translation framework that improves accuracy, particularly in the preservation of context between questions and answers. This makes MamayLM not only powerful in understanding complex Ukrainian text but also reliable when it comes to generating coherent and culturally relevant responses.

On key benchmarks like the ZNO (Ukrainian high school exams), MamayLM outperforms all similarly sized models and even surpasses much larger models, including those with 27B and 70B parameters. Its superior performance on the Winogrande and Hellaswag challenges further solidifies its place as a top contender in the AI space. The ability of MamayLM to handle multiple-choice questions, logical reasoning tasks, and general trivia, in both Ukrainian and English, is truly remarkable.

The bilingual capabilities of MamayLM also offer practical advantages for businesses and government institutions, especially in Ukraine, where efficient and culturally aware AI applications are in high demand. By operating on a single GPU, MamayLM provides a cost-effective solution for local businesses, reducing the need for expensive infrastructure and making advanced AI technologies more accessible. Moreover, it enables applications in crucial sectors such as healthcare and education, where language barriers often pose significant challenges.

Fact Checker Results

MamayLM’s impressive performance is backed by rigorous testing across multiple benchmarks, including those specific to Ukrainian contexts like ZNO.
The model’s bilingual capabilities make it versatile in both Ukrainian and English, with strong performance across a range of tasks.
Despite being relatively small (9B parameters), MamayLM outperforms models up to 10x larger, making it highly efficient for practical use in various sectors.