Alternatives to Phi-2

The TinyLlama initiative seeks to pretrain a Llama model with 1.1 billion parameters using a dataset of 3 trillion tokens. With the right optimizations, this ambitious task can be completed in a mere 90 days, utilizing 16 A100-40G GPUs. We have maintained the same architecture and tokenizer as Llama 2, ensuring that TinyLlama is compatible with various open-source projects that are based on Llama. Additionally, the model's compact design, consisting of just 1.1 billion parameters, makes it suitable for numerous applications that require limited computational resources and memory. This versatility enables developers to integrate TinyLlama seamlessly into their existing frameworks and workflows.

Introducing a remarkable family of compact language models (SLMs) that deliver exceptional performance while being cost-effective and low in latency. These models are designed to enhance AI functionalities, decrease resource consumption, and promote budget-friendly generative AI applications across various platforms. They improve response times in real-time interactions, navigate autonomous systems, and support applications that demand low latency, all critical to user experience. Phi-3 can be deployed in cloud environments, edge computing, or directly on devices, offering unparalleled flexibility for deployment and operations. Developed in alignment with Microsoft AI principles—such as accountability, transparency, fairness, reliability, safety, privacy, security, and inclusiveness—these models ensure ethical AI usage. They also excel in offline environments where data privacy is essential or where internet connectivity is sparse. With an expanded context window, Phi-3 generates outputs that are more coherent, accurate, and contextually relevant, making it an ideal choice for various applications. Ultimately, deploying at the edge not only enhances speed but also ensures that users receive timely and effective responses.

Pixtral Large is an expansive multimodal model featuring 124 billion parameters, crafted by Mistral AI and enhancing their previous Mistral Large 2 framework. This model combines a 123-billion-parameter multimodal decoder with a 1-billion-parameter vision encoder, allowing it to excel in the interpretation of various content types, including documents, charts, and natural images, all while retaining superior text comprehension abilities. With the capability to manage a context window of 128,000 tokens, Pixtral Large can efficiently analyze at least 30 high-resolution images at once. It has achieved remarkable results on benchmarks like MathVista, DocVQA, and VQAv2, outpacing competitors such as GPT-4o and Gemini-1.5 Pro. Available for research and educational purposes under the Mistral Research License, it also has a Mistral Commercial License for business applications. This versatility makes Pixtral Large a valuable tool for both academic research and commercial innovations.

Mistral 7B is a language model with 7.3 billion parameters that demonstrates superior performance compared to larger models such as Llama 2 13B on a variety of benchmarks. It utilizes innovative techniques like Grouped-Query Attention (GQA) for improved inference speed and Sliding Window Attention (SWA) to manage lengthy sequences efficiently. Released under the Apache 2.0 license, Mistral 7B is readily available for deployment on different platforms, including both local setups and prominent cloud services. Furthermore, a specialized variant known as Mistral 7B Instruct has shown remarkable capabilities in following instructions, outperforming competitors like Llama 2 13B Chat in specific tasks. This versatility makes Mistral 7B an attractive option for developers and researchers alike.

Introducing the next iteration of our open-source large language model, this version features model weights along with initial code for the pretrained and fine-tuned Llama language models, which span from 7 billion to 70 billion parameters. The Llama 2 pretrained models have been developed using an impressive 2 trillion tokens and offer double the context length compared to their predecessor, Llama 1. Furthermore, the fine-tuned models have been enhanced through the analysis of over 1 million human annotations. Llama 2 demonstrates superior performance against various other open-source language models across multiple external benchmarks, excelling in areas such as reasoning, coding capabilities, proficiency, and knowledge assessments. For its training, Llama 2 utilized publicly accessible online data sources, while the fine-tuned variant, Llama-2-chat, incorporates publicly available instruction datasets along with the aforementioned extensive human annotations. Our initiative enjoys strong support from a diverse array of global stakeholders who are enthusiastic about our open approach to AI, including companies that have provided valuable early feedback and are eager to collaborate using Llama 2. The excitement surrounding Llama 2 signifies a pivotal shift in how AI can be developed and utilized collectively.

Stable LM represents a significant advancement in the field of language models by leveraging our previous experience with open-source initiatives, particularly in collaboration with EleutherAI, a nonprofit research organization. This journey includes the development of notable models such as GPT-J, GPT-NeoX, and the Pythia suite, all of which were trained on The Pile open-source dataset, while many contemporary open-source models like Cerebras-GPT and Dolly-2 have drawn inspiration from this foundational work. Unlike its predecessors, Stable LM is trained on an innovative dataset that is three times the size of The Pile, encompassing a staggering 1.5 trillion tokens. We plan to share more information about this dataset in the near future. The extensive nature of this dataset enables Stable LM to excel remarkably in both conversational and coding scenarios, despite its relatively modest size of 3 to 7 billion parameters when compared to larger models like GPT-3, which boasts 175 billion parameters. Designed for versatility, Stable LM 3B is a streamlined model that can efficiently function on portable devices such as laptops and handheld gadgets, making us enthusiastic about its practical applications and mobility. Overall, the development of Stable LM marks a pivotal step towards creating more efficient and accessible language models for a wider audience.

Kimi K2 represents a cutting-edge series of open-source large language models utilizing a mixture-of-experts (MoE) architecture, with a staggering 1 trillion parameters in total and 32 billion activated parameters tailored for optimized task execution. Utilizing the Muon optimizer, it has been trained on a substantial dataset of over 15.5 trillion tokens, with its performance enhanced by MuonClip’s attention-logit clamping mechanism, resulting in remarkable capabilities in areas such as advanced knowledge comprehension, logical reasoning, mathematics, programming, and various agentic operations. Moonshot AI offers two distinct versions: Kimi-K2-Base, designed for research-level fine-tuning, and Kimi-K2-Instruct, which is pre-trained for immediate applications in chat and tool interactions, facilitating both customized development and seamless integration of agentic features. Comparative benchmarks indicate that Kimi K2 surpasses other leading open-source models and competes effectively with top proprietary systems, particularly excelling in coding and intricate task analysis. Furthermore, it boasts a generous context length of 128 K tokens, compatibility with tool-calling APIs, and support for industry-standard inference engines, making it a versatile option for various applications. The innovative design and features of Kimi K2 position it as a significant advancement in the field of artificial intelligence language processing.

Phi-4 is an advanced small language model (SLM) comprising 14 billion parameters, showcasing exceptional capabilities in intricate reasoning tasks, particularly in mathematics, alongside typical language processing functions. As the newest addition to the Phi family of small language models, Phi-4 illustrates the potential advancements we can achieve while exploring the limits of SLM technology. It is currently accessible on Azure AI Foundry under a Microsoft Research License Agreement (MSRLA) and is set to be released on Hugging Face in the near future. Due to significant improvements in processes such as the employment of high-quality synthetic datasets and the careful curation of organic data, Phi-4 surpasses both comparable and larger models in mathematical reasoning tasks. This model not only emphasizes the ongoing evolution of language models but also highlights the delicate balance between model size and output quality. As we continue to innovate, Phi-4 stands as a testament to our commitment to pushing the boundaries of what's achievable within the realm of small language models.

Upstage has unveiled Solar Pro 2, a cutting-edge large language model designed for frontier-scale applications, capable of managing intricate tasks and workflows in various sectors including finance, healthcare, and law. This model is built on a streamlined architecture with 31 billion parameters, ensuring exceptional multilingual capabilities, particularly in Korean, where it surpasses even larger models on key benchmarks such as Ko-MMLU, Hae-Rae, and Ko-IFEval, while maintaining strong performance in English and Japanese as well. In addition to its advanced language comprehension and generation abilities, Solar Pro 2 incorporates a sophisticated Reasoning Mode that significantly enhances the accuracy of multi-step tasks across a wide array of challenges, from general reasoning assessments (MMLU, MMLU-Pro, HumanEval) to intricate mathematics problems (Math500, AIME) and software engineering tasks (SWE-Bench Agentless), achieving problem-solving efficiency that rivals or even surpasses that of models with double the parameters. Furthermore, its enhanced tool-use capabilities allow the model to effectively engage with external APIs and data, broadening its applicability in real-world scenarios. This innovative design not only demonstrates exceptional versatility but also positions Solar Pro 2 as a formidable player in the evolving landscape of AI technologies.

Qwen2 represents a collection of extensive language models crafted by the Qwen team at Alibaba Cloud. This series encompasses a variety of models, including base and instruction-tuned versions, with parameters varying from 0.5 billion to an impressive 72 billion, showcasing both dense configurations and a Mixture-of-Experts approach. The Qwen2 series aims to outperform many earlier open-weight models, including its predecessor Qwen1.5, while also striving to hold its own against proprietary models across numerous benchmarks in areas such as language comprehension, generation, multilingual functionality, programming, mathematics, and logical reasoning. Furthermore, this innovative series is poised to make a significant impact in the field of artificial intelligence, offering enhanced capabilities for a diverse range of applications.

Pre-trained language models have made significant strides, achieving top-tier performance across multiple Natural Language Processing (NLP) applications. The impressive capabilities of GPT-3 highlight how increasing the scale of these models can unlock their vast potential. Recently, a comprehensive framework known as ERNIE 3.0 was introduced to pre-train large-scale models enriched with knowledge, culminating in a model boasting 10 billion parameters. This iteration of ERNIE 3.0 has surpassed the performance of existing leading models in a variety of NLP tasks. To further assess the effects of scaling, we have developed an even larger model called ERNIE 3.0 Titan, which consists of up to 260 billion parameters and is built on the PaddlePaddle platform. Additionally, we have implemented a self-supervised adversarial loss alongside a controllable language modeling loss, enabling ERNIE 3.0 Titan to produce texts that are both reliable and modifiable, thus pushing the boundaries of what these models can achieve. This approach not only enhances the model's capabilities but also opens new avenues for research in text generation and control.

The Megatron-Turing Natural Language Generation model (MT-NLG) stands out as the largest and most advanced monolithic transformer model for the English language, boasting an impressive 530 billion parameters. This 105-layer transformer architecture significantly enhances the capabilities of previous leading models, particularly in zero-shot, one-shot, and few-shot scenarios. It exhibits exceptional precision across a wide range of natural language processing tasks, including completion prediction, reading comprehension, commonsense reasoning, natural language inference, and word sense disambiguation. To foster further research on this groundbreaking English language model and to allow users to explore and utilize its potential in various language applications, NVIDIA has introduced an Early Access program for its managed API service dedicated to the MT-NLG model. This initiative aims to facilitate experimentation and innovation in the field of natural language processing.

Reka Flash 3 is a cutting-edge multimodal AI model with 21 billion parameters, crafted by Reka AI to perform exceptionally well in tasks such as general conversation, coding, following instructions, and executing functions. This model adeptly handles and analyzes a myriad of inputs, including text, images, video, and audio, providing a versatile and compact solution for a wide range of applications. Built from the ground up, Reka Flash 3 was trained on a rich array of datasets, encompassing both publicly available and synthetic information, and it underwent a meticulous instruction tuning process with high-quality selected data to fine-tune its capabilities. The final phase of its training involved employing reinforcement learning techniques, specifically using the REINFORCE Leave One-Out (RLOO) method, which combined both model-based and rule-based rewards to significantly improve its reasoning skills. With an impressive context length of 32,000 tokens, Reka Flash 3 competes effectively with proprietary models like OpenAI's o1-mini, making it an excellent choice for applications requiring low latency or on-device processing. The model operates at full precision with a memory requirement of 39GB (fp16), although it can be efficiently reduced to just 11GB through the use of 4-bit quantization, demonstrating its adaptability for various deployment scenarios. Overall, Reka Flash 3 represents a significant advancement in multimodal AI technology, capable of meeting diverse user needs across multiple platforms.

ChatGLM-6B is a bilingual dialogue model that supports both Chinese and English, built on the General Language Model (GLM) framework and features 6.2 billion parameters. Thanks to model quantization techniques, it can be easily run on standard consumer graphics cards, requiring only 6GB of video memory at the INT4 quantization level. This model employs methodologies akin to those found in ChatGPT but is specifically tailored to enhance Chinese question-and-answer interactions and dialogue. Following extensive training with approximately 1 trillion identifiers in both languages, along with additional supervision, fine-tuning, self-assistance through feedback, and reinforcement learning from human input, ChatGLM-6B has demonstrated an impressive capability to produce responses that resonate well with human users. Its adaptability and performance make it a valuable tool for bilingual communication.

Stability AI, along with its CarperAI lab, is excited to unveil Stable Beluga 1 and its advanced successor, Stable Beluga 2, previously known as FreeWilly, both of which are robust new Large Language Models (LLMs) available for public use. These models exhibit remarkable reasoning capabilities across a wide range of benchmarks, showcasing their versatility and strength. Stable Beluga 1 is built on the original LLaMA 65B foundation model and has undergone meticulous fine-tuning with a novel synthetically-generated dataset utilizing Supervised Fine-Tune (SFT) in the conventional Alpaca format. In a similar vein, Stable Beluga 2 utilizes the LLaMA 2 70B foundation model, pushing the boundaries of performance in the industry. Their development marks a significant step forward in the evolution of open access AI technologies.

Phi-4-reasoning is an advanced transformer model featuring 14 billion parameters, specifically tailored for tackling intricate reasoning challenges, including mathematics, programming, algorithm development, and strategic planning. Through a meticulous process of supervised fine-tuning on select "teachable" prompts and reasoning examples created using o3-mini, it excels at generating thorough reasoning sequences that optimize computational resources during inference. By integrating outcome-driven reinforcement learning, Phi-4-reasoning is capable of producing extended reasoning paths. Its performance notably surpasses that of significantly larger open-weight models like DeepSeek-R1-Distill-Llama-70B and nears the capabilities of the comprehensive DeepSeek-R1 model across various reasoning applications. Designed for use in settings with limited computing power or high latency, Phi-4-reasoning is fine-tuned with synthetic data provided by DeepSeek-R1, ensuring it delivers precise and methodical problem-solving. This model's ability to handle complex tasks with efficiency makes it a valuable tool in numerous computational contexts.

Mistral Saba is an advanced model boasting 24 billion parameters, developed using carefully selected datasets from the Middle East and South Asia. It outperforms larger models—those more than five times its size—in delivering precise and pertinent responses, all while being notably faster and more cost-effective. Additionally, it serves as an excellent foundation for creating highly specialized regional adaptations. This model can be accessed via an API and is also capable of being deployed locally to meet customers' security requirements. Similar to the recently introduced Mistral Small 3, it is lightweight enough to operate on single-GPU systems, achieving response rates exceeding 150 tokens per second. Reflecting the deep cultural connections between the Middle East and South Asia, Mistral Saba is designed to support Arabic alongside numerous Indian languages, with a particular proficiency in South Indian languages like Tamil. This diverse linguistic capability significantly boosts its adaptability for multinational applications in these closely linked regions. Furthermore, the model’s design facilitates an easier integration into various platforms, enhancing its usability across different industries.

K2 Think represents a groundbreaking open-source advanced reasoning model that has been developed in collaboration between the Institute of Foundation Models at MBZUAI and G42. Even with its relatively modest 32 billion parameters, K2 Think achieves performance that rivals that of leading models with significantly larger parameter counts. Its strength lies in mathematical reasoning, where it has secured top rankings on prestigious benchmarks such as AIME ’24/’25, HMMT ’25, and OMNI-Math-HARD. This model is part of a wider initiative of UAE-developed open models, which includes Jais (for Arabic), NANDA (for Hindi), and SHERKALA (for Kazakh), and it builds upon the groundwork established by the K2-65B, a fully reproducible open-source foundation model released in 2024. K2 Think is crafted to be open, efficient, and adaptable, featuring a web app interface that facilitates user exploration, and its innovative approach to parameter positioning marks a significant advancement in the realm of compact architectures for high-level AI reasoning. Additionally, its development highlights a commitment to enhancing access to state-of-the-art AI technologies in various languages and domains.

StarCoder and StarCoderBase represent advanced Large Language Models specifically designed for code, developed using openly licensed data from GitHub, which encompasses over 80 programming languages, Git commits, GitHub issues, and Jupyter notebooks. In a manner akin to LLaMA, we constructed a model with approximately 15 billion parameters trained on a staggering 1 trillion tokens. Furthermore, we tailored the StarCoderBase model with 35 billion Python tokens, leading to the creation of what we now refer to as StarCoder. Our evaluations indicated that StarCoderBase surpasses other existing open Code LLMs when tested against popular programming benchmarks and performs on par with or even exceeds proprietary models like code-cushman-001 from OpenAI, the original Codex model that fueled early iterations of GitHub Copilot. With an impressive context length exceeding 8,000 tokens, the StarCoder models possess the capability to handle more information than any other open LLM, thus paving the way for a variety of innovative applications. This versatility is highlighted by our ability to prompt the StarCoder models through a sequence of dialogues, effectively transforming them into dynamic technical assistants that can provide support in diverse programming tasks.

Chinchilla is an advanced language model that operates with a compute budget comparable to Gopher while having 70 billion parameters and utilizing four times the amount of data. This model consistently and significantly surpasses Gopher (280 billion parameters), as well as GPT-3 (175 billion), Jurassic-1 (178 billion), and Megatron-Turing NLG (530 billion), across a wide variety of evaluation tasks. Additionally, Chinchilla's design allows it to use significantly less computational power during the fine-tuning and inference processes, which greatly enhances its applicability in real-world scenarios. Notably, Chinchilla achieves a remarkable average accuracy of 67.5% on the MMLU benchmark, marking over a 7% enhancement compared to Gopher, showcasing its superior performance in the field. This impressive capability positions Chinchilla as a leading contender in the realm of language models.

Tülu 3 is a cutting-edge language model created by the Allen Institute for AI (Ai2) that aims to improve proficiency in fields like knowledge, reasoning, mathematics, coding, and safety. It is based on the Llama 3 Base and undergoes a detailed four-stage post-training regimen: careful prompt curation and synthesis, supervised fine-tuning on a wide array of prompts and completions, preference tuning utilizing both off- and on-policy data, and a unique reinforcement learning strategy that enhances targeted skills through measurable rewards. Notably, this open-source model sets itself apart by ensuring complete transparency, offering access to its training data, code, and evaluation tools, thus bridging the performance divide between open and proprietary fine-tuning techniques. Performance assessments reveal that Tülu 3 surpasses other models with comparable sizes, like Llama 3.1-Instruct and Qwen2.5-Instruct, across an array of benchmarks, highlighting its effectiveness. The continuous development of Tülu 3 signifies the commitment to advancing AI capabilities while promoting an open and accessible approach to technology.

DeepSeek-R1 is a cutting-edge open-source reasoning model created by DeepSeek, aimed at competing with OpenAI's Model o1. It is readily available through web, app, and API interfaces, showcasing its proficiency in challenging tasks such as mathematics and coding, and achieving impressive results on assessments like the American Invitational Mathematics Examination (AIME) and MATH. Utilizing a mixture of experts (MoE) architecture, this model boasts a remarkable total of 671 billion parameters, with 37 billion parameters activated for each token, which allows for both efficient and precise reasoning abilities. As a part of DeepSeek's dedication to the progression of artificial general intelligence (AGI), the model underscores the importance of open-source innovation in this field. Furthermore, its advanced capabilities may significantly impact how we approach complex problem-solving in various domains.

OpenEuroLLM represents a collaborative effort between prominent AI firms and research organizations across Europe, aimed at creating a suite of open-source foundational models to promote transparency in artificial intelligence within the continent. This initiative prioritizes openness by making data, documentation, training and testing code, and evaluation metrics readily available, thereby encouraging community participation. It is designed to comply with European Union regulations, with the goal of delivering efficient large language models that meet the specific standards of Europe. A significant aspect of the project is its commitment to linguistic and cultural diversity, ensuring that multilingual capabilities cover all official EU languages and potentially more. The initiative aspires to broaden access to foundational models that can be fine-tuned for a range of applications, enhance evaluation outcomes across different languages, and boost the availability of training datasets and benchmarks for researchers and developers alike. By sharing tools, methodologies, and intermediate results, transparency is upheld during the entire training process, fostering trust and collaboration within the AI community. Ultimately, OpenEuroLLM aims to pave the way for more inclusive and adaptable AI solutions that reflect the rich diversity of European languages and cultures.

Olmo 3 represents a comprehensive family of open models featuring variations with 7 billion and 32 billion parameters, offering exceptional capabilities in base performance, reasoning, instruction, and reinforcement learning, while also providing transparency throughout the model development process, which includes access to raw training datasets, intermediate checkpoints, training scripts, extended context support (with a window of 65,536 tokens), and provenance tools. The foundation of these models is built upon the Dolma 3 dataset, which comprises approximately 9 trillion tokens and utilizes a careful blend of web content, scientific papers, programming code, and lengthy documents; this thorough pre-training, mid-training, and long-context approach culminates in base models that undergo post-training enhancements through supervised fine-tuning, preference optimization, and reinforcement learning with accountable rewards, resulting in the creation of the Think and Instruct variants. Notably, the 32 billion Think model has been recognized as the most powerful fully open reasoning model to date, demonstrating performance that closely rivals that of proprietary counterparts in areas such as mathematics, programming, and intricate reasoning tasks, thereby marking a significant advancement in open model development. This innovation underscores the potential for open-source models to compete with traditional, closed systems in various complex applications.

Llama (Large Language Model Meta AI) stands as a cutting-edge foundational large language model aimed at helping researchers push the boundaries of their work within this area of artificial intelligence. By providing smaller yet highly effective models like Llama, the research community can benefit even if they lack extensive infrastructure, thus promoting greater accessibility in this dynamic and rapidly evolving domain. Creating smaller foundational models such as Llama is advantageous in the landscape of large language models, as it demands significantly reduced computational power and resources, facilitating the testing of innovative methods, confirming existing research, and investigating new applications. These foundational models leverage extensive unlabeled datasets, making them exceptionally suitable for fine-tuning across a range of tasks. We are offering Llama in multiple sizes (7B, 13B, 33B, and 65B parameters), accompanied by a detailed Llama model card that outlines our development process while adhering to our commitment to Responsible AI principles. By making these resources available, we aim to empower a broader segment of the research community to engage with and contribute to advancements in AI.

Cohere's Command A Translate is a robust machine translation solution designed for enterprises, offering secure and top-notch translation capabilities in 23 languages pertinent to business. It operates on an advanced 111-billion-parameter framework with an 8K-input / 8K-output context window, providing superior performance that outshines competitors such as GPT-5, DeepSeek-V3, DeepL Pro, and Google Translate across various benchmarks. The model facilitates private deployment options for organizations handling sensitive information, ensuring they maintain total control of their data, while also featuring a pioneering “Deep Translation” workflow that employs an iterative, multi-step refinement process to significantly improve translation accuracy for intricate scenarios. RWS Group’s external validation underscores its effectiveness in managing demanding translation challenges. Furthermore, the model's parameters are accessible for research through Hugging Face under a CC-BY-NC license, allowing for extensive customization, fine-tuning, and adaptability for private implementations, making it an attractive option for organizations seeking tailored language solutions. This versatility positions Command A Translate as an essential tool for enterprises aiming to enhance their communication across global markets.

Llama 4 Behemoth, with 288 billion active parameters, is Meta's flagship AI model, setting new standards for multimodal performance. Outpacing its predecessors like GPT-4.5 and Claude Sonnet 3.7, it leads the field in STEM benchmarks, offering cutting-edge results in tasks such as problem-solving and reasoning. Designed as the teacher model for the Llama 4 series, Behemoth drives significant improvements in model quality and efficiency through distillation. Although still in development, Llama 4 Behemoth is shaping the future of AI with its unparalleled intelligence, particularly in math, image, and multilingual tasks.

EXAONE Deep represents a collection of advanced language models that are enhanced for reasoning, created by LG AI Research, and come in sizes of 2.4 billion, 7.8 billion, and 32 billion parameters. These models excel in a variety of reasoning challenges, particularly in areas such as mathematics and coding assessments. Significantly, the EXAONE Deep 2.4B model outshines other models of its size, while the 7.8B variant outperforms both open-weight models of similar dimensions and the proprietary reasoning model known as OpenAI o1-mini. Furthermore, the EXAONE Deep 32B model competes effectively with top-tier open-weight models in the field. The accompanying repository offers extensive documentation that includes performance assessments, quick-start guides for leveraging EXAONE Deep models with the Transformers library, detailed explanations of quantized EXAONE Deep weights formatted in AWQ and GGUF, as well as guidance on how to run these models locally through platforms like llama.cpp and Ollama. Additionally, this resource serves to enhance user understanding and accessibility to the capabilities of EXAONE Deep models.

DeepSeek-V2 is a cutting-edge Mixture-of-Experts (MoE) language model developed by DeepSeek-AI, noted for its cost-effective training and high-efficiency inference features. It boasts an impressive total of 236 billion parameters, with only 21 billion active for each token, and is capable of handling a context length of up to 128K tokens. The model utilizes advanced architectures such as Multi-head Latent Attention (MLA) to optimize inference by minimizing the Key-Value (KV) cache and DeepSeekMoE to enable economical training through sparse computations. Compared to its predecessor, DeepSeek 67B, this model shows remarkable improvements, achieving a 42.5% reduction in training expenses, a 93.3% decrease in KV cache size, and a 5.76-fold increase in generation throughput. Trained on an extensive corpus of 8.1 trillion tokens, DeepSeek-V2 demonstrates exceptional capabilities in language comprehension, programming, and reasoning tasks, positioning it as one of the leading open-source models available today. Its innovative approach not only elevates its performance but also sets new benchmarks within the field of artificial intelligence.

Utilize sophisticated coding and language models across a diverse range of applications. Harness the power of expansive generative AI models that possess an intricate grasp of both language and code, paving the way for enhanced reasoning and comprehension skills essential for developing innovative applications. These advanced models can be applied to multiple scenarios, including writing support, automatic code creation, and data reasoning. Moreover, ensure responsible AI practices by implementing measures to detect and mitigate potential misuse, all while benefiting from enterprise-level security features offered by Azure. With access to generative models pretrained on vast datasets comprising trillions of words, you can explore new possibilities in language processing, code analysis, reasoning, inferencing, and comprehension. Further personalize these generative models by using labeled datasets tailored to your unique needs through an easy-to-use REST API. Additionally, you can optimize your model's performance by fine-tuning hyperparameters for improved output accuracy. The few-shot learning functionality allows you to provide sample inputs to the API, resulting in more pertinent and context-aware outcomes. This flexibility enhances your ability to meet specific application demands effectively.

The newest version in the Llama series, Llama 3.3, represents a significant advancement in language models aimed at enhancing AI's capabilities in understanding and communication. It boasts improved contextual reasoning, superior language generation, and advanced fine-tuning features aimed at producing exceptionally accurate, human-like responses across a variety of uses. This iteration incorporates a more extensive training dataset, refined algorithms for deeper comprehension, and mitigated biases compared to earlier versions. Llama 3.3 stands out in applications including natural language understanding, creative writing, technical explanations, and multilingual interactions, making it a crucial asset for businesses, developers, and researchers alike. Additionally, its modular architecture facilitates customizable deployment in specific fields, ensuring it remains versatile and high-performing even in large-scale applications. With these enhancements, Llama 3.3 is poised to redefine the standards of AI language models.

Baichuan-13B is an advanced large-scale language model developed by Baichuan Intelligent, featuring 13 billion parameters and available for open-source and commercial use, building upon its predecessor Baichuan-7B. This model has set new records for performance among similarly sized models on esteemed Chinese and English evaluation metrics. The release includes two distinct pre-training variations: Baichuan-13B-Base and Baichuan-13B-Chat. By significantly increasing the parameter count to 13 billion, Baichuan-13B enhances its capabilities, training on 1.4 trillion tokens from a high-quality dataset, which surpasses LLaMA-13B's training data by 40%. It currently holds the distinction of being the model with the most extensive training data in the 13B category, providing robust support for both Chinese and English languages, utilizing ALiBi positional encoding, and accommodating a context window of 4096 tokens for improved comprehension and generation. This makes it a powerful tool for a variety of applications in natural language processing.

Recent breakthroughs in natural language processing, comprehension, and generation have been greatly influenced by the development of large language models. This research presents a system that employs Ascend 910 AI processors and the MindSpore framework to train a language model exceeding one trillion parameters, specifically 1.085 trillion, referred to as PanGu-{\Sigma}. This model enhances the groundwork established by PanGu-{\alpha} by converting the conventional dense Transformer model into a sparse format through a method known as Random Routed Experts (RRE). Utilizing a substantial dataset of 329 billion tokens, the model was effectively trained using a strategy called Expert Computation and Storage Separation (ECSS), which resulted in a remarkable 6.3-fold improvement in training throughput through the use of heterogeneous computing. Through various experiments, it was found that PanGu-{\Sigma} achieves a new benchmark in zero-shot learning across multiple downstream tasks in Chinese NLP, showcasing its potential in advancing the field. This advancement signifies a major leap forward in the capabilities of language models, illustrating the impact of innovative training techniques and architectural modifications.

Canopy Labs has unveiled Orpheus, an innovative suite of advanced speech large language models (LLMs) aimed at achieving human-like speech generation capabilities. Utilizing the Llama-3 architecture, these models have been trained on an extensive dataset comprising over 100,000 hours of English speech, allowing them to generate speech that exhibits natural intonation, emotional depth, and rhythmic flow that outperforms existing high-end closed-source alternatives. Orpheus also features zero-shot voice cloning, enabling users to mimic voices without any need for prior fine-tuning, and provides easy-to-use tags for controlling emotion and intonation. The models are engineered for low latency, achieving approximately 200ms streaming latency for real-time usage, which can be further decreased to around 100ms when utilizing input streaming. Canopy Labs has made available both pre-trained and fine-tuned models with 3 billion parameters under the flexible Apache 2.0 license, with future intentions to offer smaller models with 1 billion, 400 million, and 150 million parameters to cater to devices with limited resources. This strategic move is expected to broaden accessibility and application potential across various platforms and use cases.

Aya represents a cutting-edge, open-source generative language model that boasts support for 101 languages, significantly surpassing the language capabilities of current open-source counterparts. By facilitating access to advanced language processing for a diverse array of languages and cultures that are often overlooked, Aya empowers researchers to explore the full potential of generative language models. In addition to the Aya model, we are releasing the largest dataset for multilingual instruction fine-tuning ever created, which includes 513 million entries across 114 languages. This extensive dataset features unique annotations provided by native and fluent speakers worldwide, thereby enhancing the ability of AI to cater to a wide range of global communities that have historically had limited access to such technology. Furthermore, the initiative aims to bridge the gap in AI accessibility, ensuring that even the most underserved languages receive the attention they deserve in the digital landscape.

On September 17, 2024, Mistral AI revealed a series of significant updates designed to improve both the accessibility and efficiency of their AI products. Among these updates was the introduction of a complimentary tier on "La Plateforme," their serverless platform that allows for the tuning and deployment of Mistral models as API endpoints, which gives developers a chance to innovate and prototype at zero cost. In addition, Mistral AI announced price reductions across their complete model range, highlighted by a remarkable 50% decrease for Mistral Nemo and an 80% cut for Mistral Small and Codestral, thereby making advanced AI solutions more affordable for a wider audience. The company also launched Mistral Small v24.09, a model with 22 billion parameters that strikes a favorable balance between performance and efficiency, making it ideal for various applications such as translation, summarization, and sentiment analysis. Moreover, they released Pixtral 12B, a vision-capable model equipped with image understanding features, for free on "Le Chat," allowing users to analyze and caption images while maintaining strong text-based performance. This suite of updates reflects Mistral AI's commitment to democratizing access to powerful AI technologies for developers everywhere.

Anthropic’s release of Claude Opus 4.5 introduces a frontier AI model that excels at coding, complex reasoning, deep research, and long-context tasks. It sets new performance records on real-world engineering benchmarks, handling multi-system debugging, ambiguous instructions, and cross-domain problem solving with greater precision than earlier versions. Testers and early customers reported that Opus 4.5 “just gets it,” offering creative reasoning strategies that even benchmarks fail to anticipate. Beyond raw capability, the model brings stronger alignment and safety, with notable advances in prompt-injection resistance and behavior consistency in high-stakes scenarios. The Claude Developer Platform also gains richer controls including effort tuning, multi-agent orchestration, and context management improvements that significantly boost efficiency. Claude Code becomes more powerful with enhanced planning abilities, multi-session desktop support, and better execution of complex development workflows. In the Claude apps, extended memory and automatic context summarization enable longer, uninterrupted conversations. Together, these upgrades showcase Opus 4.5 as a highly capable, secure, and versatile model designed for both professional workloads and everyday use.

Smaug-72B is a formidable open-source large language model (LLM) distinguished by several prominent features: Exceptional Performance: It currently ranks first on the Hugging Face Open LLM leaderboard, outperforming models such as GPT-3.5 in multiple evaluations, demonstrating its ability to comprehend, react to, and generate text that closely resembles human writing. Open Source Availability: In contrast to many high-end LLMs, Smaug-72B is accessible to everyone for use and modification, which encourages cooperation and innovation within the AI ecosystem. Emphasis on Reasoning and Mathematics: This model excels particularly in reasoning and mathematical challenges, a capability attributed to specialized fine-tuning methods developed by its creators, Abacus AI. Derived from Qwen-72B: It is essentially a refined version of another robust LLM, Qwen-72B, which was launched by Alibaba, thereby enhancing its overall performance. In summary, Smaug-72B marks a notable advancement in the realm of open-source artificial intelligence, making it a valuable resource for developers and researchers alike. Its unique strengths not only elevate its status but also contribute to the ongoing evolution of AI technology.

Alpa is designed to simplify the process of automating extensive distributed training and serving with minimal coding effort. Originally created by a team at Sky Lab, UC Berkeley, it employs several advanced techniques documented in a paper presented at OSDI'2022. The Alpa community continues to expand, welcoming new contributors from Google. A language model serves as a probability distribution over sequences of words, allowing it to foresee the next word based on the context of preceding words. This capability proves valuable for various AI applications, including email auto-completion and chatbot functionalities. For further insights, one can visit the Wikipedia page dedicated to language models. Among these models, GPT-3 stands out as a remarkably large language model, boasting 175 billion parameters and utilizing deep learning to generate text that closely resembles human writing. Many researchers and media outlets have characterized GPT-3 as "one of the most interesting and significant AI systems ever developed," and its influence continues to grow as it becomes integral to cutting-edge NLP research and applications. Additionally, its implementation has sparked discussions about the future of AI-driven communication tools.

Phi-4-mini-reasoning is a transformer-based language model with 3.8 billion parameters, specifically designed to excel in mathematical reasoning and methodical problem-solving within environments that have limited computational capacity or latency constraints. Its optimization stems from fine-tuning with synthetic data produced by the DeepSeek-R1 model, striking a balance between efficiency and sophisticated reasoning capabilities. With training that encompasses over one million varied math problems, ranging in complexity from middle school to Ph.D. level, Phi-4-mini-reasoning demonstrates superior performance to its base model in generating lengthy sentences across multiple assessments and outshines larger counterparts such as OpenThinker-7B, Llama-3.2-3B-instruct, and DeepSeek-R1. Equipped with a 128K-token context window, it also facilitates function calling, which allows for seamless integration with various external tools and APIs. Moreover, Phi-4-mini-reasoning can be quantized through the Microsoft Olive or Apple MLX Framework, enabling its deployment on a variety of edge devices, including IoT gadgets, laptops, and smartphones. Its design not only enhances user accessibility but also expands the potential for innovative applications in mathematical fields.

Phi-4-mini-flash-reasoning is a 3.8 billion-parameter model that is part of Microsoft's Phi series, specifically designed for edge, mobile, and other environments with constrained resources where processing power, memory, and speed are limited. This innovative model features the SambaY hybrid decoder architecture, integrating Gated Memory Units (GMUs) with Mamba state-space and sliding-window attention layers, achieving up to ten times the throughput and a latency reduction of 2 to 3 times compared to its earlier versions without compromising on its ability to perform complex mathematical and logical reasoning. With a support for a context length of 64K tokens and being fine-tuned on high-quality synthetic datasets, it is particularly adept at handling long-context retrieval, reasoning tasks, and real-time inference, all manageable on a single GPU. Available through platforms such as Azure AI Foundry, NVIDIA API Catalog, and Hugging Face, Phi-4-mini-flash-reasoning empowers developers to create applications that are not only fast but also scalable and capable of intensive logical processing. This accessibility allows a broader range of developers to leverage its capabilities for innovative solutions.

Training cutting-edge language models presents significant challenges; it demands vast computational resources, intricate distributed computing strategies, and substantial machine learning knowledge. Consequently, only a limited number of organizations embark on the journey of developing large language models (LLMs) from the ground up. Furthermore, many of those with the necessary capabilities and knowledge have begun to restrict access to their findings, indicating a notable shift from practices observed just a few months ago. At Cerebras, we are committed to promoting open access to state-of-the-art models. Therefore, we are excited to share with the open-source community the launch of Cerebras-GPT, which consists of a series of seven GPT models with parameter counts ranging from 111 million to 13 billion. Utilizing the Chinchilla formula for training, these models deliver exceptional accuracy while optimizing for computational efficiency. Notably, Cerebras-GPT boasts quicker training durations, reduced costs, and lower energy consumption compared to any publicly accessible model currently available. By releasing these models, we hope to inspire further innovation and collaboration in the field of machine learning.

DeepSeek-V3 represents a groundbreaking advancement in artificial intelligence, specifically engineered to excel in natural language comprehension, sophisticated reasoning, and decision-making processes. By utilizing highly advanced neural network designs, this model incorporates vast amounts of data alongside refined algorithms to address intricate problems across a wide array of fields, including research, development, business analytics, and automation. Prioritizing both scalability and operational efficiency, DeepSeek-V3 equips developers and organizations with innovative resources that can significantly expedite progress and lead to transformative results. Furthermore, its versatility makes it suitable for various applications, enhancing its value across industries.

DeepSeek V3.1 stands as a revolutionary open-weight large language model, boasting an impressive 685-billion parameters and an expansive 128,000-token context window, which allows it to analyze extensive documents akin to 400-page books in a single invocation. This model offers integrated functionalities for chatting, reasoning, and code creation, all within a cohesive hybrid architecture that harmonizes these diverse capabilities. Furthermore, V3.1 accommodates multiple tensor formats, granting developers the versatility to enhance performance across various hardware setups. Preliminary benchmark evaluations reveal strong results, including a remarkable 71.6% on the Aider coding benchmark, positioning it competitively with or even superior to systems such as Claude Opus 4, while achieving this at a significantly reduced cost. Released under an open-source license on Hugging Face with little publicity, DeepSeek V3.1 is set to revolutionize access to advanced AI technologies, potentially disrupting the landscape dominated by conventional proprietary models. Its innovative features and cost-effectiveness may attract a wide range of developers eager to leverage cutting-edge AI in their projects.

Nexusflow has unveiled Athene-V2, its newest model suite boasting 72 billion parameters, which has been meticulously fine-tuned from Qwen 2.5 72B to rival the capabilities of GPT-4o. Within this suite, Athene-V2-Chat-72B stands out as a cutting-edge chat model that performs comparably to GPT-4o across various benchmarks; it excels particularly in chat helpfulness (Arena-Hard), ranks second in the code completion category on bigcode-bench-hard, and demonstrates strong abilities in mathematics (MATH) and accurate long log extraction. Furthermore, Athene-V2-Agent-72B seamlessly integrates chat and agent features, delivering clear and directive responses while surpassing GPT-4o in Nexus-V2 function calling benchmarks, specifically tailored for intricate enterprise-level scenarios. These innovations highlight a significant industry transition from merely increasing model sizes to focusing on specialized customization, showcasing how targeted post-training techniques can effectively enhance models for specific skills and applications. As technology continues to evolve, it becomes essential for developers to leverage these advancements to create increasingly sophisticated AI solutions.