Alternatives to OLMo 2

Tülu 3 is a cutting-edge language model created by the Allen Institute for AI (Ai2) that aims to improve proficiency in fields like knowledge, reasoning, mathematics, coding, and safety. It is based on the Llama 3 Base and undergoes a detailed four-stage post-training regimen: careful prompt curation and synthesis, supervised fine-tuning on a wide array of prompts and completions, preference tuning utilizing both off- and on-policy data, and a unique reinforcement learning strategy that enhances targeted skills through measurable rewards. Notably, this open-source model sets itself apart by ensuring complete transparency, offering access to its training data, code, and evaluation tools, thus bridging the performance divide between open and proprietary fine-tuning techniques. Performance assessments reveal that Tülu 3 surpasses other models with comparable sizes, like Llama 3.1-Instruct and Qwen2.5-Instruct, across an array of benchmarks, highlighting its effectiveness. The continuous development of Tülu 3 signifies the commitment to advancing AI capabilities while promoting an open and accessible approach to technology.

Molmo represents a cutting-edge family of multimodal AI models crafted by the Allen Institute for AI (Ai2). These innovative models are specifically engineered to connect the divide between open-source and proprietary systems, ensuring they perform competitively across numerous academic benchmarks and assessments by humans. In contrast to many existing multimodal systems that depend on synthetic data sourced from proprietary frameworks, Molmo is exclusively trained on openly available data, which promotes transparency and reproducibility in AI research. A significant breakthrough in the development of Molmo is the incorporation of PixMo, a unique dataset filled with intricately detailed image captions gathered from human annotators who utilized speech-based descriptions, along with 2D pointing data that empowers the models to respond to inquiries with both natural language and non-verbal signals. This capability allows Molmo to engage with its surroundings in a more sophisticated manner, such as by pointing to specific objects within images, thereby broadening its potential applications in diverse fields, including robotics, augmented reality, and interactive user interfaces. Furthermore, the advancements made by Molmo set a new standard for future multimodal AI research and application development.

Baichuan-13B is an advanced large-scale language model developed by Baichuan Intelligent, featuring 13 billion parameters and available for open-source and commercial use, building upon its predecessor Baichuan-7B. This model has set new records for performance among similarly sized models on esteemed Chinese and English evaluation metrics. The release includes two distinct pre-training variations: Baichuan-13B-Base and Baichuan-13B-Chat. By significantly increasing the parameter count to 13 billion, Baichuan-13B enhances its capabilities, training on 1.4 trillion tokens from a high-quality dataset, which surpasses LLaMA-13B's training data by 40%. It currently holds the distinction of being the model with the most extensive training data in the 13B category, providing robust support for both Chinese and English languages, utilizing ALiBi positional encoding, and accommodating a context window of 4096 tokens for improved comprehension and generation. This makes it a powerful tool for a variety of applications in natural language processing.

Introducing the next iteration of our open-source large language model, this version features model weights along with initial code for the pretrained and fine-tuned Llama language models, which span from 7 billion to 70 billion parameters. The Llama 2 pretrained models have been developed using an impressive 2 trillion tokens and offer double the context length compared to their predecessor, Llama 1. Furthermore, the fine-tuned models have been enhanced through the analysis of over 1 million human annotations. Llama 2 demonstrates superior performance against various other open-source language models across multiple external benchmarks, excelling in areas such as reasoning, coding capabilities, proficiency, and knowledge assessments. For its training, Llama 2 utilized publicly accessible online data sources, while the fine-tuned variant, Llama-2-chat, incorporates publicly available instruction datasets along with the aforementioned extensive human annotations. Our initiative enjoys strong support from a diverse array of global stakeholders who are enthusiastic about our open approach to AI, including companies that have provided valuable early feedback and are eager to collaborate using Llama 2. The excitement surrounding Llama 2 signifies a pivotal shift in how AI can be developed and utilized collectively.

Qwen2 represents a collection of extensive language models crafted by the Qwen team at Alibaba Cloud. This series encompasses a variety of models, including base and instruction-tuned versions, with parameters varying from 0.5 billion to an impressive 72 billion, showcasing both dense configurations and a Mixture-of-Experts approach. The Qwen2 series aims to outperform many earlier open-weight models, including its predecessor Qwen1.5, while also striving to hold its own against proprietary models across numerous benchmarks in areas such as language comprehension, generation, multilingual functionality, programming, mathematics, and logical reasoning. Furthermore, this innovative series is poised to make a significant impact in the field of artificial intelligence, offering enhanced capabilities for a diverse range of applications.

OpenEuroLLM represents a collaborative effort between prominent AI firms and research organizations across Europe, aimed at creating a suite of open-source foundational models to promote transparency in artificial intelligence within the continent. This initiative prioritizes openness by making data, documentation, training and testing code, and evaluation metrics readily available, thereby encouraging community participation. It is designed to comply with European Union regulations, with the goal of delivering efficient large language models that meet the specific standards of Europe. A significant aspect of the project is its commitment to linguistic and cultural diversity, ensuring that multilingual capabilities cover all official EU languages and potentially more. The initiative aspires to broaden access to foundational models that can be fine-tuned for a range of applications, enhance evaluation outcomes across different languages, and boost the availability of training datasets and benchmarks for researchers and developers alike. By sharing tools, methodologies, and intermediate results, transparency is upheld during the entire training process, fostering trust and collaboration within the AI community. Ultimately, OpenEuroLLM aims to pave the way for more inclusive and adaptable AI solutions that reflect the rich diversity of European languages and cultures.

Qwen-7B is the 7-billion parameter iteration of Alibaba Cloud's Qwen language model series, also known as Tongyi Qianwen. This large language model utilizes a Transformer architecture and has been pretrained on an extensive dataset comprising web texts, books, code, and more. Furthermore, we introduced Qwen-7B-Chat, an AI assistant that builds upon the pretrained Qwen-7B model and incorporates advanced alignment techniques. The Qwen-7B series boasts several notable features: It has been trained on a premium dataset, with over 2.2 trillion tokens sourced from a self-assembled collection of high-quality texts and codes across various domains, encompassing both general and specialized knowledge. Additionally, our model demonstrates exceptional performance, surpassing competitors of similar size on numerous benchmark datasets that assess capabilities in natural language understanding, mathematics, and coding tasks. This positions Qwen-7B as a leading choice in the realm of AI language models. Overall, its sophisticated training and robust design contribute to its impressive versatility and effectiveness.

Large language models, often requiring extensive computational resources for training over long periods, have demonstrated impressive proficiency in zero- and few-shot learning tasks. Due to the high investment needed for their development, replicating these models poses a significant challenge for many researchers. Furthermore, access to the few models available via API is limited, as users cannot obtain the complete model weights, complicating academic exploration. In response to this, we introduce Open Pre-trained Transformers (OPT), a collection of decoder-only pre-trained transformers ranging from 125 million to 175 billion parameters, which we intend to share comprehensively and responsibly with interested scholars. Our findings indicate that OPT-175B exhibits performance on par with GPT-3, yet it is developed with only one-seventh of the carbon emissions required for GPT-3's training. Additionally, we will provide a detailed logbook that outlines the infrastructure hurdles we encountered throughout the project, as well as code to facilitate experimentation with all released models, ensuring that researchers have the tools they need to explore this technology further.

StarCoder and StarCoderBase represent advanced Large Language Models specifically designed for code, developed using openly licensed data from GitHub, which encompasses over 80 programming languages, Git commits, GitHub issues, and Jupyter notebooks. In a manner akin to LLaMA, we constructed a model with approximately 15 billion parameters trained on a staggering 1 trillion tokens. Furthermore, we tailored the StarCoderBase model with 35 billion Python tokens, leading to the creation of what we now refer to as StarCoder. Our evaluations indicated that StarCoderBase surpasses other existing open Code LLMs when tested against popular programming benchmarks and performs on par with or even exceeds proprietary models like code-cushman-001 from OpenAI, the original Codex model that fueled early iterations of GitHub Copilot. With an impressive context length exceeding 8,000 tokens, the StarCoder models possess the capability to handle more information than any other open LLM, thus paving the way for a variety of innovative applications. This versatility is highlighted by our ability to prompt the StarCoder models through a sequence of dialogues, effectively transforming them into dynamic technical assistants that can provide support in diverse programming tasks.

We are excited to unveil Codestral, our inaugural code generation model. This open-weight generative AI system is specifically crafted for tasks related to code generation, enabling developers to seamlessly write and engage with code via a unified instruction and completion API endpoint. As it becomes proficient in both programming languages and English, Codestral is poised to facilitate the creation of sophisticated AI applications tailored for software developers. With a training foundation that encompasses a wide array of over 80 programming languages—ranging from widely-used options like Python, Java, C, C++, JavaScript, and Bash to more niche languages such as Swift and Fortran—Codestral ensures a versatile support system for developers tackling various coding challenges and projects. Its extensive language capabilities empower developers to confidently navigate different coding environments, making Codestral an invaluable asset in the programming landscape.

Pixtral Large is an expansive multimodal model featuring 124 billion parameters, crafted by Mistral AI and enhancing their previous Mistral Large 2 framework. This model combines a 123-billion-parameter multimodal decoder with a 1-billion-parameter vision encoder, allowing it to excel in the interpretation of various content types, including documents, charts, and natural images, all while retaining superior text comprehension abilities. With the capability to manage a context window of 128,000 tokens, Pixtral Large can efficiently analyze at least 30 high-resolution images at once. It has achieved remarkable results on benchmarks like MathVista, DocVQA, and VQAv2, outpacing competitors such as GPT-4o and Gemini-1.5 Pro. Available for research and educational purposes under the Mistral Research License, it also has a Mistral Commercial License for business applications. This versatility makes Pixtral Large a valuable tool for both academic research and commercial innovations.

CodeQwen serves as the coding counterpart to Qwen, which is a series of large language models created by the Qwen team at Alibaba Cloud. Built on a transformer architecture that functions solely as a decoder, this model has undergone extensive pre-training using a vast dataset of code. It showcases robust code generation abilities and demonstrates impressive results across various benchmarking tests. With the capacity to comprehend and generate long contexts of up to 64,000 tokens, CodeQwen accommodates 92 programming languages and excels in tasks such as text-to-SQL queries and debugging. Engaging with CodeQwen is straightforward—you can initiate a conversation with just a few lines of code utilizing transformers. The foundation of this interaction relies on constructing the tokenizer and model using pre-existing methods, employing the generate function to facilitate dialogue guided by the chat template provided by the tokenizer. In alignment with our established practices, we implement the ChatML template tailored for chat models. This model adeptly completes code snippets based on the prompts it receives, delivering responses without the need for any further formatting adjustments, thereby enhancing the user experience. The seamless integration of these elements underscores the efficiency and versatility of CodeQwen in handling diverse coding tasks.

OpenAI's o1 series introduces a new generation of AI models specifically developed to enhance reasoning skills. Among these models are o1-preview and o1-mini, which utilize an innovative reinforcement learning technique that encourages them to dedicate more time to "thinking" through various problems before delivering solutions. This method enables the o1 models to perform exceptionally well in intricate problem-solving scenarios, particularly in fields such as coding, mathematics, and science, and they have shown to surpass earlier models like GPT-4o in specific benchmarks. The o1 series is designed to address challenges that necessitate more profound cognitive processes, representing a pivotal advancement toward AI systems capable of reasoning in a manner similar to humans. As it currently stands, the series is still undergoing enhancements and assessments, reflecting OpenAI's commitment to refining these technologies further. The continuous development of the o1 models highlights the potential for AI to evolve and meet more complex demands in the future.

This repository showcases the research preview of LongLLaMA, an advanced large language model that can manage extensive contexts of up to 256,000 tokens or potentially more. LongLLaMA is developed on the OpenLLaMA framework and has been fine-tuned utilizing the Focused Transformer (FoT) technique. The underlying code for LongLLaMA is derived from Code Llama. We are releasing a smaller 3B base variant of the LongLLaMA model, which is not instruction-tuned, under an open license (Apache 2.0), along with inference code that accommodates longer contexts available on Hugging Face. This model's weights can seamlessly replace LLaMA in existing systems designed for shorter contexts, specifically those handling up to 2048 tokens. Furthermore, we include evaluation results along with comparisons to the original OpenLLaMA models, thereby providing a comprehensive overview of LongLLaMA's capabilities in the realm of long-context processing.

Qwen3 is a state-of-the-art large language model designed to revolutionize the way we interact with AI. Featuring both thinking and non-thinking modes, Qwen3 allows users to customize its response style, ensuring optimal performance for both complex reasoning tasks and quick inquiries. With the ability to support 119 languages, the model is suitable for international projects. The model's hybrid training approach, which involves over 36 trillion tokens, ensures accuracy across a variety of disciplines, from coding to STEM problems. Its integration with platforms such as Hugging Face, ModelScope, and Kaggle allows for easy adoption in both research and production environments. By enhancing multilingual support and incorporating advanced AI techniques, Qwen3 is designed to push the boundaries of AI-driven applications.

Sky-T1-32B-Preview is an innovative open-source reasoning model crafted by the NovaSky team at UC Berkeley's Sky Computing Lab. It delivers performance comparable to proprietary models such as o1-preview on various reasoning and coding assessments, while being developed at a cost of less than $450, highlighting the potential for budget-friendly, advanced reasoning abilities. Fine-tuned from Qwen2.5-32B-Instruct, the model utilized a meticulously curated dataset comprising 17,000 examples spanning multiple fields, such as mathematics and programming. The entire training process was completed in just 19 hours using eight H100 GPUs with DeepSpeed Zero-3 offloading technology. Every component of this initiative—including the data, code, and model weights—is entirely open-source, allowing both academic and open-source communities to not only replicate but also improve upon the model's capabilities. This accessibility fosters collaboration and innovation in the realm of artificial intelligence research and development.

OpenGPT-X is an initiative based in Germany that is dedicated to creating large AI language models specifically designed to meet the needs of Europe, highlighting attributes such as adaptability, reliability, multilingual support, and open-source accessibility. This initiative unites various partners to encompass the full spectrum of the generative AI value chain, which includes scalable, GPU-powered infrastructure and data for training expansive language models, alongside model design and practical applications through prototypes and proofs of concept. The primary goal of OpenGPT-X is to promote innovative research with a significant emphasis on business applications, thus facilitating the quicker integration of generative AI within the German economic landscape. Additionally, the project places a strong importance on the ethical development of AI, ensuring that the models developed are both reliable and consistent with European values and regulations. Furthermore, OpenGPT-X offers valuable resources such as the LLM Workbook and a comprehensive three-part reference guide filled with examples and resources to aid users in grasping the essential features of large AI language models, ultimately fostering a deeper understanding of this technology. By providing these tools, OpenGPT-X not only supports the technical development of AI but also encourages responsible usage and implementation across various sectors.

The Mixtral 8x7B model is an advanced sparse mixture of experts (SMoE) system that boasts open weights and is released under the Apache 2.0 license. This model demonstrates superior performance compared to Llama 2 70B across various benchmarks while achieving inference speeds that are six times faster. Recognized as the leading open-weight model with a flexible licensing framework, Mixtral also excels in terms of cost-efficiency and performance. Notably, it competes with and often surpasses GPT-3.5 in numerous established benchmarks, highlighting its significance in the field. Its combination of accessibility, speed, and effectiveness makes it a compelling choice for developers seeking high-performing AI solutions.

PygmalionAI is a vibrant community focused on the development of open-source initiatives utilizing EleutherAI's GPT-J 6B and Meta's LLaMA models. Essentially, Pygmalion specializes in crafting AI tailored for engaging conversations and roleplaying. The actively maintained Pygmalion AI model currently features the 7B variant, derived from Meta AI's LLaMA model. Requiring a mere 18GB (or even less) of VRAM, Pygmalion demonstrates superior chat functionality compared to significantly larger language models, all while utilizing relatively limited resources. Our meticulously assembled dataset, rich in high-quality roleplaying content, guarantees that your AI companion will be the perfect partner for roleplaying scenarios. Both the model weights and the training code are entirely open-source, allowing you the freedom to modify and redistribute them for any purpose you desire. Generally, language models, such as Pygmalion, operate on GPUs, as they require swift memory access and substantial processing power to generate coherent text efficiently. As a result, users can expect a smooth and responsive interaction experience when employing Pygmalion's capabilities.

DeepSeek-V2 is a cutting-edge Mixture-of-Experts (MoE) language model developed by DeepSeek-AI, noted for its cost-effective training and high-efficiency inference features. It boasts an impressive total of 236 billion parameters, with only 21 billion active for each token, and is capable of handling a context length of up to 128K tokens. The model utilizes advanced architectures such as Multi-head Latent Attention (MLA) to optimize inference by minimizing the Key-Value (KV) cache and DeepSeekMoE to enable economical training through sparse computations. Compared to its predecessor, DeepSeek 67B, this model shows remarkable improvements, achieving a 42.5% reduction in training expenses, a 93.3% decrease in KV cache size, and a 5.76-fold increase in generation throughput. Trained on an extensive corpus of 8.1 trillion tokens, DeepSeek-V2 demonstrates exceptional capabilities in language comprehension, programming, and reasoning tasks, positioning it as one of the leading open-source models available today. Its innovative approach not only elevates its performance but also sets new benchmarks within the field of artificial intelligence.

Vicuna-13B is an open-source conversational agent developed through the fine-tuning of LLaMA, utilizing a dataset of user-shared dialogues gathered from ShareGPT. Initial assessments, with GPT-4 serving as an evaluator, indicate that Vicuna-13B achieves over 90% of the quality exhibited by OpenAI's ChatGPT and Google Bard, and it surpasses other models such as LLaMA and Stanford Alpaca in more than 90% of instances. The entire training process for Vicuna-13B incurs an estimated expenditure of approximately $300. Additionally, the source code and model weights, along with an interactive demonstration, are made available for public access under non-commercial terms, fostering a collaborative environment for further development and exploration. This openness encourages innovation and enables users to experiment with the model's capabilities in diverse applications.

The upcoming GPT-5 is the next version in OpenAI's series of Generative Pre-trained Transformers, which remains under development. These advanced language models are built on vast datasets, enabling them to produce realistic and coherent text, translate between languages, create various forms of creative content, and provide informative answers to inquiries. As of now, it is not available to the public, and although OpenAI has yet to disclose an official launch date, there is speculation that its release could occur in 2024. This iteration is anticipated to significantly outpace its predecessor, GPT-4, which is already capable of generating text that resembles human writing, translating languages, and crafting a wide range of creative pieces. The expectations for GPT-5 include enhanced reasoning skills, improved factual accuracy, and a superior ability to adhere to user instructions, making it a highly anticipated advancement in the field. Overall, the development of GPT-5 represents a considerable leap forward in the capabilities of AI language processing.

We are excited to announce the launch of our X1 large series of models. The most robust model from Giga ML is now accessible for both pre-training and fine-tuning in an on-premises environment. Thanks to our compatibility with Open AI, existing integrations with tools like long chain, llama-index, and others function effortlessly. You can also proceed with pre-training LLMs using specialized data sources such as industry-specific documents or company files. The landscape of large language models (LLMs) is rapidly evolving, creating incredible opportunities for advancements in natural language processing across multiple fields. Despite this growth, several significant challenges persist in the industry. At Giga ML, we are thrilled to introduce the X1 Large 32k model, an innovative on-premise LLM solution designed specifically to tackle these pressing challenges, ensuring that organizations can harness the full potential of LLMs effectively. With this launch, we aim to empower businesses to elevate their language processing capabilities.

Qwen2.5-Max is an advanced Mixture-of-Experts (MoE) model created by the Qwen team, which has been pretrained on an extensive dataset of over 20 trillion tokens and subsequently enhanced through methods like Supervised Fine-Tuning (SFT) and Reinforcement Learning from Human Feedback (RLHF). Its performance in evaluations surpasses that of models such as DeepSeek V3 across various benchmarks, including Arena-Hard, LiveBench, LiveCodeBench, and GPQA-Diamond, while also achieving strong results in other tests like MMLU-Pro. This model is available through an API on Alibaba Cloud, allowing users to easily integrate it into their applications, and it can also be interacted with on Qwen Chat for a hands-on experience. With its superior capabilities, Qwen2.5-Max represents a significant advancement in AI model technology.

GPT-4, or Generative Pre-trained Transformer 4, is a highly advanced unsupervised language model that is anticipated for release by OpenAI. As the successor to GPT-3, it belongs to the GPT-n series of natural language processing models and was developed using an extensive dataset comprising 45TB of text, enabling it to generate and comprehend text in a manner akin to human communication. Distinct from many conventional NLP models, GPT-4 operates without the need for additional training data tailored to specific tasks. It is capable of generating text or responding to inquiries by utilizing only the context it creates internally. Demonstrating remarkable versatility, GPT-4 can adeptly tackle a diverse array of tasks such as translation, summarization, question answering, sentiment analysis, and more, all without any dedicated task-specific training. This ability to perform such varied functions further highlights its potential impact on the field of artificial intelligence and natural language processing.

Instruction-following models like GPT-3.5 (text-DaVinci-003), ChatGPT, Claude, and Bing Chat have seen significant advancements in their capabilities, leading to a rise in their usage among individuals in both personal and professional contexts. Despite their growing popularity and integration into daily tasks, these models are not without their shortcomings, as they can sometimes disseminate inaccurate information, reinforce harmful stereotypes, and use inappropriate language. To effectively tackle these critical issues, it is essential for researchers and scholars to become actively involved in exploring these models further. However, conducting research on instruction-following models within academic settings has posed challenges due to the unavailability of models with comparable functionality to proprietary options like OpenAI’s text-DaVinci-003. In response to this gap, we are presenting our insights on an instruction-following language model named Alpaca, which has been fine-tuned from Meta’s LLaMA 7B model, aiming to contribute to the discourse and development in this field. This initiative represents a step towards enhancing the understanding and capabilities of instruction-following models in a more accessible manner for researchers.

Falcon Mamba 7B marks a significant milestone as the inaugural open-source State Space Language Model (SSLM), presenting a revolutionary architecture within the Falcon model family. Celebrated as the premier open-source SSLM globally by Hugging Face, it establishes a new standard for efficiency in artificial intelligence. In contrast to conventional transformers, SSLMs require significantly less memory and can produce lengthy text sequences seamlessly without extra resource demands. Falcon Mamba 7B outperforms top transformer models, such as Meta’s Llama 3.1 8B and Mistral’s 7B, demonstrating enhanced capabilities. This breakthrough not only highlights Abu Dhabi’s dedication to pushing the boundaries of AI research but also positions the region as a pivotal player in the global AI landscape. Such advancements are vital for fostering innovation and collaboration in technology.

Foundation models, including GPT-4, have significantly accelerated advancements in artificial intelligence, yet the most advanced models remain either proprietary or only partially accessible. In response to this challenge, the RedPajama initiative aims to develop a collection of top-tier, fully open-source models. We are thrilled to announce that we have successfully completed the initial phase of this endeavor: recreating the LLaMA training dataset, which contains over 1.2 trillion tokens. Currently, many of the leading foundation models are locked behind commercial APIs, restricting opportunities for research, customization, and application with sensitive information. The development of fully open-source models represents a potential solution to these limitations, provided that the open-source community can bridge the gap in quality between open and closed models. Recent advancements have shown promising progress in this area, suggesting that the AI field is experiencing a transformative period akin to the emergence of Linux. The success of Stable Diffusion serves as a testament to the fact that open-source alternatives can not only match the quality of commercial products like DALL-E but also inspire remarkable creativity through the collaborative efforts of diverse communities. By fostering an open-source ecosystem, we can unlock new possibilities for innovation and ensure broader access to cutting-edge AI technology.

The latest iteration of the open-source AI model, which can be fine-tuned and deployed in various environments, is now offered in multiple versions, including 1B, 3B, 11B, and 90B, alongside the option to continue utilizing Llama 3.1. Llama 3.2 comprises a series of large language models (LLMs) that come pretrained and fine-tuned in 1B and 3B configurations for multilingual text only, while the 11B and 90B models accommodate both text and image inputs, producing text outputs. With this new release, you can create highly effective and efficient applications tailored to your needs. For on-device applications, such as summarizing phone discussions or accessing calendar tools, the 1B or 3B models are ideal choices. Meanwhile, the 11B or 90B models excel in image-related tasks, enabling you to transform existing images or extract additional information from images of your environment. Overall, this diverse range of models allows developers to explore innovative use cases across various domains.

Gemma represents a collection of cutting-edge, lightweight open models that are built upon the same research and technology underlying the Gemini models. Created by Google DeepMind alongside various teams at Google, the inspiration for Gemma comes from the Latin word "gemma," which translates to "precious stone." In addition to providing our model weights, we are also offering tools aimed at promoting developer creativity, encouraging collaboration, and ensuring the ethical application of Gemma models. Sharing key technical and infrastructural elements with Gemini, which stands as our most advanced AI model currently accessible, Gemma 2B and 7B excel in performance within their weight categories when compared to other open models. Furthermore, these models can conveniently operate on a developer's laptop or desktop, demonstrating their versatility. Impressively, Gemma not only outperforms significantly larger models on crucial benchmarks but also maintains our strict criteria for delivering safe and responsible outputs, making it a valuable asset for developers.

Stable LM represents a significant advancement in the field of language models by leveraging our previous experience with open-source initiatives, particularly in collaboration with EleutherAI, a nonprofit research organization. This journey includes the development of notable models such as GPT-J, GPT-NeoX, and the Pythia suite, all of which were trained on The Pile open-source dataset, while many contemporary open-source models like Cerebras-GPT and Dolly-2 have drawn inspiration from this foundational work. Unlike its predecessors, Stable LM is trained on an innovative dataset that is three times the size of The Pile, encompassing a staggering 1.5 trillion tokens. We plan to share more information about this dataset in the near future. The extensive nature of this dataset enables Stable LM to excel remarkably in both conversational and coding scenarios, despite its relatively modest size of 3 to 7 billion parameters when compared to larger models like GPT-3, which boasts 175 billion parameters. Designed for versatility, Stable LM 3B is a streamlined model that can efficiently function on portable devices such as laptops and handheld gadgets, making us enthusiastic about its practical applications and mobility. Overall, the development of Stable LM marks a pivotal step towards creating more efficient and accessible language models for a wider audience.

The newest version in the Llama series, Llama 3.3, represents a significant advancement in language models aimed at enhancing AI's capabilities in understanding and communication. It boasts improved contextual reasoning, superior language generation, and advanced fine-tuning features aimed at producing exceptionally accurate, human-like responses across a variety of uses. This iteration incorporates a more extensive training dataset, refined algorithms for deeper comprehension, and mitigated biases compared to earlier versions. Llama 3.3 stands out in applications including natural language understanding, creative writing, technical explanations, and multilingual interactions, making it a crucial asset for businesses, developers, and researchers alike. Additionally, its modular architecture facilitates customizable deployment in specific fields, ensuring it remains versatile and high-performing even in large-scale applications. With these enhancements, Llama 3.3 is poised to redefine the standards of AI language models.

We are thrilled to present DBRX, a versatile open LLM developed by Databricks. This innovative model achieves unprecedented performance on a variety of standard benchmarks, setting a new benchmark for existing open LLMs. Additionally, it equips both the open-source community and enterprises crafting their own LLMs with features that were once exclusive to proprietary model APIs; our evaluations indicate that it outperforms GPT-3.5 and competes effectively with Gemini 1.0 Pro. Notably, it excels as a code model, outperforming specialized counterparts like CodeLLaMA-70B in programming tasks, while also demonstrating its prowess as a general-purpose LLM. The remarkable quality of DBRX is complemented by significant enhancements in both training and inference efficiency. Thanks to its advanced fine-grained mixture-of-experts (MoE) architecture, DBRX elevates the efficiency of open models to new heights. In terms of inference speed, it can be twice as fast as LLaMA2-70B, and its total and active parameter counts are approximately 40% of those in Grok-1, showcasing its compact design without compromising capability. This combination of speed and size makes DBRX a game-changer in the landscape of open AI models.

Teuken-7B is a multilingual language model that has been developed as part of the OpenGPT-X initiative, specifically tailored to meet the needs of Europe's varied linguistic environment. This model has been trained on a dataset where over half consists of non-English texts, covering all 24 official languages of the European Union, which ensures it performs well across these languages. A significant advancement in Teuken-7B is its unique multilingual tokenizer, which has been fine-tuned for European languages, leading to enhanced training efficiency and lower inference costs when compared to conventional monolingual tokenizers. Users can access two versions of the model: Teuken-7B-Base, which serves as the basic pre-trained version, and Teuken-7B-Instruct, which has received instruction tuning aimed at boosting its ability to respond to user requests. Both models are readily available on Hugging Face, fostering an environment of transparency and collaboration within the artificial intelligence community while also encouraging further innovation. The creation of Teuken-7B highlights a dedication to developing AI solutions that embrace and represent the rich diversity found across Europe.

The multilingual T5 (mT5) is a highly versatile pretrained text-to-text transformer model, developed using a methodology akin to that of T5. This repository serves as a resource for replicating the findings outlined in the mT5 research paper. mT5 has been trained on the extensive mC4 corpus, which encompasses 101 different languages, including but not limited to Afrikaans, Albanian, Amharic, Arabic, Armenian, Azerbaijani, Basque, Belarusian, Bengali, Bulgarian, Burmese, Catalan, Cebuano, Chichewa, Chinese, Corsican, Czech, Danish, Dutch, English, Esperanto, Estonian, Filipino, Finnish, French, Galician, Georgian, German, Greek, Gujarati, Haitian Creole, Hausa, Hawaiian, Hebrew, Hindi, Hmong, Hungarian, Icelandic, Igbo, Indonesian, Irish, Italian, Japanese, Javanese, Kannada, Kazakh, Khmer, Korean, Kurdish, Kyrgyz, Lao, Latin, Latvian, Lithuanian, Luxembourgish, Macedonian, Malagasy, Malay, Malayalam, Maltese, Maori, Marathi, Mongolian, Nepali, Norwegian, Pashto, Persian, Polish, Portuguese, Punjabi, Romanian, Russian, Samoan, Scottish Gaelic, Serbian, Shona, Sindhi, and many others. This impressive range of languages makes mT5 a valuable tool for multilingual applications across various fields.

We are excited to announce the launch of Phi-2, a language model featuring 2.7 billion parameters that excels in reasoning and language comprehension, achieving top-tier results compared to other base models with fewer than 13 billion parameters. In challenging benchmarks, Phi-2 competes with and often surpasses models that are up to 25 times its size, a feat made possible by advancements in model scaling and meticulous curation of training data. Due to its efficient design, Phi-2 serves as an excellent resource for researchers interested in areas such as mechanistic interpretability, enhancing safety measures, or conducting fine-tuning experiments across a broad spectrum of tasks. To promote further exploration and innovation in language modeling, Phi-2 has been integrated into the Azure AI Studio model catalog, encouraging collaboration and development within the research community. Researchers can leverage this model to unlock new insights and push the boundaries of language technology.

Yi-Large is an innovative proprietary large language model created by 01.AI, featuring an impressive context length of 32k and a cost structure of $2 for each million tokens for both inputs and outputs. Renowned for its superior natural language processing abilities, common-sense reasoning, and support for multiple languages, it competes effectively with top models such as GPT-4 and Claude3 across various evaluations. This model is particularly adept at handling tasks that involve intricate inference, accurate prediction, and comprehensive language comprehension, making it ideal for applications such as knowledge retrieval, data categorization, and the development of conversational chatbots that mimic human interaction. Built on a decoder-only transformer architecture, Yi-Large incorporates advanced features like pre-normalization and Group Query Attention, and it has been trained on an extensive, high-quality multilingual dataset to enhance its performance. The model's flexibility and economical pricing position it as a formidable player in the artificial intelligence landscape, especially for businesses looking to implement AI technologies on a global scale. Additionally, its ability to adapt to a wide range of use cases underscores its potential to revolutionize how organizations leverage language models for various needs.

LLaVA, or Large Language-and-Vision Assistant, represents a groundbreaking multimodal model that combines a vision encoder with the Vicuna language model, enabling enhanced understanding of both visual and textual information. By employing end-to-end training, LLaVA showcases remarkable conversational abilities, mirroring the multimodal features found in models such as GPT-4. Significantly, LLaVA-1.5 has reached cutting-edge performance on 11 different benchmarks, leveraging publicly accessible data and achieving completion of its training in about one day on a single 8-A100 node, outperforming approaches that depend on massive datasets. The model's development included the construction of a multimodal instruction-following dataset, which was produced using a language-only variant of GPT-4. This dataset consists of 158,000 distinct language-image instruction-following examples, featuring dialogues, intricate descriptions, and advanced reasoning challenges. Such a comprehensive dataset has played a crucial role in equipping LLaVA to handle a diverse range of tasks related to vision and language with great efficiency. In essence, LLaVA not only enhances the interaction between visual and textual modalities but also sets a new benchmark in the field of multimodal AI.

Mistral Large stands as the premier language model from Mistral AI, engineered for sophisticated text generation and intricate multilingual reasoning tasks such as text comprehension, transformation, and programming code development. This model encompasses support for languages like English, French, Spanish, German, and Italian, which allows it to grasp grammar intricacies and cultural nuances effectively. With an impressive context window of 32,000 tokens, Mistral Large can retain and reference information from lengthy documents with accuracy. Its abilities in precise instruction adherence and native function-calling enhance the development of applications and the modernization of tech stacks. Available on Mistral's platform, Azure AI Studio, and Azure Machine Learning, it also offers the option for self-deployment, catering to sensitive use cases. Benchmarks reveal that Mistral Large performs exceptionally well, securing its position as the second-best model globally that is accessible via an API, just behind GPT-4, illustrating its competitive edge in the AI landscape. Such capabilities make it an invaluable tool for developers seeking to leverage advanced AI technology.

Falcon 2 11B is a versatile AI model that is open-source, supports multiple languages, and incorporates multimodal features, particularly excelling in vision-to-language tasks. It outperforms Meta’s Llama 3 8B and matches the capabilities of Google’s Gemma 7B, as validated by the Hugging Face Leaderboard. In the future, the development plan includes adopting a 'Mixture of Experts' strategy aimed at significantly improving the model's functionalities, thereby advancing the frontiers of AI technology even further. This evolution promises to deliver remarkable innovations, solidifying Falcon 2's position in the competitive landscape of artificial intelligence.

Qwen LLM represents a collection of advanced large language models created by Alibaba Cloud's Damo Academy. These models leverage an extensive dataset comprising text and code, enabling them to produce human-like text, facilitate language translation, craft various forms of creative content, and provide informative answers to queries. Key attributes of Qwen LLMs include: A range of sizes: The Qwen series features models with parameters varying from 1.8 billion to 72 billion, catering to diverse performance requirements and applications. Open source availability: Certain versions of Qwen are open-source, allowing users to access and modify the underlying code as needed. Multilingual capabilities: Qwen is equipped to comprehend and translate several languages, including English, Chinese, and French. Versatile functionalities: In addition to language generation and translation, Qwen models excel in tasks such as answering questions, summarizing texts, and generating code, making them highly adaptable tools for various applications. Overall, the Qwen LLM family stands out for its extensive capabilities and flexibility in meeting user needs.

EXAONE Deep represents a collection of advanced language models that are enhanced for reasoning, created by LG AI Research, and come in sizes of 2.4 billion, 7.8 billion, and 32 billion parameters. These models excel in a variety of reasoning challenges, particularly in areas such as mathematics and coding assessments. Significantly, the EXAONE Deep 2.4B model outshines other models of its size, while the 7.8B variant outperforms both open-weight models of similar dimensions and the proprietary reasoning model known as OpenAI o1-mini. Furthermore, the EXAONE Deep 32B model competes effectively with top-tier open-weight models in the field. The accompanying repository offers extensive documentation that includes performance assessments, quick-start guides for leveraging EXAONE Deep models with the Transformers library, detailed explanations of quantized EXAONE Deep weights formatted in AWQ and GGUF, as well as guidance on how to run these models locally through platforms like llama.cpp and Ollama. Additionally, this resource serves to enhance user understanding and accessibility to the capabilities of EXAONE Deep models.

Smaug-72B is a formidable open-source large language model (LLM) distinguished by several prominent features: Exceptional Performance: It currently ranks first on the Hugging Face Open LLM leaderboard, outperforming models such as GPT-3.5 in multiple evaluations, demonstrating its ability to comprehend, react to, and generate text that closely resembles human writing. Open Source Availability: In contrast to many high-end LLMs, Smaug-72B is accessible to everyone for use and modification, which encourages cooperation and innovation within the AI ecosystem. Emphasis on Reasoning and Mathematics: This model excels particularly in reasoning and mathematical challenges, a capability attributed to specialized fine-tuning methods developed by its creators, Abacus AI. Derived from Qwen-72B: It is essentially a refined version of another robust LLM, Qwen-72B, which was launched by Alibaba, thereby enhancing its overall performance. In summary, Smaug-72B marks a notable advancement in the realm of open-source artificial intelligence, making it a valuable resource for developers and researchers alike. Its unique strengths not only elevate its status but also contribute to the ongoing evolution of AI technology.

Reka Flash 3 is a cutting-edge multimodal AI model with 21 billion parameters, crafted by Reka AI to perform exceptionally well in tasks such as general conversation, coding, following instructions, and executing functions. This model adeptly handles and analyzes a myriad of inputs, including text, images, video, and audio, providing a versatile and compact solution for a wide range of applications. Built from the ground up, Reka Flash 3 was trained on a rich array of datasets, encompassing both publicly available and synthetic information, and it underwent a meticulous instruction tuning process with high-quality selected data to fine-tune its capabilities. The final phase of its training involved employing reinforcement learning techniques, specifically using the REINFORCE Leave One-Out (RLOO) method, which combined both model-based and rule-based rewards to significantly improve its reasoning skills. With an impressive context length of 32,000 tokens, Reka Flash 3 competes effectively with proprietary models like OpenAI's o1-mini, making it an excellent choice for applications requiring low latency or on-device processing. The model operates at full precision with a memory requirement of 39GB (fp16), although it can be efficiently reduced to just 11GB through the use of 4-bit quantization, demonstrating its adaptability for various deployment scenarios. Overall, Reka Flash 3 represents a significant advancement in multimodal AI technology, capable of meeting diverse user needs across multiple platforms.

OpenLLaMA is an openly licensed reproduction of Meta AI's LLaMA 7B, developed using the RedPajama dataset. The model weights we offer can seamlessly replace the LLaMA 7B in current applications. Additionally, we have created a more compact 3B version of the LLaMA model for those seeking a lighter alternative. This provides users with more flexibility in choosing the right model for their specific needs.