Alternatives to Falcon-40B

OpenLLaMA is an openly licensed reproduction of Meta AI's LLaMA 7B, developed using the RedPajama dataset. The model weights we offer can seamlessly replace the LLaMA 7B in current applications. Additionally, we have created a more compact 3B version of the LLaMA model for those seeking a lighter alternative. This provides users with more flexibility in choosing the right model for their specific needs.

Instruction-following models like GPT-3.5 (text-DaVinci-003), ChatGPT, Claude, and Bing Chat have seen significant advancements in their capabilities, leading to a rise in their usage among individuals in both personal and professional contexts. Despite their growing popularity and integration into daily tasks, these models are not without their shortcomings, as they can sometimes disseminate inaccurate information, reinforce harmful stereotypes, and use inappropriate language. To effectively tackle these critical issues, it is essential for researchers and scholars to become actively involved in exploring these models further. However, conducting research on instruction-following models within academic settings has posed challenges due to the unavailability of models with comparable functionality to proprietary options like OpenAI’s text-DaVinci-003. In response to this gap, we are presenting our insights on an instruction-following language model named Alpaca, which has been fine-tuned from Meta’s LLaMA 7B model, aiming to contribute to the discourse and development in this field. This initiative represents a step towards enhancing the understanding and capabilities of instruction-following models in a more accessible manner for researchers.

Foundation models, including GPT-4, have significantly accelerated advancements in artificial intelligence, yet the most advanced models remain either proprietary or only partially accessible. In response to this challenge, the RedPajama initiative aims to develop a collection of top-tier, fully open-source models. We are thrilled to announce that we have successfully completed the initial phase of this endeavor: recreating the LLaMA training dataset, which contains over 1.2 trillion tokens. Currently, many of the leading foundation models are locked behind commercial APIs, restricting opportunities for research, customization, and application with sensitive information. The development of fully open-source models represents a potential solution to these limitations, provided that the open-source community can bridge the gap in quality between open and closed models. Recent advancements have shown promising progress in this area, suggesting that the AI field is experiencing a transformative period akin to the emergence of Linux. The success of Stable Diffusion serves as a testament to the fact that open-source alternatives can not only match the quality of commercial products like DALL-E but also inspire remarkable creativity through the collaborative efforts of diverse communities. By fostering an open-source ecosystem, we can unlock new possibilities for innovation and ensure broader access to cutting-edge AI technology.

Easily train and deploy large-scale AI models with just a single command by pointing to your S3 bucket—then let us take care of everything else, including orchestration, efficiency, node failures, and infrastructure management. The process is straightforward and scalable, allowing you to utilize MosaicML to train and serve large AI models using your own data within your secure environment. Stay ahead of the curve with our up-to-date recipes, techniques, and foundation models, all developed and thoroughly tested by our dedicated research team. With only a few simple steps, you can deploy your models within your private cloud, ensuring that your data and models remain behind your own firewalls. You can initiate your project in one cloud provider and seamlessly transition to another without any disruptions. Gain ownership of the model trained on your data while being able to introspect and clarify the decisions made by the model. Customize content and data filtering to align with your business requirements, and enjoy effortless integration with your existing data pipelines, experiment trackers, and other essential tools. Our solution is designed to be fully interoperable, cloud-agnostic, and validated for enterprise use, ensuring reliability and flexibility for your organization. Additionally, the ease of use and the power of our platform allow teams to focus more on innovation rather than infrastructure management.

Introducing the next iteration of our open-source large language model, this version features model weights along with initial code for the pretrained and fine-tuned Llama language models, which span from 7 billion to 70 billion parameters. The Llama 2 pretrained models have been developed using an impressive 2 trillion tokens and offer double the context length compared to their predecessor, Llama 1. Furthermore, the fine-tuned models have been enhanced through the analysis of over 1 million human annotations. Llama 2 demonstrates superior performance against various other open-source language models across multiple external benchmarks, excelling in areas such as reasoning, coding capabilities, proficiency, and knowledge assessments. For its training, Llama 2 utilized publicly accessible online data sources, while the fine-tuned variant, Llama-2-chat, incorporates publicly available instruction datasets along with the aforementioned extensive human annotations. Our initiative enjoys strong support from a diverse array of global stakeholders who are enthusiastic about our open approach to AI, including companies that have provided valuable early feedback and are eager to collaborate using Llama 2. The excitement surrounding Llama 2 signifies a pivotal shift in how AI can be developed and utilized collectively.

Falcon-7B is a causal decoder-only model comprising 7 billion parameters, developed by TII and trained on an extensive dataset of 1,500 billion tokens from RefinedWeb, supplemented with specially selected corpora, and it is licensed under Apache 2.0. What are the advantages of utilizing Falcon-7B? This model surpasses similar open-source alternatives, such as MPT-7B, StableLM, and RedPajama, due to its training on a remarkably large dataset of 1,500 billion tokens from RefinedWeb, which is further enhanced with carefully curated content, as evidenced by its standing on the OpenLLM Leaderboard. Additionally, it boasts an architecture that is finely tuned for efficient inference, incorporating technologies like FlashAttention and multiquery mechanisms. Moreover, the permissive nature of the Apache 2.0 license means users can engage in commercial applications without incurring royalties or facing significant limitations. This combination of performance and flexibility makes Falcon-7B a strong choice for developers seeking advanced modeling capabilities.

We are excited to present MPT-7B, the newest addition to the MosaicML Foundation Series. This transformer model has been meticulously trained from the ground up using 1 trillion tokens of diverse text and code. It is open-source and ready for commercial applications, delivering performance on par with LLaMA-7B. The training process took 9.5 days on the MosaicML platform, requiring no human input and incurring an approximate cost of $200,000. With MPT-7B, you can now train, fine-tune, and launch your own customized MPT models, whether you choose to begin with one of our provided checkpoints or start anew. To provide additional options, we are also introducing three fine-tuned variants alongside the base MPT-7B: MPT-7B-Instruct, MPT-7B-Chat, and MPT-7B-StoryWriter-65k+, the latter boasting an impressive context length of 65,000 tokens, allowing for extensive content generation. These advancements open up new possibilities for developers and researchers looking to leverage the power of transformer models in their projects.

Magic’s LTM-1 technology facilitates context windows that are 50 times larger than those typically used in transformer models. As a result, Magic has developed a Large Language Model (LLM) that can effectively process vast amounts of contextual information when providing suggestions. This advancement allows our coding assistant to access and analyze your complete code repository. With the ability to reference extensive factual details and their own prior actions, larger context windows can significantly enhance the reliability and coherence of AI outputs. We are excited about the potential of this research to further improve user experience in coding assistance applications.

Mistral 7B is a language model with 7.3 billion parameters that demonstrates superior performance compared to larger models such as Llama 2 13B on a variety of benchmarks. It utilizes innovative techniques like Grouped-Query Attention (GQA) for improved inference speed and Sliding Window Attention (SWA) to manage lengthy sequences efficiently. Released under the Apache 2.0 license, Mistral 7B is readily available for deployment on different platforms, including both local setups and prominent cloud services. Furthermore, a specialized variant known as Mistral 7B Instruct has shown remarkable capabilities in following instructions, outperforming competitors like Llama 2 13B Chat in specific tasks. This versatility makes Mistral 7B an attractive option for developers and researchers alike.

Vicuna-13B is an open-source conversational agent developed through the fine-tuning of LLaMA, utilizing a dataset of user-shared dialogues gathered from ShareGPT. Initial assessments, with GPT-4 serving as an evaluator, indicate that Vicuna-13B achieves over 90% of the quality exhibited by OpenAI's ChatGPT and Google Bard, and it surpasses other models such as LLaMA and Stanford Alpaca in more than 90% of instances. The entire training process for Vicuna-13B incurs an estimated expenditure of approximately $300. Additionally, the source code and model weights, along with an interactive demonstration, are made available for public access under non-commercial terms, fostering a collaborative environment for further development and exploration. This openness encourages innovation and enables users to experiment with the model's capabilities in diverse applications.

Mistral Small 3.1 represents a cutting-edge, multimodal, and multilingual AI model that has been released under the Apache 2.0 license. This upgraded version builds on Mistral Small 3, featuring enhanced text capabilities and superior multimodal comprehension, while also accommodating an extended context window of up to 128,000 tokens. It demonstrates superior performance compared to similar models such as Gemma 3 and GPT-4o Mini, achieving impressive inference speeds of 150 tokens per second. Tailored for adaptability, Mistral Small 3.1 shines in a variety of applications, including instruction following, conversational support, image analysis, and function execution, making it ideal for both business and consumer AI needs. The model's streamlined architecture enables it to operate efficiently on hardware such as a single RTX 4090 or a Mac equipped with 32GB of RAM, thus supporting on-device implementations. Users can download it from Hugging Face and access it through Mistral AI's developer playground, while it is also integrated into platforms like Google Cloud Vertex AI, with additional accessibility on NVIDIA NIM and more. This flexibility ensures that developers can leverage its capabilities across diverse environments and applications.

The Mixtral 8x7B model is an advanced sparse mixture of experts (SMoE) system that boasts open weights and is released under the Apache 2.0 license. This model demonstrates superior performance compared to Llama 2 70B across various benchmarks while achieving inference speeds that are six times faster. Recognized as the leading open-weight model with a flexible licensing framework, Mixtral also excels in terms of cost-efficiency and performance. Notably, it competes with and often surpasses GPT-3.5 in numerous established benchmarks, highlighting its significance in the field. Its combination of accessibility, speed, and effectiveness makes it a compelling choice for developers seeking high-performing AI solutions.

Introducing Mistral NeMo, our latest and most advanced small model yet, featuring a cutting-edge 12 billion parameters and an expansive context length of 128,000 tokens, all released under the Apache 2.0 license. Developed in partnership with NVIDIA, Mistral NeMo excels in reasoning, world knowledge, and coding proficiency within its category. Its architecture adheres to industry standards, making it user-friendly and a seamless alternative for systems currently utilizing Mistral 7B. To facilitate widespread adoption among researchers and businesses, we have made available both pre-trained base and instruction-tuned checkpoints under the same Apache license. Notably, Mistral NeMo incorporates quantization awareness, allowing for FP8 inference without compromising performance. The model is also tailored for diverse global applications, adept in function calling and boasting a substantial context window. When compared to Mistral 7B, Mistral NeMo significantly outperforms in understanding and executing detailed instructions, showcasing enhanced reasoning skills and the ability to manage complex multi-turn conversations. Moreover, its design positions it as a strong contender for multi-lingual tasks, ensuring versatility across various use cases.

Falcon 2 11B is a versatile AI model that is open-source, supports multiple languages, and incorporates multimodal features, particularly excelling in vision-to-language tasks. It outperforms Meta’s Llama 3 8B and matches the capabilities of Google’s Gemma 7B, as validated by the Hugging Face Leaderboard. In the future, the development plan includes adopting a 'Mixture of Experts' strategy aimed at significantly improving the model's functionalities, thereby advancing the frontiers of AI technology even further. This evolution promises to deliver remarkable innovations, solidifying Falcon 2's position in the competitive landscape of artificial intelligence.

This repository showcases the research preview of LongLLaMA, an advanced large language model that can manage extensive contexts of up to 256,000 tokens or potentially more. LongLLaMA is developed on the OpenLLaMA framework and has been fine-tuned utilizing the Focused Transformer (FoT) technique. The underlying code for LongLLaMA is derived from Code Llama. We are releasing a smaller 3B base variant of the LongLLaMA model, which is not instruction-tuned, under an open license (Apache 2.0), along with inference code that accommodates longer contexts available on Hugging Face. This model's weights can seamlessly replace LLaMA in existing systems designed for shorter contexts, specifically those handling up to 2048 tokens. Furthermore, we include evaluation results along with comparisons to the original OpenLLaMA models, thereby providing a comprehensive overview of LongLLaMA's capabilities in the realm of long-context processing.

Teuken-7B is a multilingual language model that has been developed as part of the OpenGPT-X initiative, specifically tailored to meet the needs of Europe's varied linguistic environment. This model has been trained on a dataset where over half consists of non-English texts, covering all 24 official languages of the European Union, which ensures it performs well across these languages. A significant advancement in Teuken-7B is its unique multilingual tokenizer, which has been fine-tuned for European languages, leading to enhanced training efficiency and lower inference costs when compared to conventional monolingual tokenizers. Users can access two versions of the model: Teuken-7B-Base, which serves as the basic pre-trained version, and Teuken-7B-Instruct, which has received instruction tuning aimed at boosting its ability to respond to user requests. Both models are readily available on Hugging Face, fostering an environment of transparency and collaboration within the artificial intelligence community while also encouraging further innovation. The creation of Teuken-7B highlights a dedication to developing AI solutions that embrace and represent the rich diversity found across Europe.

In honor of Cleopatra, whose magnificent fate concluded amidst the tragic incident involving a snake, we are excited to introduce Codestral Mamba, a Mamba2 language model specifically designed for code generation and released under an Apache 2.0 license. Codestral Mamba represents a significant advancement in our ongoing initiative to explore and develop innovative architectures. It is freely accessible for use, modification, and distribution, and we aspire for it to unlock new avenues in architectural research. The Mamba models are distinguished by their linear time inference capabilities and their theoretical potential to handle sequences of infinite length. This feature enables users to interact with the model effectively, providing rapid responses regardless of input size. Such efficiency is particularly advantageous for enhancing code productivity; therefore, we have equipped this model with sophisticated coding and reasoning skills, allowing it to perform competitively with state-of-the-art transformer-based models. As we continue to innovate, we believe Codestral Mamba will inspire further advancements in the coding community.

Canopy Labs has unveiled Orpheus, an innovative suite of advanced speech large language models (LLMs) aimed at achieving human-like speech generation capabilities. Utilizing the Llama-3 architecture, these models have been trained on an extensive dataset comprising over 100,000 hours of English speech, allowing them to generate speech that exhibits natural intonation, emotional depth, and rhythmic flow that outperforms existing high-end closed-source alternatives. Orpheus also features zero-shot voice cloning, enabling users to mimic voices without any need for prior fine-tuning, and provides easy-to-use tags for controlling emotion and intonation. The models are engineered for low latency, achieving approximately 200ms streaming latency for real-time usage, which can be further decreased to around 100ms when utilizing input streaming. Canopy Labs has made available both pre-trained and fine-tuned models with 3 billion parameters under the flexible Apache 2.0 license, with future intentions to offer smaller models with 1 billion, 400 million, and 150 million parameters to cater to devices with limited resources. This strategic move is expected to broaden accessibility and application potential across various platforms and use cases.

The MiniMax‑M1 model, introduced by MiniMax AI and licensed under Apache 2.0, represents a significant advancement in hybrid-attention reasoning architecture. With an extraordinary capacity for handling a 1 million-token context window and generating outputs of up to 80,000 tokens, it facilitates in-depth analysis of lengthy texts. Utilizing a cutting-edge CISPO algorithm, MiniMax‑M1 was trained through extensive reinforcement learning, achieving completion on 512 H800 GPUs in approximately three weeks. This model sets a new benchmark in performance across various domains, including mathematics, programming, software development, tool utilization, and understanding of long contexts, either matching or surpassing the capabilities of leading models in the field. Additionally, users can choose between two distinct variants of the model, each with a thinking budget of either 40K or 80K, and access the model's weights and deployment instructions on platforms like GitHub and Hugging Face. Such features make MiniMax‑M1 a versatile tool for developers and researchers alike.

GPT-J represents an advanced language model developed by EleutherAI, known for its impressive capabilities. When it comes to performance, GPT-J showcases a proficiency that rivals OpenAI's well-known GPT-3 in various zero-shot tasks. Remarkably, it has even outperformed GPT-3 in specific areas, such as code generation. The most recent version of this model, called GPT-J-6B, is constructed using a comprehensive linguistic dataset known as The Pile, which is publicly accessible and consists of an extensive 825 gibibytes of language data divided into 22 unique subsets. Although GPT-J possesses similarities to ChatGPT, it's crucial to highlight that it is primarily intended for text prediction rather than functioning as a chatbot. In a notable advancement in March 2023, Databricks unveiled Dolly, a model that is capable of following instructions and operates under an Apache license, further enriching the landscape of language models. This evolution in AI technology continues to push the boundaries of what is possible in natural language processing.

Qwen2.5-1M, an open-source language model from the Qwen team, has been meticulously crafted to manage context lengths reaching as high as one million tokens. This version introduces two distinct model variants, namely Qwen2.5-7B-Instruct-1M and Qwen2.5-14B-Instruct-1M, representing a significant advancement as it is the first instance of Qwen models being enhanced to accommodate such large context lengths. In addition to this, the team has released an inference framework that is based on vLLM and incorporates sparse attention mechanisms, which greatly enhance the processing speed for 1M-token inputs, achieving improvements between three to seven times. A detailed technical report accompanies this release, providing in-depth insights into the design choices and the results from various ablation studies. This transparency allows users to fully understand the capabilities and underlying technology of the models.

IBM® Granite™ comprises a suite of AI models specifically designed for business applications, built from the ground up to prioritize trust and scalability in AI implementations. Currently, the open-source Granite models can be accessed. Our goal is to make AI widely available to as many developers as possible, which is why we have released the essential Granite Code, as well as Time Series, Language, and GeoSpatial models as open-source on Hugging Face, under the permissive Apache 2.0 license, allowing extensive commercial use without restrictions. Every Granite model is developed using meticulously selected data, ensuring exceptional transparency regarding the sources of the training data. Additionally, we have made the tools that validate and maintain the quality of this data accessible to the public, meeting the rigorous standards required for enterprise-level applications. This commitment to openness and quality reflects our dedication to fostering innovation in the AI landscape.

We present the Granite series of decoder-only code models specifically designed for tasks involving code generation, such as debugging, code explanation, and documentation, utilizing programming languages across a spectrum of 116 different types. An extensive assessment of the Granite Code model family across various tasks reveals that these models consistently achieve leading performance compared to other open-source code language models available today. Among the notable strengths of Granite Code models are: Versatile Code LLM: The Granite Code models deliver competitive or top-tier results across a wide array of code-related tasks, which include code generation, explanation, debugging, editing, translation, and beyond, showcasing their capacity to handle various coding challenges effectively. Additionally, their adaptability makes them suitable for both simple and complex coding scenarios. Reliable Enterprise-Grade LLM: All models in this series are developed using data that complies with licensing requirements and is gathered in alignment with IBM's AI Ethics guidelines, ensuring trustworthy usage for enterprise applications.

Large language models, often requiring extensive computational resources for training over long periods, have demonstrated impressive proficiency in zero- and few-shot learning tasks. Due to the high investment needed for their development, replicating these models poses a significant challenge for many researchers. Furthermore, access to the few models available via API is limited, as users cannot obtain the complete model weights, complicating academic exploration. In response to this, we introduce Open Pre-trained Transformers (OPT), a collection of decoder-only pre-trained transformers ranging from 125 million to 175 billion parameters, which we intend to share comprehensively and responsibly with interested scholars. Our findings indicate that OPT-175B exhibits performance on par with GPT-3, yet it is developed with only one-seventh of the carbon emissions required for GPT-3's training. Additionally, we will provide a detailed logbook that outlines the infrastructure hurdles we encountered throughout the project, as well as code to facilitate experimentation with all released models, ensuring that researchers have the tools they need to explore this technology further.

Introducing an open-source AI model that can be fine-tuned, distilled, and deployed across various platforms. Our newest instruction-tuned model comes in three sizes: 8B, 70B, and 405B, giving you options to suit different needs. With our open ecosystem, you can expedite your development process using a diverse array of tailored product offerings designed to meet your specific requirements. You have the flexibility to select between real-time inference and batch inference services according to your project's demands. Additionally, you can download model weights to enhance cost efficiency per token while fine-tuning for your application. Improve performance further by utilizing synthetic data and seamlessly deploy your solutions on-premises or in the cloud. Take advantage of Llama system components and expand the model's capabilities through zero-shot tool usage and retrieval-augmented generation (RAG) to foster agentic behaviors. By utilizing 405B high-quality data, you can refine specialized models tailored to distinct use cases, ensuring optimal functionality for your applications. Ultimately, this empowers developers to create innovative solutions that are both efficient and effective.

Falcon Mamba 7B marks a significant milestone as the inaugural open-source State Space Language Model (SSLM), presenting a revolutionary architecture within the Falcon model family. Celebrated as the premier open-source SSLM globally by Hugging Face, it establishes a new standard for efficiency in artificial intelligence. In contrast to conventional transformers, SSLMs require significantly less memory and can produce lengthy text sequences seamlessly without extra resource demands. Falcon Mamba 7B outperforms top transformer models, such as Meta’s Llama 3.1 8B and Mistral’s 7B, demonstrating enhanced capabilities. This breakthrough not only highlights Abu Dhabi’s dedication to pushing the boundaries of AI research but also positions the region as a pivotal player in the global AI landscape. Such advancements are vital for fostering innovation and collaboration in technology.

Pixtral Large is an expansive multimodal model featuring 124 billion parameters, crafted by Mistral AI and enhancing their previous Mistral Large 2 framework. This model combines a 123-billion-parameter multimodal decoder with a 1-billion-parameter vision encoder, allowing it to excel in the interpretation of various content types, including documents, charts, and natural images, all while retaining superior text comprehension abilities. With the capability to manage a context window of 128,000 tokens, Pixtral Large can efficiently analyze at least 30 high-resolution images at once. It has achieved remarkable results on benchmarks like MathVista, DocVQA, and VQAv2, outpacing competitors such as GPT-4o and Gemini-1.5 Pro. Available for research and educational purposes under the Mistral Research License, it also has a Mistral Commercial License for business applications. This versatility makes Pixtral Large a valuable tool for both academic research and commercial innovations.

Fullmoon is an innovative, open-source application designed to allow users to engage directly with large language models on their personal devices, prioritizing privacy and enabling offline use. Tailored specifically for Apple silicon, it functions smoothly across various platforms, including iOS, iPadOS, macOS, and visionOS. Users have the ability to customize their experience by modifying themes, fonts, and system prompts, while the app also works seamlessly with Apple's Shortcuts to enhance user productivity. Notably, Fullmoon is compatible with models such as Llama-3.2-1B-Instruct-4bit and Llama-3.2-3B-Instruct-4bit, allowing for effective AI interactions without requiring internet connectivity. This makes it a versatile tool for anyone looking to harness the power of AI conveniently and privately.

Janus-Pro-7B is a groundbreaking open-source multimodal AI model developed by DeepSeek, expertly crafted to both comprehend and create content involving text, images, and videos. Its distinctive autoregressive architecture incorporates dedicated pathways for visual encoding, which enhances its ability to tackle a wide array of tasks, including text-to-image generation and intricate visual analysis. Demonstrating superior performance against rivals such as DALL-E 3 and Stable Diffusion across multiple benchmarks, it boasts scalability with variants ranging from 1 billion to 7 billion parameters. Released under the MIT License, Janus-Pro-7B is readily accessible for use in both academic and commercial contexts, marking a substantial advancement in AI technology. Furthermore, this model can be utilized seamlessly on popular operating systems such as Linux, MacOS, and Windows via Docker, broadening its reach and usability in various applications.

The upcoming GPT-5 is the next version in OpenAI's series of Generative Pre-trained Transformers, which remains under development. These advanced language models are built on vast datasets, enabling them to produce realistic and coherent text, translate between languages, create various forms of creative content, and provide informative answers to inquiries. As of now, it is not available to the public, and although OpenAI has yet to disclose an official launch date, there is speculation that its release could occur in 2024. This iteration is anticipated to significantly outpace its predecessor, GPT-4, which is already capable of generating text that resembles human writing, translating languages, and crafting a wide range of creative pieces. The expectations for GPT-5 include enhanced reasoning skills, improved factual accuracy, and a superior ability to adhere to user instructions, making it a highly anticipated advancement in the field. Overall, the development of GPT-5 represents a considerable leap forward in the capabilities of AI language processing.

Falcon 3 is a large language model that has been made open-source by the Technology Innovation Institute (TII), aiming to broaden access to advanced AI capabilities. Its design prioritizes efficiency, enabling it to function effectively on lightweight devices like laptops while maintaining high performance levels. The Falcon 3 suite includes four scalable models, each specifically designed for various applications and capable of supporting multiple languages while minimizing resource consumption. This new release in TII's LLM lineup sets a benchmark in reasoning, language comprehension, instruction adherence, coding, and mathematical problem-solving. By offering a blend of robust performance and resource efficiency, Falcon 3 seeks to democratize AI access, allowing users in numerous fields to harness sophisticated technology without the necessity for heavy computational power. Furthermore, this initiative not only enhances individual capabilities but also fosters innovation across different sectors by making advanced AI tools readily available.

Phi-4 is an advanced small language model (SLM) comprising 14 billion parameters, showcasing exceptional capabilities in intricate reasoning tasks, particularly in mathematics, alongside typical language processing functions. As the newest addition to the Phi family of small language models, Phi-4 illustrates the potential advancements we can achieve while exploring the limits of SLM technology. It is currently accessible on Azure AI Foundry under a Microsoft Research License Agreement (MSRLA) and is set to be released on Hugging Face in the near future. Due to significant improvements in processes such as the employment of high-quality synthetic datasets and the careful curation of organic data, Phi-4 surpasses both comparable and larger models in mathematical reasoning tasks. This model not only emphasizes the ongoing evolution of language models but also highlights the delicate balance between model size and output quality. As we continue to innovate, Phi-4 stands as a testament to our commitment to pushing the boundaries of what's achievable within the realm of small language models.

Code Llama is an advanced language model designed to generate code through text prompts, distinguishing itself as a leading tool among publicly accessible models for coding tasks. This innovative model not only streamlines workflows for existing developers but also aids beginners in overcoming challenges associated with learning to code. Its versatility positions Code Llama as both a valuable productivity enhancer and an educational resource, assisting programmers in creating more robust and well-documented software solutions. Additionally, users can generate both code and natural language explanations by providing either type of prompt, making it an adaptable tool for various programming needs. Available for free for both research and commercial applications, Code Llama is built upon Llama 2 architecture and comes in three distinct versions: the foundational Code Llama model, Code Llama - Python which is tailored specifically for Python programming, and Code Llama - Instruct, optimized for comprehending and executing natural language directives effectively.

CodeQwen serves as the coding counterpart to Qwen, which is a series of large language models created by the Qwen team at Alibaba Cloud. Built on a transformer architecture that functions solely as a decoder, this model has undergone extensive pre-training using a vast dataset of code. It showcases robust code generation abilities and demonstrates impressive results across various benchmarking tests. With the capacity to comprehend and generate long contexts of up to 64,000 tokens, CodeQwen accommodates 92 programming languages and excels in tasks such as text-to-SQL queries and debugging. Engaging with CodeQwen is straightforward—you can initiate a conversation with just a few lines of code utilizing transformers. The foundation of this interaction relies on constructing the tokenizer and model using pre-existing methods, employing the generate function to facilitate dialogue guided by the chat template provided by the tokenizer. In alignment with our established practices, we implement the ChatML template tailored for chat models. This model adeptly completes code snippets based on the prompts it receives, delivering responses without the need for any further formatting adjustments, thereby enhancing the user experience. The seamless integration of these elements underscores the efficiency and versatility of CodeQwen in handling diverse coding tasks.

Sky-T1-32B-Preview is an innovative open-source reasoning model crafted by the NovaSky team at UC Berkeley's Sky Computing Lab. It delivers performance comparable to proprietary models such as o1-preview on various reasoning and coding assessments, while being developed at a cost of less than $450, highlighting the potential for budget-friendly, advanced reasoning abilities. Fine-tuned from Qwen2.5-32B-Instruct, the model utilized a meticulously curated dataset comprising 17,000 examples spanning multiple fields, such as mathematics and programming. The entire training process was completed in just 19 hours using eight H100 GPUs with DeepSpeed Zero-3 offloading technology. Every component of this initiative—including the data, code, and model weights—is entirely open-source, allowing both academic and open-source communities to not only replicate but also improve upon the model's capabilities. This accessibility fosters collaboration and innovation in the realm of artificial intelligence research and development.

Yi-Large is an innovative proprietary large language model created by 01.AI, featuring an impressive context length of 32k and a cost structure of $2 for each million tokens for both inputs and outputs. Renowned for its superior natural language processing abilities, common-sense reasoning, and support for multiple languages, it competes effectively with top models such as GPT-4 and Claude3 across various evaluations. This model is particularly adept at handling tasks that involve intricate inference, accurate prediction, and comprehensive language comprehension, making it ideal for applications such as knowledge retrieval, data categorization, and the development of conversational chatbots that mimic human interaction. Built on a decoder-only transformer architecture, Yi-Large incorporates advanced features like pre-normalization and Group Query Attention, and it has been trained on an extensive, high-quality multilingual dataset to enhance its performance. The model's flexibility and economical pricing position it as a formidable player in the artificial intelligence landscape, especially for businesses looking to implement AI technologies on a global scale. Additionally, its ability to adapt to a wide range of use cases underscores its potential to revolutionize how organizations leverage language models for various needs.

Hippocratic AI represents a cutting-edge advancement in artificial intelligence, surpassing GPT-4 on 105 out of 114 healthcare-related exams and certifications. Notably, it exceeded GPT-4's performance by at least five percent on 74 of these certifications, and on 43 of them, the margin was ten percent or greater. Unlike most language models that rely on a broad range of internet sources—which can sometimes include inaccurate information—Hippocratic AI is committed to sourcing evidence-based healthcare content through legal means. To ensure the model's effectiveness and safety, we are implementing a specialized Reinforcement Learning with Human Feedback process, involving healthcare professionals in training and validating the model before its release. This meticulous approach, dubbed RLHF-HP, guarantees that Hippocratic AI will only be launched after it receives the approval of a significant number of licensed healthcare experts, prioritizing patient safety and accuracy in its applications. The dedication to rigorous validation sets Hippocratic AI apart in the landscape of AI healthcare solutions.

Tülu 3 is a cutting-edge language model created by the Allen Institute for AI (Ai2) that aims to improve proficiency in fields like knowledge, reasoning, mathematics, coding, and safety. It is based on the Llama 3 Base and undergoes a detailed four-stage post-training regimen: careful prompt curation and synthesis, supervised fine-tuning on a wide array of prompts and completions, preference tuning utilizing both off- and on-policy data, and a unique reinforcement learning strategy that enhances targeted skills through measurable rewards. Notably, this open-source model sets itself apart by ensuring complete transparency, offering access to its training data, code, and evaluation tools, thus bridging the performance divide between open and proprietary fine-tuning techniques. Performance assessments reveal that Tülu 3 surpasses other models with comparable sizes, like Llama 3.1-Instruct and Qwen2.5-Instruct, across an array of benchmarks, highlighting its effectiveness. The continuous development of Tülu 3 signifies the commitment to advancing AI capabilities while promoting an open and accessible approach to technology.

We are excited to announce the launch of our X1 large series of models. The most robust model from Giga ML is now accessible for both pre-training and fine-tuning in an on-premises environment. Thanks to our compatibility with Open AI, existing integrations with tools like long chain, llama-index, and others function effortlessly. You can also proceed with pre-training LLMs using specialized data sources such as industry-specific documents or company files. The landscape of large language models (LLMs) is rapidly evolving, creating incredible opportunities for advancements in natural language processing across multiple fields. Despite this growth, several significant challenges persist in the industry. At Giga ML, we are thrilled to introduce the X1 Large 32k model, an innovative on-premise LLM solution designed specifically to tackle these pressing challenges, ensuring that organizations can harness the full potential of LLMs effectively. With this launch, we aim to empower businesses to elevate their language processing capabilities.

The latest iteration of the open-source AI model, which can be fine-tuned and deployed in various environments, is now offered in multiple versions, including 1B, 3B, 11B, and 90B, alongside the option to continue utilizing Llama 3.1. Llama 3.2 comprises a series of large language models (LLMs) that come pretrained and fine-tuned in 1B and 3B configurations for multilingual text only, while the 11B and 90B models accommodate both text and image inputs, producing text outputs. With this new release, you can create highly effective and efficient applications tailored to your needs. For on-device applications, such as summarizing phone discussions or accessing calendar tools, the 1B or 3B models are ideal choices. Meanwhile, the 11B or 90B models excel in image-related tasks, enabling you to transform existing images or extract additional information from images of your environment. Overall, this diverse range of models allows developers to explore innovative use cases across various domains.

OLMo 2 represents a collection of completely open language models created by the Allen Institute for AI (AI2), aimed at giving researchers and developers clear access to training datasets, open-source code, reproducible training methodologies, and thorough assessments. These models are trained on an impressive volume of up to 5 trillion tokens and compete effectively with top open-weight models like Llama 3.1, particularly in English academic evaluations. A key focus of OLMo 2 is on ensuring training stability, employing strategies to mitigate loss spikes during extended training periods, and applying staged training interventions in the later stages of pretraining to mitigate weaknesses in capabilities. Additionally, the models leverage cutting-edge post-training techniques derived from AI2's Tülu 3, leading to the development of OLMo 2-Instruct models. To facilitate ongoing enhancements throughout the development process, an actionable evaluation framework known as the Open Language Modeling Evaluation System (OLMES) was created, which includes 20 benchmarks that evaluate essential capabilities. This comprehensive approach not only fosters transparency but also encourages continuous improvement in language model performance.

The QwQ-32B model, created by Alibaba Cloud's Qwen team, represents a significant advancement in AI reasoning, aimed at improving problem-solving skills. Boasting 32 billion parameters, it rivals leading models such as DeepSeek's R1, which contains 671 billion parameters. This remarkable efficiency stems from its optimized use of parameters, enabling QwQ-32B to tackle complex tasks like mathematical reasoning, programming, and other problem-solving scenarios while consuming fewer resources. It can handle a context length of up to 32,000 tokens, making it adept at managing large volumes of input data. Notably, QwQ-32B is available through Alibaba's Qwen Chat service and is released under the Apache 2.0 license, which fosters collaboration and innovation among AI developers. With its cutting-edge features, QwQ-32B is poised to make a substantial impact in the field of artificial intelligence.

DeepSeek-V2 is a cutting-edge Mixture-of-Experts (MoE) language model developed by DeepSeek-AI, noted for its cost-effective training and high-efficiency inference features. It boasts an impressive total of 236 billion parameters, with only 21 billion active for each token, and is capable of handling a context length of up to 128K tokens. The model utilizes advanced architectures such as Multi-head Latent Attention (MLA) to optimize inference by minimizing the Key-Value (KV) cache and DeepSeekMoE to enable economical training through sparse computations. Compared to its predecessor, DeepSeek 67B, this model shows remarkable improvements, achieving a 42.5% reduction in training expenses, a 93.3% decrease in KV cache size, and a 5.76-fold increase in generation throughput. Trained on an extensive corpus of 8.1 trillion tokens, DeepSeek-V2 demonstrates exceptional capabilities in language comprehension, programming, and reasoning tasks, positioning it as one of the leading open-source models available today. Its innovative approach not only elevates its performance but also sets new benchmarks within the field of artificial intelligence.

QwQ-Max-Preview is a cutting-edge AI model based on the Qwen2.5-Max framework, specifically engineered to excel in areas such as complex reasoning, mathematical problem-solving, programming, and agent tasks. This preview showcases its enhanced capabilities across a variety of general-domain applications while demonstrating proficiency in managing intricate workflows. Anticipated to be officially released as open-source software under the Apache 2.0 license, QwQ-Max-Preview promises significant improvements and upgrades in its final iteration. Additionally, it contributes to the development of a more inclusive AI environment, as evidenced by the forthcoming introduction of the Qwen Chat application and streamlined model versions like QwQ-32B, which cater to developers interested in local deployment solutions. This initiative not only broadens accessibility but also encourages innovation within the AI community.

StarCoder and StarCoderBase represent advanced Large Language Models specifically designed for code, developed using openly licensed data from GitHub, which encompasses over 80 programming languages, Git commits, GitHub issues, and Jupyter notebooks. In a manner akin to LLaMA, we constructed a model with approximately 15 billion parameters trained on a staggering 1 trillion tokens. Furthermore, we tailored the StarCoderBase model with 35 billion Python tokens, leading to the creation of what we now refer to as StarCoder. Our evaluations indicated that StarCoderBase surpasses other existing open Code LLMs when tested against popular programming benchmarks and performs on par with or even exceeds proprietary models like code-cushman-001 from OpenAI, the original Codex model that fueled early iterations of GitHub Copilot. With an impressive context length exceeding 8,000 tokens, the StarCoder models possess the capability to handle more information than any other open LLM, thus paving the way for a variety of innovative applications. This versatility is highlighted by our ability to prompt the StarCoder models through a sequence of dialogues, effectively transforming them into dynamic technical assistants that can provide support in diverse programming tasks.