Average Ratings 0 Ratings
Average Ratings 0 Ratings
Description
Recent breakthroughs in natural language processing, comprehension, and generation have been greatly influenced by the development of large language models. This research presents a system that employs Ascend 910 AI processors and the MindSpore framework to train a language model exceeding one trillion parameters, specifically 1.085 trillion, referred to as PanGu-{\Sigma}. This model enhances the groundwork established by PanGu-{\alpha} by converting the conventional dense Transformer model into a sparse format through a method known as Random Routed Experts (RRE). Utilizing a substantial dataset of 329 billion tokens, the model was effectively trained using a strategy called Expert Computation and Storage Separation (ECSS), which resulted in a remarkable 6.3-fold improvement in training throughput through the use of heterogeneous computing. Through various experiments, it was found that PanGu-{\Sigma} achieves a new benchmark in zero-shot learning across multiple downstream tasks in Chinese NLP, showcasing its potential in advancing the field. This advancement signifies a major leap forward in the capabilities of language models, illustrating the impact of innovative training techniques and architectural modifications.
Description
SubQ 1.1 Small is the second iteration of Subquadratic’s long-context AI model, built to help enterprises solve problems that require reasoning across entire artifacts rather than isolated chunks. The model is designed for use cases involving large code repositories, document libraries, legal agreements, financial reports, contracts, and other complex information sets. Its Subquadratic Sparse Attention architecture reduces the compute burden of traditional dense attention, making it more practical to process multi-million-token contexts. SubQ 1.1 Small achieves near-perfect performance on needle-in-a-haystack retrieval tests up to 12M tokens, despite being trained primarily at 1M tokens. It also performs strongly on RULER, GPQA Diamond, LiveCodeBench, and AutomationBench Finance, showing a balance between long-context retrieval and general reasoning ability. At 1M tokens, the model uses 64.5x less compute than dense attention and runs 56x faster than FlashAttention-2 on a single attention layer. This efficiency makes long-context training and inference more scalable for enterprise AI applications. SubQ 1.1 Small is especially valuable for teams that need to analyze relationships across full documents, trace logic across codebases, or connect information across extensive collections. The model is intended to help organizations reduce dependence on complex retrieval workarounds and reason more directly over large-scale data.
API Access
Has API
No
API Access
Has API
Yes
Screenshots View All
No images available
Integrations
Claude Code
No
OpenAI
No
OpenAI Codex
No
PanGu Chat
Yes
SubQ
No
Integrations
Claude Code
Yes
OpenAI
Yes
OpenAI Codex
Yes
PanGu Chat
No
SubQ
Yes
Pricing Details
No price information available.
Free Trial
No
Free Version
No
Pricing Details
No price information available.
Free Trial
No
Free Version
No
Deployment
Web-Based
Yes
On-Premises
Yes
iPhone App
No
iPad App
No
Android App
No
Windows
No
Mac
No
Linux
No
Chromebook
No
Deployment
Web-Based
Yes
On-Premises
No
iPhone App
No
iPad App
No
Android App
No
Windows
No
Mac
No
Linux
No
Chromebook
No
Customer Support
Business Hours
No
Live Rep (24/7)
No
Online Support
No
Customer Support
Business Hours
No
Live Rep (24/7)
No
Online Support
Yes
Types of Training
Training Docs
Yes
Webinars
No
Live Training (Online)
No
In Person
No
Types of Training
Training Docs
Yes
Webinars
No
Live Training (Online)
No
In Person
No
Vendor Details
Company Name
Huawei
Founded
1987
Country
China
Website
huawei.com
Vendor Details
Company Name
Subquadratic
Founded
2026
Country
United States
Website
subq.ai/subq-1-1-small-technical-report