DeepSeek's new chatbot boasts an impressive introduction: "Hi, I was created so you can ask anything and get an answer that might even surprise you." This AI, a product of the Chinese startup DeepSeek, has rapidly become a major market competitor, even contributing to a significant drop in NVIDIA's stock price. Its success stems from a unique architecture and training methodology incorporating several innovative technologies.

Multi-token Prediction (MTP): Unlike traditional word-by-word prediction, MTP forecasts multiple words simultaneously, analyzing different sentence parts for enhanced accuracy and efficiency.

Mixture of Experts (MoE): This architecture leverages multiple neural networks to process input data, accelerating AI training and improving performance. DeepSeek V3 utilizes 256 neural networks, activating eight for each token processing task.

Multi-head Latent Attention (MLA): This mechanism focuses on crucial sentence parts, repeatedly extracting key details from text fragments to minimize information loss and capture subtle nuances.

DeepSeek initially claimed to have trained its powerful DeepSeek V3 neural network for a mere $6 million using only 2048 GPUs. However, SemiAnalysis revealed a far more substantial infrastructure: approximately 50,000 Nvidia Hopper GPUs, including 10,000 H800s, 10,000 H100s, and additional H20 GPUs, distributed across multiple data centers. This translates to a server investment of roughly $1.6 billion and operational expenses estimated at $944 million.

DeepSeek, a subsidiary of the Chinese hedge fund High-Flyer, owns its data centers, granting complete control over AI model optimization and faster innovation implementation. This self-funded approach enhances flexibility and decision-making speed. Furthermore, the company attracts top talent, with some researchers earning over $1.3 million annually, primarily from Chinese universities.

While DeepSeek's initial $6 million training cost claim appears unrealistic—referring only to pre-training GPU usage and excluding research, refinement, data processing, and infrastructure—the company has invested over $500 million in AI development. Its compact structure facilitates efficient innovation implementation compared to larger, more bureaucratic competitors.

DeepSeek's example showcases a well-funded independent AI company successfully competing with industry giants. However, its success is undeniably linked to substantial investments, technical breakthroughs, and a strong team, making the "revolutionary budget" claim somewhat misleading. Nevertheless, the company’s costs remain significantly lower than competitors; for example, DeepSeek spent $5 million on R1, while ChatGPT4 cost $100 million. This cost difference, even considering DeepSeek's actual expenditure, highlights a significant competitive advantage.