Unlocking AI Mysteries: DeepSeek LLM Open Source Release Dominates Llama 2 and Claude-2

China's DeepSeek Releases Powerful DeepSeek LLM Model

DeepSeek, a Chinese company dedicated to unraveling the mysteries of AGI, has open-sourced its DeepSeek LLM model on GitHub, Hugging Face, and AWS S3. This model, with 67 billion parameters, was carefully trained from scratch on a massive dataset of 2 trillion tokens.

Model Versions

• Two versions: Base and Chat
• Available in English and Chinese
• Accessible under the MIT license

Performance Highlights

• 67B Base outperforms competitors like Llama 2 in reasoning, coding, mathematics, and Chinese comprehension
• Coding: HumanEval Pass@1 score of 73.78
• Mathematics: GSM8K 0-shot: 84.1, Math 0-shot: 32.6
• Generalization with a score of 65 on the Hungarian National High School Exam
• Open-sourced on GitHub, Hugging Face, and AWS S3 with access to intermediate checkpoints during training

Evaluations

• Rigorous evaluations against benchmarks like LLaMA-2, GPT-3.5, and Claude-2
• Superiority in English and Chinese languages, even in unique exams like the Hungarian National High School Exam
• Use of 20 million Chinese multiple-choice questions in DeepSeek LLM 7B Chat enhances benchmark performance in MMLU, C-Eval, and CMMLU.

Pre-training

• Transparent pre-training process, emphasizing openness
• Model architecture similar to LLaMA, using auto-regressive transformer decoder models with unique attention mechanisms

Alibaba's Entry

• Alibaba introduces its LLM called Qwen-72B, trained on 3T tokens
• Smaller model, Qwen-1.8B, released as a gift to the research community.