Unlocking Potential: LLM360 Open-Source Framework by Cerebras Systems, Petuum, and MBZUAI

Cerebras Systems, Petuum, and MBZUAI Unveil LLM360 Framework

Cerebras Systems, Petuum, and Mohamed bin Zayed University of Artificial Intelligence (MBZUAI) have collaborated to introduce LLM360, an open-source framework for large language models (LLMs). Developed in partnership with MBZUAI’s Institute of Foundation Models, LLM360 aims to streamline the development of LLMs, providing developers with insights and methodologies to simplify, expedite, and reduce costs.

Key Models Released:

• Amber:
• 7 billion parameter English-language model
• Trained on 1.2 trillion tokens
• Released under the Apache 2.0 license
• CrystalCoder:
• 7 billion parameter model
• Trained on 1.4 trillion tokens
• Designed for English language and coding tasks
• Released under the Apache 2.0 license
• Specifically trained on Python and web programming language
• Diamond:
• Upcoming release
• 65 billion parameters

Architecture and Performance:

• Models built on Meta’s LLaMA architecture
• Amber performs similarly to LLaMA-7B and OpenLLaMA-v2-7B, outperforming Pythia-6.7B

Training Details:

• CrystalCoder incorporates a blend of text and code data for enhanced effectiveness
• Introduction of code data occurs early in the pretraining stage
• CrystalCoder specifically trained on Python and web programming language

AI Developments in UAE:

• UAE working towards becoming an AI superpower
• Recent AI company, A171, launched to compete with AI giant OpenAI
• Emphasis on open-source models to promote research initiatives

Source: LLM360 Blog