Stability AI unveils Stable Beluga LLMs on LLaMA and LLaMA 2
TL;DR
- Stability AI released Stable Beluga 1 on Meta's LLaMA 65B and Stable Beluga 2 on LLaMA 2 70B, both instruction fine-tuned.
- As of July 27th, 2023, Stable Beluga 2 ranked #1 and Stable Beluga 1 ranked #4 on the Open LLM Leaderboard.
- Training used a synthetic 600,000-point Orca-style dataset, roughly 10% of the original Orca paper's sample size.
Stability AI and its CarperAI lab released two open-access instruction fine-tuned language models: Stable Beluga 1, built on Meta's LLaMA 65B, and Stable Beluga 2, built on LLaMA 2 70B. Both were "trained via supervised fine-tuning on Stability AI's internal Orca-style dataset", according to the Stability AI announcement, in mixed-precision BF16 with AdamW.
The recipe was "directly inspired by the methodology pioneered by Microsoft" in the Orca paper on progressive learning from GPT-4 explanation traces, but scaled down: a synthetic corpus of "600,000 data points (roughly 10% of the dataset size used in the original Orca paper)". The announcement credits datasets curated by Enrico Shippole and evaluation via EleutherAI's lm-eval-harness with AGIEval added.
As of July 27th, 2023, the company said, "Stable Beluga 2 ranked #1 and Stable Beluga 1 ranked #4 on the Open LLM Leaderboard". Both were originally codenamed FreeWilly and ship under a non-commercial community license as research artifacts.
Shared on Bluesky by 1 AI expert
Originally reported by stability.ai
Read the original article →Original headline: Meet Stable Beluga 1 and Stable Beluga 2, Our Large and Mighty Instruction Fine-Tuned Language Models — Stability AI