Google's Gemini: A Swift Launch and Unveiling
Google recently launched Gemini, a new model, after facing delays in delivering its promised fall release. This move seems hurried, possibly in response to competitors like OpenAI and Microsoft unveiling their own products.
- Among Google's three Gemini models, Gemini Ultra gained attention for outperforming OpenAI's GPT-4 on various benchmarks, including MMLU, a crucial metric assessing language model capabilities across different subjects.
- However, a closer look reveals that Google used COT@32 instead of the standard 5-shot learning to boost Gemini's perceived performance on the MMLU benchmark. 5-shot learning involves training the model with five examples of each class, while COT focuses on providing a series of reasoning steps to guide the model in problem-solving.
- Bindu Reddy, founder of Abacus AI, pointed out that Google's claim of Gemini beating GPT-4 is based on a different methodology and that GPT-4 remains superior.
The Landscape of Language Models
- Even if Gemini Ultra outperforms GPT-4, it may not be a game-changer. New open-source language models, like Llama 2 and TII's Falcon 180B, are emerging with comparable or superior performance.
- AI Advisor Vin Vashishta emphasizes that model metrics should connect with customer outcomes, not just benchmarks. He notes that companies spend millions on benchmarks that customers often overlook.
- Reddy echoes this sentiment, emphasizing that in applications like ChatGPT, user experience matters more than benchmarks. If a language model is uninteresting or generates dull content, it won't succeed.
Gemini Ultra's Demo: Reality Check
- Google showcased Gemini Ultra's multi-modal capabilities in a demo video, but it was later revealed that the video was staged. The demonstration didn't happen in real-time or with voice interaction, and Google had used still images and text prompts to create the illusion.
- This isn't the first time Google has used marketing tactics, as seen when it displayed a Google Cloud ad during the AWS re:Invent event.
The Future Outlook
- Gemini Ultra is not yet released, and its actual performance compared to GPT-4 remains to be seen, with Google hoping that OpenAI doesn't release GPT-5 in the meantime.