Unveiling Google's Gemini: A Closer Look at the Surprising Launch

Google's Gemini: A Swift Launch and Unveiling

Google recently launched Gemini, a new model, after facing delays in delivering its promised fall release. This move seems hurried, possibly in response to competitors like OpenAI and Microsoft unveiling their own products.

Gemini's Performance

• Among Google's three Gemini models, Gemini Ultra gained attention for outperforming OpenAI's GPT-4 on various benchmarks, including MMLU, a crucial metric assessing language model capabilities across different subjects.
• However, a closer look reveals that Google used COT@32 instead of the standard 5-shot learning to boost Gemini's perceived performance on the MMLU benchmark. 5-shot learning involves training the model with five examples of each class, while COT focuses on providing a series of reasoning steps to guide the model in problem-solving.
• Bindu Reddy, founder of Abacus AI, pointed out that Google's claim of Gemini beating GPT-4 is based on a different methodology and that GPT-4 remains superior.

The Landscape of Language Models

• Even if Gemini Ultra outperforms GPT-4, it may not be a game-changer. New open-source language models, like Llama 2 and TII's Falcon 180B, are emerging with comparable or superior performance.
• AI Advisor Vin Vashishta emphasizes that model metrics should connect with customer outcomes, not just benchmarks. He notes that companies spend millions on benchmarks that customers often overlook.
• Reddy echoes this sentiment, emphasizing that in applications like ChatGPT, user experience matters more than benchmarks. If a language model is uninteresting or generates dull content, it won't succeed.

Gemini Ultra's Demo: Reality Check

• Google showcased Gemini Ultra's multi-modal capabilities in a demo video, but it was later revealed that the video was staged. The demonstration didn't happen in real-time or with voice interaction, and Google had used still images and text prompts to create the illusion.
• This isn't the first time Google has used marketing tactics, as seen when it displayed a Google Cloud ad during the AWS re:Invent event.

The Future Outlook

• Gemini Ultra is not yet released, and its actual performance compared to GPT-4 remains to be seen, with Google hoping that OpenAI doesn't release GPT-5 in the meantime.