Hot Take on Google Gemini 2.5

Well our model point five it is available now in such as inside Google in the model user and even in the version of the menu here. This is a really real model, much more capable than pre versions of Gemini. Google to make this thing work really, really, really well. So let's give it a bench test of uh one of my favorite prompts just to see how well it does. I'll uh we'll work inside of AI Studio because that is my preferred environment for testing the model.

We have here it has awareness of structured output code execution, function calling, and grounding with search. This is one is impressive because this is the pro model, which is typically smarter, but it has access to search, and as you're about to see, which is the part that just blew my mind, it's a reasoning model. So you can see it's a reasoning model. You can see it's got a million tokens uh content, which is relatively, you know, it's smaller than the regular Gemini Pro, which was two million tokens. And here's why it's gonna reason through this.

So this is my egg substitute prompt, the one of the the more difficult prompts that requires uh a lot of lateral thinking and things to how do we make a substitute for it? And it has flash thinking's architecture, but it has pros brains, and this is interesting. This is just incredible. So it's going through, of course, you can see here it's in its thinking aloud all the different pieces as needed and pull from the uh the internet and we can give the consumer we could give the consumer version uh the exact same prompt but the results are essentially identical and then it's gonna spit out its results for what a good egg substitute be now this prompt is intentionally a little bit wacky and it requires you to think to understand eggs so requires you you know some domain knowledge it's going through its thought experiments to figure out uh what are the wet and dry mixes and what are the the isolates here so we have chick pea flour pea protein uh protein content brought up why other ingredients were excluded here is the dry mix so mostly chicken flour uh one pea one tablespoon protein flour that's about right because eggs are about set the well the the challenge with eggs is that they are um they're different the yolk is different than the white and it thought this through really well combine your ingredients combine your wet ingredients whisk thoroughly rest is that part previous reasoning models have forgotten that part which is that the proteins have to hydrate and so it it nailed that that's why I say this is a very very smart model because it's figuring things out that other reasoning models did not get right let's take a look at some of its scores to to show you how big a deal this model is go to El Marina overall, it is currently topping the leaderboard. Uh, it's got a score, an arena score of 1443, 40 points higher than GROK3.

Uh, and uh the the uh open AI for uh GPT 4.5. Where I think this is really interesting is we go over to Ader. So Ader is a coding tool. Their polyglot leaderboard is notoriously difficult to scroll on because they they make it do hundreds of exercises in coded, and those exercises are not trivial. Look at the scores for percent correct, meaning it it got the code correct and was able to return the correct flat to the coding tool.

Claude 3.7 Sonic until yesterday was the best coding tool available by a significant by a decent amount. So you can see Deep Seek plus Claude, 64% correct, 100% uh return format, 65% for 3.7, 01, which is OpenAI's flagship thinking model, 61%. Uh, and then you go down the list here. The previous version of Gemini, uh regular pro uh was 38.2. Uh the the February version was 35.6.

Those were you know pretty low scores, and and the community, the code community is like, yeah, Gemini, not so good at coding. Look where Pro 2 is Pro 2.5 is 72.9% correct, about a 90% edit format, uh defensed. The gap between Clawed and Gemini is massive. When you look at the difference between Claude, say and oh one, that's like a what a three percent difference. It's a huge, huge gap between these two models.

Gemini 2.5 on a very difficult test has done extremely well. Now, where this is really interesting is when you start to put it into practice, when you start uh using this thing for real to see just how well it can do. Let's go ahead and let me make a new folder here. We'll call and let's put in into test here a actually let's yeah, let's have it. Let's let's fire up a I need to uh put keys and things in in a separate window because I don't want to display my API key world.

That would be stupid. All right, cool that windows open. All right, make our git repo there. And let's do chat mode architect. Uh if you're not familiar with A is paragraph, so you use it in concert with uh another coding environment.

Uh let's go ahead and open up a digital open up my test folder in VS Code. We've got which so far is nothing. And we're gonna start with a prom asking giving this a the requirements for PRD. The requirements for a PRD is is a product requirements document. I'm gonna go into Gemini 2.5 and I'm gonna say I want you to create a for this app.

And this is an app to basically read an inbox that uh thunder. And these are all the requirements for my app. Go ahead and build me this product requirement document. And so it's gonna think through the pieces, flesh them out. And this is still in the thinking phase.

It has not actually technically done anything yet, although we can see the actual tokens that it's using for its thinking. And here we are. Then there's functional requirements, non-functional requirements, technical specifications, things to add a scope. Copy. Let's go to PS Code.

Save this as PRD.md. Go to Ader and say read only PRD.md. So we're a chat mode architect, we've read the PRD. Say, and now let's say from back to Gemini, now build a file by file work plan with the develop requirements for each file. List them.

That's good enough right there. So let's build a work plan. And again, we're gonna have it just think through what should what should be in step by step plan to build this CS software. So it's got the file structure, config, logger, so and it's it's enumerated nicely and box processing logic. Got our main application orchestrated, our requirements file, node Pepsi's.

We're gonna copy that. Go back into here and call this paste save planty. Trim it off just so we have that clean. Should do this thing in PRD back to here. We only work plan.

Using the PRE from PRD.md as our guide and the work plan from work plan md as our file by file work plan. Begin coding step one from the work plan, which is config YML. And what is just so good about about the way that this model think and works is it is so capable that it can it can nail this down really quickly. Let's check the config up. Does it look okay?

Does it look like you would want a decent YAML file? So there's our path, there's our start there, there's our end date. So far, so good. Okay, great. Move on to step two.

Auger.py your rate limited. Says come back later. Okay. So you get the idea. Within this within the Gemini model.

Oh, hey, look, it decided it to automate limit. Let's see, let's see how it looks. Uh here is the new thing. Do you want to do this? Yeah, okay.

Well, let it, yep, it's gonna be again for a little while. But I it automatically just retries, so eventually I'll just come back later. That's the the speed at which you can build stuff in like a pair. So cursor, client, aider, any of the ones that can connect to multiple APIs. This model is available via API.

For the moment it's free. I don't tell anyone else for the moment it is free, which means that you could build state of the art apps today right now, and not get a bill for it until Google takes it out of experimental and puts it into production. That is nuts. I can't begin to explain just just how crazy that is that it is it is free for now. World class to build stuff that you want in the language you want.

No, there isn't obviously an environmental cost in the processing cost. That that's where this is. So here's the bottom line. This is an extremely capable model. This is a very smart model.

It is can do coding, it can do all the things that language models can do. It's already in, it's already available in the consumer interface, it's available in the developer interface, it's available by API. Try it out. One of the things that you should have is a benchmark prompt or ideally a set of benchmark prompts. The very difficult things that you want models to try and do.

Like mine was one of mine was the developer scramble egg substitute. Another is build an app from a PRD. Do that every time a new model comes out, test it out because we're expecting you know Deep Seek R2 anytime now. Um we're gonna see how it does as well on the on these different tests. But for right now, this is phenomenal.

Try out, build with it, and try using environments. Like if you have access to uh N8N, an N8N server, you can tie the API as long as you've got your API key from AS Studio, you can tie it there and like end stuff with it. If you have um ACPs or or APIs you can connect to it again, in any of these uh age agenda frameworks, you can start passing stuff to them, get results. Go and try it out, see how it works. I'll talk to you all next time.

Take care.