Evaluating OpenAI GPT OSS 20B and GPT OSS 120B

So it's not a fireside chat. It I just needed a countdown timer. We're going to take a look today at OpenAI's new open models. Uh, this was an announcement that was made about uh five hours ago now on the open A website, their new model series called GPT OSS. Continuing the fine tradition of naming models completely incomprehensible names.

Uh I'm guessing the OSS refers to open source software, which is a misnomer because they, unless they release the training data, it's not open source, it's open weights. Anyway, this is a big deal because this is the first model since GPT 2 in 2020 that OpenAI has actually released the the weights for, which is their name, you know, open, the open and open AI really hasn't been the case. They've been closed models ever since GPT 3. So this one is their first uh open weights model since then. It's kind of a big deal.

And it comes in two sizes. It comes in GPT OSS 120B, 120 billion parameter, well, 117 billion parameters with 5.1 billion parameters activated per token. That tells you that this is a mixture of experts model. And you can see that the total number of experts it supports is uh 1284 active experts per token with a context length of 128k. This smaller version is GPT OSS 20B, 24 layers, 21 billion parameters, of which 3.6 billion are active per token, uh 32 total experts, four active per token, 128 uh thousand tokens uh context window.

So if that's a bunch of techno babble, um here's what this means. Basically, there's a big one, there's a medium-sized one. If you have, for example, a really good gaming machine, a really good gaming laptop, you can probably run the the small one, right? Which is gonna take up probably around 15, 16 gigabytes of video RAM. So if you've got a really nice gaming laptop that has a 16 gigabyte uh video graphics card, a GPU, you're gonna be able to run this model in memory.

The bigger model, you're gonna need have your duty hardware. So you'll need like a Mac Pro or MacBook Pro with uh like at least 96 gig of RAM, probably closer to 128. When they talk about active parameters per token, imagine that the live this thing is a library, right? And you have 21 billion parameters, it's called 21 billion pages right on it. Um 3.6 billion are active at a time.

So if you give it a task, instead of the library having to search through the whole library, they can search through just a section of the library, and it's it's much faster. These mixture of experts models have been the gold standard for mid and small size models, and even the big ones like Gemini for several years now. It's it's very interesting. There's uh been this dichotomy. Um OpenAI has typically stuck with what are called dense models, whereas everyone else has gone to a mixture of experts, companies like Deep Seek, Gemini, etc.

Um experts per token active. This means that um every time you send a word through, four different little mini librarians go to work trying to figure out where in this section of the library they need to spend their time. And the context window, 120,000 tokens, roughly equivalent to about 80,000 words. So my friend Ann Hanley wrote this wonderful book, everybody writes, about 75,000 words. In theory, the model can hold this much in its memory.

Well, if you believe OpenAI, um, and you believe their benchmarks, uh, these open models score about as well on code, on uh reasoning. You can see uh there's 03, which is their flagship model, OSS 120, the the big model, but the big open model scores an 80, the small model scores a 71. Um the the on um questions across academic disciplines, it scores a 90 versus their flagship model at 93, and so on and so forth. And and so these are all the same benchmarks that everybody pretty much uses, which means, and this is a point of speculation for later, if they're releasing model open models like this that are competitive to their current flagship models, then the assumption is in the AI nerd herd community that GPT 5 will have to be substantially better because you wouldn't release a model that is competitive to your current closed model offerings. Um, how do you use these things?

This is where it starts to get kind of messy. There's a good chance that a lot of the infrastructure providers, like Groc with a Q, G R O Q, uh, Deep Infra, Cerebrus, etc., will have these models available for you to use. But if you have the hardware, then you can run them on your own computer. So I'm gonna pull up here LM Studio. LM Studio is for is a Mac local model, hence the name LM Studio, local model environment.

And you can download these models and put them on your machine if your machine has enough memory. So today, what we're gonna do is we're gonna put these new models through their paces. I'm gonna go and we're gonna start with the smaller model. We'll start with GPT OSS 20B. I'm gonna crank up the context window here.

Let's go to 16,000 tokens. We're gonna leave everything else as is number of experts active. And we hit load model. Now, what's important to note here is that we are using this model locally. I could turn off, well, I I mean I could turn off the internet and it would still run.

I would obviously stop streaming. Um you'll note that in the MCP selector for example here. I have no extensions active whatsoever. Um, this one does have the ability to choose what level of reasoning effort you want, low, medium, or high. Let's dial this up to medium.

And it looks like the model has loaded. Let's put this through some tests to see how well it does. First thing we'll test is a timeliness query. I'm gonna say, as of your most recent knowledge, what is the status of the Ukraine-Russia war? Who controls what?

And we will see. Here comes its thinking. This is a big deal for OpenAI. They typically have not shown the thinking phase of what tokens uh are being produced. Um, so its knowledge here, uh, the knowledge cutoff 2024.06 for this particular query, and it's spitting out uh with excess.

So that this is the model's knowledge for this particular model is about a year old because it is obviously uh August of 2025. So that's our first prompt. Let's go ahead and just delete that. But you would notice though, this is really fast. If I scroll all the way down to the bottom here, escape.

This is cranking out 68 tokens a second, 2006 tokens, uh four tenths of a second to the first token. That's really actually quite fast. Um let's move on to our next question, which is a bit more niche, as of your most recent knowledge. Create a new chat, make sure reasoning effort is medium. As of your most recent knowledge, what is the most recent issue of Christopher Penn's almost timely newsletter?

Again, remember there's no web search involved, so it does it has to rely solely on the knowledge it has. Um, let's see if it's just gonna completely hallucinate. Uh it says, I can't pull the data, so I can tell you the sleep scorecard. So it is completely hallucinated the answer here. Not a huge surprise.

Um, my newsletter isn't that big enough, uh big of a deal uh to make it into the train data for this small model. 76 tokens per second, nice and fast. Let's go ahead and delete this chat. Next question, we're gonna do a mathematical reasoning question. And the mathematical reasoning question is a recipe question.

So we're gonna crank the reasoning effort to high. This is gonna consume more tokens, so it will it'll have more to say. And I'm gonna say rescale this recipe for three servings and convert it to metric. So it's not just simple multiplication to go from one serving to three, but it also has to convert it to metric along the way. Here we go.

We're kicking this off. Again, I'm very impressed with the speed of this this 21 billion parameter model, just cranking out stuff. So it says first step, calculate the quantities in US customary and then convert two thirds of a cup of sugar is about you know so it's converting its numbers, and then conversions, and let's see if it comes up with the answer here. The correct answer would be 402 grams of sugar, 710 milliliters of milk, 135 milliliters of olive, hot 133, nine large eggs, that's correct, a pinch of salt, still the same, 15 milliliters of vanilla, and uh 1.4 liters with so okay. So it went with grams still liters, 1.4 liters of uh this so it has done a pretty credible job of the conversion to metric, um, which is not bad.

Um I've had other models that are bigger size models just do a completely terrible job of this. So that's that's actually really good. Let's go ahead and delete that. Next chat, we're gonna do a logic and math reasoning problem, and we're gonna say, hey, you need to convert, you need to think through how to replace eggs in a recipe. I'm gonna turn the reasoning up high, paste this in.

This is if you say you have an egg shortage. What can you use to substitute for eggs? So it's going to do its thinking. Let's see, we got I've given it some ingredients like mushroom powders and stuff. While it's thinking, one of the most important things you can do when you're dealing with AI models of any kind is to have benchmarks, benchmark prompts that you use that you can assess the smartness of a model to see if it does what you want it to do.

Because all of these academic benchmarks that are that are out online, they're good, right? There's nothing wrong with them, but they don't necessarily test the way that you work or the way that you think. So using a uh a set of benchmark prompts that you've built yourself, terrific way to make use of this stuff. So let's see what we got here. Uh, it is thinking through producing multiple experiments, adding tapioca, replacing the all-purpose wheat flour with rice flour for lower gluten.

Uh, it's coming up with ingredients that it can't use and it is recognized that correctly. And it's scored that recipe is 78. And what are we at here? We're at 12 gigabytes of RAM, 4% CPU usage. So, this is actually really efficient.

It's taking into account the water content of the egg. Now, here's the thing this one, this part here is kind of a a giveaway. Um, if you know cooking, eggs are the yolk and the white are are chemically different, and they have different water content. So, a model that really reasons well is going to separate that out and say, well, you have to account for the water content being different in those places. Let's see what we got for our experimental design.

Protein, 10% of the final weight, starts in viscosity, gluten network, uh, no chewiness, very little all purpose foul, xanthem gum plus oil, flavor of salt plus MSC Pitashi, optional source was ketchup mustard. Okay. So it did a pretty decent job for a very complicated problem. Um I would encourage and the it's its recipe comes up with a 78 on the score, which is is pretty credible. Our final exercise is going to create have it create try to create a game.

So let's give it a prompt. We're gonna give it high reasoning effort. And there is no canvas, so I'll just say build a simple HTML game that uses HTML CSS tailwind phaser IO to create a simple side scrolling shooter that uses the rocket ship emoji as a sprite along with um a couple of the other emoji. And so it's gonna hopefully reason through and figure out what it needs to do to actually build this HTML. While it's gonna do that, I'm gonna set up a uh container called Testa HTML on my desktop so when it does spit out um the appropriate HTML, we can test to see if it actually runs.

Yep, there's the lives counter movements. This is so far pretty capable. Um I'm just looking at the code as it's going by. This is pretty decent. Like this is I would say so.

The a comparable model to this size-wise would be like Mistral Small or the Quen 3 38 30B model. This the reasoning of the logic and the fact that you can turn up and turn down the reasoning is pretty impressive. Also in the notes, it said this was built by OpenAI using the MX FP4 compression, which supposedly gets you better performance out of the same number of layers than other uh other model training. So we'll see what it comes up with here. Uh question in the chat.

How are we supposed to compare GPT OSS to Deep Seek and Quen? That's exactly what we're doing right now. So you can look at, and while this is cranking away, let me bring up a browser here. Um there are some really good sites like uh artificial analysis that you can put um you can put models side by side once they are they're available. So I don't think yeah, none of it's not available in here yet.

Oh, wait, no, they do have benchmarks in here for it. So let's take that question and choose 03 03 mini high 04 mini uh and put in the two new the GPT OSSs. They may not have data for them yet. Let's put them alongside Llamaforce Scout and Maver Scout and Maverick, and let's go down to GLM, which just came out this past week. The new versions of Quen.

And let's see what's on the screen. The yeah, the GPT OSS ones and they're not benchmarks in here yet. Um, if I had to guess, they're probably gonna come in right around where the Alibaba models are. All right. It has finally gotten to the stage of writing the HTML.

It might run out of tokens because I only gave it 1600 to work with and just as a point of comparison this should be if if it does this well it should be about 300 lines of HTML and CSS together. So we'll see if it can oh look it's doing error checking well that's nice but it's fine this should be okay this is actually really smart like in terms of reasoning this is really a smart model it might I might have turned the reasoning up too high let's see test scenario start the game okay we can keep it potential improvement the final code needs adjustment oh so it's at it's interesting it's not using the native emoji it's it's actually using PNG files spawn rates and let's check as it's working here, how it's doing on memory. Really well. So I've my entire system right now is only 54 gigabytes. LM Studio.

This is a surprisingly efficient. Now it's going through the physics. Oh my goodness. I would I would classify this as overthinking. It is now overthinking this.

Now the final code's ready. How many tokens have you produced? You have gotta be coming up close on the token limit for this chat. Okay, so I think I can safely conclude the reasoning effort high is very likely going to throw it into chasing its tail over and over again. Because it is now rechecking the CD, the C D N JS stuff.

I'm gonna I'm just gonna stop you because you're you're let's dial this down to medium and try that again. My goodness. It really just went totally off the rails there and just went perpetually chasing its tail. Let's try that again. Wow, that okay.

That went that went straight to coding. So I wonder what happens if I set it to no low reasoning effort. Does it just like wing it and YOLO? Because that required no thinking at all. We were four minutes in on the previous one, and it hadn't produced a single line of code.

This one we just YOLO. Uh let's copy, bring this into our editor, trim off the goods. I will be stunned if this worked out of the box. Mostly because I would don't think it can. So let's go ahead and take our file and see if it produces anything.

Huh. Look at that. So for okay, it's one of the oops, it just crashed. Um but it it got a somewhat working game on on a one-shot, which is pretty impressive for the loading reasoning. I don't know if that was gonna uh hold up.

So let's do a comparison now. So I'm gonna ditch this. We're gonna delete it. Let's eject the 20 billion parameter model, and let's see if we can even load the 120 billion parameter model, which is MXFP4, 59 gigabytes. Let's see if we can get the full GPU offloaded.

Number of experts is 32. Let's take that back down to four, which is where it typically should be. CPU pool is 12. And let's see if my computer's grinds to a halt, which is a non-zero possibility. Let's watch the memory meter here.

So far it's doesn't look like it's loading at all. Maybe because the memory meter is not changing a bit. The fact that it's it says it's loading. Oh, there it goes. 40 gigabytes, 41.

At least within LM Studio, I can see that it is starting to consume greater and greater amounts of memory. Let's quit activity modder. There we go. You can see the ramp up as it starts to just gobble system memory. 91 gigabytes, 99, 101.

It has loaded. It has loaded at 101 gigabytes of system memory. Alright, let's see. Let's see if this thing will even run at all. I'm gonna switch to medium first prompt as of your most recent knowledge.

Thinking okay. June 2024. Okay, so it has the same cutoff date. Huh. We are at sixty-eight gig of man of memory consumed in its footprint.

Look how fast it's going for a 119 billion parameter model. Yes, at activation, it's only got um 17 or 18 billion parameters, right? What is it at activation? It is at 5 billion parameters activated. That's why it beh it's behaving like a a much smaller model.

Okay. 42 tokens a second. One 1.4 seconds to first token. That is really fast for a model this size. Let's delete.

Let's go on to our next test following the same benchmark. Let's go and see if it knows anything about my newsletter. I would guess it's going to come up with the exact same answer. Uh what GPU is this? I am using a Mac.

So this is the M4, which is has an integrated GPU. 54 tokens per second, 300 tokens, 1.4 seconds to first token. Let's see, issue 210, AI generated content, how to spot the real fix. So interesting, it's more on target in that it is like the last one said I was I data has sleep and wellness log, which is completely wrong. Um this one at least has the right general topic.

It's still wrong, but it's it's a more accurate hallucination. Let's go on to our third test. We're gonna rescale our BlackBerry recipe, set it to medium, and we're gonna convert it to metric at the same time, see how it thinks it through. Alright, so our the recipe is coming up with sugar, 400 grams, that's about correct. Milk 710 milliliters, so it's slightly off there.

Olive oil is 133 milliliters, slightly off there, but it's fine. Nine eggs, that's logical. Um, it got the salt correct. The last one just said it was uh still a pinch of salt. Vanilla is 15 milliliters, and it went with grams for the blackberries rather than uh liters this time through.

So it would it did a good job there. Let's do the egg test. So we're gonna copy, start a new prompt here. The egg test to see how well it reasons an egg substitute. Again, we are looking for if it can think through, for example, differences in the water content of the yolk versus the white.

So water, it's yep, it still does did the what the previous version of the smaller version of the model did. Umgredients. Now the the medium reasoning effort doesn't seem like a ton of reasoning. When we switched it to high, it just overthought itself into oblivion. I wonder what the low version looks like.

We'll have to try that in just a second to see how well it worked, it works. Okay. Let's give it the exact same prompt, but switch the reasoning effort to low and see if it just does it. I'm gonna guess it's just gonna do a one-shot and not reason at all. We need to propose, yeah.

Thanks. So it basically the low means don't bother reasoning. Okay, let's do the last one, which is building the simple HTML. Let's give this a try. We're gonna switch the reasoning to medium.

And it's going through and thinking through. There's definitely more reason reasoning on this and the large model than it did on the small model for how much to think through the stuff, but it still is a reasonable response. Alright, here comes its text for its its game. Let's get our file container open. Basically, with these models, the amount of disk space it consumes is the amount of memory it's going to consume.

As a general good rule of thumb. So there's absolutely no way this is going to fit on inside 36 gigabyte. All right, so the HTML is done. Let's grab it and copy it. Put it in our shell and see if this one runs any better.

So bring up a browser window. And hit refresh. And nothing. Absolutely nothing. Which is weird.

So it thought more. It reasoned more. And it produced something that was completely non-working as opposed to the previous version. Now as a point of comparison, uh so that was still that was still 37 tokens a second, 3,600 tokens of 1.5 seconds to the first token. Let's eject the large model.

And just as a point of comparison, let's try swapping down to the Quen30 uh model. This is one the one that came out uh this week. It's Quen's version make sure at 16,000 16,384 and hit load. Start a new chat. So this Quen model would be comparable to the small version of the GPT OSS.

I still do not love those names. Let's give the Quinn model the same general test questions we started with earlier. We're gonna ask it once it finishes loading here. Interesting that the GPT OSS model has a seems to have a relatively fast load time. All right, let's ask it.

What is your most recent knowledge? Alright, so this is mid-2024, so June of 2024 as well. 97 tokens per second. So this so the Quen model is definitely faster. Uh the GPT OSS was at 70 something tokens, I believe.

Uh let's start a new chat. Let's move on to the knowledge question about my website. You can see here there is no reasoning button though. Um doesn't have the capacity for that. Uh it says I can't access that, so it's just a refusal, which is fine.

Um, let's do the mathematical reasoning on rescaling the recipe. So it has rescaled metric measurements. 100 grams of sugar, wrong. 300 mil grammiliters of milk, also wrong. Uh 45 milliliters of olive oil, wrong.

So it basically converted to metric, uh, but it completely hosed that. So let's go into our thinking through the egg. So it's thinking through this is also really wrong. So 75% of a uh egg is water, and so 54 grams is not 75%. So this is very clearly quite wrong.

And now let's do the coding test. Um bearing in mind the first version of the GPT OSS small model did create a kind of sort of working version. All right, so we've got HTML. Let's give this a try, copy it into a document, paste it in, trim off the superfluous stuff. Now let's see if the Quen version runs.

Well, it kind of got there. At least got the interface, but nothing else. So it did not. Okay. So here's where we landed.

The GPT OSS models are pretty good. Um we just did a side by side with Quen, uh, the Quen3 model. And the GPT OSS model, the small model, did a really, really good job. And it's fast, 70 tokens per second on my machine. Uh consumed had a memory footprint about 16 gigabytes of RAM.

Uh so as advertised. And this is MLX, uh, so it's optimized for Apple uh MPS. Uh this is the four-bit quantization. I would say that's that's a good workhorse model. The fact that the 20 billion parameter model got working HTML and JavaScript, uh, even ahead of the 120 billion parameter model was was pretty impressive.

Now, from a I would say that's a really, really good daily workhorse model, the 20 billion parameter model. Um, and it's super fast. I can only imagine it once it goes up on Cerebros or Groc or um Deep Infra, how fast it's going to be on that infrastructure. I mean that's probably gonna be like 2,000 tokens a second, uh, which would be crazy. But I would recommend this.

I would say this is a good model. The 120 billion parameter model, if you have the memory to support it, I could definitely see that being useful in like a small office, or if you have like the NVIDIA DGX Spark, um, where you've got a dedicated machine in the office, you could definitely use the 120 billion parameter model as a as a day-to-day workhorse model for a small group of people if you wanted to keep it uh um in-house. Uh I don't have uh I see Mahid as asking, I don't have any MCP setup in my LM studio, um, so I can't test that. Um, but given that it was optimized for tool calling, it's probably going to be just fine at that. And I think OpenAI has released a really good model, which again brings us back to so what's the GPT 5 going to be if they're willing to release an open weights model that is competitive with their small like the 03 mini and 04 mini, what's GPT 5 gonna be?

Thanks for tuning in. I hope that uh you found this comparison useful. Uh, I would recommend if you're using and running open models, I would recommend to trying out the GPT OSS models despite the stupid name. OpenAI, please hire a product manager who can name things well. And uh we'll talk to you all on the next one.

I'm going to put up the uh the closure. Uh, this was not a fireside chat per se, but here we go.