So What? The Local AI Model Roundup

Happy Thursday, everyone. This is So What, the Marketing Analytics and Insights live show from Trust to Insights. Katie is staring at the mountains of uh Vermont. Does robot even have mountains? Vermont, Green Mountain.

That's what it means. I mean, yeah, but are they actually mountains? Yeah, yeah, there's uh a few, you know, well over five. I think, but uh Bromley is over 8,000. I always think of like rolling hills personally, but okay.

The green hill. Uh so I'm Chris. Uh we have John uh here, and uh Katie is out staring at apparently actual mountains. Uh this week we are talking about the local AI model roundup. So this is something that's by popular request.

In fact, uh Brian and our us uh analytics for marketers slack group, which if not if you're not a member of is uh Trust Insights.ai slash analytics for marketers, uh was asking, like, what's the latest and greatest about local AI models? So before we begin, uh John, when you are looking at the the AI world and stuff. Do local models even factor in on your radar, or do you are you strictly cloud only? Or you're like, you know what? I'm just gonna bypass this whole AI thing.

No, you know, cloud is good enough for what I've in fact. I I'm not gonna lie about this. Kelsey and you and Katie are like I'm just like, I need this fixed, make this happen. And you guys just show up with the finished file. It's like here it is.

And I'm like, oh my god, that's awesome. And and it works because you guys have been testing and playing with stuff, and then uh yeah, the stuff I've been seeing in like the last two or three months has just been amazing. Um, but yeah, I love to, you know, I play around cloud, is definitely enough for me. I am not, and it'll be interesting to have you tell us. I mean, for everything I've seen, it's basically if you get to a point where you're having to ration your work because every week you're burning through all your tokens.

That's when you really want to start looking at this, or of course, security is the other thing if you don't want to go to the outside world. Um, and yeah, so I guess that hamstrings me. Because there are a lot of things that I don't do because I'm like, yeah, I'm not putting my data in there. I don't want that to go there, you know. So that's that's part of the mix.

But no, I'm excited about this because this is the same thing with every new technology. You know, there's you can play around with it and learn how to use it and do different stuff, but until you cross that line of actually setting up and configuring it yourself, like that's a whole different level of learning how things work and being able to do stuff too. It opens your eyes to possibilities when you have to dig down in the blood and guts. And then I'm super thrilled most of all, because normally this is the kind of thing that people just you have to do at two in the morning by yourself and like find help in Reddit groups, and it's like completely and entirely painful every step of the way. So I'm getting a guided tour here, so I'm just thrilled to death about this.

Okay. Um, we should probably do a bit of table setting. We've done it in the past on past episodes of the show, which if you want to catch, you can go to trustinsights.ai slash YouTube. You'll find all of our past shows, and you can look for local models. We should differentiate because there's a bunch of terminology.

Um there's cloud models, there's open weight models, closed weight models, local models. So uh let's talk about open versus closed first. Closed weight models are models where you're not allowed to download them. Uh, they are locked behind a uh closed doors. This is like Claude Opus.

This is uh G open AI GPT 5.5. This is Gemini, Google's Gemini. These models, they're good, but you don't get access to them. Now you can you can buy like API access to them, but you will never have them on you know on hardware of your own. They are closed, um, they are they are under lock and key.

Open models are models that where the the training weights, and when we talk about models, they all ai models are basically made of statistics, and that big pile of statistics are called weights, like basically how but you know what's the probability of this or that or whatever. It's a big it's almost like kind of like a big spreadsheet. Um those are the weights. OpenAI and Gem and Google and Anthropic, they ain't giving you the spreadsheet. Um, whereas open weights models like uh uh Kimi K2.6, Mini Max, uh Deep Seek, Alibaba Quinn, etc.

The manufacturers' models have said yes, you can have the spreadsheet. You you can have it, and then if you want, you can make changes to it. So that's the difference between closed weights and open weights. Is with open weights models, you're allowed to download them, you're allowed to modify them. You don't get the training data they were made with, but you do get the spreadsheets, if you will, that they use to assemble things.

Now, in terms of cloud versus local, local in our for today's show means you can download it and run it on hardware that you own. And this is gonna vary based on on how much money is spent on hardware. Um at the low end, there are some models that can run on things as small as a phone. On the high end. You could blow 50 grand on like a DGX supercluster in your house.

And there are people who do. There's this one dude on YouTube who's got like a hundred grand worth of Mac minis all chained together so he can run Deep Seek at home. And I'm like, well, it's your money, man. Um, whereas cloud means the models being hosted somewhere else that has the computational facilities. Um this is where it can get messy because open weights models can be served up by third parties called inference providers, companies like Groc GROQ with the Q, not Elon Musk's thing, Cerebrus, uh Deep Infra, etc.

So, what these companies will do is download open weights models, and they have like several million dollars worth of hardware, and you can use their hardware essentially, kind of like hosting your own website on their service. You're still using an open weights model, but you're just not putting it on your machine. So today we're talking about local models that you can run on nice hardware. Let's call it nice hardware. So you're we're talking, you know, a nice MacBook with decent memory.

Uh essentially, if you can play like Call of Duty at like full ultra resolution, you can run local models pretty easily. Right. And so, but when you're buying this, you pretty much you're gonna dedicate this machine to it, right? I mean, because if you're gonna be doing that much stuff, it's do we assume it's just gonna be kind of running all the time and you're not gonna be able to get any other use out of this box? It depends on how nice a computer you have.

So, like if you have a top-of-the-line MacBook, you could be doing other things. Like the fans will be spinning, but you can do other things. On the other hand, if you're doing it like on a MacBook Air, yeah, that that that's all that computer is gonna be doing. Um, and there are a lot of options for that hardware if you want to invest in it. Uh, but from what I've seen, you're not going to find anything less than four thousand dollars uh for a device because GPUs are in very short supply, and every AI company on the planet's trying to buy more of them.

Uh so the you know, good old good old economics. In general, closed models are always cloud, because you're never gonna get uh a company release those, and open models can either be cloud or local. And today we're gonna be focusing on on what those are. When we talk about local models, there's a lot of there they come in a bunch of different sizes. So all models that are that are local models are measured in a size called parameters.

Uh, this is basically how big that spreadsheet is. Um in general when you look at the parameter counts, the easiest way is when you go to download one of these things, if you look at how much disk space it takes up, that's how much memory it's gonna need, plus like an extra 20%. So if you go to a model like Deep Seek and you see, wow, this thing's 500 gigabytes of disk space, you know it's gonna you're gonna need a lot of memory to run that thing. Okay. The next thing is how do you know which model to use?

Because spoiler, uh, if you go to Hugging Face, which is the probably the planet's best repository of models, and you tap on the models selector, you can see that as of today there are 2.87 million models. Just a few. And we see everything from like Supertonic, which is uh we're gonna talk about that in a bit, uh, to on sloths, quen threes with MTP uh and all these really cool things that are are uh lots of jargon in them. In general, um, there are there are some really good review sites. One of my favorites is called artificial analysis that looks at the big families and and the headline models.

So this is their set of benchmarks. First thing we have to figure out is what do you want to use it for? And there are models for text, for coding, for writing specifically, especially if you're doing fiction writing, uh, for running agents for generating music, for generating uh the spoken word, for doing speech recognition and transcription. There's all these different things you want to do. So the first part, uh, I'm gonna channel Katie here, is we have to figure out like what you would be doing, and that goes back to the 5P framework part, Justin.

What do you want to do? Uh, do you because there are different models for different things. So when you're thinking about how you might use local AI, John, of all the different use cases, I just broadly mentioned, which ones you're like, oh yeah, I want to do that. Yeah, that's a good question. You know, the stuff that we've seen, um, really text analysis is uh the most useful stuff so far, you because we get transcripts of reports, and then we'll get um, you know, requests for proposal, and then we'll have our scopes and our agreements and all of our processes, and getting all that data together synthesized, and so it can just say, hey, here's a scope of work that's done.

That's uh a useful application for us. Um, but yeah, so it seems like all this model selection stuff, it is all about the use cases. Like you kind of need to know which one goes with what you want. And then is it the same with normal software on this front? And that there's a whole range of, you know, these are the ones that have been running for a year and are pretty much error-free.

And this is the one that was released yesterday that like might burn your whole house down, and you have to kind of know which level of how much you can stomach as far as that. Essentially, um, the the extra challenge though is that as new technologies come out, um, model makers adapt them very, very quickly. And so if you say, I'm gonna standardize on this model and I'm not gonna change it for a year because I don't want to deal with like the crazy pace of change, and then someone comes up with a new paper that says, Hey, I can make your model six times faster, you're like, crap, should I change? Um, but yeah, text analysis is probably the easiest one to get started with, and there's a lot of choices, but it is you know, text analysis and coding are probably the two most useful things. And if a model's good at both of those, by default, it's also gonna be good at at being what's called an agentic model.

That is a model that can uh run agents well. There's other there's two more considerations. In addition to the model, with it's just kind of like the engine of a car, there is the harness, which is the rest of the car. So depending on the kind of harness you have, depending on the kind of uh car you have, sometimes if the harness is good enough, you can put in a smaller engine and still get good performance. Like, so if you're running, if you're driving NASCAR, yeah, you probably need the the beefiest engine you can cram into your Honda Civic.

Um, but if you're not doing NASCAR, if you just go to the grocery store, you don't need an eight-cylinder engine in your Honda Civic, right? You can you can run on probably like a two-cylinder engine if you just go to the grocery store because you're not gonna be going like 89 miles an hour. So that means that the container that we put the model in is just as important. And these are containers like um clawed code, clawed co-work, etc. With local models, there's an additional twist, which is you need a server, a piece of software that will serve up the model, uh, which you don't have with cloud providers, because cloud providers by default are the servers.

So these would be tools like LM Studio, Olama, which is very popular uh because it's so easy to use, uh, VLLM. If you're a company and you want to make a model available to multiple employees, uh anything LLM. Uh so there's all these, so there's the model, there's the server, and then there's the harness that uses the model for things like you know, text analysis, uh loading documents and things. And the better your server and harness are, the smaller, lighter, and faster a model can be. Because one of the catches is of the the bigger a model gets, the smarter it gets, but the slower it gets and the more memory it takes up.

So there's a lot of optimization going on in that. You you kind of need to you know play around and see what kind of results. And so I can see I understand that the harness is actually calling all kinds of stuff then, and it has the UI. Is there anything else in the harness that it, you know, like what are the other functions of that that you have to worry about or tweak? Um the things that it connects to.

So for example, if you're using uh a harness like uh open code or clawed code, you might have you know web search utilities, you might have knowledge graphs, you might have all these things that will make AI function better because it it provides more guardrails. A great harness has lots of guardrails built into it that you configure. So when you're using clawed code from Anthropic, you're configured, you know, it's got thousands of guardrails and and little tricks and stuff to make things work better. Whereas if you were to use the model, you know, you know, raw off of like uh a llama uh llama server, it's gonna go pretty wild. It's gonna be very unpredictable.

It's like you know, uh a tornado in a factory. So the more rules you have, the the the better the harnesses, the the better the model's gonna perform. In general, what we recommend is if you are just getting started out with local models, uh Olama, the free open source package, is probably the easiest server to use. And then if that's available, then pick a robust harness. Uh and for most people, that's going to be clawed code.

Um or clawed cowork, yeah, from the claw desktop app, is probably gonna be the easiest one to use because there's so much that's been done to optimize it and to make it as non technical as possible, that it uh it's it's terrific. So that's the server and that's the harness that we'd recommend. Now that brings us back to the model. Um in general, for a tool like Claude Cowork or similar, you want a model that is the most skilled at agent tasks, because that's really what that that system is. So if we go back to um the artificial analysis lovely bar chart here, you'll notice there's they have three indices.

They have the how smart is the model, how good is it coding, and how good is it at running agents. Um currently, the absolute best model, and it will cost you an arm and a leg to run it as an agent tool would be OpenAI's GPT 5.5, right? This is it it does things really well, and it it spins the meters and and your build just goes up and up and up. Um the first model on here that is an open weights model, meaning that you could download it, is uh Xiaomi's MIMO uh 2.5. However, this is not a local model because it requires a lot of hardware to run.

You're talking uh 10 grand. The first model on here that you could realistically run on a laptop is Alibaba's Quen 3.6, uh the 27 billion parameter model. This came out three weeks ago now. Um, and it is an absolutely incredible model, and it will run on a on a decently sized MacBook. So if your MacBook has 64 gig of RAM uh and a modern processor, you know, M2, M3, M4, M5, 64 gigab, you can run this and it will do really well, and it'll be really it'll be nice and fast.

Uh the next model that you could run conceivably uh on a on a laptop would be uh Quen 3.6, their their mixture of experts model. One of the things I think it's worth pointing out here on this index is I put last summer's OpenAI GPT 5, like their flagship model. Like this is this was the state of the art. And if you look, you now have models that you can run on a laptop that beat the pants off of like the super big, super expensive model from last summer. And that's how far these things have come.

Yeah, yeah, that's amazing. Because I was noticing, I mean, it's not even a 15% difference based on this scale between what you can run on hardware and the best thing you can buy with unlimited hardware. I mean, that's crazy. Yep. Uh, I have also uh Nemotron Super, that's the one that powers open claw for uh Nemo Claw, which a lot of people use Google's Gemma just came out not too long ago.

But really, the the Alibaba Quen model is probably that family is the most accessible. Uh and this is an important point because a lot of people get this confused. Yes, it is made by Alibaba. And if you use Quen in their web interface in the cloud, that cloud runs inside the people's republic of China. Um, and if you were to work with any kind of sensitive data or private data, you should not use that confidential data with it.

However, if you download the model and run it on your own hardware, it is perfectly safe to use. Um, there's there's no harm in it. Yes, you will still get some refusals like asking it about you know what happened in Shannon Square. Like it will say, Ah, I'm not gonna answer that question. Um, but that's no different than the censorship in any other models.

But there's a difference between a Chinese cloud model run in China, don't use their cloud, and a Chinese model you run in your own hardware, totally safe. All right, so then Quen is our choice, then we're gonna fire that up and see what we can do. Yep. So this is so now we get into even messier stuff. Um you have to decide what size you want because there's many different flavors of this.

If I go into Hugging Face and I type in Quen 3.6, uh there are 1,949 choices. And now how does that work? Is this is it open source too, or is it just, you know, there's the official version that comes from the maker, and then there's all these forks. Yeah, so there's the official version, which is right here. You can see there's the 27 billion parameter model, there's the 35 billion parameter model uh from Quen AI itself, and then you have all of the variants that the open weights community has made of this that have been tuned for specific architectures or tuned to do very specific things.

So, for example, there's one here. This is from the group uh called UnSloth. These this is two Korean dudes that have come up with a absolutely insane laboratory uh where they take models and they do what's called quantization, where they make them smaller and lighter while trying not to reduce their accuracy. And for the most part, they do such a good job that their quantizations are better than the manufacturers. So you can run a model for that's been modified by these guys, and it's faster, it's just as smart, it takes up less memory, um, and and they've they have started implementing some really cool uh new technology.

Um so there, if you're going to use a local model, see if Unsloth has a version of it. That would be my tip there. And so the the basic idea with that is that they're optimizing for what's not the best hardware on earth, like they're they're making a human version. Is that that that's yeah, a good way of putting it, yeah. They're they're basically taking a model and trimming the fat.

Uh, because there's a lot in in the base model that you may or may not need. Um so that's that's you know, the probably one of the preferred vendors. Their stuff is very trustworthy, uh, and very, very, very high performing. When we talk about quantization, it's it it really is it's like taking a model and applying some compression to it, trimming out stuff that you don't need, and there's specialized tools for that. Your average person's not going to do that, but you do want to look at uh if I if I tap into here and look at the different versions, you will see all the different quantizations, and you'll see that they get smaller that in terms of file size as the numbers go down, but as the numbers go down, they get dumber.

Uh, so like the smartest version in this folder is gonna be this uh eight-bit version here, 39 gigabytes. So you're gonna need a lot of memory to run it. Um, but it's gonna be absolutely pristine in its capabilities to to to to behave smartly. Once you get below four bits, and you get into the three and two and one bits, like, yeah, they they start getting really stupid. They're fast, but they're stupid.

So, what that would be if there was some specific, you know, very, very narrow use case that would be what you would use that thing for. Yep, or you would use like their studio software to to train it to tune it for just one specific task. Um, because a lot of the knowledge has been lost. So that's sort of the the text space in general. Take a look at artificial analysis for the family of model you're thinking about using and see how it scores on the specific type of tasks you're doing.

So is it coding? Is it general intelligence? Um the other thing you want to look at here is let's see, the you want to look at uh where did the GDP val go? So omniscience is how badly does this model make stuff up, right? The lower bars are better.

So for example, Deepseek, GPT OSS, uh, these models they hallucinate, like Deep Seek hallucinates 95% of the time. The more the higher the hallucination rate is, the more you know you need to be bringing your own data to the party. Um it's astonishing when you look at like the Quen 3.6 models, you know, 48 to 50 percent. That's half of what a big model like Deep Seek or OpenAI's uh GPT OSS hallucinates at. So none of these are still reliable enough that you would ever want to use them without a harness that incorporates a web search.

So that when the model says, I this is the quick answer, like go check out go check that out. Go check that out. Uh or make sure that you are providing all the data. So as a simple example, let me show you this. Uh let's start a new chat.

I'm gonna use Quen 3.6. Let's make the interface bigger here. I'm gonna turn off web search. So this model no can only rely on its own knowledge. I'm gonna say, who is this?

Oh, there we go. And it's gonna ask itself, it's gonna go through logic, and it completely hallucinates this, right? I already know who Laura Thompson is. Right, Laura Thompson. Right?

Um, completely wrong because it doesn't have access to outside knowledge. If I turn web search back on, as it goes through and thinks, it will say, um, maybe I should search this, right? So it knows web search is available. Hey, and it says Katie Robert and CO co-friend. What's interesting is the language is the same, right?

It says the CEO is also the co-founder, so it knew enough, it had enough, it's seen enough training data to know that, but it's still 100% needed the web search to be able to pull this off. So that's when we're talking about local AI, you have to have a harness that incorporates things like web search and and background data because otherwise, boy, is it gonna make things up? Um, all right, so that is the one of the scores to look at on here. Another one to look at is it's called GDP values. Can a model execute an economically valuable task?

So this is where eight like agents again looking down the board here. This is where last summer, GPT 5 where it it landed, it was it scored a 1292 on this test. Um, and if we look in the results here, there's Quen 3, 3.6 with score 200 points higher on that test. So the little model that you run on your machine scores significantly higher. This is insane that it has these capabilities.

But all of these models are very, very smart. Like we use the Singapore-based uh Mini Max, which again, they because it's a cloud provider, they are a Chinese company, they're based in Singapore. Uh the well, their data center is based in Singapore, so you should not put sensitive information into that, but we use that one a lot on their on their cloud because it requires a lot of compute. So that's text generation. Now there's a whole bunch of other models out there, and rather than uh uh you know running through them all, we have a handy page on the Trust Insights website.

If you go to the instant insights section, of which models should we should you use today? And in general, for open weights, if uh if you have the hardware, GLM 5.1. If you don't have like $10,000 worth of hardware, use Quen for 3.6, the 27 billion for animal model. It's so smart. It is a great coding model, it's a great text model.

There are models that can generate video. The current best model that you can download and run on your computer is LTX 2.3. It's not great, it's got some work. Um for image generation, the the best local model is Black Forest Labs Flux2 Dev. So this is a model that uh very capable, it generates really decent images.

Um I can actually demo this one. Uh we use also use a provider called Deep Infra. They're a local, they're an they're one of those cloud-based providers. I will use it for today because it's way faster than waiting for this stuff to start up on my laptop. I can run on my laptop.

Boy, does it take a while? So let's look at Flux2 Dev. And let's go into their demo. And let's put in. Anyway, a photo realistic image of a CEO and their sidekick in a modern office conference room.

This was the prompt that I'd used for uh LinkedIn the other day. Um and we're gonna do 1024. Let's use 1024 by 768. So landscape format. And hit go.

And we'll see what it comes up with in terms of its ability to generate. And that it came up with a dude and a dude with a man button. Yeah, man button. All right. Um, that's that's a choice.

Um, but you can see for an open weights model, that's not terrible. Right. So this is run on your computer. Definitely. Um so that's what flux two dev looks like.

For text to speech, one of the uh shocking new ones in this space is um super tonic. So this is a model that is uh it just came out a couple weeks ago, and it runs entirely locally, and it comes with 10 different voices, five uh voices that are coded as male, five voices that are coded as female, two of one of each that is coded with a a UK accent. Uh and it runs entirely locally, and it's very, very fast. So I was trying it last night, and I generated a very short clip. So let's give this a quick listen here.

The world's best marketing podcast with Christopher Penn and John Wall. You're listening to marketing over coffee. The world's right. That's not terrible for a something that runs entirely locally on your device. So if there's text content that you wanted to turn into like spoken word, it the the best in class for that is Google's uh Gemini Flash 3.1, but you get charged for it.

And uh so we did uh the gender of AI for SEO and PPC book. We made it in an audiobook version, and that cost about 10 bucks. Now, granted, that's not a huge amount of money. But if you had a lot of text you wanted to read aloud, that adds up after a while because that that was only like 40,000 words, right? So if you've got a lot of text content that or you want to be using it with like a voice interface agent, that that adds up fast.

Super tonic was uh uh pretty terrific. The downside is you need to be using it with Python code, it doesn't have a harness. Right? So you with with like a clawed code or an open code, you have to build the harness yourself. So, how much of a lift is that?

I mean, I still imagine you're just throwing commands in the IDE if it's Python driven, or is there all kinds of other stuff you have to configure? There's there's so there has robust documentation. So, what you would typically do, uh again, huge surprise is you'd use the 5P framework to build a project plan to say, like, hey, I need a uh a Python application that will use Supertonic to render this text. I'll put the text in this folder. You know, here's how you'll process it.

Here's who the audience is, uh, and who's gonna listen to it. Here's the the you know, how you're gonna clean up the text first to make it speakable, because that's a big part uh that a lot of people do not do well. Um, and then here's how you know you did it right, right? You'll be able to uh you would give it to uh transcription software like Parakeet MLX, for example, which we'll talk about in a second, and you'll check how many words do you screw up in the speaking allowed of it. So you can use these two two kinds of tools in parallel to say, did you do it right?

Oh, that's crazy though. So I I didn't understand where you're coming from originally with that. So it's not that you drive it with Python, it's like you need to use Python to build the thing to make it work. Yes. Okay.

But if you do it, it it is pretty fantastic. Uh, I have actually have it uh was using it earlier today for a client project because the client wants to turn a bunch of their text content into audio. And we're talking thousands of pages. Um, and that would give them a bill of hundreds of dollars to do that. And instead, I can literally put their page content into the Python app and say, just make MP3s, and then they'll be able to take that and load it, uh, you know, uh put it with a cover image and load it to YouTube and stuff like that, and and and get that done at very, very low cost.

Yeah, right, not have to pay to have the cloud chew all that stuff up. The best current music generation model for that you can run is one called AC Step. Uh, and it's pretty terrible. Um, it's it's not great right now. However, it is better than it the open source open weights models used to be.

Open weights models used to generate stuff that sounded like cats screaming. It was so bad. A step sounds like where Suno was maybe two years ago. Suno is currently the best in class there, but it is cloud based. A step you can run locally or will run on a on a MacBook or a decent gaming laptop.

It's got a ways to go still but it it it has made some serious strides. Well how about I mean Suno is so good what kind of use case would make you want to run that locally like what could you benefit out of that? If you want to do a lot of it's a ton. Yeah because Suno you you get charged like 20 credits per you know per song and you know you run you can burn through this pretty quickly. Whereas A step it's you're limited by computational power.

For speech recognition one of the best fastest models is a model called Parakeet by NVIDIA. And this is one of those rare ones where it is not only the open weights choice it's also the first choice. It is so good insanely fast. And the tool that I use it in the most uh is an open weight open source tool called type whisper this is I would if you are a Mac owner and you want like nice text to speech transcription completely for free this is the tool to use because you download it and then you specify I want to use the Parakeet engine. And that will download the Parakeet model on your computer.

And this is the harness for that model. If you've seen like whisperFlow and Super Whisper and all these different uh people, you know, with these these typing models that you like, you just dictate knit types, and they're all paying 20 bucks a month for their software. You could be like, Yeah, I'm gonna do this for zero dollars a month. Install type whisper, which uses the parakeet model, and and enjoy the same utility. I would I reckon I you know I can't speak highly enough of it.

It's it's so good and it's worth doing. There's two versions. Version two for parakeet is English only, but has extremely low word error rates, meaning it screws up the least. Version three has slightly higher word error rates, but also is multilingual and speaks like eight different languages. So if you were working in like French uh or German uh and you wanted to voice recognition in those languages, that would be the model to use.

That's interesting. That opens up all kinds of doors with agents as far as translating and all kinds of things. Yep. For agentic operations, um, Quen 3.6 is the openweights model to use. It is, you know, if we go back to the artificial analysis and we look at the uh at that leaderboard uh at the very top for a gentic uh capabilities, this model is insanely good.

Like it is up there with it, is a it is a peer of um Claude Sonnet, right? Which is anthropic's second best model. Um, and if you look at Claude Opus 4.5, which was best in class in January of this year, it's better than that. Like that is the part that is difficult to for a lot of people to wrap their heads around this. It is so good that it it beats a foundation model from four months ago.

Here's the use case for that. If you would be using a system like Hermes Agent or OpenClaw or something like along those lines, you would run the Quen model locally in a tool like LM Studio because it it has its own uh API, and then you would hook your agent into this. And then instead of spending a gazillion dollars on anthropic uh costs to run an agent or spending you know hundreds of dollars on token plans or agent plans, it's the cost of electricity to run this, particularly if you have good hardware to run it fast. Um I'm using the the Quen 3.6, the the mixture of experts model, the 35B model, which again is if we go back to that chart, is still pretty darn smart. It's it's down here, it's it's scored to 58.

Um, so it's still on par with Claude Sonnet. And I can run my Hermes agent off of it and spend no money at all. So if you go back, if you remember last week's live stream when we were talking about using Hermes Agent for like sales prospecting, and we and we built that a little agent to do that, I could power that whole thing by a model running on my laptop and pay no money at all and have you know, other than electricity, and have it be working 24-7 just doing sales prospecting for me. Right, yeah, yeah. I can totally see you just have the thing running continually in the background.

And as long as it's plugged in, you're good. You're not you're not getting paying these huge expenses the whole time. Exactly. Um, so those are the models, that's sort of the local model roundup these days. Uh Quen 3.6, hard to beat, hard to beat on uh across the board.

Um flux2 dev for image generation, super tonic for text to speech, A step for audio generation, Parakeet for speech recognition, um LTX 2.3 for video. Oh, it's not like I said, it's not great. Um I would say it is you know, this this list changes weekly. Uh Keith is asking, can I ask a basic AI question? Of course, you can ask any question you want.

Uh whether we answered or not will will depend, but feel free to to ask questions. And while Keith asked that question, John, in terms of uh utility. Our our friend uh Tom Webster bought himself a uh a DGX Spark, which is a high-end piece of AI hardware, it's about five thousand dollars. Um do you see yourself buying hardware like that to run your own models on the on the farmstead? No, five, because you know, I mean, if you're talking about five grand, so then you know you can easily say, well, how long is it gonna take me to burn a thousand bucks in the cloud?

You know, I mean, I would throw down major money in the cloud. Um, yeah, I I think I would really have to hit some kind of use case where either I had something that was running continually, or I was in some situation where I was burning tokens at just some insane rate and it is holding me up because that's also a workaround. Like I can set up three cheap boxes with all different names and you know, buy some cheap tokens and still make some stuff work. So, yeah, I don't see any um reason for me to cough up five grand yet. Now you're talking about a thousand dollar price point, then everything changes.

Then it's like, well, I can cough up a thousand bucks to start playing and just to see what kind of trouble I get into. But yeah, unfortunately, I guess I'm showing my card of where I am in the economic cycle. I am not in the five grand does not play around money. Yeah. Well, in the early days of um open claw, a few folks got bills from Anthropic for eight, ten thousand dollars because the meter was running and the agents were just going crazy, uh, which was was pretty intense.

So yeah, that five grand box suddenly looks real cheap. You taking your job suddenly looks awesome. Exactly. Uh Keith's question was I often hear about protecting data, but how do I use formula and organize that data? So that's a data governance question.

Uh, in terms of how do you manage your data? And uh the short answer is you have to create, you have to have your digit in formats that generative AI tools can read, especially if you're gonna use local models because they're not as smart as their cloud counterparts. So they need uh to have it be more friendly to you. So that means text files are your friend, especially if you can put them into a uh format called markdown, which is uh very use machine friendly. It's plain text with some special characters in it.

If you're doing tabular data, uh format you're looking for is format called YAML, Y A M L. Uh rhymes with Camel. And again, that takes what is uh like a CSV file and turns it into a really long list instead that an AI model is is able to understand. Uh, what you generally want to avoid is you want to avoid uh Andy Crestedino talked about this in an article recently, which I thought was a good analogy. If you have to open up an application other than a text editor, uh, it is a file format that AI is not going to be efficient at.

So if you never open up Microsoft Word or PowerPoint or Photoshop, whatever, that's not a good format to work with AI. If you can open up notepad uh and work with the file uh and and have it work, that's that's gonna work for you. So I would say, in terms of you know, formulating and organizing your data, convert it to formats that AI can work with, and then you'll have a much easier time. Yeah, and there's tons of markdown tools. So it's not it's not like difficult to get to MD.

So no, exactly. Um, and if you don't have any, ask Claude or the uh agent of your choice to write you one, uh, and they can do that. All right, John. So any final parting thoughts on local AI models and and what's next. No, you know, it's interesting to think about use cases and when you could play with it.

But yeah, unfortunately, the punchline for me is uh 5,000 bucks is still too much. One thing to think about for an organization, especially if you're a small business that has like five employees. If you want guaranteed AI that is yours, you might spring with that $5,000 box and then use uh a technology called Tail Scale. It's actually in the Trust Insights newsletter this week, which uh you can find uh over at TrustInsights.ai/slash newsletter. Whoops, there you go.

Um Tail Scale allows you to create a VPN no matter where you are in the world. So you could create this for like we could create it for all four of us at Trust Insights, have our computers connect to that network, and then have the box living in like you know, my basement or Katie's basement or something. And then it would, we could all connect to it and use it as though we were using a cloud-based server, but it would be uh ours. And we and and if anthroopic went bankrupt next week, you know, or open it or Sam Altman and opening I got arrested or whatever. Uh and that provider went out of business, we would still have options.

Yeah, that's a super idea. You know, networking a single box like that, that's very easy. Because now you're you're talking fractional cost. That that's a real interesting proposition. Yep.

The other thing is, and this is highly experimental. So if you like living on the edge, uh if you have a lot of nice computers at your company, uh again with Tail Scale, there is a model harness going back to the beginning of the show. The harness called XO EXO that looks on your LAN and provides a model harness that it uses all the GPUs of all the computers in it. So if you had a company where you had four people with four really nice MacBooks, you could load some of the bigger models and it would share the load across all the machines. That works best on the LAN, but even on you know, as long as everybody who is on the network has good internet, uh, you could you could conceivably load really big models that way too.

That's nuts. And I imagine there must be some crazy pirate stuff where you could just buy 65 Xboxes and like put them in a closet. That but it always seems like the there's also a commensurate level of headache with the lower price. You're just you're just trading one for the other. Yeah.

So again, some of those YouTubers who have like 10 Mac Studios all chained together, that's how they're doing it. They're they're wiring wall together. And yeah, they've got a closet with a cooling fan. Stuff. So it's pretty crazy.

That sounds good. I won't be running up my heating bill. Yeah, I won't be heating my house with servers this winter, but we'll we'll see how it goes. Yeah, we'll see how it goes. Although, this is kind of a funny thing in terms of uh hosted AI.

This is not available anywhere outside of France, but apparently this is a thing in France. The French AI companies um essentially give a wall-mounted unit that contains a couple of GPUs to people, and they and if they pay for the electric electricity for it to pay for people to have it in their houses as distributed processing so that the heat is and people use it as a literal space heater in their houses, and so they're using they're intelligently using the waste heat from these things to heat houses. Um obviously it causes issues in the summertime, but not in the wintertime. But I'm like, this is such a fantastic idea to have your you know a data center that you just put in like some French farmhouse, and they're like, oh, this is our this is our space heater that is an AI node. Yeah, right.

There's not heat is not a waste product in the right market. Exactly. So that's gonna do it for this week. Uh next week, uh, we will be back. Uh Katie, well, I should say Katie will be back on the show.

Uh, I believe next week we are doing uh getting started with Paperclip, the AI agency. So that'll be next week's show. And so until next week, thanks for tuning in, and we'll talk to you on the next one. Thanks for watching today. Be sure to subscribe to our show wherever you're watching it for more resources and to learn more.

Check out the Trust Insights podcast at trustinsights.ai slash TI podcast and a weekly email newsletter at trustinsights.ai slash newsletter. Got questions about what you saw in today's episode? Join our free Analytics for marketers Slack group at trust insights.ai/slash analytics for marketers. See you next time.