You Ask, I Answer: Why Do Open Source Large Language Models Matter?

In today's episode, Amy asks, can you explain in layman's terms why open source models like Llama 2 matter so much? Can you break it down so that we can understand what it's what's important about it? Okay, so the Lama 2 model, which was released by Meta, I still want to say Facebook, but it's Meta, is an open source model in that they have given away the model itself and the model weights. What that means is that anyone who agrees to the license can take this language model and run it on their own hardware or software. And if you have some fewer than 700 million users per month, you don't have to pay Facebook Meta.

You don't have to pay Meta a dime. It's just free to use. Now, when you look at existing models, like the models that power ChatGPT or the models that power Google Bard, we have access to the interface, right? We can log on to ChatGPT and we can start typing, and we'll get some results for free. And if you want more or better stuff, you pay the 20 bucks a month for the pro account.

Um but you don't get the model itself. You get an interface to it. That means that you are perpetually using someone else's model on their terms in their system, in their services. And at any point in time, they can turn off the switch and you no longer have access to it. As we've seen many, many times with Chat GPT when their demand is very, very high.

They're like, sorry, we're having performance issues, so the service is not running right now. You can't do anything about it. You're like, uh, okay, well, I guess I'll just get a sandwich because um it's not working. That's a model owned by someone else. An open source model, and it's there's some there's some debate about whether you can truly call it open source because Meta has not released the training data used to make the model.

They've only released the the fully baked product itself. However, there's a whole very arcane system of decompiling and retuning models. We're not going to get into. Effectively, though, functionally, it is very much like an open source product. You can take this thing, you can modify it, you can run it, you can do all sorts of stuff with it, and you don't have to pay Meta for it until you reach that 700 million a month user mark.

Why is this a big deal? Because if you want to integrate generative AI, uh large language models into your product, into your product or service, if you use a big provider like OpenAI or Microsoft Azure or Google, you have to pay for it. Every time someone uses a chat bot in your in your app, you have to pay a small amount of money to open AI or to Microsoft or to Google. And that adds up real fast, right? It's it's fine in testing.

Oh, it's just three tenths of a cent to run this query. Okay, great. What happens when it's 100 million users a day asking those questions? What happens when it's you know, people asking 80 questions uh at a time? Those bills add up real fast.

Also, at any point, you know, Microsoft or Google or OpenAI can say, well, you know, I really don't like how you're using my software, so I've cut off your account. Which is really problematic when you think about it, because it means that access to this technology is heavily controlled by large technology companies. When you have an open source model, you download this onto your laptop, onto your servers, onto your compute infrastructure, and you run it with open source infrastructure like Cobalt or Llamas.cpp or any of the many, many, many, many engines that can run a large language model that are all open source and free. At that point, no one can control you anymore. No one can say, hey, you can't use the model for that, because it's on the privacy of your own laptop.

There's no data going across the internet. If you want to have a conversation with uh Llama Instruct, which is the the chat version of it, um, entirely about like clowns covering themselves in catch-up, you can. You can have that conversation that's completely private, it is completely up to you how you you interact with the model. From a business perspective, again, as long as you're under that 700 million user a month requirement, that means that instead of paying hundreds or thousands or tens of thousands of dollars to open AI because you've integrated their software into your product, you run this model instead, and you pay the cost of your compute infrastructure running it instead. So if you've already got a room full of servers, um, it's not a huge stretch to add more and to run the model yourself.

Then it's yours. If chat GPT is down that day, you don't care because your servers and your infrastructure are running your version of the model, and no one can take that away from you. No one can censor you, no one can tell you you can't do this or that. I mean, they can tell you, but you can't that they can't enforce it. And so it's that's a re it's it's about independence, it's about freedom of choice, it is about control over the AI experience.

You can also take these models and tune them. You can you can modify them. If you have if you download the Llama 2 base model, you can retune it and say, okay, I want you to be specifically expert in accounting. You're gonna know general ledgers, double entry bookkeeping, um, generally account uh accepted accounting procedures, you'll know uh tax uh structure, you'll know all this stuff, and you'll re-weight the model. You will influence it, you'll modify what it can and can't do.

It will get worse at some things, right? It will as you the more you tune a model towards a specific task, the less it can do other tasks. So it will stop being able to tell jokes as well. But boy, will it answer any question you have about any accounting scenario or any marketing scenario or any uh uh HR scenario, you name it, you can retune these models. And so these are the reasons why, at least from my point of view, these open source models are really, really important.

They allow you to build your own infrastructure, they allow you to tune a model to very specific uh uh circumstances, and they give you more control, they give you more control. If your company deals with uh personally identifying information, sensitive protected information or uh protected health information, you absolutely should not be using a third-party model for those. You should under no circumstance be handing that to open AI or Google or whatever. When you download and run the Llama model, for example, from Meta, yes, Facebook made it, but it's not on their servers, it's on your laptop, and no data is being transmitted to Meta. It's all being run locally, which means that you have control over the data that you're giving to a model, what it does with it, and where that output goes.

And for ensuring privacy and data safety, that is the only choice that you can uh use in large language models and still comply with all the laws that you're required to. So that's why it's a big deal. It will it's getting to be a bigger and bigger deal every day, and it's something that people need to be aware of and to have in their mix of their generative AI solutions. So, great question. Thanks for asking.

If you like this video, go ahead and hit that subscribe button.