You Ask, I Answer: Open Weights, Open Source, and Custom GPT Models

In today's episode, Joseph asks if I wanted to dabble in an attempt to make my own custom like GPT, a language model, using something that is open source, do I need to use something like Llama to accomplish that goal? Okay, so this is a little bit tricky. The Llama models are what we would call open models in the sense that you can get the model itself, the model weights and download them and use them and you can fine-tune them and manipulate them and things like that. They are not strictly if you want to be adhere to what open source is really about. They are not open source models.

And here's why. Open source requires the disclosure of the source code, not the compiled binary. So if you write uh a piece of software that you compile in C, if you want it to be open source, you have to give away the C source code itself and not just the compiled, you know, and end product, the app itself. With language models, extending that analogy. If I give away the Llama model, I'm giving away open weights.

Here are the weights that you may use to uh manipulate and change into a model that performs with the tasks you want to perform. To for it to be truly open source, the training data that the model was made from would also have to be given away. Right? So this would be things like Common Crawl, for example, or archive and Stack Exchange and Reddit and the books are uh the online books archive and Project Gutenberg and all that stuff. If you wanted to do a true open source language model, you would need to open source the training documents themselves.

And some of these exist. Um for example, the repository that like 90% of language models are trained on is called Common Crawl. You can go visit it at commoncrawl.org. This is a massive, massive archive of essentially the public internet. It's a web crawler that goes around and scrapes the web.

And anything that it can see, uh it puts in there, unless people specifically tell it not to. That huge common crawl archive is what a lot of model makers use as sort of their base starting recipe. There is definitely opportunity for someone to look at that archive and selectively pull pieces out of it to train and build a transformer-based model, a pre-trained transformer model from scratch, from absolute scratch. So you'd say, here's we're not going to use LAM as a starting point, we're going to make our own. This requires, however, an enormous amount of compute power and time.

When Lama 2 was put together, I think it was something like several roomfuls of A100 GPUs and about $2 million worth of compute time to build this thing over, I think it was 12 weeks was how long it took roomfuls of servers to build the Llama model. Most of us do not have that kind of firepower. Most of us we just can't afford it. It's going to be a toy. If you want to build a custom GPT-like system, yes, you could start with something from the Lama 2 family, because Llama 2 is open source and open weights and it is commercially licensable.

And then you would uh do one of a couple different ways of customizing it. One would be fine-tuning it, where you would give it additional instruction sets and essentially uh alter the weights in the model so that it performs some some instructions better, right? So you might have thousands of examples like, hey, um, when a customer says this, do this. When a customer says do this, do this. You might have thousands of those things, and you would then essentially retune Llama to follow instructions like that better.

That's what fine-tuning does. You might also want to add new knowledge to Llama. And that's where something like retrieval augmented generation would come into play, where you would say, here's a library of of extra data. You should look in this library first before you go into your your general library, so that you get better answers. Those would be methods for customizing it.

When you look at something like OpenAI's custom GPT, that is uh a model that is that is a system that is largely custom instructions, so you give it specific prompts and retrieval augmented generation. You upload files to it and it can talk to those files, or you can make a function call to call to external data sources. Um it's not a fine-tune, right? You're not getting you're not convincing it to uh learn certain instructions better, not really. So that would be how you would accomplish that goal of making that custom-like thing.

You would uh you would do the do a fine-tune if the llama model just doesn't answer the questions the way you want them answered from an instructions following perspective, like it just doesn't follow directions well, or if it doesn't have the knowledge, you would give it access to some kind of vector database that would have the knowledge you want in it that it could then reference. If it can follow instructions fine and just makes up answers, retrieval augmented generation is the way to go. If it can't even follow instructions, fine-tuning is the way to go. So that's how you would approach that. I would suggest, as a starting point, trying OpenAI's custom GPTs just to see if your idea is even feasible first.

Because if you can't get it working in a no-code environment that's pretty simplistic, there's a good chance that you would spend a lot of time and money and effort on a more custom example that probably wouldn't work much better. So give that a shot. As always, if you have additional questions, feel free to ask them at any time. You can leave them in the comments or whatever. Thanks for tuning in.

I'll talk to you next time. If you enjoyed this video, please hit the like button. Subscribe to my channel if you haven't already. And if you want to know when new videos are available, hit the bell button to be notified as soon as new content is live.