You Ask, I Answer: Large Language Model Capability Limits?

In today's episode, Omar asks, is there a limit to the capability of large language models? Well, yes, of course there is. There's limitations to any kind of technology. The limitations occur in a couple of different places. So the first place a limitation occurs on a large language model really is on the training data on how the model was made, what content was used to build that model or fine-tune it.

And depending on whose model you're using, this might not even be clear, right? When you look at something like stable LM, stable LM uses uh the pile, which is a uh data repository created by the Ulithra AI. And so it's pretty clear what's in the pile and what's not. So if there's content that you're creating that was not in the pile, the model's gonna have a harder time generating that. Other models, like, for example, um the GPT family from OpenAI, we don't know what's in there.

We don't know what's in there, so we don't really know what its limitations are. We do know there are certain prompts that work better and worse. Uh, and that's likely a reflection of the fact that the content that's in there is kind of a mishmash. We don't really know for sure what it does and does not know. So that's one aspect of limitations.

The second set of limitations are actively imposed limitations. There are uh limitations that model makers, particularly closed source models, put on their models so that they can't do certain things. Uh if you were to ask OpenAI how to uh in the chat GPT interface, how to make a certain type of um homemade chemistry experiment that you under no circumstances should do at home. It will say, hey, I can't do that, right? Even things that seem innocuous, like saying, hey, I need to uh I'm getting ready to do a roast for my friend's uh thing or a bachelorette party.

I need you to write 20 insults about my friend. And this the software will say, nope, you uh we can't do that. That's not respectful. Um, we should always treat people with respect and kindness. Well, yes, that's true in the abstract, but in reality, I'm trying to have some lighthearted fun at my friend's bachelorette party, and the software is not helping.

So those that's an example of an imposed limit on these systems, and there are many. And again, with closed source models in particular, we don't know what those limitations are until we hit them, until we run face first into them. The third place that there's limitations is us, the people, the operators, right? So the models are very capable, us less so. The quality of your prompt matters a great deal for being able to determine what output you get, right?

Whether the output is any good, whether it's usable or not. And people who are less skilled at writing prompts are going to get lower quality results. So it's not necessarily that the model's capability is limited, it's the person's capability to get useful information out of the model. Um that is that can be challenging. And finally, of course, there's the inherent technical limitations of the models.

Uh, today, large language models still struggle with reasoning, they still struggle with with true logic. Everything that they do is probability-based, which means that abstract thinking and abstract reasoning is just not in the cards because by definition, that goes against probabilities, right? If you ask somebody to create something that's never been seen before, the way a human does that is going to be very different than the way a machine does that. Now, machines will probably not do that nearly as well because they still need to draw upon knowledge of what has gone previously instead of being able to create something that is truly net new. Uh, that's a limitation of the fact that that's how artificial intelligence models work.

So there's a lot of different limitations on these models, and the limitations vary from model to model. Stable LM has fewer limitations than GPT-4. Llama, the was it snooze Snoozy, the 13 billion uh Llama derivative, that has fewer limitations than stable LM. So there's a lot of these different models that are out there that have greater or fewer restrictions based on their architecture, how they were made, uh the types of prompts they need. Uh so there's it really depends.

It depends on the model. We it used to be sort of a one-trick pony world, right? Everybody used chat GPT. Those days are gone. Um now it's like saying there's cars on the road, right?

And you might some people might have a Ferrari, some people might have a Tesla, some people might have a Hyundai. And it it varies. It's gonna vary. So there are limitations. The limitations you should be concerned about are the ones that that are inherent to the model that you're using.

So getting to know the model that you're using and being skilled at knowing what its limitations are and working around them is really a critical skill to be able to succeed in the use of large language models. Good question. Thanks for asking. If you like this video, go ahead and hit that subscribe button.