You Ask, I Answer: What Does ChatGPT Learn?

In today's episode, Carol asks, does ChatGPT learn from my prompts and the text I feed it? And does it feed that knowledge to others? What research value do they get from our interactions? So with any AI service, it's always a good idea, always a good idea to read the terms of service, to read the frequently asked questions and any technical documentation that's provided. OpenAI, which is the company that produces chat GPT, has a good amount of actual documentation and disclosures on its website about what the model is and how it works.

So from uh Bullet Point 2 in their frequently asked questions, uh, ChatGPT is fine-tuned from GPT 3.5, a language model trained to produce text. Chat GPT was optimized for dialogue by using reinforcement learning with human feedback, a method that uses human demonstrations to guide the model towards the desired behavior. So what this means, when you're talking about reinforcement learning, reinforcement learning is when you train a machine learning model to perform a task of some kind, chat, uh, score things, guess things, categorize things. And then you essentially take the uses of that model and you know, thumbs up, thumbs down, whether it did its job, and you feed that back to the original data set, and then you retrain the model. You basically have it rebuild itself and you keep doing this over and over and over again so that over time, as long as the responses are intelligible and well curated, the model gets smarter.

The model gets better at doing what it's supposed to do. Now, chat GPT uh does absolutely uh log everything you type into it. Uh in fact, uh, when when you read terms of service, term number six. Will you use my conversations for training? Yes.

Your conversations may be reviewed by our AI trainers to improve our systems. So everything you type into this system is being logged. And in fact, there is a um a specific note in here in the terms of service, like, hey, don't put confidential information in here because this is not, you know, this is not the place for that. Uh it is being logged. So is it learning from your prompts and the text feed it?

Yes, it is. The most overt way you can give feedback to uh open AI is that thumbs up, thumbs down. When you are using the service, you'll see the little thumbs up, thumbs down icons right next to its responses, and you're reading each response. Yes, this was a good response, no, this was not a good response, helps the model learn, right? It provides that that training feedback for them to retrain their software.

That said, you can tell a lot about um someone's responses by how much refinement there is, right? Uh so even if you don't use the thumbs up, thumbs down, if you say, write me a poem in the style of Edgar Allan Poe, but about the 2020 presidential election, and it does its thing, and then you keep asking for refinement after refinement after refinement. That's a pretty good indicator that the model's not doing what you intended it to do because it didn't nail it on the first shot or the second shot or the third shot and so forth. So even in cases where you're not using that built-in ratings feature, there are plenty of behavioral signals that would indicate, yeah, this this thing is is not going right. Um I don't know this.

Now, this is purely speculation on my part, but it would completely not surprise me if the the outputs and then our inputs were basically being added to the training data set uh at in hold, right? So when you write a prompt and submit that, that goes into training data, right? For the next iteration of the model. So I would absolutely assume that that knowledge, any knowledge that we're creating with this software, is being incorporated into that reinforcement learning system, uh, the human feedback. Uh some folks in the AI community will call it active learning, where it's it's retraining itself on a regular and frequent basis.

OpenAI seems to release a new version about every month or so. So my guess is they they collect data uh for a certain period of time, they then retrain the model and they roll out the newly retrained model. And obviously, if it goes sideways for some reason, they can just roll back to the previous model. But that's that's what I'm pretty sure is going on underneath the hood. So what research value are they getting from our interactions?

They're rebuilding the model, right? They're improving the model. Open AI makes these really large language models, the GPT series of models, um generative uh pre-trained transformers. So they had GPT 2, then three, now 3.5. Later this year they're going to be releasing four, which will, despite all the hype online, it's just going to be more of the same, right?

It'll be better at what it does because it'll have more data. And critically, this is the part that I don't think people understand about these map these models. Critically, our interactions with it provide richer training data than they can get just by scraping the internet itself. Because if you scrape like a Reddit forum thread, yeah, you have some context, but you don't have that that thumbs up, thumbs down, that behavioral data, as if as opposed to when we work with the model directly and say, write me a poem about um, you know, uh uh slicing cheese, but in the style of uh, I don't know, some poet. Um, um I can tell my betraying my my lack of studies in English class.

Bath. These are very clear pairs of information. You know, uh, a prompt response, prompt response, and that's better quality training data for someone who's building a large language model. So that's what's happening with the data we're feeding into this. Um, it is where we know for sure from what's disclosed, it is being used to retrain the model.

It would not be surprised is without surprise me in the slightest if it was being used to train the next iteration of the big model GPT-4, right, with all the conversations, because this thing is taken off like like wildfire online. And so thousands, if not millions of people are freely giving it a lot of information. Um, and that you as a researcher, as someone trying to build software, as um someone trying to acquire high quality data, you couldn't ask for a better, better way to do that than to have a bunch of people eagerly running in to provide you with more training data. So that's what's happening, but expect no privacy. It's in the terms of service, expect no privacy.

If you are putting in like, you know, you have this rare medical condition, I don't know that I would put that into a system like this that is going to be reviewed in in some part by the AI team that that builds this. So good question. Thanks for asking. If you like this video, go ahead and hit that subscribe button