Mind Readings: Build Your Own Generative AI Benchmark Tests

In today's episode, let's talk about benchmarking and AI and knowing whether an AI tool is meeting the it is capable of meeting the hype about it. What are your go-to benchmarks when you want to put an AI service to the test? Here's the thing. Every time an AI company releases something they claim is state of the art, we all kind of nod, like, yeah, yeah, state of the art, good, good job. But we don't really have an agreed upon set of metrics about what constitutes state of the art.

There's a ton of synthetic benchmarks in AI. You'll hear terms like MMLU and the LSAT test and uh human preference and and um all sorts of different synthetic benchmarks that people use to test AI models. But these tests have a lot of problems, one of which is that the models have learned the tests themselves. And so they're really good at at uh testing well, but they don't necessarily adapt outside that, kind of like a uh academic genius who doesn't do well in the real world. Another major problem with synthetic benchmarks is the benchmark may test for things you don't care about or or things you you don't do.

If you want a uh a tool that reads your email and replies to it uh appropriately, that's a real world test that has very specific parameters, but synthetic model tests, they're not gonna measure that. So one of the useful practices you should have on hand are your own benchmarks for how well a model or a service or a vendor can do. Um earlier, uh not too long ago, someone dropped a pitch in my in my DMs about the they were saying this is the ultimate voice isolation AI. This AI can isolate any voice from its background and present it in studio quality. Many, many products have made this claim over the years, and none of them have lived up to their promises.

None of them have gotten even close. So I have a benchmark test with for this. That is a piece of video. So it's a short interview with uh actress Katie McGraw from um shows like Superworld. She did an interview at Sandy Group Comic-Con.

The interview was clearly not done by professionals. Um fans, which is great for this test because the interview is filled with background noise. And critically, it's filled with background noise of other human voices. Humans can understand this. Machines cannot.

Most noise removal mechanisms or noise generative mechanisms filter on non-speech frequencies. So they can take out, you know, a jackhammer in the background, because a jackhammer and a human voice are very different frequencies. Or they'll extract speech frequencies and pass them through a generative model and essentially reconstruct the voice. But with this interview, there's no way to do that. In fact, let me play a clip of it.

I guess heroes and villains are heroes that have a chance of being evil and villains that always have a hope of intention. I have tested this clip against every vendor that says that they've got state of the art, you know, amazing quality. None of them, not a single AI tool, not a single sound cleaning tool has ever made this interview studio quality. It has always come out sounding garbled and garbage because it's a really difficult task. And so that's a that's a great benchmark.

Uh our tools getting better, but this particular use case, not really. Um this is my gold standard. If you have a tool that you claim is state of the art can do a perfect job isolating a voice, this is the the test. If you can clean this up and truly make Katie McGraw's voice sound studio quality with no priming and no reference data, then you've got a winner. Another test I use is for coding.

In the R programming language, there is there's a library called HTTR that for years and years was the gold standard for doing uh web requests inside of R, the particular type of function. About three years ago, Hadley Wickham and the Tidyverse crew, who are uh amazing contributors to the R language, uh, did a ground up rewrite of its new library, uh calling it HTTR2. Now, a model's level of sophistication in coding is whether it knows whether to use HTTR or HTTR2. Their their function calls are similar, but not the same. And this was released three years ago, so it's not new information.

So it this is a test of a model. When I'm coding and I ask a model, hey, help me do some web requests in R to see which library it uses. Is it smart enough to know that HTTR2 supersedes HTTR? And you shouldn't use the old one anymore. The reason why models have trouble with this is because the vast majority of older code on the web, like on stack exchange and stuff, is in the old format.

And so a model that knows to prefer HTTR2, understands not only code, but understands the age of code and the logic and the reason, the the sensibility of using newer libraries. Older models, they don't know that, or less less skillful models don't know that. And that's a a really helpful test just to understand how smart is this model. In Python, there's a there was a newsletter, uh, there's an application or no, it's a package called Newspaper 3K. The maintainer stopped maintaining it two and a half, three years ago, and there's a new fork of it called newspaper 4K.

Now, if you're a human programmer, you would go to the newspaper 3K package if you saw it and say, hey, this package is no longer maintained, but someone else has taken up a and forked it and started a new version over here. And you would know, as a human, I'm gonna go over there to the new one. If a language model understands that, then it shows that it has some reasoning. And I'll tell you, as of right now, of all the state of the art models that are in existence that you use for coding in Python, none of them know this. They all are still relying on the older one.

So that those are two examples of benchmark tests. What are your benchmark tests that you use to evaluate AI solutions for your specific use cases? What are the things that used to use to stump AI that you know defy and and and maybe bring down to reality some of the claims made by different AI tools and vendors? If you don't have that list, it's a good time to build it. In fact, one of the best times to build is before you issue an RFP.

And in the RFP, you're just saying, you know, a model vendors will be evaluated based on a series of tests, but you don't tell them what the tests are because you don't want them teaching to the test. But a set of objective tests like that can really help you understand what the capabilities of a model actually are and whether they can solve your problems. So I hope you build that list for yourself. That's gonna do it for today's episode. Thanks for tuning in.

I'll talk to you soon. If you enjoyed this video, please hit the like button, subscribe to my channel if you haven't already, and if you want to know when new videos are available, hit the bell button to be notified as soon as new content is live.