Saturday Night Data Party: Analyzing Text Metrics

Okay. Welcome to Saturday Night Data Party. It is Saturday, the 18th of July. Tonight we're taking a look at um some data from the popular fanfiction website, Archive of Our Own. Now, if you're unfamiliar with this site, uh, it is a essentially a fictional place that you can go and find essentially people writing about pretty much every topic under the sun, including fiction that is was set in other universes like uh music and bands and and movies and television shows.

When you dig around in here, let's go ahead into TV shows. Uh let's pick a TV show. What's a popular TV show? Um I don't know anything about TV shows. Uh let's just look for yeah.

Okay, um, let's what's got a lot of data? Uh the A team. Yeah, that's good old 80s stuff. You'll notice that there's a lot of metadata. There's a title, there's an author, there's tags, there's the language it's in, the words, the chapters, um, kudos, which is essentially like you know, votes or likes, and then hits, which is the number of views.

One of the first things we have to tackle when we deal with any kind of data extraction, is we have to figure out how we're gonna get the data out of uh a site like this, because there's no API, there's no um there's no uh provided means to to extract this information. And what we really want to get a sense of is where are where's the data gonna come from? So the good news is that for any even moderately popular site on the internet, someone somewhere has already done the hard work of building a piece of code that will extract the data for us. In this case, uh we're gonna use a uh uh a Python scrape. So let's do Python scraper, search for this, and of course, our first result comes up on GitHub.

This is a script by uh Radiolarian, and it is open source, it is free to use. It looks like it uses beautiful soup. Um, and then we feed our we feed the the Python script the URL that we want and have it ingest this data. Now this we're using archive of our own because it's a fun site to spend some time on, but you can use this uh this type of technology on places like Medium, uh Forbes.com, uh, you know, business inside, any place where there's again a a popular enough it's a popular enough sort of website that somebody will have been motivated at some point to say, hey, I would like to look at some of the data behind this. I'm gonna extract it out.

Now to do this, you have to download the scripts to your your personal computer, make sure that you have Python installed, uh, and then run them, follow the directions, run them and extract the information. Uh I took the liberty of doing this in advance for a couple reasons. One, uh it takes a really long time. The the authors of these scripts do it generally a pretty good job of being compliant with the terms of service, which means you know not trying to grab all this data all at once and uh you know overwhelming the site and and and doing bad things to it. And two, um, we then need to go and clean it.

So what I've done is I've taken that data uh from the the script, and you can see I've got about a thousand different pieces of story here. I've got ratings, published, date, the the date it was published, the date uh the status was last changed, number of words, number of chapters, comments, kudos, bookmarks, hits, and then using the R library quantita, we've got four readability scores, uh Coleman Lau grade level, flesh cancade score, del Chow score, and smog simple measure of gobbledygooker. How how many essentially do you use a lot of syllables in your text? There's also four measures of lexical diversity. So lexical diversity is a term for how broad is the language you use.

This is a really good term for uh not only fiction writing, but also business writing too. Lexical diversity means that you're using a wide variety of words, right? You're not just using the same 25 words over and over again. I mean it's sort of a uh I guess a proxy for having a broad vocabulary. So we've got for each story of those, and then using the sentiment library, sentimenter, uh, we've gone ahead and pulled the uh sentiment for the the eight major core emotions, positive and negative.

All this gives us essentially a very large table of each of these stories. And by the way, if you're here, say hello in the comments. Otherwise, I have no idea who's who's here. And we can see there's our ratings, dates, things like this. Now, the objective we care about is probably going to be something along the lines of what makes a story popular?

What gets hits, if you will. So let's uh get started with that. We've got our data frame here. Let's do uh let's do something pretty simple to get started. And I want just numerics, meaning the numeric data.

So I'm gonna boil this down. This gets rid of anything in our table that was not a number. Because anytime you're gonna be doing like any kind of correlation, um you wanna you want to get rid of dates and other formats. The other thing we want to do is we want to get rid of what are called near zero variables. And these are variables that have uh essentially no variance to them, uh which means that from predictive purposes they're not gonna be very helpful.

So let's go ahead and get rid of those. And what do we have here? We're down to 17 variables that have enough variation that we have meaningful information about them. Okay, cool. Let's now go ahead and run a correlation.

I'm gonna use a spearment correlation because we have no guarantee that this is a meaningful distribution, and zoom this in. So when we look at hits here, we see you know moderate correlations, you know, anything over 0.25 in a spearman uh p-value is gonna be you know moderate. Anything above 0.5 is is good. So comments, right? Is uh bookmarks at 0.78, it's very strong, and then kudos 0.79.

So this is uh useful. Here's the challenge, and this is something that um you have to be very careful when we do this kind of analysis. Hits or views of the content is a is a number that occurs first, right? Something like uh a kudos or a bookmark or comment are things that happen afterwards. You cannot in no place can you can you get a comment on a story without someone reading it, right?

So there's so sequentially there's there's a pattern of time that has to happen before. Whereas things like the number of words in a story, that happens before user interactions. So while it would be great to say, yeah, the more kudos a story gets, the more hits it's gonna get, it's actually probably the other way around. The more hits that a story gets, the more kudos it's gonna get because logically, you the more people who read the story, the more people who can interact with it. So of the variables we have here that are uh predecessors to user interaction, which is all these emotions and things like that, um, we see that of those two here, the word count uh numbers, and and those are actually tightly correlated, the word count and negative emotions have the highest uh correlation.

So essentially we're saying that longer stories may result in more uh traffic, more views on it. So the next question is well, is there a way that we could conceivably prove that um the way we're gonna try and tackle that is now that we know that there's some issues uh in the data set, right? Uh particularly with comments, uh bookmarks, and kudos, we need to remove those in order to be able to do an analysis of them so let's do uh clean Watson data frame here and we're gonna take our AR3 data and we're going to select and remove hits we're gonna I'm sorry we're gonna remove kudos we're gonna remove bookmarks and we're gonna remove comments we are going to keep in some of those other fields okay so we have we have we got through and removed anything that is user okay next let's go ahead and write this out to a text file on our desktop because what we're gonna do going to want to do is take this to a different system and we're gonna take it to IBM Watson. Now what are we gonna do here? We're going to feed this data set into Watson and ask it based on the information we have provided using the auto AI functionality can we design an experiment that can judge better those different characteristics we were talking about like uh lexical diversity and things and get a better sense of uh did that not write out oh duh.

There we go. Okay, let's call this AR3 test. We're gonna create ourselves a new experiment here. Let's load our data. And we want to predict on hits.

Run our experiment. Now, for those of you who've never seen auto AI before, the name's kind of misleading. It's it is automated machine learning, but the vast majority of what it does is the data preparation cleaning and readying a data set and then essentially testing different algorithms on it to see does uh does a particular type of data or does a particular type of algorithm work better for for this particular data set. So uh it has just begun. It is going to start looking at the data, doing uh some analysis on it, it's going to try cleaning it if there's anything that's egregiously wrong with it, and then it's going to start selecting things like uh what measures should we use, what algorithms, what models would give us the best overall result for essentially predicting those hits, uh those those page views.

Again, this is something that you can do uh with any numerical data. So if you had Google Analytics data, if you have um social media data, you can do any of those data types in here as long as you take the time up front to think about the variables that you do not want to include. Because you can use the built-in data refinery to try and clean it up, but it's not really the best. Um it's it's difficult to use. So if we can do that in the prep phase, uh we can eliminate uh that step here and go right into uh picking algorithms.

Now what you're seeing is uh it's gone through and done some hyperparameter optimization. It is now doing uh feature transformation where it adds together different data points to see if uh any of those are relevant, and then it's gonna do another round of hyperparameter optimization. Um which if you think of if you think of uh machine learning model as a stove, right? Um, and you're baking cookies. Hyperparameter optimization would be things like trying to bake the uh a batch of cookies at every degree of temperature between like 200 and 400 um for every amount of time between five minutes and 30 minutes to see which combination of time and temperature gives you the best cookie, right?

Uh and again, that will depend on some things like you know, the dough, uh the ingredients in the dough, etc., which is the data in this case. But hyperparameter optimization is literally just fiddling with the knobs on the stove, every possible variation to see uh what combination of those external parameters uh makes the best cookies. So in this case, makes the uh the best possible outcome for the data. Let's take a look while it's it's still thinking, uh, and take a look at what is come up with feature importance here. So this is really interesting.

Um what this is saying is that the rating, you know, uh general, uh mature, explicit teen, etc., uh, has an outsized impact and a pretty s sizable one on on whether something gets it gets hits or not, right? So depending on why somebody might be viewing this particular uh type of content, um we could potentially build a model to predict, like, okay, this uh the the this rating or this type of story gets the most interest. If some people who want very explicit content, this may be the thing. So now that we have that knowledge, we could go back to our data and analyze it and say, okay, well, what does that break down look like? What is uh what are the different ratings, and you know, how do we bucket the number of hits for e uh on average in each of those ratings to see which rating gets the highest number?

And that will tell us then if we were content creators here, write more stories like this. Um we could also, for example, like uh explode different uh themes or fandoms. If I wanted to, you know, we we started jokingly with the A team because I couldn't find anything else uh very quickly there. But if there's like five popular TV shows that people are writing fan stories in, would that make a difference? Probably would.

Um we'd want to know that. That's the power of these systems, the ability for us to very, very quickly get to an answer of okay, here's what looks like has an outsized impact on this objective, now we can go back and and start refining our data and get to a point where we can build a hypothesis. That's the the the the data science portion of this is if we say and we find out, for example, the explicit content is the thing, then our hypothesis is if I create content that is uh explicit, it will get more views. Um chip asks you can drilling to see which ratings are driving it, not in here. Again, we have to go back to our original data set and parse that out separately uh in order to to make that determination in fact let's see if we can do that in a reasonably timely fashion let's start with our straight data set here and we'll do a slim AO3 start with AO3 data and we just want to use reading and hits I don't need that second parent there and let's do a group by rating and a summarize uh let's do mean average hits equals mean hits.

Let's see if okay so we've done done our summarize let's take a look at our slim data set here. Five observations two variables and as suspected yep explicit followed by mature um teen and up not rated general audience so two chips question um yes we can very quickly go in now and see what it is that gets that uh now again think about how you would use this with like data from say medium if you have like content topics like what topics get the most clicks on medium if you have the data you can then do exactly what we just did say like I I've got to write political stuff if I want to be popular on medium um if you uh do it with YouTube and use YouTube's categories I need to make videos about um I don't know uh a Kardashian uh in order to be popular on YouTube. Whatever the case is um we can find that data very very quickly by first running it through Watson to give us that massive massive regression analysis and then jumping back into our data to look here and now we have a hypothesis that we can go test. So good question chip. So that's the tonight's exercise tonight's fun is looking at how do we use these tools together to build a better hypothesis that we can then go test because a lot of the times we struggle in marketing with just we just test things randomly because we don't have a data driven hypothesis.

This is the process for doing that exploratory analysis to find out what should we be hypothesizing in order to create uh a meaningful outcome. So uh if you have follow-up questions uh please join the the free Slack group that comes with this stuff uh go to trust insights.ai slash analytics for marketers uh totally free to join again trust insights.ai slash analytics for marketers and you ask questions about this or any of the other shows that we've done uh if you have uh wondering like is there a way to do it with you know LinkedIn data spoiler alert there's not LinkedIn is notoriously bad about locking well good and for them locking locking down their data like crazy um but other things you could do the exact same process, say with emails in your customer service inbox uh or chat logs on your your company website. Exact same process, just with a different data set. So thanks for watching. I'll talk to you folks soon.

Take care. Want help solving your company's data analytics and digital marketing problems? Visit trustinsights.ai today and let us know how we can help you.