Saturday Night Data Party: Text Mining

Not bad. Alright. So this week for what we're doing is playing with some text mining stuff. So text mining, if you're unfamiliar, is the process of using uh data science and machine learning tools to disassemble text to understand it better. Um and you can do all sorts of really cool stuff.

You can uh unpack text, see what words and phrases are most used, um, you can identify parts of speech. This there's a lot of interesting things that are fun to do with uh the software. In this case, uh one of the things I've been trying on our quarantine is uh just doing some free form writing for fun, uh not business related. And I wanted to see how what I write differs from uh other authors in terms of other things that we can determine uh based on the data that's contained within writing. So we're gonna do is let's go ahead and get started here.

We're gonna use the R programming language. The library that we're gonna use is a library called Quantata. Uh this is open source software. Uh you install it as part of the R programming language, and we're gonna go ahead and load in uh some of the text. I've got one piece of text here that is I think about uh 70,000 lines long.

Uh there's a lot of text in it. We're gonna clean it up a little bit and make it distinct, and then we're going to do uh what's called cleaning and tokenization. So in order for uh hey Marcus, uh yeah, this is this is uh text mining stuff. So we're gonna go through and make sure that all the text is in uh ASCII format. We're going to go through uh and uh t tokenize it, which means splitting each set of words and phrases into uh individual essentially numeric vectors, and then remove some common stop words.

The processing of this, not super exciting. The first thing we could we'll do is do what's called co-locations. Co-locations are words and phrases that occur together. Um this is stuff that uh if you were to look, for example, in news articles, you would see the words like coronavirus and pandemic together a lot. That's a co-location.

You would see things like stock and market together. Um co-locations are a good way to understand a piece of text. Uh I use it a ton with social media monitoring because you can very quickly understand a large amount of text. It's a it's a processor cheap way of uh looking at text as opposed to something that's process expensive that consumes a lot of compute time. So in this case, I've got some co-locations.

Uh I'm doing bigrams first, which are two word combinations, and then uh we're gonna do three-word combinations, which will be uh trigrams. And these for this type of text you're gonna see more uh just very basic stuff. So here's an example of what this comes out. This is obviously the script to um uh the froze Disney frozen movies, right? And we've got some numbers, we've got the Z scores, which is sort of the overall relevance to the body.

Again, not super helpful for uh for writing, unless you want to see like what phrases an author might use a whole bunch. Where we're gonna get some good stuff is in what is called lexical diversity. Now, this is uh a language concept that basically means um how diverse are the words that you use. Um what you'll see here is the text number and then the lexical diversity with two different scores. The the documentation on this stuff is very, very, very arcane.

Uh quanteta has a reference library, and it's really terrific. They explain which measures, uh, type token ratios, uh corrected ones. Uh there's a whole bunch of different ratios, and then it's incumbent upon you as the writer to decide which algorithm, which formula do you want to use for lexical diversity? So the there's the original ratio, the sort of the the log logarithmic version, and then the corrected version. So I'm going to use the logarithmic version and the corrected version.

As with anything in R, it's going to go through each, if you look in the in the original script, it's each of these essentially represents one line uh in the script. So in order to make this easier to understand, we're going to summarize it down. And now we see it has a mean uh C, that's the log uh TTR of 0.93, and a mean CTTR of 2.53. Now, these numbers mean nothing, right? They're they're they're just benchmarks.

So now the next thing we want to do, I'm gonna take a just a quick screenshot of this so I can remember it. Uh actually, let's do this. Instead of doing that, let's assign that to a variable, a unique linear variable. And now for the next time through, I'm gonna go into my data folder and swap out that one script with a piece of writing that I wrote. And now I'm gonna repeat the process.

I'm gonna rerun this entire thing, but we're now gonna assign that data to line two, to variable two. So it's gone through, tokenized it, cleaned it up. I've got biograms and trigrams signed. And now let's see what the lexical diversity summary score is.94.713. So that's interesting because when I was working on this piece of uh writing, uh I felt like I felt like my writing was not, I guess, diverse enough in the in the types of words and phrases I was using.

And apparently the opposite is true. What we're seeing here with this analysis is no, actually, um it my writing is slightly more diverse in this. And so I can now go back and look at my writing with a more critical eye and go, well, okay, there's something here. I mean, I maybe my writing isn't as bad as I thought. This is uh I should uh uh put a disclaimer, this is fiction writing, which is not one of my strong points.

I'm trying to to teach myself the the basic skill as a way to become a better overall writer. Um so that's actually not terrible. Now, if I were to put in say four or five of my favorite works in this particular genre, uh might end up with a a very different look. Let's do that. Uh put them in, let's do summary three.

We're gonna notice this is gonna take a lot longer because when you start the the more data you throw into this uh sort of thing, the longer it will take to do the crunching. Uh you can see that took almost four times as long to to get through that. And here that you can see now the diversity numbers have shot up because we've got bigger, more diverse works. Uh 0.96, uh 2.96 there. So this tells me that from a from a perspective of uh my writing isn't terrible in at least according to this measure, measure of lexical diversity.

Um it may still suck. Um but from this one particular measure I can at least see that I'm not at the bottom of the heap. I mean the the classic example um for for poor lexical diversity is actually uh the scene from Forrest Gump when uh when Bubba is going, uh you can make shrimp kebab, shrimp creole, shrimp gumbo, pan-fried shrimp, deep fried shrimp, stir-fried shrimp, pineapple shrimp, lemon shrimp, coke shrimp coconut shrimp, you know. And what you're seeing is that like that word shrimp just repeated over and over and over again. There's very little lexical diversity because 50% of the text is the word shrimp.

Um of the things that I've noticed that I do in my own writing, and I've read in other people's writing, is that I there's a subset of words that I'm used to, that I'm comfortable with uh when describing uh like what a character is thinking or feeling, and it gets stuck in a rut. So part of the thing, the challenge that I face is how do I measure that? And that's what this lexical diversity measure would be good for, and then can I improve that? And so one of the tools that I would trot out for this would be the related words tool. So when you're and you're doing writing, you type in the word said, you know, there are all these other ways to use something to write it in such a way that you get more diversity, so it's just not so repetitive a reader.

Now, where the benefit is for us in more of like a marketing context is you look at all your social posts, you look at all your blog posts, you'll get all the content you create white papers and things. Are there words and phrases that you use so often and so frequently within a piece of text that your lexical diversity scores are like you know zero because you're just saying the same thing over and over again? This would be one of the many, many measures you could use to benchmark yourself. Can I go back through when I'm editing a white paper or whatever and and fix things up a little bit, make uh phrase things differently. Now, this is a very programmatic way to look at writing.

Uh, if if you want more of the human way to look at it, I would definitely check out um my friend Ann Hanley's book, Everybody Writes. I think that's a fantastic uh reference, especially if you're on the non-technical side. But um this is a technical way to pick apart some of those things that are in your writing as a diagnostic tool. Now, there are many, many, many, many diagnostic tools out there. Um you've seen tons of them, you know, grammarly and Hemingway uh and stuff like that.

Those are all good for like grammar and things. This is more of the what words and phrases are you putting in your text and are you stuck? Um, there are other measures too. You can do uh frequency counts, uh, which is super helpful when you're doing um parts of speech frequency. So do you use the same adjectives over and over again?

Do you use the same adverbs, the same verbs, the same nouns? Uh how often do you use certain words? Uh the quantative library actually doesn't do much of that, at least not natively. Uh for that, there's a library called UDP. Uh that also coincidentally works in like 27 different languages.

So if you're using something, if you're doing that in like um you know French Canadian, for example, or uh Danish, you would have to use UDPipe because a lot of the natural language processing tools are very biased towards the English language. I think quantity works with some other languages. I'm not sure if if it works with most. UDPipe works with like 20 some odd languages, including non-Western character set languages like Chinese. So takeaway from this session is use lexical diversity uh and other text mining measures to look at your text, look at your competitors' texts, and it would be we're not going to do it here, but it would be interesting to see if you ran lexical diversity scores, say on a hundred blog posts, and you measured returning you uh as you measure traffic to those posts to see if there is a statistical relationship between better written stuff uh and stuff that performs well, particularly when it comes to search.

Remember now that now that search engines are looking for topics as opposed to individual keywords, it would be interesting to run this analysis and see if more well written, diverse content performs better. We're not gonna do that here because I think that would take probably a couple hours to put that code together, but it'd be kind of fun, kind of a fun thought experiment to see if that's the case. In any event, making your writing more diverse and not beating to death the same words and phrases over and over again. It's a good thing to do. So take that away.

Play with this stuff, see how it works for you. If you don't code in R, uh, you can do this. I know there's a ton of really good natural language processing libraries in Python as well. Uh I don't know that there are a ton of off the shelf software products that would do stuff like this in this way. So you'll probably have to stick with the the coding versions for now.

But uh yeah, play with it. See if it helps improve your writing. Thanks for watching. Hope you uh are having a relaxing time and uh staying uh staying safe and staying home and uh trying some new things if you have the the bandwidth to do so. I like I said, I'm trying some fiction writing for fun.

Thanks for watching. I'll talk to you soon. Want help solving your company's data analytics and digital marketing problems? Visit TrustInsights.ai today and let us know how we can help you.