You Ask, I Answer: Scaling Content Curation?

In today's episode, Jen asks, how do you curate content for your social media channels? That's a complicated question to answer. Um, a few years ago, I was curating content, you know, reading through Feedly and blogs and stuff like that, and uh noticed that I kept doing the same thing over and over again, which took a really long time, a couple hours a week. And I said, This is silly. Uh, this is something that can clearly be automated.

One of the benchmarks for automation is if you do the same thing over and over and over again, and you do it more than twice, it's time to start thinking about automating it. When you are doing the same thing every single day, it is definitely time to start automating. So the process for automating this thing is relatively straightforward. It just is a lot of custom code. Now there are paid services that do this kind of automation, and they are all reassuringly expensive.

The reason I don't use paid services is twofold. One, I'm cheap. Uh, and two, more importantly, I don't like the black box algorithm of how these services uh find and recommend content. I don't know what goes into the box, and therefore I don't know how it makes its decisions. And when you ask vendors, they say it's a proprietary algorithm.

I get that. But I still want to know how it works. So I ended up rolling my own. Uh what I did and how you can start thinking about doing this for yourself if you have the requisite technology skills. Um, is first you need a uh source of articles of content.

So what I'm gonna show you is bring this up here on screen. Um, this is the RF's SEO tool, one of my favorites. Um type in the the topic of your choice, and you'll see, of course, a large collection of articles on your topics. What you want to do is not focus on the articles, but focus on the sources. Also, if you already subscribe to some great blogs and like Feedly or whatever servers you use, export that list of logs.

You will need then a scraper to uh go out and ri and retrieve those pieces of content and put them in some sort of storage mechanism. I use a SQL database, uh, which you can see here that pulls in each article and then uh by URL from those RSS feeds. Now, the important thing here is that I don't want just the article title, and I don't want just the URL. I want the full article itself. So one of the things that um the software I wrote does is vacuum up the actual content of the article itself.

And you can see here on screen the a number of these that have the full text uh coming in. That's important because the next step in this process is to figure out is this an article that I would want to share? Is a topic irrelevant? So there's a stage uh there's a uh another piece of software on this database server that goes through and identifies this is something that I care about. Uh it also pulls in social shares, SEO data from RFs, things like that.

In this case, we can see uh there are a number of articles that are about data. There's one here about uh analytics and things, and you could you can just uh spot check very quickly just by looking at the titles, like is this a uh relevant article? Um here's what you know, data-driven planning for city resilience, quantifying sea level rise. Okay, that sounds like something that I would share. Uh there are also flags in here for things that I don't want.

Um see, where's that column? There's oh yeah, there's what's called blacklist. Uh, and that is essentially if I sort this column here. Uh oh, I delete anything that's below a certain point. Um articles about politics, especially certain politicians, don't want them, don't want to share them, so they automatically get blacklists knocked out.

Uh they never see the light of day. Uh the next step after that is to design them with social sharing links. Um I have my own link shortener because I got tired of of getting ads from the service I was using to hand over a lot of money a month uh for them. So have a link shortener connected there. And all this database processing happens um on the database itself, and that prepares essentially about fifteen, twenty thousand articles a month um for uh processing.

And this this script runs the this system here runs every 15 minutes or so, so it's not like it you know once a month, um, because it does take time for the the software to go out and do all this processing and scoring. At the end, what you end up with is a scoring system, right? So at the at the very end, there is resort this here. Um you can see the these are the highest ranked articles based on those topics, uh not containing things they don't want. What is social media management?

Instagram revenue and usage statistics, right? 41 best data science programs. These are all things that are perfectly on target for the kind of stuff I share. So now the last step is to use uh another piece of software that I wrote uh that goes in and takes all these articles, blends in a series of social shares of things that are essentially are ads, right? Um things that I want to promote, like my newsletter, uh like the uh the trust Insights newsletter, uh, whatever uh oddities I want to promote, and mixes them in too.

So there's uh from a content curation perspective, there's 25 ads. There are 50 articles, so you know, about two to one ratio there, and then there's a a thank you section as well, uh, where uh I pull in an additional 25 articles that are uh all things that other people have written about Trust Insights. I want to make sure that we're sharing the love, thanking people for covering the company, right? That's an important thing to do. This will sew together a at the end of the process, one single uh CSV file.

And it looks kind of like this. This then goes into Agora Pulse, Buffer, uh, Sprout Social, whatever uh system you want to use to share your content. This is all pr uh built out, and this is uh fresh content. One of the restrictors on the system is it has to be only content that's been shared in the last uh seven days. And what the summary is is like cleaning, loading, the different social shares, topic scans, link shortening, content scan.

At the end of this process, as of right now, there are 321 articles that I could be sharing within that are published within the last seven days that are topically relevant out of the you know 5,000 or so each week that are raw inputs. Now, this system is very technology heavy, and you can see the user interface kind of sucks. Uh actually, you know, it it it does suck if you uh if you're used to having uh a really nice polished interface, uh this is this is not something that that is gonna do you any good. Um, which is one of the reasons why it's not for sale, right? It's it it is not a product that you could just buy off the shelf and run on your own servers.

Uh Trust Insights does offer it as a service if you want or help to to get a file. We have a couple of paying clients who get weekly files from us. But for those folks, you know, we work with them to tune their topics and tune their stuff so that they have input into the algorithm, but ultimately they're not maintaining the algorithm or the infrastructure. Like I said, the reasons I do this are twofold. One, I'm cheap.

And two, I know how the articles are getting chosen. And when something comes up that I'm like, uh, I don't like that kind of article. I don't want that kind of content in my social shares. I can go in under the system itself and write exceptions, write um rules or or change the code around to say like this is not something I want anymore. Now, there is some but not a ton of machine learning in this.

And uh one of my goals for 2020 is to upgrade the article selection process to instead of using manual tagging to use uh supervised learning as a as a way to to process the articles and get even better targeting. But that's going to require a lot of uh of work, so that's gonna be one of those things that probably gets done, you know, when when it even when it gets slow. Um but that's how I do the the content creation. This process and this system has taken probably four years to write and tune over time, and there are constantly new changes coming in as you know, new clients come on who want this service, or as I see things and learn things that I want to improve on it, that changes the system too. And it it's it's ongoing.

If I were going to start over from scratch, I'd probably do some of the back end architecture a lot differently. Because it was built um with my skills at that time, and as my skills evolve, the system evolves, but it's still not uh it's still not where it could be yet, and not where it needs to go. To build something like this yourself, you need SQL database skills, you need a scripting language that is web compatible like PHP, Python, etcetera. Um, and you need data processing uh language skills like R or Python in order to be able to create the scripts that you need and sew them all together into one system. So those are sort of the three sets of skills you'll need to implement a system like this.

I would strongly recommend that um you come up with your own algorithms and it and you may want a a user interface. Uh I don't, I don't particularly need one, but uh you may want a user interface uh if you're gonna do this yourself. But that's how it works, that's how the system works. It's it is been a pet project for years, um, and it continues to grow, and uh I hope you found this useful for thinking about how you could build your own system like this. As always, please subscribe to the YouTube channel and the newsletter.

I'll talk to you soon. Want help solving your company's data analytics and digital marketing problems? Visit TrustInsights.ai today and let us know how we can help you.