So What? How to process unstructured survey data

Well, happy Thursday. Welcome to So What the Marketing Analytics and Insights live show. If you are subscribed to the Trust Insights newsletter or you're in our Analytics for Marketers Slack group, you may have seen that we recently asked our one question survey that we do once a quarter. And the question that we asked was around which marketing channels you plan on using in 2022 and why. And so today what we want to do in this episode is we want to cover how to process that unstructured data.

So this is you still have time to respond to the one question survey. And so for us, we were curious to know the rationale behind using certain channels. And the the responses were really interesting. And so we're going to take a look at it in terms of what you, the user, can do with the data, and then what we as Trust Insights, the people asking the question, will be doing with the data. So Chris, where do you want to start?

Um, why did we run the survey? Like what was the what what are we trying to get out of it? So, what what we're trying to get out of it as a marketing consulting company is understanding a couple of things. Uh, number one, we wanted to understand where people were spending their time and energy um in a different marketing channel and for what purpose. So, what their customer journey looks like, digital or not digital.

Uh, so we wanted to understand that just purely from a curiosity perspective, but then as a company, the more channels people listed, the more commonalities we saw based on what they were saying from awareness or conversion. We wanted to make sure that our services and our content aligned with what the questions that people had about, you know, how where do I build awareness or what channels are best for conversion? And so making sure we have that data to then create content to help other people understand that same question. Got it. Okay.

So let's start uh just with the survey itself. This is the landing page that we asked of on things. And what you'll notice is it really is one question. Um there's there's no like 15 parts or anything like that. And so what we get out of this is two pieces of information.

We just get the the person's email address and we get a block of text, whatever uh somebody wants to type in here. This is good for discovery. This is good for you know um because people will type in things that you didn't anticipate, as opposed to having like a drop-down menu or something like that. Um this is not good if you don't have the ability to process the responses and and and turn them into something because you'll get a lot of interesting anomalies. So let's look at what the raw results look like.

This is Excel. I've removed people's identities so that you know we're not like sharing PII on the air. That's not fun for anyone. Um, and this is what you would get out of any kind of survey software. Just a nice spreadsheet with your question and the answers.

Now here's the challenge. Um, as you can see, there's a lot of variation in here. And if you wanted to start picking this apart, and all you had was Excel, uh or the spreadsheet software of your choice. Um, there are a few different ways that you could do that, but one of the easiest is just to start counting stuff, and in fact, counting stuff's pretty much what you're gonna do. So I'm gonna go ahead and just in uh cell B2 here, I'm gonna start typing a formula.

I'm gonna close this warning that I don't care about. I'm gonna say I want to count if in cell b2, uh2, and then I want to put a s an asterisk here, which means anything, and I see already you know uh linked in. So I'm gonna put just type in LinkedIn star close the quote, and this tells me how many times LinkedIn appears in that thing. So I might put a peddless column LinkedIn, right? And then you can drag that down the rest of the way you can see how many times LinkedIn has appeared in entries.

And my next one might be say Facebook, and I might go uh same exact thing, count oops, equals count if A2 star Facebook star and so on and so forth. Now, fix my quote. Oh, quotes around the stars. Oh, yes, thank you. All right.

And again, just drop that down. Now, here's the challenge with this uh particular method. Um, you do have to anticipate or or be able to browse through and see what are some of the variations. So if we scroll down here, uh you'll notice that some folks, like in this response here, uh, shorten things up, right? They typed in F B, Ig, L I, and uh, and so on and so forth.

So there's actually two mentions of Facebook, but we only counted the one. So again, if you're doing this in Excel, you would have to add a plus sign here and say, I also want to count if an A2 star or the quote that time, FB star quote. Now I've got a compound formula that's going to count one for every for the occurrences of Facebook and another for the occurrence of FB and so on and so forth. And this is a admittedly a pretty rudimentary approach, but it's not entirely bad, right? Because you can at least uh if I just highlight column B here, I've summed this up, and just from this approach, uh, not kind of all those variations, we got 71 occurrences of that.

That's so that tells us LinkedIn occurs at least 71 times in this spreadsheet. Now, from there, you could start totaling things up and trying to figure out okay, how many times do it do individual words appear? What this doesn't get you is doesn't get you a more nuanced understanding of you know how many of the different types of occurrences. So, Katie, to what you were saying, um, this is what I would say the the the non-coder for type of person would use. Um any questions on this before we move to the coding version?

Yeah, I've got an easy one. How'd you do the uh paste down through the rest of the column? What's the shortcut for that? Uh you just double-click on the corner of the box. Really?

I'd see this is the problem when you're using Excel 2008. Um that's been like an Excel since like 2000, though. And maybe I'm using 97 then. No, I you know, I for a marketing team or a marketer that doesn't have access to writing code or machine learning. I think this is actually a really great approach because at the very you know uh at the very like core of it, you can just understand here are the different social channels people plan on using next year, or here are the different digital marketing channels.

And then so for us, so we don't we don't do a lot with LinkedIn in general. One because getting analytics out of LinkedIn is tough, but in thinking about you know our own company strategy, what more could we as a company be doing with LinkedIn? Because that seems where everybody's moving to. And so I think there's a lot of things that as a marketer you could be thinking about doing with this data, even just from account. Exactly.

Any other Excel tips, John? Yeah, no, we don't want to get into pivot table or v lookup. That's that's a road we don't want to travel there. Yeah, I think that's a different episode. That is that is definitely a different episode that I will recommend.

Uh my friend Oz Du Soleil is sort of an Excel MVP. So look him up on LinkedIn if you want a really good, really like amazing. Like I didn't know Excel could do that. All right. So we're gonna move into uh a programming environment.

Uh, this is the programming language R. And the first thing I'm gonna highlight here is I've written I've taken uh some time. This is like a cooking show. I've I baked this in advance, um, and I've tried to accommodate for many of the different variations of the way that terms appear in the text. You know, people like using those two little uh FB and then essentially rewriting uh those terms so that they correspond to the appropriate uh uh language in here.

So I'm gonna go ahead and run through this, and let's go ahead and see what we come up with. Chris, that seems very similar to uh the way that you would write the script for channel grouping in uh Google Analytics. You want to try to capture all of the different variations and then rewrite it so that it's if you see FB, then it means Facebook. That's correct. That's exactly right.

Um other thing we're gonna highlight here is you just have a bit of very basic um analysis here, just looking at how long are the responses. So the mean response in our survey is 184 characters, which is uh just slightly less than you know a tw uh a full-size tweet. Um the shortest response was eight characters. That was other person I believe said don't know. Um and the longest response was a person who wrote a book at 1820 characters and stuff.

So there's there's is a lot of variation. Uh if we look at the histogram, though, it definitely leans towards the shorter size. So people did uh as we promised in the email, say uh yes, this is 30 seconds or less. They they took the appropriate amount of time. What we're doing is we're taking apart the text and we're first altering those responses to you know to fit those terms like we talked about.

And then we're breaking apart the test text, breaking it down by words, removing what are called stop words, a and the, etc. Words that don't offer any any value. And then we group them all together, and we say, okay, I want to see how many of these words all cleaned up and fixed there are. So let's take a look what we've got here. Now, having gone through all this, we see that there were 105 mentions of email.

Right. We have a uh which is number our our number one. Um we have LinkedIn is number two, Facebook is number three, Instagram is number four, social in general. Uh we have Twitter down there. We then uh start getting into stuff like YouTube, uh, social media in general, SEO, uh, Google Ads, TikTok, which surprised me, you know, which made the the top 30, direct mail, which also made the top 30, I thought was interesting, and events.

So these are sort of the the major channels and things that uh people had identified in our survey and said this is this is what we think is is um going to be in our 2022 plans. So, Katie, when you look at this, what does this tell you? It tells me that there's a lot of emphasis on social media in 2022, is that's where people want to be posting information and finding their audience, and I find it both surprising and unsurprising. Unsurprising uh in terms of ease of use, it's very easy to post something on social media. Um, surprising, I think more so for myself personally, just given um recent events with certain social media platforms that it's still so widely popular, and having read through just about every single response, a lot of people saying that's where my audience is, that's where my customers are.

Um, so I was definitely interested to see that social media still tops the list. Um, but even more pleased to see that email uh is at the very, very top. And so what that says to me, because I'm anticipating your next question, is as you know, a company as Trust Insights, our responsibility then is to make sure that we are creating as much helpful content about like the top 30 channels. I would agree with that. Uh and if you look uh it's certainly not uh uh anything differently revelatory, but we do see that in in the word cloud.

Uh now, one of the next steps that we want to take is to do a lot of um processing to figure out not just the terms in general, but how do these terms relate to each other? Um this is where that's probably a little too much. Let's let's dial that down to maybe 30 times. Because not all these entries are you know, have all these terms in them. We want to see how things kind of cluster together.

So we see uh your social channels, LinkedIn, Facebook, uh, Instagram, organic social over here, paid search, uh email, direct mail trade shows, and it uh a network map of words like this helps us understand what are the terms and things that are are near each other in responses and what are the terms where uh things are far away. So it's interesting to me that uh direct mail trade shows and emails are sort of over here, um, email and social over here. On one side. You have all your social channels, LinkedIn, Twitter, Facebook uh over here, uh, Facebook and Instagram in the middle, and then you have just regular email at the heart of the network, which really, even though you know, we were saying with um the other parts that LinkedIn was was really critical, we see that email is literally the heart of this Word network, which means that in terms of the most important things that people care about, um, it really is uh email is sort of the the center of the strategy. So for us, you're right, we've got to spend an awful lot of time with it.

Um one thing I thought was interesting that is absent on here, we see a lot of stuff, you know, we do we do see content in here, but we don't see organic search, you know, slash SEO, even though we know from many of our clients' reports that that is the primary traffic driver for a lot of different companies. And so uh what's your perspective on that? Why do we not see people saying we're gonna put a lot of focus on organic search because it's our number one channel? You know, I it's a good question, and this is pure speculation. Um, because the first question I would say even before that is what is it that's being posted on social media?

And so are you writing content and then posting it on social media and that's the way that you're using the channel, or are you just posting your thoughts and you know reflections on social media to bring in people to your website? And so I think there's you know, this idea, and again, pure speculation. I don't know that this is true at all, but that SEO is more difficult, that it's hard to, you know, rank within Google, but you already have your audience baked into a social platform, so why not create the content and share it there versus relying on Google to serve it up? John, what do you think based on the customers you've talked to? Well, one thing, and we were talking about this last week was this kind of gap where in a lot of marketing departments, a lot of SEO stuff, they just throw over the cubicle wall to the IT department.

You know, like Search Console is more of the IT department's problem. And I think that's a a massive gap, to be honest. I think you know, we've seen over and over again where somebody goes in and cleans up Search Console and suddenly they're getting like 20% more traffic every day. And that's you know, as opposed to the crack pipe of social ads, you know, you fix something on the organic side and that delivers traffic for the rest of your business's life. Um it's a massive uh it's a massive difference.

It's a different kind of hit. But um yeah, I just don't understand why companies don't jump on that, you know. It's it's bizarre. And and maybe it's a bias with our audience too, you know, because we tend to there's a lot of folks in the A4M group that are especially talking about problems on certain ad platforms. And so maybe we're we're skewing in that direction.

Yeah, I would I would agree with that. Um for sure. It we constantly see people under investing in SEO. Mike, one of our largest clients, um, you know, they spend millions of dollars on paid social and advertising, and they have like one person working on SEO and and not even full time. And it always baffles me to see that because you look in the activation model, 60% of your leads come from search.

So we're like, how do we help people bridge that gap? Not that we're, you know, we're not an SEO agency, we're not gonna become an SEO agency, but in terms of just understanding how people are resourced properly, even here, you know, yes, email important, and you know, obviously uh we like that because we have some specialization in that and we know what we're doing with that. But to your point, Katie, so many of so much of the stuff is social. SEO is way down here, and yet we know that from a an effectiveness perspective, probably most of the people who responded to the survey, organic is their organic search is their top driver. I think one of the things that we might want to do um is redefine what SEO looks like.

Um, and so the different ways that you can create content that gets SEO value on your website. So it goes back to that transmedia framework that we're so fond of where you create one piece of content and you use it multiple different ways. And so for this live stream, for example, you know, not only are we live, but we're also recording it. And then what happens is tomorrow I will turn it into a blog post using the transcription from uh this episode. That then becomes the long form content with the right terms in it.

You know, I can clean up the transcript a little bit, make sure I'm hitting certain terms, and so that becomes the content. And all we've done is sit here and talk at each other for you know 30 minutes or so, but we have a full piece of content that is hopefully helpful and explains the thing, and then we can also then share it on social. We can also then share it in email, and so we've done one thing and we've used it multiple different ways. Then we can cut down little you know snippets of the video, post it on TikTok, post it on Instagram, you know, whatever we want to do with it. So that's I think that's something that we need to really put a focus on and re-educate how you can think about SEO, it's not just sitting down, you know, with a bunch of keywords and writing a piece of content that hits the keywords six different times.

One of the next steps that I think is important in this process is that this all we've boiled down the data to quantitative stuff so we can see priorities and things. We actually want to go back to our spreadsheet now and look at some of the qualitative stuff. Um let's take SEO, because I think that's a really good example. Um I'm gonna rename this to SEO. And John, yeah, magic, double click, boom.

Unbelievable. And then I'm gonna filter this column uh only for the only for the ones, take out the zeros. And what this does is it essentially filters out only the answers that people have given where they have uh used SEO in their response. And now this is where we start to go from quantitative to qualitative. For the people who answered SEO, what else did they have to say?

Uh are there any similarities in here? We see uh I see just in terms of length alone, people who answered SEO had a lot uh of stuff to say, right? We see these are not short answers. Well, it seems that those are the marketing teams that have more well-rounded strategies versus just relying on one channel. Um, you know, something that I found really interesting, especially for people who were talking about social media as their primary channel in 2022, was that's where my audience is, or that's where my prospects are.

And that naturally brings up more questions for me of how do you know that? How do you know where your audience is? And that's not me saying you don't know. That's me really wondering, how did you get to the point of understanding where your audience is to know? Because so many people responded Facebook, but I personally, and again, pure speculation, I have a hard time believing that that much audience is on Facebook, other than maybe just for awareness.

You know, one other thing with that is on the qualitative side. It was interesting how many of them were sort of apologizing, like, well, we use Facebook because that's where they are, and language like that, where they were obviously kind of signaling a bias that they feel like whether it's ethical or for whatever reason, they shouldn't be there. And that's you know, that's a really good point because you're trying to meet your customers where they are, and if your customers are on Facebook and you hate Facebook, you know, you're kind of stuck. But again, that's a whole different episode. Yeah.

Facebook, but it's still king of social, even if it gets a meta weird next year. That's true. I appreciate the the uh the sense of humor. So at this point now, we can start looking to see, like you're saying, what some of the sentiment around these things. We didn't do any sentiment analysis uh on the the text.

We certainly can if we want to. Um, but looking at the responses themselves, I'm not sure that we would actually get a whole lot of value that because a lot of these things are just lists and they're not um there's I don't see strong opinions being listed here as just more uh sort of factual statements. If you were doing an opinion-based thing like um, hey, this person's running for office in 2024. Um, what do you think about that? You'd get some obviously very strong opinions, and that's the case where sentiment analysis would make a great deal of sense.

Or um, you know, we're renaming our company to hide our transgressions. Uh what do you think about our new name? You would get sentiment analysis very, very important there. So we now we've done qu uh quantitative, we've done qualitative. The next step, if we wanted to refine this further, uh, would be to circle back.

If you think about survey analysis is sort of like that yin and yang um idea, you start with one qualitative or quantitative. In this case, we started with qualitative, which is asking a bunch of stuff. Uh now with our data from R, we have quantitative data data as a starting point. Now we can run another survey if we wanted to, uh, saying, please tell us about your, you know, how important is email marketing to you on a scale one to five? How important is LinkedIn to you on a scale of one to five, and so on and so forth.

So now we've we've boiled down the things that we think are important and we can start putting numbers to that to get a sense of the actual importance. See what I would do, I think that that's a good approach. The other approach I would take is where in your sales funnel or where in your journey do these channels fall? Are are they awareness? Are they engagement?

Because I think that again, what we're trying to get out of it is an understanding of where our community is at so that we can then provide additional helpful content. And so we can write about LinkedIn, we can write about Facebook, but getting into that more specific use of why, like where in your journey are you using Facebook, or how do you know your audience is there, I think would help us get more specific about the advice that we're able to give. Yep, I would agree with that. Um that is uh relatively straightforward is Google Analytics 4. So uh for those who have not seen uh previous episodes, you can find them on a YouTube channel.

But let me pull up our Google Analytics for attribution model and let's look at source, because I don't trust their channel groupings. Um and now we are starting to be able to pick apart in terms of go away. Um early, middle, or or late funnel. Like for us, early funnel, Google is the second biggest source besides email. So even though our survey folks said SEO is not on there, at least we know from our own data for ourselves, our own marketing, it's uh it's the number two source at the top of the funnel, it's the number three source in the middle of the funnel, and it's the number two, well, technically number three, but we don't count directly, um, as source as at the bottom of the funnel for our journey.

Um we see referral traffic, uh, you know, my website uh is number three at the at the start, number four in the middle, uh, and then drops off, it's not part of the of the last touch. So, from uh a a you know, some folks had talked about affiliates and partnerships. Um this is it would be an interesting thing for those folks to look at in their own attribution modeling because I would have the question well, if you think at affiliates and partners are important, are you measuring affiliates and partners' impact at each stage in your funnel? Because if you have a if you have a bottom of the funnel problem and your affiliates and partners aren't showing up there, there they might not be working for you for your biggest need. That's not to say stop doing it, because if you stop putting stuff at the top of a funnel, nothing comes out the bottom.

Just know that from a priority perspective that you if you're you know CMO is saying, hey, we need more leads, you might need to change some of your focus. So John, I'll give you your you know weekly to do. Um, you know, as our business development guru, you need to go back to our Christopher S. Penn partner and yell at him as to why he's not producing more conversions. Get a better rate, yeah, definitely.

That's right. So I'll let you and Chris out. The good news is the almost timely newsletter publication doing okay. Yeah. Um, but no, I mean, I feel like this is really interesting because to me, I was looking at attribution and the survey feedback as complementary.

And so it, you know, a lot of people listed just the channels, but some people went into the detail of I use this for awareness, I use this for engagement, I use this for conversion. And it's interesting to me to see how they've stacked their channels so that we can then think about, you know, uh educating on what that channel what that customer journey looks like. One of the tools that I used to really enjoy, which I don't think exists anymore, uh, Chris, you introduced it to me, you know, five or six years ago, was Google had their customer journey. Um it was just on their website, and you could say, This is my kind of business, this is the you know, size that we are, and it would give you your expected customer journey for each of the channels, and we would be able to see, okay, Google says you should use social for awareness, you should use direct for this, you should use SEO for this. And then we could map that against what we were actually seeing.

Yeah, unfortunately, that's been gone for three years now, may it. We keep hoping Google will bring it back, but no. But I that would be that's something that's the first thing that came to mind when I when we started getting all the survey responses is I would love to have that, you know, Google's perspective of what the customer journey should look like to map it against these responses of well, here's what you know they say you should do, and here's what you're doing. What's the discrepancy in the middle? Because we obviously don't have all of these people's data to say this is what's actually happening.

Yeah, I I don't disagree. I I wish we had access or had the ability to build our own uh that was generalized. We certainly have our are the ones that we use for our uh for ourselves that are specific to us, but for the industry as a whole, that uh that sort of thing that doesn't exist. The other thing I think is really important is uh when you look at your attribution modeling, you know, this is our this is ours. Um we can see you know, Gmail is in there, email in general is just a massive uh probably too much.

Um then you have LinkedIn, Twitter, and social and the other socials in there. One of the things that I think is important about this survey data is that we see, like you said, we saw a lot of people putting a lot of emphasis on social channels. I would be looking at your your multi-touch attribution report to say are the results we are getting commensurate with the effort we're putting in, right? I I heard somebody call it return on effort, and I want to stab them with like a haddock. Um, but return on anything is a financial equation, it's not anything else return on influence, return on blah blah blah, whatever.

Um but there is something to be said for are you over investing in a channel that isn't giving you enough return? So um kind of what we were saying earlier, very few people put SEO out of uh 200 somewhat responses, 21 had SEO in there. So 10% what people put that as one of their channels. And yet when we look at our attribution models for a bunch of different companies, it's 60, 70, sometimes 80%. In fact, we had a one client earlier we were on the phone with today, 92% of their conversions come from organic search.

And so I would say to those folks who are putting like LinkedIn and Twitter and Facebook and stuff, yeah, okay, if your attribution model looks like ours, and two percent of your conversions are coming from LinkedIn, but 20% of your budget and time is going into it, you might want to reprioritize. Well, and I I feel like so one of the things that we talked about, and I think it was on last week's episode, was reconfiguring our Google Analytics goals so they follow that sales funnel of awareness, engagement, and conversion, because this really only gives you the view of conversion. And so I would be interested to see if you break it out into into those three separate pockets, do those social channels change their position, and therefore, you know, you're a little bit more justified in spending that time and energy in those channels because you're right, without awareness, the rest of the funnel falls apart. So people have to find out about you, but you have to be using the channels in the right context. And this is why we share the customer journey information and why I really always liked that Google tool of you know the awareness to engagement to conversion because like for us, social really is an awareness tool, it's not a conversion tool.

It's not where people are going to say, I want to buy something from Trust Insights. They're going to our email newsletter or they're coming directly to our website to our contact form because they know enough about us. And so I think that's the context that's missing from some of this is not just what channels, but how those channels are being used, what information is being communicated, what the messaging is, uh, and what the audience is, you know, how they're reacting to it, what their engagement is with that information. Yep. So we've taken the survey data, we've applied natural language processing, showed a couple different versions of it, cleaned up the data.

Um, what do we do next, Katie? Um, all right. So, John, get out your notepad. Um, you know, what we do next is we we have a couple of options. One is we can run a follow-up survey to get a little bit more information.

Uh, so that's one thing we can do. I don't know that it's necessary. I think that we've been talking about the importance of SEO. So I think that it's an opportunity for us to publish more content around rethinking SEO and how it's not just sitting down to write long form content, making sure you're hitting a certain amount of keywords. That's obviously important, but that's not the only way to approach it.

So I think there's an education opportunities for us with what we've learned. And so it's just more ideas for us with our own content creation. Do you think marketers, at least the ones that and you know, as you said, there's a bias because it's our audience, but do you think our audience is on the right track or the wrong track for 2022? You know, I think I said this last week as we were getting the responses, and I'm a little concerned with the amount of social media I see in terms of the channels being used. I think social media is important to have in your mix, but the emphasis and the sole focus on social media that we saw come through with the responses to me is a little bit concerning because it's one of those platforms you have the least control over.

Um it's one of those platforms, those digital channels that people come and go, and you know, their opinions of the channel can change with the you know the way the wind changes. And so obviously, like you know, Facebook is going through a lot, and so people are moving away from Facebook. So if you're doubling down on Facebook in 2022, that may be something to consider of what else could we be doing? It's easy to pick on Facebook because obviously they're who they are. But LinkedIn is another one of those channels that people are saying that's where my audience is, that's where I'm gonna be spending my time and energy.

But we know from our own use of Facebook, you don't get great metrics out of Facebook out of uh sorry, I keep saying Facebook, LinkedIn. So people are doubling down on LinkedIn, you don't get great metrics out of LinkedIn. Um, you know, so I think that I was concerned with the doubling down on social media in general as the only strategy that people had. Yeah, I agree. Well, I guess maybe on uh a future episode of either so what or our podcast in ear insights, we can talk about uh reimagining SEO for uh the current future and what's coming next year, because guess what, folks?

Google's multi uh use uh multipurpose uh model is on its way and it's going to change an awful lot for us. But that's that is literally uh for another show. Any final thoughts, John? No, I think we're good to go, but just uh yeah, get your SEO stuff in order. Stop throwing money down the drain.

And if you have questions about SEO, we can help. Exactly. Questions or if you want some help with the analytics, we can do that too. Thanks for tuning in, and we'll see you next week. Thanks for watching today.

Be sure to subscribe to our show wherever you're watching it. For more resources and to learn more, check out the Trust Insights Podcast at Trustinsights.ai slash TI podcast and a weekly email newsletter at TrustInsights.ai slash newsletter. Got questions about what you saw in today's episode? Join our free analytics for markers Slack group at TrustInsights.ai slash analytics for marketers. See you next time.