| Tweet | Buffer |
For this week’s R&D post, we ask a simple question spawned from a discussion we had in our research lab: How does social media impact user engagement?
Point: Social media makes users more engaged because people they know through networks like Facebook, Twitter and LinkedIn referred them to the website.
Counterpoint: Social media does not make users more engaged with a website because they were directed by a friend, but weren’t necessarily looking for that information. Referrals from organic search results, PPC ads, other non-social channels and banner ads will have higher user engagement metrics because those users were actively looking for that information.
You might already agree with one of the statements above, but we thought we would set up a simple hypothesis test with data from 30 of our clients to see if we can arrive at any interesting conclusions. This is not a study about the importance of social media on ranking factors or how it should be employed as an SEO strategy. Rather, it is a small-scale test to see how users interact with a site from social media by examining pages per visit, average time on site, and +1 metrics, tweets and likes.
Average Time on Site
We sampled 30 websites from our client database and ran a comparison of means using paired samples since each pair (social vs. non-social) is observed from the same particular website and is not independent. Each website’s unique social strategies make it difficult to generalize this to a higher population. It is important to note these were not independent random samples; there is a convenience bias based on the fact that we chose clients that had integrated some form of social strategy for their campaign in order to have enough referral data from social networks. We were also biased by choosing clients that had higher traffic so that we could have more data points.
For each website, we looked at the average time on site from those referred by social networks and the average time on site from non-social referrals.
The social networks included in this test were:
- Facebook
- Twitter (including shortened URLs like t.co, tinyurl, snipurl and bit.ly; twitthat, twittergadget)
- Delicious
- HootSuite
- StumbleUpon
- MySpace
- Bright Kite
- Digg
- Tumblr
- Technorati
- WordPress Blogs
- Live Journal
- Blog Lines
- Net Vibes
- News Gator
- Pr Web
- Lifestream.aol
- Faves
- Posterous
The time range for the test was September 1–November 30, 2011.
The null hypothesis for this T-test is that there will be no significant difference in average time on site between social referrals and non-social referrals.
The alternative hypothesis that we are trying to prove is that non-social referrals will have a higher average time on site than social referrals (agreeing with the counterpoint above). We can try to prove this by rejecting the null hypothesis.
μ1 = Average time on site from social referrals
μ2 = Average time on site from non-social referrals
Ηo: μ1 = μ2
ΗA: μ2 > μ1
α = 5% (significance level)
At this 5% significance level, we would reject Ho if the value of t exceeds 1.64 based on normal approximation.
where
= sample mean of differences
= hypothesized population mean difference
sd = standard deviation of differences
n = number of observations
Using the statistical package StatPlus to run the analysis, we found the test statistic to be:
t = 1.56403
Therefore, we failed to reject our null hypothesis at the 5% significance level. We failed to prove there is a significant difference in time on site between social referrals and non-social referrals.
For reference:
- The average time on site from all social referrals was roughly 1 minute and 14 seconds.
- The average time on site from all non-social referrals was roughly 1 minute and 49 seconds.
Pages per Visit
Similarly, we can set up a hypothesis test for number of pages per visit between social and non-social referrals.
μ1 = Mean number of pages per visit from social referrals
μ2 = Mean number of pages per visit from non-social referrals
Ηo: μ1 = μ2
ΗA: μ2 > μ1
α = 5% (significance level)
Again, we would need a t-statistic >1.64 to reject the null hypothesis.
Our test-statistic was found to be:
t = 1.71723; p-level = .0483
However, even though our test statistic is greater than 1.64, our confidence intervals overlap. Therefore, we cannot reject the null hypothesis that the means are the same.
A 95% confidence interval shows the difference in mean pages per visit to be between (-.1024 and 1.17438).
Again, for reference:
- The mean number of pages per visit from social referrals was 2.89.
- The mean number of pages per visit from non-social referrals was 3.426.
Going Forward
So, was this all just a waste of time? What can we learn from this?
Well, even though we failed to reject our hypothesis that user engagement is the same between social and non-social referrals, we are hoping to run more tests like this to better understand user behavior.
We did find an interesting breakdown of social referrals for our sample of 30 clients:
Average Percentage of all Social Visits by type of Social Network
43.89% Facebook
21.54% StumbleUpon
14.48% Twitter
11.22% LinkedIn
7.59% Other
0.77% HootSuite
0.51% Delicious
Our clients see the majority of social referrals from Facebook at 43.89%. Keep in mind that the time range for this was only three months (September through November).
Perhaps we will conduct a proper study on this by including more clients and using a longer time range to compare. Also, user engagement is more than just average time on site and pages per visit. What other metrics do you think are important to measure user engagement?
We would love to hear your feedback on this and any ideas on what you would like to see tested, so feel free to leave a comment!
Casey Szulc
Latest posts by Casey Szulc (see all)
- July’s Biggest Changes In SEO - August 2, 2012
- The Impact of Social Media on User Engagement - January 3, 2012
- Cyber Monday Recap: A Comparison of Before and After - December 2, 2011
I’m interested in knowing how this applies to the scientific community. I’d also like to know how it applies to business owners.
My first question: What is the correlation, if any, between time on site and number of pages visited and actual conversions? I often argue with ‘old school webmasters’ about bounce rate, time on site, and pages visited because we often find there’s no correlation between those variables and actual conversions. In fact, if you’re a publication site with advertising, you WANT people to bounce so that you get paid for the click. If you’re an ecommerce site, you WANT less pages per visit to increase the conversion rates.
My second question: This is an instant in time that you aggregated the data for. That’s not a social media strategy. Social is a progression whereby trust issues are overcome and authority is built with the fan or follower. While a search visitor often has intent to purchase, a social visitor is researching over time. Your data should split out the data series by the number of visits per visitor and I believe you’ll find different findings over the course of multi-month or year-long strategy. And you should be measuring conversion, not data that may not matter.
Another analogy. You go to the bank and put money in your retirement fund, half the money in high risk, half in low risk. You come back a week later and find that the difference in your return is negligible… less than 5%. Is that how you measure the return of a retirement? Nope. Social is a long-term investment. It’s an amplifier to get word of your authority outside your immediate network. I don’t expect any of our customers to post a tweet and get a sale. But I do expect that, as they grow a following and communicate effectively, the investment will compound and eventually pay off in sales.
Advertising is an event, marketing is a strategy.
Hey Doug, these are great points. We will look into how to make this a more comprehensive full-blown study with a much larger date range and include more actionable metrics, such as conversions.
I’m not too sure that these results would mean much. For examples: traffic from StumbleUpon is COMPLETELY different than traffic from Facebook. There is an entirely different mindset between the two users. This goes for pretty much every different social network out there. While it may be convient to group them all together, it really doesn’t provide any useful data. Each network creates, attracts, and facilitates it’s own type of user. Just because these sites are all categorized as “social media” doesn’t mean that the traffic will be similar.
Scott, thank you for your comment. You make a very interesting argument. It’s true that we’ve grouped these social networks together even though each has a completely different user base. The goal of the study was not to show the difference between these well-known social networks but to show the difference between these social networks and users coming from search engines, banner ads, and other non-social referrals. I believe that certain sites can still be classified as “social” even if they have different user goals. But this raises an interesting question: Could all sites be considered social (in some way or another)? As we continue to develop this study we will have to define what a “social” website is and determine which sites fall under this category.
Thank God for analytics, I would quit when the math started.
I think this research is informative… but it’s definitely missing an important aspect. Can we compare quantitive results to qualitative results? For example, there may not be any noticeable change in time spent on the particular website, but how many more times is that user going to revisit the site or purchase something from the site? When they are redirected to a website via word of mouth marketing from a dear friend or colleague, is the quality of that particular visit of higher value than someone who simply types in the web address and clicks “enter?” Douglas Karr summed this sentiment up nicely, and I absolutely agree that this type of study takes time. I think there are so many more details that have to be considered with this research, and I don’t believe the results will ever be 100% sound. That’s human interaction for you.