For this week’s R&D post, we ask a simple question spawned from a discussion we had in our research lab: How does social media impact user engagement?
Point: Social media makes users more engaged because people they know through networks like Facebook, Twitter and LinkedIn referred them to the website.
Counterpoint: Social media does not make users more engaged with a website because they were directed by a friend, but weren’t necessarily looking for that information. Referrals from organic search results, PPC ads, other non-social channels and banner ads will have higher user engagement metrics because those users were actively looking for that information.
You might already agree with one of the statements above, but we thought we would set up a simple hypothesis test with data from 30 of our clients to see if we can arrive at any interesting conclusions. This is not a study about the importance of social media on ranking factors or how it should be employed as an SEO strategy. Rather, it is a small-scale test to see how users interact with a site from social media by examining pages per visit, average time on site, and +1 metrics, tweets and likes.
Average Time on Site
We sampled 30 websites from our client database and ran a comparison of means using paired samples since each pair (social vs. non-social) is observed from the same particular website and is not independent. Each website’s unique social strategies make it difficult to generalize this to a higher population. It is important to note these were not independent random samples; there is a convenience bias based on the fact that we chose clients that had integrated some form of social strategy for their campaign in order to have enough referral data from social networks. We were also biased by choosing clients that had higher traffic so that we could have more data points.
For each website, we looked at the average time on site from those referred by social networks and the average time on site from non-social referrals.
The social networks included in this test were:
- Twitter (including shortened URLs like t.co, tinyurl, snipurl and bit.ly; twitthat, twittergadget)
- Bright Kite
- WordPress Blogs
- Live Journal
- Blog Lines
- Net Vibes
- News Gator
- Pr Web
The time range for the test was September 1–November 30, 2011.
The null hypothesis for this T-test is that there will be no significant difference in average time on site between social referrals and non-social referrals.
The alternative hypothesis that we are trying to prove is that non-social referrals will have a higher average time on site than social referrals (agreeing with the counterpoint above). We can try to prove this by rejecting the null hypothesis.
μ1 = Average time on site from social referrals
μ2 = Average time on site from non-social referrals
Ηo: μ1 = μ2
ΗA: μ2 > μ1
α = 5% (significance level)
At this 5% significance level, we would reject Ho if the value of t exceeds 1.64 based on normal approximation.
= sample mean of differences
= hypothesized population mean difference
sd = standard deviation of differences
n = number of observations
Using the statistical package StatPlus to run the analysis, we found the test statistic to be:
t = 1.56403
Therefore, we failed to reject our null hypothesis at the 5% significance level. We failed to prove there is a significant difference in time on site between social referrals and non-social referrals.
- The average time on site from all social referrals was roughly 1 minute and 14 seconds.
- The average time on site from all non-social referrals was roughly 1 minute and 49 seconds.
Pages per Visit
Similarly, we can set up a hypothesis test for number of pages per visit between social and non-social referrals.
μ1 = Mean number of pages per visit from social referrals
μ2 = Mean number of pages per visit from non-social referrals
Ηo: μ1 = μ2
ΗA: μ2 > μ1
α = 5% (significance level)
Again, we would need a t-statistic >1.64 to reject the null hypothesis.
Our test-statistic was found to be:
t = 1.71723; p-level = .0483
However, even though our test statistic is greater than 1.64, our confidence intervals overlap. Therefore, we cannot reject the null hypothesis that the means are the same.
A 95% confidence interval shows the difference in mean pages per visit to be between (-.1024 and 1.17438).
Again, for reference:
- The mean number of pages per visit from social referrals was 2.89.
- The mean number of pages per visit from non-social referrals was 3.426.
So, was this all just a waste of time? What can we learn from this?
Well, even though we failed to reject our hypothesis that user engagement is the same between social and non-social referrals, we are hoping to run more tests like this to better understand user behavior.
We did find an interesting breakdown of social referrals for our sample of 30 clients:
Average Percentage of all Social Visits by type of Social Network
Our clients see the majority of social referrals from Facebook at 43.89%. Keep in mind that the time range for this was only three months (September through November).
Perhaps we will conduct a proper study on this by including more clients and using a longer time range to compare. Also, user engagement is more than just average time on site and pages per visit. What other metrics do you think are important to measure user engagement?
We would love to hear your feedback on this and any ideas on what you would like to see tested, so feel free to leave a comment!