A large percentage abandon proper analysis when presented with statistics

We’re sure that many of you have been presented with the following claim online:

“40% of visitors will abandon a site that takes more than 3 seconds to load”.

But is it actually true?

It’s everywhere: on countless websites from solution vendors and on blogs run by SEO consultants and marketing specialists. It turns up in SlideShare presentations and has been included in numerous pretty infographics (and some not so pretty).

Despite it’s venerable age we saw the claim once again in a recent presentation and we decided that not only did it sound questionable, but that it would be an interesting exercise to check where it came from and whether it was accurate.

Not everybody cites the source but, to be fair, many do and before long we were in possession of the original study from August 17, 2009:

We don’t necessarily recommend that you sign up and download it, as it is rather old and rather brief. What’s more interesting are the issues that arise when we perform even a cursory analysis of the claim that it continues to spawn.

The report was compiled from on an online survey of 1,048 online shoppers who were asked a number of questions.

The 40% claim appears to be based on the following question in the study:

“Typically, how long are you willing to wait for a single web page to load before leaving the website?”

The results are reported for the following timeframes:

“Less than 1 second”, “1 second”, “2 seconds”, “3 seconds”, “More than 4 seconds”.

The most obvious thing we can note here is that no actual testing of any sort appears to have been done and no website analytics data was used. People were simply asked a few questions.

So what we actually have are 40% of the respondents saying they will abandon a website if it takes more than 3 seconds to load…but this doesn’t mean that they actually will abandon it, only that they say they will.

There is a difference between what people say they will do and what they actually do.

Further, it’s highly unlikely that people will accurately measure the passage of time when waiting for a page to load so the claimed 3 seconds is unlikely to be a reliable figure; in fact 1% of the respondents said they wouldn’t even wait 1 second for a page to load (in 2009!) which seems to suggest some issues with the timeframes.

Finally, and this is an important point, the study gives no indication of whether the respondents were allowed to give freeform answers or whether they were offered a limited choice of options. If, as seems likely, the answers to the question were limited to a small set of options then there might have been a response bias. This is where the range and wording of the allowed answers has an effect on the outcome. So in this case if the options were simply “less than 1 second”, “1 second”, “2 seconds”, “3 seconds” or “more than 4 seconds” then this might have tended to make those answering the question feel that 1 – 4 seconds was the usual range. Now we don’t know for certain how the study was done – but the point is that in order to assess the claims we need to know.

This may all seem trivial, and certainly people WILL abandon your site if it takes too long to load, but the study being cited simply does not justify the claim that 40% of visitors WILL abandon a site that takes more than 3 seconds to load.

If we are going to make informed decisions then it’s vital that we are accurate and precise; that we analyse the “facts” that we are presented with. Even though the very broad thrust of the claim is clearly valid, we mustn’t simply take it on board without a proper assessment.

“It will be found, in fact, that the ingenious are always fanciful, and the truly imaginative never otherwise than analytic.”
Edgar Allan Poe 

Now, the original Akamai/Forrester study is simply what it is, and it clearly informs us about how it was performed. We have little argument with it, but it does however use unfortunate language (“Forrester found that 40% of consumers will wait no more than 3 seconds for a Web page to render before abandoning the site”) which fails, especially when quoted out of context, to clearly state that we are dealing with mere sentiment and estimate rather than measured and verified fact. As a result a questionable claim has spread all over the web… even into the most reputable and supposedly rigorous marketing specialists’ online material.

Of course, there is data available that will tell us what the relationship between page load speed and abandonment actually is – but we need to be sure it is based on measurement rather than the notoriously unreliable answers that we all give to questions about our behaviour.

What should you take away from all this?

• Always check the original source of a claim.
• Many statistics are generated by people who are trying to sell you something.
• Statistics have not always been generated using proper controls and methodologies.
• There are complex nuances that should be considered even when dealing with apparently simple claims.
• In the content marketing era, where everyone is scrambling to create/curate content and everyone wants an infographic, there is a growing tendency for people to adapt and re-purpose content without much regard for accuracy, especially if it’s convenient or it seems to be broadly true.

Most of all, beware consultants bearing statistics when what you really need is a proper, deep analysis.