Mistakes We Are All Guilty of Making When Analyzing Data

Graph showing mistakes we make in analysis of data

If you run your own ecommerce store, then you will undoubtedly spend a lot of time analyzing the statistics for your site. Whether you have just released a new product and want to know how popular it is, or want to know which products are selling best overall, statistics can give you almost any information you want. However, there are some pitfalls that many website owners make when analyzing their website statistics. Pim Bellinga, Founder of I Hate Statistics has come up with some of most-often made mistakes that website owners make.

Pim Bellinga, comments on the reason for releasing this information:

“What we saw is that many startups do A/B testing without a deep understanding of what is going on. Our mission is to help people make better decisions based on data. We currently focus on teaching students, but we thought startups like us might like a concise overview of what to look out for as well.”

Make conclusions before finishing testing

As humans there is nothing we hate more than waiting, as store owners we are notoriously impatient. When we start testing something, we want results and we don't want to wait a while for the results, we want them right away. This means that a lot of people check their tests every day, to see the results, and then as soon as they notice a statistically significant result, they stop the test and declare a winner.

However as Pim Bellinga states, the problem with this is that “each time you check and make a conclusion, your chances on drawing the wrong conclusion double. After five times, your chances on a wrong conclusion have jumped from 5% to 23%!”

What does this mean?

Once you start testing, leave the test results until after the testing has been completed, and only then should you come back and look at the results. In other words, during the testing phase you need to fight the urge to look at the data.

Interpreting variations in your data as something you caused

As a store owner, we are quick to try new methods of increasing revenue, we are also just as quick to interpret our statistics as something we did.  Many of us are guilty of saying “Oh I changed or did XYZ and therefore I am seeing this spike in the data”. In ecommerce however, variations can often be random, there will be some days that are busier than others without any particular reason. Therefore, it would be a mistake to jump the gun and say “I must have caused this!”

What does this mean?

When analyzing your data, be very careful with what you attribute a spike in your traffic to, consider the fact that it could be random before saying the spike was caused by something you did. Only if there is a very clear indication that your peak is caused by something other than random coincidence, then you should study this data, otherwise wait and see if your changes really have an impact.

Feature image curtsey of Alexander Stotskii


Catalin Zorzini

I'm a web design blogger and started this project after spending a few weeks struggling to find out which is the best ecommerce platform for myself. Check out my current top 10 ecommerce site builders.