Qanuk BlogA/B testing - how to measure the effectiveness of recommendations in your store?

A/B testing - how to measure the effectiveness of recommendations in your store?

by Bartek Kwiatkowski Jan 13, 2022

Today’s world of eCommerce is defined by the laws of competition. Truth be told, they’re not really laws, but rather companies’ ability to adapt, change and expand in the right direction. Slip-ups or missed opportunities can often have serious repercussions and that’s why businesses cannot afford to take guesses when it comes to decision-making. A/B testing is a great way to ensure it never happens. Data is power and it’s paramount for companies to realize its potential not only for facilitating sustainable growth but most importantly, for making sure you stay competitive. In this article, we will focus on using A/B testing to measure the effectiveness of recommendations in online stores. This is, after all, what we specialize in at

What is A/B testing?

Picture this: you’re a baker, tasked with preparing a cake for your best friend’s wedding. You can’t decide between a chocolate cake with peanut butter and brownie filling or a classic sponge with buttercream and fresh berries. Both are great, but which one is the best for this situation? Since you can’t make the decision yourself you serve both to some of the guests prior to the reception and watch their reactions. Did they finish their portions? Are they going back for more? Are they nodding approvingly? The thing is, they don’t know the purpose of your ‘experiment’. They don’t mind, though - it’s free cake after all! You, on the other hand, know which of the two cakes is the best choice for the wedding.

That is precisely what A/B testing is all about. In real life, the baker is actually a website developer and the cakes are different versions of a website. Each of them is displayed to visitors at random to see, which one does better according to a chosen metric, for example, amount of money spent, CTR, or the number of sign-ups.

Does it work, though? Should you join in? In short - yes. In more detail - do you use Spotify, Facebook, LinkedIn or eBay? They, along with many other big players in the online world, are using A/B testing to constantly and continuously improve their services. This is one of the main reasons behind their sustained success. Even back in 2008 to find out, which type of website layout produced the most sign-ups. The team produced 24 different variants of the homepage and tested it across 300,000 visits. The results were tangible - a 40% increase in sign-ups, 2.8 million leads and $60 million in donations. The numbers speak for themselves. The real benefit of A/B testing is that it allows companies to measure the effectiveness of several versions of their website easily and very accurately so. It produces real data that can be used to enact changes that work.

What are recommendations?

Nowadays, A/B testing is widely used in eCommerce environments to find out the most suitable solutions. The industry is particularly competitive, as many shops don’t vary that much with their offer. Their task is to keep the visitors in the store when they visit, as opposed to leaving the website and picking another one. At, we implement solutions that help with just that, namely visual or behavioral recommendation systems. What are they exactly? Go to any bigger online store and click on any product. Scroll down and you’ll see an area titled something like: ‘recommended products’ or ‘customers who bought this were also interested in this’, along with several products that are either similar or related to the one you’re currently browsing. For example, with visual recommendations, when browsing a new blouse, the store will recommend a bunch of similar ones, showcasing a wider array of products, and acting like a virtual shop assistant by giving you more options to choose from. Behavioral recommendations take into account the behaviors of visitors on your pages. They analyze the data and then suggest products accordingly. For example, if you’re looking for a fishing rod, the algorithm will suggest adding a box of fish bait.

Where does A/B testing fit in with this? Let’s go back to the Obama website example. His team tested 24 different options containing six graphics and four CTA boxes. The goal was to find the best combination of the two elements that would secure the most sign-ups. Being elements of websites as well, recommendations work the same in the context of A/B testing. You’re aiming to find the most suitable placement, style and type of recommendation panels that will translate to conversions. Every store has its own unique needs and tests will allow you to precisely pinpoint them.

How to measure the effectiveness of recommendations?

Figure out what to measure.

The main metric you pick in order to put your experiments into a frame is called a North Star metric. The name is not accidental - just like the star guided ancient sailors, this value will guide not only your tests, but most importantly, the design decisions you’ll make. It might be obvious to start measuring conversions. But what does it actually mean? Each online store is different and has a different client base. A successfully converting website might mean something as simple as the number of products bought for a plant store. But for a wine and cheese shop, it might mean whether visitors were inclined to add a lovely triangle of brie they saw being recommended on a product page for merlot. Here are some of the metrics that can be measured using A/B testing:

  • Time spent on the store website,
  • The amount of money spent,
  • Click-through rate (CTR),
  • Time spent on each page,
  • Patterns of interactions on pages,
  • The number of clicks onto recommendation panels,
  • Number of sign-ups,
  • Number of returning customers.

Design good, scalable experiments that will give you tangible results.

Any good scientific experiment is thoroughly designed and capable of providing reliable and scalable results. A/B tests are no different. Make sure to pick metrics that will be capable of clearly showing, which design decision or system of recommendations is working the best for you, as well as the visitors of your pages. In the context of measuring the effectiveness of recommendations, a good measuring stick would be the number of clicks on recommended panels per the number of visitors, the money they spent on products related to the one they were browsing, or the patterns of behavior on your pages. These will likely allow you to find out the ways to improve your online store’s recommendation strategy.

A/B tests can be designed using widely available tools, such as Google Optimize or Firebase - our personal choices at It makes it easier to get started, as the framework for operations remains the same and is widely recognizable. You also need to keep in mind, which platform you’re testing for. Mobile is chasing or even surpassing desktop in many aspects and apps are frequently customers’ go-to when it comes to online purchases. Testing for different platforms looks different in the same way the layout of a page on your laptop differs from one on your smartphone. Similarly, what proves to be successful for one platform is not guaranteed to work for the other. That’s why, your best bet is to run two parallel experiments, one for desktop and one for mobile, so that you can adapt each to its unique needs.

Share this post:

Ready to increase conversions with machine learning?