What works in e-commerce A/B experiments
[The investigators] categorise roughly 2,600 experiments into a set of 29 categories, and measure a few statistics, such as the average uplift. The categories that perform best in terms of average uplift are :
- scarcity (stock pointers) +2.9% uplift
- urgency (countdown timers) +2.3% uplift
- social proof (informing users of others’ behaviour) +1.5% uplift
- abandonment recovery (messaging to keep users on-site) +1.1% uplift
- product recommendations (suggesting other products to purchase) +0.4% uplift
Most simple UI changes to websites are ineffective. For example
- colour (changing the colour of elements on a website) +0.0% uplift
- buttons (modifying website buttons) -0.2% uplift
- calls to action (changing the wording on a website to be more suggestive) -0.3% uplift
We find that 90% of experiments have an effect of less than 1.2% on revenue, positive or negative. However, we find that overall our clients benefit from A/B testing campaigns, some greatly.
source: http://www.qubit.com/sites/default/files/pdf/qubit_meta_analysis.pdf