Test #2: “Multiple layout and design changes can improve the UX”



Implement a layout which simplifies the process for selecting and adding ancillaries while removing products which are not available or are dependent upon other selections will increase revenue per experience.

Key Performance Indicator (KPI):

Purchased Premium/Total Offer Premium

Traffic Source:

Desktop traffic only

Summary of difference(s) between variations:

Figure 1: Variation A – Original  (75% of traffic)

Figure 2 – Variation B (Test) (25% of traffic)
*** WINNER ***


Sample Size & Test length:

Approx. 4,800 visitors were randomly shown either variation over the same 25-day period.


By implementing the changes in layout and design below that each work together to simplify the user experience, we will increase purchases and revenue.

  1. Call attention to selected products by adding a green border, with a checkmark and green font
  2. Display products which are available and hide those not available (interactivity based on selections)
  3. Implement Add/Remove buttons in different colors to call additional attention to the available products
  4. Utilize a slider to enable easy selection of the Excess product


The test variation B increased revenue by  $581,483 annualized which is +21.09% with 92% significance.

Actionable Takeaways:

The test results supported the original hypothesis: Customers positively responded to hiding products and messaging that did not apply to them indicating this test met the customer’s usability needs in simplifying their experience.  Sometimes testing incremental changes can prove beneficial because you can learn from each change individually, but it takes more time.  When you A/B test multiple changes at once, you can not point to the specific drivers of the lift, but does it matter?  Test & Learn methodology is a good strategy to be successful in continuous testing.  But let’s say you don’t have the time or resources to do continuous testing… you can lump several ideas together and get test results faster because you can measure more significant changes — for better or for worse!