What is a testing culture? This may vary from organization to organization, but in my mind it means that testing is at the center of every customer-facing change an organization makes. If there is a pending change to a website or mobile application, then we will always look to test the change before making it permanent. This sounds good, but in order for an organization have a true testing culture, there are several foundational pieces that the organization needs to have in place to be successful on this front.

First off, the organization needs a definition of what their testing culture will be, and the staff should be educated on this so that it is clear. If an organization defines their testing culture as one where all changes to the company website are tested before being put in, then they need to be sure the process for testing is communicated, understood and followed.

This leads to the need for a testing process. A repeatable process for testing that is well organized and clearly defined is key to success in developing a testing culture. A few steps in a testing process would be:

  1. Define the Key Performance Indicator (KPI), this is the metric that you intend to improve by testing.
  2. Define the test hypothesis: If I do X, then I think Y will happen. This should be something that you think will move your KPI positively.
  3. Determine the Sample needed in order to measure the lift you hope to achieve with your test. Estimate the time frame that the test will run.
  4. Obtain internal approvals needed before running the test.
  5. Setup, Run and Monitor the test.
  6. Analyze the test.
  7. Make recommendations and communicate results.
  8. Implement the winner.
  9. Start the process again.

Seeing this fairly simple test process may bring to mind the resource needs for running a process like this. First off, you’ll need a software tool that can manage the testing process for you. The tool will ensure that your intended population randomly sees the test offer and will keep track of the results for you. Secondly, you need staff resources that can strategize, design and analyze test results. We have many different roles who contribute to our testing culture including Account Managers, Optimization Managers, UI/UX Designers and Developers, Configuration Analysts, Marketing Analysts and Statisticians.

Finally, to really give your testing culture the boost it needs, Senior Leaders in your organization need to be bought in on the testing process. Senior Leaders need to ensure you have the people, tools and technology to support the process and they need to show interest and engagement in the process so that the rest of the organization sees that it is a priority.

One way to help your Senior Leaders understand the positive impact that testing brings to your organization is to quantify the impact. By quantifying the impact, you provide them with a measure of success that they can relate to other initiatives. It is one thing to communicate that the most recent test run had a lift in Revenue of 7%, it is quite another thing to translate that 7% number into an actual dollar amount so that everyone is clear on the Revenue associated with the effort. The practice of converting your result to Incremental Revenue will also help everyone understand the relative impact of a test. You can have a low revenue product that drives a large lift and can accurately compare it to a high revenue test that generated a low lift. The lower lift test may have driven a more substantial impact and this is important for your leaders to understand.

Creating a testing culture in your organization will require intention, commitment and discipline. Consistency in supporting the process and allowing the process to run is very important to the long term success that will come from a culture of testing. It will be time and effort well spent as the development of this culture will be rewarded with better decision-making and ultimately stronger results.