Five requirements to build a world-class digital experimentation program

September 21, 2020 By: Guest Author

This guest post is part of our Continuous Product Design (CPD) Evangelist series. 

Steve Harris is an experienced data product, data science, and data analytics leader in digital transformations and data transformations across industries. He has developed multiple large-scale, self-sustaining data platforms and ecosystems, and built multiple cross-functional global data organizations from the ground up.

In the digital world, we’re lucky to have the ability to collect and analyze massive amounts of data. We can understand performance, causes, impacts and drivers at a level that’s difficult to replicate in other areas.

One of the big advantages that this provides is the ability to apply this science and to experiment at scale. A digital experiment involves giving different populations different digital experiences and then measuring the results. You’re experimenting with ways for your users to engage with the end goal of improving their experiences.

At its best, experimentation is a great intersection of product, IT, analytics and user experience. The best companies constantly experiment. The FAANG companies come to mind—Facebook, Apple, Amazon, Netflix and Google. Compare your experience to others’ on their sites and mobile apps. Is there a difference? Does your experience change seemingly all the time? Chances are that you’re always in one or more experiments when you use these companies’ products.

In general, I prefer the term “experimentation” to “testing.” Experimentation is intentionally meant to convey the idea of a tinkerer who’s always experimenting and trying to find a better way to do things. The idea of experimentation also helps to promote the use of the scientific method. After all, we can’t just see if something is right—we have to prove it!

A successful digital experimentation program can be built from five key tenets—team, technology, governance, culture, and innovation. Let’s talk about each:


Having the right team, people and organizational structure in place is crucial to make sure that all the facets of testing are covered and to build a true testing program. You’ll likely start small with lean versions of the teams below and maybe people doing some of these roles on the side.

  • Experimentation team: This is the team that thinks through and leads the broad experimentation program.
  • IT: This is the team that builds the site, including some of the more complex assets that get tested. Make sure to have IT as a partner from the outset.
  • Stakeholders/Lines of Business: Typically, these are the end customers of an experimentation program. They ultimately will recommend experiments, will own the products that see lift from experiments and will know the business inside and out. As the program progresses, stakeholders tend to take a more and more active role in a successful experimentation program as they understand it better and benefit from the measurable improvements to their business.

It’s important to think about where in the organization the experimentation program would best fit. Organizations are very different, and there’s no one-size-fits-all answer to this question. I’ve seen experimentation programs be successful in organizations such as Analytics, Digital, Marketing, Product and IT. It’s important to know your organization.

It’s also likely that the experimentation organization will evolve over time, and this is absolutely true when a program is successful. Over time, the program might move from a part of the company that has new programs to a place where things are more operational or business as usual (BAU). It’s also common to see the program shift to have more input and influence from stakeholders as it gets to be more successful, even to the point of potentially moving from a centralized model to a decentralized model.


You need a way to actually run digital experiments. How do you give some populations different experiences? There are many tools out there from some of the biggest analytics vendors that allow you to create, implement, run, and analyze tests, such as Adobe Target, Optimizely, and Monetate (all Quantum Metric experimentation partners). Chances are, if you have an analytics vendor, they have a testing tool that can integrate well with the measurement you already have in place.

If you have a home-grown analytics environment, you can still use a SaaS platform such as Adobe Target or Optimizely to start, though you’ll likely want to eventually build something in house to match as your experimentation program gets more sophisticated. Some experimentation tools are WYSIWYG and allow you to manipulate the page and its elements in the tools themselves, while some require more development—this is probably the same as your experiments themselves, especially as you get better and better.


What does someone in your organization need to do to run a test? Setting up a reusable process that can be constantly iterated on and improved is key to being well-managed and to helping stakeholders understand what needs to be done to run a proper test. For example:

  • Experiments need to be calendared and recorded. For instance, if you are in financial services and testing two different offers of credit, you’re making an offer of credit to all people in the experiment. You need to record this for regulatory purposes. Similar considerations exist across industries. You also need to record the experiment to make sure that experiments don’t interfere with each other and so you know what was running when. You can create and provide a calendar that can both allow stakeholders to sign up for tests and can record all the different tests that have run and when they ran.
  • You need to make sure that tests run to the significance (and seasonality) that your organization declares. Once a test is run, there is a great temptation to view the results early and constantly. When one treatment is performing better early on, you might want to stop the experiment and implement the winner, but don’t do it! Statistical significance is just that—a way to know if the change you’re seeing is really happening or just a matter of chance.
  • Consider how your experimentation program will interact with IT. Experimentation tools can give great power to democratize the ability to make changes to the site. That can be great for experimentation, but if hundreds or thousands of people can make changes, that can quickly descend into chaos! IT should be a great partner from the very beginning to help figure out who can make changes, how they can make those changes, and who has to sign off for an experiment to go live. For instance, even a well-meaning product manager might want to make a change that is unintentionally against company policy or even illegal (think about a packaged foods company wanting to increase conversion by removing reference to the sugar content of its food). Intentional, thoughtful governance will give you the tools to stop this from happening.
  • Don’t let your experimentation tool become a de facto Content Management System. Many testing tools allow you to make changes for the test in the tools themselves, sometimes, the temptation can be there to save time and money by leaving the winning treatment live through the platform. Resist this temptation at all costs. You can quickly have an unwieldy system with less record of what has been implemented. You can also add weight to the page, and you can potentially have things so intertwined that you could never swap out your experimentation tool! IT will have a vested interest here and should be a great partner.


One of the best ways to get momentum in an experimentation program is to create a great culture around this work. Get people to engage, and make this something that people will want to engage in!

One way I’ve found is to get people to predict winners for tests. You can get people together and have them vote on what they think will win for an individual experiment and then let everyone see how they do. Leaders can and should participate! This can both create a sense of competition and also show that the highest paid person’s opinion (HIPPO) is often no more valuable than people on the ground. Make sure to tell your leaders this first so they’re prepared!

In addition, constantly take every opportunity possible to share results and to elevate those who are creating and running experiments. Let your stakeholders bask in the limelight, and let them see that being part of the experimentation program is rewarding, both from a cultural standpoint and from the perspective of building better products for your users.

Driving the culture can be a top-down and a bottom-up approach. Both are excellent ways to create momentum.


We’re driving a culture of experimentation, and we have to experiment too. One of the best ways to do that is to always perform experimentation on your experimentation—how meta! A high-functioning experimentation program is never static. There are always new ways to think about things and to do things better. And just like the experiments themselves, limit yourself to trying just a few things at a time so you don’t spread yourself and your team too thin.

There are many things you can test as you build up the maturity of your program. Some of the possibilities include: multivariate testing, limiting interference among tests, target shuffling, and implementing one-armed bandits. Experimentation can also be the gateway to a targeting and personalization program that can allow for even greater lift among diverse populations.
_ _

This doesn’t even begin to scratch the surface. Each of these points above could be a program itself, with so many aspects to it. But don’t let that deter you. If you haven’t already, just get started. Dive right in—the water’s warm. And remember to always experiment your way through your experimentation program!

If you liked this post, check out Steve’s last post on the Quantum Metric blog: The Rise of the Data Product Management Discipline.

Interested in Learning More?

Get a demo