Turbocharge your testing program by listening to your customers

(Or another title could be, “Ideation is a real word I swear and it’s important guys”)

When I first heard the word ideation used in a meeting, I immediately chalked it up as “consulting-speak,” it just didn’t strike me as a real word. In my defense, I was working for a consulting company at the time and heard it used by a consultant in reference to a testing and optimization program we were helping stand up for a new client. Though the word was new, the concept was not, and it’s so vital to a successful product testing program that Optimizely has an entire section of their knowledge base dedicated to it. The reason it’s so important is that every testing and optimization effort, regardless of scope, follows a well worn path.

The Blast Off
At the start, there is a flurry of activity as a backlog of ideas are tested, many of them have been laying dormant for months if not years. It’s the inheritance of not having a product testing program or capable solution for a period of time; you’ve had a whole bunch of very smart people across 100’s of meetings individually wondering “if only I could test that!” 

As soon as the project starts or the technology is implemented, these backlogged ideas are quickly documented, evaluated, queued up and implemented. This process is often guided by experienced people, either the testing vendor or an outside consultant, who has seen this before, knows what will have the biggest impact first, and has their own ideas to add as well. And then the worst possible outcome occurs… success! 

Test after test is delivering significant uplift. Tests are broad reaching and highly visible and exciting, and conversion rates are sometimes increasing by points at a time. The entire team, or even company, is giddy with the potential to transform their online channel. Now other parties who see the success you are having come up with their own ideas of what to test. These new test ideas can’t exactly compete with the original backlog of ideas you’ve been working with up till then, but now you have the momentum and the capability to test out the nuggets among them, so you do. 

The Apex, and Facing Re-Entry

And things are… good. Not great, but good. You are seeing uplift, mostly, but it’s not as dramatic. The ideas are typically edge cases, and that means they may be focused on smaller parts of the overall experience, or less critical changes, or a smaller segment of visitors. As a result, potential uplift is naturally capped. You miss the big wins, but still everyone is excited, “we just need more ideas like the first ones we had!” everyone thinks, hopefully. 

This is where things can fall off a cliff rapidly. You hear ideas that are just variants of ones that worked in the past. You try to mine your analytics data to point you in the right direction, but you are reading into second-hand signals, so it’s very easy to misinterpret the “why” behind the “what” you are seeing in the data and come up with a faulty hypothesis. You start exploring testing other aspects of the experience like server-side code, multiple messaging variants, or native mobile apps, but the technical difficulty ramps up and the risk is higher, so you can’t move as quickly and test as much. And now, not only are you not seeing as significant an uplift, you actually start seeing misses. Tests are moving the needle in the wrong direction. 

There is no fault here, as this is a natural part of the process. Even the best vendor services and consultants can’t help all that much at this point, as you are missing the signals you need to tell you what to test next. The catch 22 of testing is that the amount of tests you run over a given time directly correlates to the program’s success. But the more you test, the less you can think of to test. It’s like owning a Ferrari capable of nailing breathtaking track times all day long, but all the gas stations around you are running dry and closing one by one. Eventually all that amazing capability will be sitting there doing nothing. Sure it’s still going to look pretty, and it will still perform when you do find gas for it, but if you can’t use the potential, why are you making the payments?

How to Achieve Orbit

When I joined Quantum Metric, one of the first things that hit me about the teams using the platform was the palpable excitement at collaborating and viewing the truth of the users experience. When seeing visitors struggle, the people responsible naturally want to help! They want to improve their product and eliminate struggles and give their customers what they want. And invariably, almost every time an opportunity or an issue is explored, you get to a point where someone exclaims something to the effect of “Ok, well why don’t we just test X and see if it helps?” It could be people not scrolling down far enough to see vital information, or a popup obscuring a button on a specific mobile screen resolution, or a message that’s not prominent enough, but it’s nearly always something that can be tested.

Crawling through reams of analytics data, or sitting around watching replays all day, is no one’s idea of fun. Which is where the magic of Continuous Product Design, aligning teams on a single version of the truth about every customer’s experience, really shines. Now you can add the following capabilities to your product testing program:

• Visualization of every aspect of the user’s journey to inspire empathy and motivation within the testing team.
• Automatically calculating true business impact to opportunities and friction points within your site to prioritize what and when to test.
• Near-unlimited segmentation and exploration capabilities to identify test audiences and cohorts.
• Real-time signals of visitor frustration and confusion to trigger or suppress intelligent test variants.

In summary, instead of 1000’s of things to potentially test, with no way to prioritize, now your teams are aligned on the dozens of solid gold nuggets that will have an impact. 

And now the magic is back. Teams that have been starved for ideas are instantly transported back to the very first heady days of the program. Tests are fun and provide real business impact again, and you are directly helping your visitors, all because you finally have a way to listen to the customer signals directly, and the intelligence and power to do it at scale. If you thought the organization took notice before, you are in for a pleasant surprise. It is not about features or capabilities, but rather how organizations are able to align their teams to a common purpose, so they can get to delivering their best product possible.

Interested in learning more about how Continuous Product Design can enable your testing program? Reach out for a custom demo.