Growth Marketing Considered Harmful

1 Introduction

Growth marketing, conversion rate optimization, and user experience (or UX) do not cease to be hot topics, particularly in today’s e-commerce industry. But what are they, exactly? And how do they relate to each other? According to a blog post by Julie Ley [4], growth hacking is a term first coined by Sean Ellis in 2012. She explains that it enables massive growth (especially in the startup world) by focusing on the entire sales funnel, being agile, and “unafraid to take risks.” It seems a significant amount of businesses swear by this concept since it supposedly contributed to the success of “many great companies including Hotmail, Dropbox, and Spotify” [4].^[1]

Now, in addition to growth hacking, growth marketing is another term that seems to have gained traction lately. Julie Ley describes it as “a more mature version of growth hacking” that “applies to a much larger set of companies” [4]. According to her blog post, it is a mix of growth hacking and traditional marketing that involves “data and agility to scale revenue” [4]. Data and revenue are two important keywords here since they explain the connection to two other techniques that come into play: quantitative experimentation (enter A/B testing) and conversion rate optimization (self-explanatory). These two are also closely intertwined since A/B testing is a crucial means to maximize conversions (purchases, registrations, newsletter signups, etc.) as a target metric.

In short, to establish the relationship between the different concepts: A/B testing—often referred to as experimentation—is an integral part of conversion rate optimization, which, together with growth hacking, is a key ingredient of growth marketing.

All in all, as of now, growth marketing and conversion rate optimization seem to be quite the success stories: papers are being written (e. g., Lopez Kaufman, Pitchforth, & Vermeer [5]), conversion rates are being optimized (e. g., McFarland [7]), and conferences are being held (e. g., Riddle [12]).

Recently, I attended one of those growth marketing conferences and in a fully occupied keynote, listened to the following three statements, which I found rather striking: “Conversion optimization is [...] user-centric”, “Only do ‘real’ quantitative analyses [...] don’t ask five people on the street!”, and “don’t ask for your customer’s opinion” (because it is just an opinion). Starting from the hypothesis conversion rate optimization equals user experience optimization, in this essay, I will challenge these three statements by presenting counterexamples from real-world use cases.

2 “Conversion Optimization Is […] User-Centric”

Beginning with the first statement, already the name suggests that conversion optimization is inherently business- and conversion-centric rather than—at its core—oriented towards users’ needs. Still, this does not mean the two are mutually exclusive, but overlapping concepts. In a lot of cases, maximizing conversions and providing a better user experience (or usability) go hand in hand, e. g., when optimizing the flow of a checkout process. However, if we approach this from a theoretical perspective and have a look at the hypothesis “conversion optimization is user-centric”, it is relatively easy to find a counter-example. Consider the following real-world case I witnessed: Changing the label of a button in an online shop from “buy now” to “reserve and continue.” Conversions increased because users did not understand anymore that they are already making a commitment. Yet, this is clearly not user-centric (Table 1, case #1). Another example is the application of the scarcity heuristic on e-commerce websites (“only 2 pieces available; 5 other users are currently looking at this”). It has been well-established in numerous case studies that this is good for conversions (e. g., Brebion [1]; Fernandez [2]). Yet, this practice exploits a cognitive bias since humans perceive things as more valuable or useful if they are (or appear to be) scarce [6]. This potentially pressures users into buying products they do not actually want or need, which again is clearly not user-centric if it goes beyond simply informing the user about the current stock.

On a different note, Jim Lewis and Jeff Sauro investigated the connection between System Usability Scale (SUS) and Net Promoter (NPS) scores and found that “SUS scores explained about 36 % of the variation in [likelihood to recommend]” [13]. That is, usability has a direct impact on customer loyalty and recommendations (Table 1, case #2). This might tempt one to reverse the above statement and say, “user centricity improves conversion rate”. However, like the original statement, this is not universally true as well. One can easily find examples in which an improved user experience can negatively influence conversions. Assume the following case: A customer considers buying a new phone that is not compatible with their computer, but they assume compatibility because they simply expect it and the product description does not explicitly state otherwise. If you leave the description like it is, there is a fair chance the customer will buy the phone. Now, if you add more information about compatibility, thus improving the user experience, you will most probably lose the customer. Therefore, a better user experience—in this case—might decrease conversions (Table 1, case #3). Another, very recent example I know of first-hand: During the current COVID-19 crisis, delivery times at a large German online shop increased drastically. Since users were informed about this only after having completed a purchase, they faced many angry customers. Hence, they decided to inform about slower delivery earlier in the funnel. The number of angry customers decreased, but conversions also measurably dropped (Table 1, case #4). Yet, from my experience, the cases in which an improved user experience has a positive impact on conversions largely outnumber the negative cases [15], and user-centricity can have a massive return on investment (ROI) (e. g., Nielsen [9]).

It is very important to understand that conversion optimization and user-centricity are intertwined, but separate concepts. One should not assume that means to improve conversions automatically lead to a better user experience, or vice versa. Therefore, when implementing measures to increase conversions, it is crucial to always keep an eye on usability and user experience (be it through, e. g., SUS questionnaires or qualitative studies) and make sure they do not get adversely affected.

3 “Only Do ‘Real’ Quantitative Analyses”

This statement is directly connected to the last one since naturally, conversion rates can only be measured in a quantitative manner. During the keynote I attended, the argument was that only quantitative analyses with “real”, i. e., large amounts of data (mostly referring to A/B testing on live websites) are worthwhile and one should not waste time conducting research with smaller quantities of users, i. e., less data. This, however, not only neglects the plethora of existing methods, but is also directly contradictory to research, e. g., by Jakob Nielsen and Rolf Molich, who found that only five evaluators are sufficient to find a majority of usability problems in a user interface [10].

Table 1

Cases from e-commerce and how they affect conversions as well as UX and usability.

Case	Conversions	User Experience	Usability
#1: “reserve and continue” instead of “buy now” on call-to-action button			—
#2: loyalty–usability correlation		—
#3: providing more information about a product’s compatibility			—
#4: informing customers about current delays in delivery			—
#5: inconclusive A/B test of product descriptions	±0	?	—
#6: after improving product descriptions			—
#7: negative A/B test with unknown UX impact		?	—

A/B testing clearly has its place for certain types of research questions, but the problem here is that A/B testing only tells you about the “what”, but not about the “why” [8]. This means that while you know that a concept you implemented works and has a positive impact on conversion rate (and potentially other revenue-related measures), you can only guess what the reasons are (given you had a good test hypothesis), or whether the conversion optimization had a positive or negative impact on usability and user experience. Therefore, it is absolutely crucial to complement those “real” quantitative analyses with something that tells you more about what is going on behind the curtain. This can be done by implementing metrics for usability and/or user experience that can be monitored alongside conversions, a topic I have extensively investigated in my Ph. D. thesis [14], or by conducting additional qualitative research with relevant customers. To illustrate this, let me give you another real-world example. We once tested a concept for better product descriptions where we introduced a new layout and interactions, but kept the existing information. We performed an A/B test on our live website, but we did not meet the required uplift in conversion rate for the new feature to be considered for implementation (Table 1, case #5). Yet, the numbers looked promising, so we conducted an additional think-aloud study in which we found out that users loved the new concept itself, but the provided information was rather useless. Following this, we implemented the feature and are now working on providing it with better content. In this case, the “real” quantitative analysis was not able to tell the whole truth and almost led us to throwing something away that was an improvement for our customers—in terms of conversions and user experience (Table 1, case #6).

As a takeaway, it is important to remember that data-driven decisions, just like conversion optimization, are not inherently customer-centric. Qualitative research and research with smaller batches of data will always have their place alongside “large” experiments, in order to paint a complete picture.

4 “Don’t Ask for Your Customer’s Opinion”

The argument made here was that opinions are inherently subjective and therefore not as useful as objective data from experimentation such as A/B testing. Exaggeratedly speaking: Only if you run an A/B test in which you compare two colors of a “buy” button and the green one leads to more conversions, you should implement the green button, but not when your customers tell you that they would find a green button better visible.

It is indeed true that humans often do not know what they want or need. Henry Ford supposedly said: “If I had asked people what they wanted, they would have said faster horses” [16]. Then again, the decision to buy something or not is subjective, as are the problems users encounter on a website, the latter being a thing customers are very good at describing [16]. In addition, in “The Design of Everyday Things”, Don Norman states that users are not designers and you should therefore only ask them for problems, but never for solutions [11]. Hence, it is extremely important to ask for your customers’ opinions—but in the right way. A/B tests are extremely valuable and the quantitative output they produce show you, as objectively as possible, the effect of a change on your website. However, they fail at uncovering problems users encounter on your website. We are currently running an A/B test that we were absolutely convinced would produce a significant uplift because customers repeatedly indicated they would really like the concept we are testing, but all the numbers are negative right now. The only thing we know is that what we implemented for the test is not working, but we have no clue whether the concept is just not good after all, or if the test introduced new problems that outweigh the positive impact (Table 1, case #7). It will be inevitable for us to conduct some follow-up qualitative research after the test.

The latter also illustrates an, in my experience, very effective strategy for research that addresses both, user experience and conversion optimization: Create test hypotheses based on customer feedback, then A/B-test appropriate solutions in a quantitative way, and finally conduct a qualitative analysis to see why the test turned out the way it did. As before, you need a suitable combination of quantitative and qualitative research as well as objective and subjective insights for a complete picture and the best possible insights.

5 Conclusion

So, why do I consider growth marketing to be harmful? From many conversations at the growth marketing conference I attended as well as within my professional (e-commerce) environment, I got the impression that an increasing number of people tend to believe that growth marketing, conversion rate optimization and user experience are, more or less, interchangeable concepts. To me, this is a very worrying trend since they are clearly not. Conversion rate optimization is a part of growth marketing (at least according to the definition given earlier) and user experience is a completely different thing, which just happens to have a large overlap with conversion rate optimization. Therefore, we need a better understanding of and better communication about the relationships between these three interrelated, but independent concepts:

Figure 1

Relationships between Growth Marketing, Conversion Rate Optimization, and User Experience; and where A/B testing is to be located.

Based on the above, it should become clear that the initially posed hypothesis conversion rate optimization equals user experience optimization cannot hold. Yes, users trigger conversions, so you should not treat the two entirely separately, but a conversion does not care about what exactly happened earlier in the funnel and what the user’s experience was, as long as they purchased something in the end. Hence, a sole focus on optimizing conversion rates clearly neglects the “X” in UX. Optimizing the user’s experience is much more than just data-driven experiments. It involves choosing the right strategy, designing, exploratory methods, and a mix of qualitative and quantitative research. A/B testing is just a very small part of user experience research and also happens to be the main instrument for professionals who optimize conversion rates. Therefore, it is crucial to understand—and always make clear—that good user experience can be a major driver of growth, but is inherently different from both, growth marketing and conversion rate optimization.

Ultimately, to conclude, I want to suggest three revised statements:

1. Conversion optimization must be user-centric.

2. Always try to complement quantitative analyses with qualitative insights—and vice versa.

3. Always ask for your customer’s opinion to uncover problems—but not to find solutions.