Grow your store with our weekly podcasts, videos, and letters. JOIN NOW
Ethercycle logo

This is a guest post by Nick Disabato.

Having run over 500 tests for over 3 dozen businesses over the past 6 years, I know a thing or two about CRO – and where things can go wrong.

First, though, here are some places where things rarely go wrong:

  • Getting enough traffic for statistically significant results. For online stores, you can at least run a new round of tests every month on the 2-3 pages that you get the most traffic, as well as your cart and checkout. Other lines of business are not so lucky!
  • The design & development process for a prototype. If you’re ready to start A/B testing, you already know how to actually build software, and you probably have design & development resources retained.
  • Focusing on the right metrics. You already know that AOV, CLTV, conversion rate, reorder rate, upsell take rate, and ARPU matter. Nobody is going around maximizing subscriptions to their mailing list, or views of their blog posts. Store owners care about getting a positive ROI out of their CRO activities, and that means revenue generation.

Which is great! But this isn’t enough. Most of the things that trip up store owners are around mindset, and following best practices that aren’t really great after all.

The main reason anyone tests is to move the needle for your store. So, how do you increase the likelihood that you’ll build tests that win?

Here are the 5 biggest blind spots that hold store owners back from an outsize win rate and significant gains in profit, and what you can do to not fall prey to each one of them:

Blind spot 1: Acting on instinct

Tests are most likely to win if they respond to customer needs and fix specific, observable issues. Put another way, you should have a process that researches how customers are acting on your store, figure out what motivates them to buy, and then create tests that respond to those behaviors and needs.

Evidence is a great way to find new test ideas. Instinct isn’t. “What if we tested this now?” is a great way to run tests that lose. But most stores do this, in the absence of a concrete plan.

Let’s say you have an add to cart button. You want to run a test that changes its text or appearance in some capacity. What do you change it to? Why? What do you think will happen if you make that change?

Good tests are intentional and deliberate, and every design decision that you make should have a clear reason. By researching any potential changes, prioritizing them to be built out, and creating hypotheses that connect directly to new experiments, you’re more likely to run tests that win.

Blind spot 2: Running “popular” A/B tests

Everyone has heard of the button color test that made a zillion dollars. Same with testing headlines. In fact, button colors & headlines are the most popular two kinds of tests – not only because they are well-known examples of A/B testing, but because they’re extremely easy to put together.

Popular A/B tests are generally popular because they have a popular perception of being low-risk and high-reward. And to be sure, Draft has run button color tests that have won. But those button color tests were also researched! For example, when running tests at the Wirecutter, we ran a test that changed every product’s buy button to the color of the store that it pointed to: Amazon orange, Walmart blue, Apple gray, etc.

We did this because customers often didn’t notice that they were going to other stores, and were thrown off by the disconnect between the Wirecutter’s experience and (for example) Amazon’s. Clickthrough rate increased as a result, because people knew where they were going and what they could do when there.

Rather than making your buttons brighter, change your buttons for a reason. Look into what motivates customers, and then change your headline to something that clearly meets their needs.

Blind spot 3: Not following statistics

Let’s say you run a test and find that your ARPU went up by 83¢ at 85% confidence. Does that mean you should roll the decision out to everyone, and expect that your ARPU will increase by 83¢ into perpetuity?

Not necessarily. Think of that 83¢ figure as the center of a bell curve with, in this case, a rather shallow slope. That bell curve represents the full range of expected long-term outcomes that you might get from rolling out your variant to all customers. In practice, you might do better or worse than 83¢. And an 85% confidence level is not generally what most optimizers use to determine success; in fact, most winners are rolled out if they are called at 95% confidence or above.

It’s also dangerous to peek at an experiment’s data while the test is still active. You should calculate your expected sample size and experiment duration ahead of time, check in only to fix bugs with your goal reporting, and only take action on the results after the experiment has run its course.

The best explanation of the math behind this is “How Not to Run an A/B Test” by Evan Miller. Here at Draft, we use his sample size calculator every day.

Blind spot 4: The HiPPO

The HiPPO, or “highest paid person’s opinion,” can severely harm the progress of your testing program.

Test ideas should never be prioritized by the rank of who suggested them. Good ideas can come from anywhere in the organization. In fact, the best ideas often come from people who are in the trenches every day, understanding the real problems that customers are facing. At Draft, we’ve come to trust customer support staff more than the CEO when it comes to figuring out the best things to test. They suggest ideas that win more often!

We solve this problem by giving the entire team access to our Trello board for project management, and point everyone to suggest new ideas in a specific column. New ideas are then discussed, researched, prioritized, built, and tested. This way, the CEO is given precisely as much power as the newest contractor.

Blind spot 5: Following the leader

Ecommerce has a habit of copying. Someone does something that works well, and others follow the same playbook.

This makes sense on the face of things, because nobody has any clear sense of what works at any time. But if you don’t actually lead on your own store’s user experience, you’re leaving your store’s fate in the hands of people who are playing at a higher level of the game.

Following the leader is a bad idea because of, you guessed it, a lack of research. Instead of copying what “works” for someone else, investigate why, what that means for you, and how – and if – you should respond to it.

It’s always good to pay attention to what other stores are doing. But it’s dangerous to implement others’ ideas wholesale without determining how they fit int your own big picture. Trust, but verify.

The goal is test ideas that win

The industry-wide success A/B test success rate is about 12.5%, or 1 in every 8 tests. Do you want a failure rate of 87.5%? No, of course you don’t. You’ll waste time and money on a bunch of tests that fail to move the needle for your business.

At Draft, our success rate hovers around 65% as of press time. How? Because we research our ideas, carefully prioritize new tests, and listen to the whole team.

Fundamentally, value-based design is a matter of abandoning your ego. You may know the kind of product that your customers need, but you won’t know how your customers behave – and why they buy – until you investigate them. By avoiding these blind spots, you should be able to improve your business’s optimization maturity, and create an ROI-positive optimization program in the process.

About the author
Nick Disabato is a designer & writer from Chicago. He runs Draft, an optimization consultancy for online stores. His most recent book is Value-Based Design, the definitive how-to guide for getting a positive ROI out of any design work. He's spoken at SxSW, O'Reilly Web 2.0, and eCommerce All-Stars, among a bunch of other places, and he thinks your dog is very good.

Shopify Plus merchant Tactical Baby Gear is disrupting the booming world of baby products — estimated this year to produce more than $11 billion in revenue worldwide — by specializing in military-style baby gear. Founder Beav Brodie and COO Alex Alexander Kristoff join us to discuss their seven-figure success.

Join this Q&A recorded live in front of an audience at the Chicago Shopify Plus Meetup.

What to do when you're feeling overwhelmed with busywork and not working on growing your business.

One of the top topic requests I get is around general business productivity.

People want to know how can I get more done?

Recently, Dylan Kelley from Wavebreak asked a wonderful inverse of this question:

"I'm constantly overworking myself... what are your top strategies for doing less?"

In today's installment of my Sunday Drive series, we talk about how to get your shit done (by doing less.)

I even reveal all of my inbox zero secrets since that's where most of us are losing our time.

You'll want to listen if you're feeling overwhelmed, overworked, or busy but not productive.

Shopify’s Unite partner conference hit hard in June bringing us the future of commerce.

Connect with Shopify leadership and partners to hear what's making waves.

(There was one common theme: Watch out, Amazon.)

The Sunday Drive is back—

It’s a bonfire:

I’m burning down the top three misconceptions about Shopify.

We’re back from Shopify Unite!

Learn:

  1. Shopify’s vision of the future
  2. What’s it mean for small business?
  3. Your new competitive advantage over Amazon

Don’t miss out on the podcast where I sit down with Shopify CEO Tobi Lutke. Subscribe in iTunes.

Ever get stuck at home with nothing to do?

That happened to me last Friday.

My wife was napping with Prince Valium after having LASIK that morning, our kids were at school, our baby was napping, and my car was at the dealership for annual service and a couple of upgrades. And because I am clinically incapable of relaxing, I decided to do invent some work myself.

I'm fortunate to be in a position to give help, so I posted on Twitter and Facebook that I'm open to questions with hope that it might make a difference for a few people.

I thought you might benefit from some of the Q&A, so here are the most helpful or thought-proviking questions...

Is a Shopify store a good fit for a mix of physical and digital goods? A knitting friend of mine is thinking about selling kits but also digital stuff (e.g., patterns in PDF format, and "how to" video instruction)

Yes, absolutely. Shopify integrates with several services to perform the fulfillment of digital goods automatically. In fact, one of the very first Shopify apps, Fetch, did just that.

What is the largest quantity of skus that Shopify can effectively handle? I.e many auto part sites aren’t on shopify because of xxx,000’s of skus

Officially it’s unlimited. The real limitation is being able to sort through it! I recently worked on a project for store with 10,000 SKUs. We were able to easily sort it using “power tools filters” app

What are the top Ecommerce marketing strategies or tactics you'd recommend to get more *repeat* customers?

Use auto-responders to cross-promote products post-purchase. Use Klaviyo to automate "win-back" email campaigns. Offer subscriptions. Segment purchasers and retarget them separately on Facebook with cross-sells.

You have 280 characters to send a single tweet to *yourself* in 2016 about Ecommerce/Shopify marketing to grow a store.

"Your most impactful efforts all evolve around audience-building. Gain focus by fiercely shedding distractions that don't support that goal. Teach everything you know, work in public, share your journey, live your truth, be your authentic self. Profit and be fulfilled."

What do you feel are the main things that separate "successful" (7 figures) eCommerce shops from those that fizzle out?

The successful ones have a total clarity of purpose. They know (through data, not best guesses) exactly who their customers are, how to reach them, and how best to serve them. They delegate instead of micromanage to scale via standardized processes. Don't let ego interfere!

What's the service-side benefit of Shopify Plus? Besides the additional features and functionality, what type of support and resources do I get from Plus that I don't get on Advanced?

Aside from access to Plus-exclusive apps like Scripts, Launchpad, and Flow to streamline and accelerate your business operations, with Shopify Plus, you'll get:

  • A dedicated point of contact at Shopify called a Merchant Success Manager.*
  • Reduced merchant service fees with the lowest possible credit card rate offered by Shopify Payments.
  • Peace of mind knowing Plus' 24/7 priority support means someone always has your back.
  • Up to nine clones for selling in different currencies, locations, or wholesale.
  • Collaboration with other high-end merchants in the exclusive Plus Facebook community.

* I’ve seen Shopify’s merchant success managers suggest promotions and outside the box growth tactics that have made our clients up to a half million with a single suggestion

How many Shopify sites get 500 unique visitors a day? I'm struggling to get traffic to my site. Any secrets to get to 500 to 1000 uniques per day?

Use Google Analytics to determine your highest-revenue traffic source and double down on that. Each month, pick one strategy to try then evaluate at the end of the month.

Most stores get their traffic from Facebook ads though.

I think you’re a fascinating guy. I’d love to know, Where does Kurt see himself in 5 years?

Thank you, Jay! I'm fully convinced you're one of the nicest people on the planet.

I have total confidence that I have pitched my carts to the right horse with Shopify, so in five years I still envision myself within the Shopify ecosystem.

Maybe I'm being small minded in my vision, but I truly hope to be doing this at an even larger scale in 2023. It has been consistently rewarding to provide so much value to people's lives that I want to keep going. I started this business in 2009 and I'm in no way bored of it yet. The last two years we've seen incredible growth, and I want to keep that rocketship going!

What's your favorite Facebook strategy(s) to attract "cold audience traffic" to new stores?

This is flat out the hardest part of marketing a store. It's why we always say "start building an audience on day1!" We've found the best results are buying using a lookalike audience of past purchasers and then segmenting it to improve relevance when combined with highly-targeted messaging.

For example, if I was selling baby products, I would create a lookalike audience from past purchasers, then segment that to just "new moms"

I’m a podcast listener and know some of your background and your ecommerce journey. I’d love to hear more about your business/personal growth and what you focused on to level up and how you plan to continue?

That's a great question. I'm going to answer it candidly.

I've been working on e-commerce projects my entire adult life. In college, I ran not one but two gray market resale businesses. I was so inexperienced I didn't even know they were gray market.

After school, I got my first job as an e-commerce product manager for a local aftermarket auto parts drop shipping company.

After that, I tried to start my own e-commerce platform for bike shops which failed but became my current e-commerce agency. (I suppose it's not a failure if you pivot and succeed, right?)

My personal growth is a different but equally important story. Entrepreneurs are so hard on themselves and can all benefit from self-care. I was no different. Fortunately, I met a wonderful woman who became my wife...but before she was willing to marry me, she made me go to therapy with her. It was absolutely one of the best things I ever did for myself. Had I not gone to see a therapist, There's no way I'd be where I am now.

I now spent a considerable amount of time every day thinking about my mindset and mental health.

Klaviyo or MailChimp? And why?

Klaviyo because their core focus is e-commerce and Shopify. They're specialists. MailChimp is great but they're generalists. Why wouldn't you use the best tool for the job? Who do you want to perform your brain surgery: a family physician or a brain surgeon?

Is Twitter marketing at all effective?

While I have greatly enjoyed Twitter for networking, I’ve never found it to be a good source of leads, qualified traffic, or anything else that would move the needle on a growing business. I say all of that with love. I greatly enjoy Twitter, but I sincerely doubt if we all deleted our accounts today, it would make any difference in anyone’s business.

Finally, to wrap things up, Joe Glennon writes, "Don’t have a question, just want to say I love this thread as much as your podcast! Thanks Kurt."

You're welcome! Hope this helps.

Want to increase revenue? Increase conversion, AOV, and CLTV.

Trick question. Of course, you do. No one was ever, "Ugh, no, I won't have anywhere to put all that money unless I get a bigger mattress."

My go-to apps to do that are:

  1. Bold Upsell
  2. Bold Brain
  3. Bold Recurring Orders
  4. Klaviyo
  5. ShopMessage
  6. Hotjar
  7. FOMO
  8. Privy
  9. ShipTimer*

*one of mine

Just search in the app store for any of the above.

Somebody ran a negative SEO campaign against us.

Apparently, I founded the ecommerce community’s go-to source for "buy viagra/Cialis online." 😂

In today's drive, I'll explain what happened, how we found out, what impact it had, and exactly how we fixed it.

Watch now.

Hope that helps!

Talk soon,
Kurt

We all know that entrepreneurs have a different mindset.

What sets a successful entrepreneur’s mindset apart from everyone else?

Is it a penchant for risk-taking, big thinking, or a preternatural charisma?

And if we can identify it, can you cultivate it for yourself?

2017 has been, without a doubt, my best year ever. And it wasn’t an accident. It was largely the result of a concerted focus on my own mindset.

As you begin planning for 2018, I’ve put together simple, practical tactics you can use to achieve your business goals.

Hope this helps!
—Kurt