The Traps You'll Fall Into When Validating a Market

19 September 2020

On September 17, 2020, Mozilla announced the end of life for two products that many people thought were game-changers. Firefox Send and Firefox Notes were the answer to Google's Drive and Docs. The antithesis of the data-mining economy. Make no mistake, these two products were always experiments—something Mozilla spun out of their now-defunct Test Pilot program. The program was designed for experimentation, user feedback, and product validation. It's that last part that drives the main thesis of this entire post.

Mozilla clearly thought they had something with both Send and Notes. They must have, otherwise they would not have continued development of those applications as standalone products even as Test Pilot was shut down. And here comes the brutal fact that product "experts" won't tell you. Market validation is fucking hard. It's so hard, in fact, that if you build products long enough, you will invariably fall into the trap of believing you've validated a market.

It happened to me with Graphite. Much like it seems Mozilla thought they had found and validated a market with Send and Notes, I thought I had found and validated a market with Graphite. I had a ton of early interest. I had a loyal community. Graphite got press, was highly touted on product aggregation sites and blogs, and seemed to establish itself as a known quantity.

But I was fooled.

What actually happened was I had built a niche product for a niche community while thinking I had build a product for a generalized market and had validated demand for that market. I had actually only validated demand in the niche community. Creating a product for a niche market is a really good way to build a business, but you have to be intentional about it. You can't accidentally do it and then expect the product to work long-term. Unfortunately, that's exactly what happened with Graphite.

And I believe that's what happened with Firefox Send and Firefox Notes.

Mozilla's Firefox browser has less than an 8% market share and that has been holding steady for a long time now. Firefox, at this point, serves a niche audience itself. So, any product released by the parent organization—Mozilla—should be considered a niche product. Perhaps that is how Mozilla views all of their experiments, but they prepared widespread announcements around their Send product. They had product launches on the most popular sites. It felt like it was designed to be a commercially viable, secure alternative to file sharing tools currently on the market.

Unfortunately, either through abuse as Mozilla has indicated or a lack of adoption, the product was not viable as a widespread solution. So, the question is how do we avoid outcomes like this? Is it possible to avoid?

Validating a market must be an ongoing process. You are never done. That first taste of validity should be met with skepticism. If you did not set out to build a niche product, you have to, at this point, ask yourself if the market validation you're seeing is validation from a much smaller market than you intended. This is hard. It's not going to be obvious, but there are ways to analyze the reception of your product that will help.

For Graphite, I had a pretty good grasp on who was using Graphite in the abstract "persona" sense. I did not have actual user data, so I could not do as much analysis as other companies, but I had a fully formed user persona based on the traction I had seen.

Yet, I ignored it.

It's really easy to ignore the personas being formed and assume that capture one particular segment of a wider market means the other segments will be captured as well. So, in your analysis, you have to understand the personas of the people using the product. If you see a trend, that's when you must ask yourself if you should be scaling your expectations back and focusing on a niche.

There is a flip-side to this process, though. If you spot a trend in user personas and prematurely scale your expectations back, you have no chance of capturing the wider market. So through the process of analyzing user adoption, you need to be on the lookout for signs or hooks into other segments of the market. Do you have a few personas that indicate your user base is highly-technical but you have small, growing pockets of users who seem to be new to tech? Perhaps scaling back to a techno-focused niche is premature. However, if you see those small pockets of non-technical users churning, that is a sign that you can capture initial interest, but your product might not be suited for that group.

As you can probably gather from the preceding paragraphs, analyzing your market is difficult. Predicting a market ahead of time—i.e. during a fundraise—is mostly a guessing game. Educate yourself as much as possible so that your guesses are as accurate as possible.

But most importantly, don't allow yourself to be fooled into thinking traction within a segment of your wider target market means you have validated your product within that market. You have to always ask yourself what is real and what is a trap.