How to measure the impact of product launch or key product change?

Think ahead

Defining the metrics before you even have something solid is quite a challenge. Yet we must think ahead. First of all, we should focus on data collection, and then — on how and if that data makes sense. But how and where to start if we have a new product idea or want to make a big change in one? We’ve even already asked the serious questions like: “How do you know that’s good for users?” or “Does it work?”. It’s easier to leave them behind and focus on product promotion. Or, what might be even more dangerous, look for help in vanity metrics to please investors or sponsors.

Vanity metrics

The difficult part is already behind you. You defined your product (or pushed a significant change to it). Now, you have to focus on proving that what you’ve come up with actually makes sense. Content might be king in social media but here, data is taking over control. Let’s start with what not to do. The so-called vanity metrics are those you should get rid of in the first place. For instance page views or downloads. This is great for printing in marketing leaflets. But in the world of data, there is no correlation between product success and page views. General metrics don’t carry enough useful insights. Using this kind of “easy” measurements you can sell your product. Yet, to make your sponsor happy we should dive deep into stronger proof. To squeeze out more meaningful information, start by staggering page views over time. This way, you’ll see who and when is actually creating these views. You can learn more about vanity metrics here.

Single metric approach

One of the common metrics promoted by Croll & Yoskovitz in “Lean Analytics” is OMTM — One Metric That Matters. It’s also known as the North Star Metric. This might not be one variable but a more complex value. Yet, adding more levels of complexity won’t necessarily let you make more sense of it. For SaaS startups, this can be the LTV/CAC ratio (we will get back to that).

Defining a single metric for a new product or significant feature also has business value. It gives you a fine goal to pursue. If it’s not too elaborate, others will follow and soon, the whole team will be crazy about the metric you’ve defined. Seeing it fluctuate will be like watching the stock market or scoreboard.

Simple metrics

On top of OMTM, you can use other metrics to support your decisions. Start by taking what you’ve already collected and try to segment those chunks of data. Cohort analysis, because that’s what you are doing here, will give you something to start with. Then, if you’re launching a big feature, you have to be sure that its impact on users is highlighted. To do so, you can stretch a cohort of your interest on a timeline “before” and “after” users have first used the new feature. If you do that instead of a standard timeline, you will see a closer correlation between cause and effect. Other useful metrics could be business and user engagement.

User engagement metrics

Sessions per user. In other words, “how often do users log in or open the app. Here, it’s better to use the median over average. Why? Consider this example:

“A company hiring 10 employees is spending $50k on their salaries. 1 person in the cleaning staff earning $500, 8 accountants getting $1500 and 1 manager who’s paid $37500 (!). The average salary in this company is $5000 but the median is $1500.”

Which one sounds closer to reality?

% Users Who Complete a Key Workflow. If you define a successful purchase at the end of your key workflow, you want to know how many users make it to the end.

Key user actions per session. Define your key actions in the customer engagement pyramid and set goals. If we take social media as an example, only 1% of users make up for key action — creating content. 10% is interacting, and 100% is passive observing. 

Session duration (remember to use it for a cohort, over time). How much time a user segment spends on interacting with the app or website.

Business metrics

Customer Lifetime Value (LTV). How much can you get from your user before they churn or stop paying. There are few methods on how to do this, you can read more about it here.

Customer Acquisition Cost (CAC). Meaning how much you have to pay to acquire a single user. Take all your marketing costs and divide them by the number of paying customers.

LTV/CAC ratio. I promised we will get back to this. The LTV/CAC ratio is especially important for your sponsors. For a healthy startup ration, starting from 3 is considered good.

Monthly Recurring Revenue. This one is quite self-explanatory.

Logo churn. It is the % of users you are losing every month. Statistics say that only 23% of users come back to an app within 3 days after installing it. This metric should be negative for your product.

Revenue churn rates. Another way to show the % of revenue that a company loses every month due to leaving users.

Customer engagement pyramid

So far, we’ve mentioned single metrics and high-level metrics. Some people say that they are all symptomatic and don't give you real value. However, having one serious OMTM and supporting business and user metrics divided by cohort and proper time frames can be really informative. If you want to dig even deeper into core factors that drive those metrics, you can build your customer engagement pyramid. To do so, you need to follow 2 steps:

Define the top of the pyramid as actions of the most engaged users. It can be publishing content, making a reservation or making a purchase. Those actions should be the foundation of your product.

Consider all actions that a user can make within your app or website. Then, place them in decreasing order from the most engaging to those requiring the least energy.

Here’s an example of a customer engagement pyramid for a car-sharing app:

What is the takeaway of this pyramid? First, you can test if the data that you’re collecting is relevant. That’s a good pitstop to make sure if what you’re measuring makes sense. The top of the pyramid gives you information about which user actions are key. If you plan to improve your UX/UI, consider starting from those.

Complex metrics

If you want to dive even deeper into the world of data, you can start by scratching the surface of econometrics. The first step will be learning more about propensity score matching. Using this method is useful when you’re measuring the impact of new features on user engagement. The results of comparing 2 groups of users, based on their behavior after we have implemented a big change in the app, will most likely be inaccurate. One group won’t have the opportunity to use it, and this group has to be defined somehow. Randomization won’t be very helpful here. This is caused by many factors influencing the user and lack of information. To give you a simplified example:

“Assume you have 2 types (A and B) of clinical treatments for people with the same disease. There are 200 patients in group A and 50 patients in group B. You want to measure the influence of post-treatment medicine on people with this disease. If you give medicine to everyone, how would you know that it’s not the treatment that caused the change? You won’t. Using propensity matching, you can create 50 homogenous groups. Each consisting of 1 patient from group B and 4 from group A. This way, comparing the medicine’s influence will be more accurate. Even more so, if we also use other statistical methods).”

As you can see, the better the measurement you aim for, the more complex these calculations get. Don’t try to read more out of data that isn't really there. Start slowly by defining your pyramid and supportive metrics. Then, build or choose your OMTM. And you’re ready to go, whether you are launching a new product or making a change in the current one!

Define your own metrics

At this point, you should have enough information to define what metrics suit your product launch (or change) best. If you think there’s a strong correlation between fewer or more other metrics and combining them makes sense — do so. That will give you a personalised measurement. In the corporate world, it’s pretty common to define those metrics as KPI. Yet, organizations tend to forget why those were defined over time. Don’t fall into this trap. Also, don’t forget to collect the necessary data over time. Then, even if you change your metrics (after learning that one doesn’t make sense anymore), you will have a benchmark. Remember that feedback from these metrics should act as guidance, not validation. The very popular Net Promoter Score has not been proven to be correlated with company success. It is, however, a strong indicator of being on the right path to making your customers happy.
 

Future readings:

“Lean Analytics: Use Data to Build a Better Startup Faster” by Alistair Croll and Benjamin Yoskovitz