A brief guide on constructing activation metrics for a product

In a previous article, I talked in regards to the Input > Output > Consequence framework, and the way “output” was the central piece, but not necessarily easy to define — simply because you would like it to be moved by your inputs, but at the identical time, it’s essential to have a causal link together with your final result.

User activation metrics fall under this category of metrics. “Activation” is the third stage of the Pirate metrics framework designed by Dave McClure (the famous AAARRR framework — Awareness, Acquisition, Activation, Retention, Referral, Revenue), and it is frequently defined as when your user passed the primary set of frictions, began using your product, received some value from it, and is now more prone to be retained in the long term.

Some examples of product activation metric:
Loom: Sharing a loom¹ 
Zappier: Setting a zap¹
Zoom: Completing a zoom meeting inside 7d of signup¹
Slack: Sending 2,000+ team messages in the primary 30 days²
Dropbox: Uploading 1 file in 1 folder on 1 device inside 1 hour²
HubSpot: Using 5 features inside 60 days²
¹2022 product benchmark from Open View 
https://openviewpartners.com/2022-product-benchmarks/
²Stage 2 Capital: the science of scaling:
https://www.stage2.capital/science-of-scaling

Measuring activation is significant since it helps you understand how well your product is resonating with latest users and whether you’re effectively getting them to develop into “lively” users. It’s the very first step toward user loyalty — that is the stage where in case your users are prone to stick around for the long haul. If activation is low, it may possibly indicate that there’s a problem with the product or the onboarding process, and it might be obligatory to make changes to enhance the user experience and increase activation.

• You would like Activation to be predictor of Retention, but at the identical time, you would like it to be easy enough as this needs to be a straightforward first step your users are following.
• Mainly, you’re searching for the smallest motion a user can take that can showcase the product’s value for them, but you would like this small motion to have a causal link with retention (nevertheless you define it).
• As with every ‘leading’ indicator, the causality piece (“doing motion Y results in long-term retention”) is difficult. You often start with observational data, and traditional data evaluation won’t offer you the complete picture, as it may possibly overlook confounding aspects that may impact activation/retention.

Using a cohort evaluation, you possibly can start constructing some intuition around what user actions could good candidate in your activation metric.

The concept is to:

• Group your users based on where they signed-up for youu product
• Separate them based on in the event that they made it to the retain stage or not
• Search for the actions which are overwhelming done by the users you made it to the retain stage, but not a lot by the users you didn’t.

Let’s say you run a fitness app. You begin creating monthly cohort, and also you notice that 70% of users that upload at the very least one workout throughout the first week of signing up are still engaged with the app a yr later, vs 40% in the event that they don’t. This could be a first idea for an activation metric.

A pre-requisite here is so that you can get the concept of which motion to check. In the instance above, you needed to have the concept to have a look at who tracked their workouts. That is where quant meets qual, and when your ‘user acumen’/common sense comes into play. Or your networking skills if you should ask the assistance of other material experts.

Some advice:

• It is advisable to provide you with just just a few ideas of potential actions, not necessarily look into too a lot of them, simply because because the adage goes: “should you torture the info long enough, it is going to confess to anything” (Ronald H. Coase). The more actions you choose, the more likely you can see something, but you might be at high risk of it being a false positive. So sticking to what is smart and shouldn’t be too far-fetched will be rule of thumb.
• It is advisable to adopt a principled approach to this, and only search for things that you just consider you’ll find a way to maneuver. If you happen to provide you with something too complicated/area of interest, you would possibly not find a way to maneuver it, and so this may defeat the aim of the entire exercise.

With propensity rating matching, you possibly can confirm or infirm your previous intuitions

When you’ve identified your potential activation signals, the following step is to be certain that they’re accurate. That’s where propensity rating matching can come in useful — to know if the correlation you found previously could actually be causation. Although this shouldn’t be the one solution existing, and it does require to have a bit of information around your users (which won’t all the time be the case) it may possibly be relatively easy to implement and may offer you more confidence in your result (until possibly further triangulation, with more robust approaches similar to A/B testing).

The concept behind propensity rating matching is the next:

• In an effort to find the causal link between taking the motion and retainment, ideally you’ll clone your users that took the motion and have the clone not take the motion — to check the result.
• Because it shouldn’t be possible (yet?), the following neatest thing is to look inside your data, find users which are very similar (almost similar) to your users that took the motion — but who didn’t take the motion.

Propensity rating matching is a strategy that permits you to find those very similar users and pair them. Concretely speaking, it’s about:

• Training a model to predict the likelihood of your users to take the motion you defined (their propensity).
• Matching users based on the previously found likelihood (the matching part)

(Note: you have got other ways to go about each steps, and a few great guidelines can be found online regarding find out how to select a model, find out how to select the best variable, what matching algorithm to pick out, etc. — for more information, see “Some Practical Guidance for the Implementation of Propensity Rating Matching”)

Taking our fitness app example again:

• You’ve identified that 70% of users that upload at the very least one workout throughout the first week of signing up are still engaged with the app a yr later, vs 40% in the event that they don’t.
• You train a model to predict the likelihood of your user to upload a workout inside every week of signing up — and you discover out that the chances are very high for users which downloaded the app via a referral link from a big fitness website
• You rank your users based on the likelihood, and begin doing a straightforward 1:1 matching (the first users by way of likelihood that took the motion is matched with the first users by way of likelihood that didn’t take the motion, and etc.)
• Post-matching, you see the difference drop greatly, but still being necessary for you to contemplate it as a possible candidate for an activation metric!

Cohort evaluation + Propensity rating matching can enable you isolate the impact of a particular motion on user behavior, which is important for outlining accurate activation metrics.

But this technique shouldn’t be a panacea —there are a bunch of hypothesis that comes with the methodology, and you have to to fine-tune it / have some validation to be certain that it really works in your use-case.

Specifically, the efficacy of PSM might be highly depending on how well you possibly can predict the self selection. If you happen to are missing key features, and the bias from unobserved characteristics is large — then the estimates from PSM will be very biased and never be really helpful.

All this being said — using this technique, even in an imperfect way, may also help having a more data-driven approach for metric selection, get you began on ‘what to concentrate on’, until you get to the stage of running A/B testing and have a greater understanding of what drive long run success.

Hope you enjoyed reading this piece! Do you have got any suggestions you’d wish to share? Let everyone know within the comment section!

And If you should read more of me, listed below are just a few other articles you would possibly like: