As I equipped for my Product Data Scientist interviews, I scoured the net for suggestions and frameworks on handling the “Success Metrics” interview query. Despite finding bits and pieces, an entire, end-to-end guide was missing. That’s why I’m excited to share the final word framework I crafted during my preparation, which landed me a suggestion from Meta! Dive in, and I hope it’s going to give you the results you want too!
Framework — Assume you’re a part of the DS team for Facebook Groups, how would you define success metrics?
Clarifying Questions — All the time start by asking clarifying questions. Make certain that you just flesh out every word of the query and most significantly the product in scope. For those who don’t ask any questions, that’s definitely a red flag, so please do!
Let me ask just a few clarifying questions. Are we taking in regards to the Groups within the Facebook core app? Is my understanding correct that Groups may be private or public?
Once the query is evident, then take an enormous breath and begin your answer by taking an enormous detour. Yes, you read that right — don’t just jump into answering the query. It’s paramount, and most significantly expected, to speak in regards to the product, the mission of the corporate and the way these two tie together before anything. So please make sure that that you just talk in regards to the points below.
Start your answer by taking an enormous detour — don’t just jump into answering the query instantly but talk in regards to the product, the mission of the corporate and the way these two tie together before anything.
Company’s Mission
To begin with, Meta’s mission is to bring people closer together and provides them the facility to construct communities.
Goal of the Product + the way it ties to Company’s Mission
FB Groups goal is to bring individuals with common interests closer together and it’s an important product of Facebook since its goal ties with the general mission of Meta to bring people closer together.
Users — All the time talk in regards to the users. Almost for each product there are 2 sides of users: the producers and the consumers. It is rather necessary to speak in regards to the user journey of each and much more necessary to present metrics later in your answer that cover either side.
For FB Groups we’ve two sides of users, the admins of the groups who’re the producers and the members of the groups which might be the consumers.
The admins create a gaggle, resolve whether it’s going to be pubic/private, send invitation for users to affix, post links/media/info and get the conversation began within the group.
The members either see on feed, search or get invited to a gaggle (depending on if it’s public or private) and once joined, they will engage with the group through posting, commenting, liking, sharing etc.
Advantages+Costs (each for the Users and Company) — before jumping into the metrics is nice to briefly talk in regards to the advantages and costs of the product each for the users and the corporate.
Certainly one of the fundamental advantages of FB Groups is that users with common interests can get together, which ties with Meta’s mission and contributes to FB app overall engagement. FB Groups also allow Meta to get a greater understanding of the users’ interests by the groups they’re members of, which in turn might help in making higher recommendations and more engaging feed for the users.
On the flip side, FB Groups could potentially make users to have interaction less with FB’s Newsfeed, which is the “heart” of the FB app and the place where revenue is generated through ads. One other potential con is when groups don’t get sufficient members or engagement, which could discourage the admins and create “empty shell” groups.
Sorts of Metrics to give attention to — Now it’s time to begin taking about metrics. Select 2 out of Acquisition/Activation/Retention/Engagement/Monetization to give attention to.
Now going to metrics, since FB Groups is a mature product, I consider it is sensible to give attention to Engagement + Retention.
Mention Company’s NSM (North Star Metric) + reporting numbers — so as to select the first metric, it’s necessary to have the NSM and reporting numbers behind our mind.
Before jumping into the success metrics of the product, let me quickly discuss Meta’s NSM, which is the variety of sessions per user per day. On top of that, Meta reports to Wall Street DAUs and MAUS. Hence once we discuss success metrics, and specifically when picking the first metric, it’s necessary to maintain the above in mind.
Metrics — Now it’s finally time to present the metrics. We’ll give metrics from the 2 areas we mentioned above we’d give attention to and it’s necessary to not over do it — 2 or 3 to-the-point metrics per area are good enough. Please note how we make sure that to have metrics that cover each the producer and consumer sides of users.
It’s necessary to have metrics that cover each the producer and consumer sides of users.
Engagement:
# of groups created per week with a minimum of 3 members per user
# of interaction inside Groups per user per week
# of sessions that involved Groups per user per week
Retention:
# of energetic days per user inside the past 7 days (Lively = used FB Groups)
2nd week retention = # of users that were energetic a minimum of once per week for two consecutive weeks / # of users energetic only the primary week
** I selected the week as timeframe since I consider FB Groups just isn’t meant for use every day.
Select Driver/Secondary/Guardrail metrics — Now it’s time to decide on the first, secondary and guardrail metrics from our list of metrics above. Don’t forget to speak in regards to the trade-offs!
Driver Metric:
# of sessions that involved group per week
[# of sessions is easy to explain, captures users behavior and ties with Meta’s NSM. If sessions involving groups are up then more interaction among users → creating communities.]
Secondary Metrics:
# of groups created per week with a minimum of 3 members per user (to capture supply)
# of interaction inside Groups per user per week (to capture demand)
# of users energetic 2 weeks in a row (this + last week) / # of users energetic last week (2nd week retention)
Guardrail Metrics:
# of users faraway from groups
# of groups shut down or reported
# of offensive/inappropriate posts posted on Groups per week
time spent on Newsfeed (we don’t want users to stop using Newsfeed as much)
Wrapping up — All the time a superb idea to quickly undergo the story you simply put together and show how/why it answers the query.
In order that’s it — that is the framework that worked for me and helped me land a suggestion from Meta! The identical framework may be used for any Product/Success Metrics questions asked for Data Science positions. Having a well-structured response that thoroughly scopes out all of the necessary components and takes the interviewer along your thought process is essential. Hope you enjoyed it and would love to listen to your feedback within the comments below!
