I currently lead a small data team at a small tech company. With every thing small, we’ve got a variety of autonomy over what, when, and the way we run experiments. On this series, I’m opening the vault from our years of experimenting, each story highlighting a key concept related to experimentation.
And here we’ll share a surprising result from an early test on our referral-bonus program and use it to debate how you would possibly narrow your decision set for experiments (at the least after they involve humans).
Background: It’s COVID and we’d like to rent a zillion nurses
IntelyCare helps healthcare facilities match with nursing talent. We’re a glorified nurse-recruiting machine and so we’re at all times trying to recruit more effectively. Nurses come to us from many sources, but those that come via referrals earn higher reviews and stick with us longer.
The yr was 2020. IntelyCare was a baby company (still is by most standards). Our app was latest and most features were still primitive. Some examples…
- We had a way for IntelyPros to share a referral link with friends but had no financial incentives to achieve this.
- Our application process was a serious slog. We required a small mountain of documents for review along with a phone interview and references. Only a small subset of applicants made it right through to working.
During a recruiting brainstorm, we latched onto the concept of referrals and agreed that adding financial incentives can be easy to check. Something like, “Get $100 when your friend starts working.” Zero creativity there, but an idea doesn’t need to be novel to be good.
Knowing that many individuals might refer time and again in the event that they earned a bonus, and knowing that our application process was nothing in need of a gauntlet, we also wondered if it may be higher as an alternative to present clinicians a prize when their friends an application.
A small prize for something easy vs a giant prize for something difficult? I mean, it depends upon many things. There’s just one approach to know which is best, and that’s to try them out.
The referral test
We randomly assigned clinicians to certainly one of two experiences:
- The clinician earns an additional $1/hour on their next shift when their referral starts a job application. (Super easy. Starting an application takes 1–2 minutes.)
- The clinician earns $100 when their referral completes their first shift. (Super hard. Some nurses race through it, but most applicants take several weeks and even months in the event that they finish in any respect).
We held out an equal third of clinicians as a control and let clinicians know the principles via a series of emails. There’s at all times a risk of spillovers in a test like this, however the considered one group stealing all of the referrals from the opposite group gave the look of a stretch, so we felt good about randomizing across individuals.
Decidedly non-social: Many individuals hear these two options and ask, “Did you’re thinking that of trying prosocial incentives?” (Example: I refer you, you do something, we get a prize). Studies show they’re often higher than individual incentives they usually’re quite common (instacart, airbnb, robinhood,…). We considered these, but our finance team became very sad at the concept of us sending $1 each to a whole bunch of people that may not ever turn into employees.
I assume Quickbooks doesn’t like that? In some unspecified time in the future, you only accept that it’s best to not mess with the finance team.
Because the $1/hr reward couldn’t be prosocial without becoming a serious headache, we limited payouts in each programs to the referring individual only. This offers us two referral programs where the important thing differences are timing and the payout amount.
Seems timing and presentation of incentives matter. Quite a bit. Social incentives also matter. Quite a bit. For those who’re attempting to growth-hack your referral program, you can be smart to contemplate each of those dimensions before increasing the payout.
Nerdy aside: Fascinated by things to check
Product data science, with minimal exception, is involved in how humans interact with things. Often it’s an internet site or an app, however it may very well be a physical object like a pair of headphones, a thermostat, or an indication on the highway.
The science comes from changing the product and watching how humans change their behavior in consequence. And you possibly can’t do higher than watching your customers interact together with your product within the wild to learn whether a change was helpful or not.
But you possibly can’t test . There are infinite things to check and any group tasked with experimenting can have to chop things right down to a finite set of ideas. Where do you begin?
- Start with the product itself. Ask people who find themselves accustomed to it how they prefer it, what they want was different, Sean Ellis, NPS, the Mom Test, etc. That is the common start line for product teams and nearly everyone else.
- Start with human nature. For a lot of a long time Behavioral Scientists have documented particular patterns in human behavior. These scientists go by different names (behavioral economists, behavioral psychologists, etc.).
In my humble opinion, the 2nd of those starting points is severely underrated. Behavioral science has documented dozens of behavior patterns that may inform how your product might change most effectively.
A couple of honorable mentions…
- Loss aversion: people hate losing greater than they like winning
- Peak-End: people remember things more favorably after they end on a positive note
- Social vs Market Norms: every thing changes when people pay for goods and services as an alternative of asking for favors
- Framing: people make decisions based on information is presented
- Left-digit Bias: perceptions of a price are disproportionately influenced by the leading digit ($0.99 = 🔥, $1.01 = 🥱)
- Present Bias: people hate waiting
These mental shortcuts aren’t a silver bullet for product and Marketing. We’ve tested a lot of these in several settings to no effect, but some have worked. Perhaps you’ll find $160M under your couch cushions.
Zooming in on present bias
Many humans reveal a behavior pattern where they like small, immediate rewards over larger rewards in the long run. Social scientists call this present bias and measure it with tough questions like…
Individuals who select immediate, smaller rewards have some measure of present bias.
Present bias can vary across people and circumstances. You might be a patient person in most situations. But once you’re hungry or drained… not a lot. That is one reason why A/B tests are so useful in comparison with, say, surveys or interviews.
For our experiment, the query isn’t so different from the lab questions:
- Would you slightly have ~$8 sometime this week or $100 sometime in the approaching weeks and possibly never?
Experiment results
A lot of us (present company included) thought the $100 offer would do best. It sounds larger. It looks higher in an email. It’s easy to elucidate.
I also thought we’d get more from referrals with the $1/hr program, but didn’t think that may carry through to more referrals. I figured people would take the cash and run.
To my surprise the $1/hr program delivered more applications via referrals , all at a lower cost.
The $100 offer led to a 65% increase in referrals in comparison with our control group. This is big!. Many individuals like the sort of program because we’re only giving rewards when a referral is successful. It appears like the cash is well spent.
The $1/hr offer, nevertheless, is even higher. Referrals increased by 81% in comparison with our control group. And regardless that the rewards were smaller and paid out earlier, referrals from these IntelyPros still went on to begin working for us at concerning the same rate because the $100 group.
Yes, many IntelyPros earned rewards for referrals that applied after which did not cross the finish line, but the mathematics still worked out. Even with an imperfect conversion rate from application to working, the whole cost per latest working IntelyPro was lower than half of the $100 group’s cost.
ABT: All the time be testing
The role that referrals play for a business can change for a lot of reasons. As an organization becomes larger and more familiar, referrals turn into less vital. You don’t need people to spread the word since the word is already out.
We ran this test in 2020 during a worldwide pandemic. We ran a smaller-scale test a yr later and saw similar results. Would we see the identical results now, in 2025? Hard to say. Things are much different within the healthcare staffing space nowadays.
That is yet one more reason why testing is so vital. Something like referral bonuses could work wonders for company Some time being a dud for company B. The upside of product experiments is that the world is your laboratory. The downside is that it’s hard to know what generalizes to other products.
Key takeaways for many who made it this far
- Whenever you’re running experiments to enhance your product and collect data from the humans using your product, behavioral science is a fruitful place to search for test ideas.
- Sometimes the ideas that appear less great end up to have the very best impact!
- Timing and presentation matter for incentives. Before you increase the dimensions of some payout, perhaps put some thought into the effort and time required to earn that payout.
