Grant Schneider, Vice President of Machine Learning

Over my eight and a half years at Upstart, I’ve had the pleasure of working on a whole lot of interesting and difficult problems that were essential to our business. (See here for just what number of!) Recently, the Machine Learning team has been working on yet one more: quantifying one of the vital dynamic macroeconomic environments seen in a long time.

The past yr reinforced the massive impact that changing macroeconomic conditions like inflation and unemployment can have on Americans’ funds, and in turn, the credit performance of loans.

So, we wondered: Is there a strategy to measure the impact of the changing macro on Upstart-powered loans in order that our bank and credit union partners could higher manage their lending?

Seems there’s. We call it the Upstart Macro Index (UMI). We launched it yesterday, and I’m going to elucidate the way it was built.

First, we would have liked a strategy to quantify the macro impact on our borrowers across different months in a way that was each rigorous and actionable. As a classically trained statistician, my first instinct can be to run an experiment holding every thing constant except the month. But that doesn’t work because no such constants exist. We will’t give the identical person the identical loan in multiple months with a view to A/B test the effect of a given pair of months. Even when the identical person got here back for a second loan, they’d be in numerous financial circumstances and, on the very least, have a further loan.

In theory, we could as a substitute randomly sample a gaggle of loan applicants, decelerate the loan process for them to get a set of loans that spanned multiple months, and track their performance in relation to the macroeconomy during that period. Nonetheless, we all know our borrowers really love our industry-leading speed, and intentionally slowing things down is near heresy around these parts.

So, we will’t run an experiment. Could we try an easy comparison of the identical cohort’s loan performance in two different months? This won’t work, either, because the chance profile of a loan will change while it’s in payment. This “month on book” factor — the variety of months for the reason that loan was originated — is an incredibly essential feature for predicting risk, and we’ve spent a big period of time modeling and enthusiastic about the connection between them at each the person loan and portfolio level.

But what about comparing the performance of various cohorts in the identical month on book? For instance, August 2022 loan originations in November 2022 and October 2022 loan originations in January 2023 would each be scheduled for his or her third payments. The issue with this approach is that upgrades to our underwriting model, changes in our marketing efforts, macroeconomic shifts, seasonality, and any variety of other aspects could make monthly cohorts very different, especially when there’s a protracted time frame between them.

You might also ask if we could have a look at a subset of borrowers inside these cohorts based on their credit rating or income. But that’s too simplistic. Our entire business is premised on the concept that an individual is greater than only a three-digit credit rating and that artificial intelligence will be used to raised determine if someone is creditworthy.

If none of those methods work, then how did we do it?

At a high level, we evaluate our underwriting model’s predictions compared to observed defaults to evaluate any uncaptured effects in a given month. Fortunately, our revolutionary (and patented!) “loan-month model” already provides us predictions on a per-loan, per-month basis. That is in contrast to more traditional models, which predict losses at the person loan level and even just the portfolio level.

We model default hazard ratios (for those of you aware of survival evaluation) because the dependent variable here. These ratios are the expected probabilities of default at a given time within the lifetime of a loan conditional on having not already defaulted or fully paid off the loan, and there are multiple ways to regulate them to attempt to quantify the macro effect. The best can be to use a post-hoc adjustment to match the actual losses in a given month we’re tracking — called an “remark month” — to those predicted by the model. But in circumstances where monthly macro effects are correlated with other inputs to the model, equivalent to credit or employment variables, this sequential approach of first fitting the model after which estimating the consequences could cause the macro effects to be under or overweighted and subsequently incorrect.

As a substitute, we add an indicator variable for every remark month to the model inputs in our model-training pipeline and retrain it. The indicator variable assigns a particular value to the remark month in such a way that the model won’t make that mistake in correlation. In consequence, we will jointly estimate the effect of a given month alongside the greater than a thousand other variables we track, allowing the model to detect nuanced relationships between them.

model_pl = Pipeline([
('features', feature_eng_step),
(
'add_obs_month',
ColumnTransformer([
(
'observation_month',
convert_to_obs_month_cat_feature,
['month_on_book', 'first_payment_month']
),
('other_features', 'passthrough', other_features)
])
),
('estimator', estimator)
])
model_pl.fit(X, y)

We then take a sample of two million historical loan payment events (meaning whether a payment was made or not and what month on book) and regenerate predictions for the whole set of possible remark months since January 2017. Why this particular cut-off date? Since it was one in all the earliest time points where we had enough loan volume for the predictions to be stable. For instance, say that in our sample of two million payment events we isolate payments two and five for a particular loan originating in May 2022, corresponding to remark months of July 2022 and October 2022. We’d then recompute the expected hazard ratios for every of those two payment events for every of the possible remark months thus far (January 2017 to February 2023), yielding 62 months × 2 million sampled payments for a complete of 124 million predictions.

for first_pmt_month_i in payment_months_of_interest:
X[:, first_pmt_month_idx] = first_pmt_month_i
preds = model_pl.predict(X)

Next we take the typical of the two million predicted hazard ratios for every of the calendar months, leading to a vector of length 62, or the typical of that 2 million for every of the 62 months. Finally, we normalize these average predicted hazards by a long-term average hazard ratio. This makes the number more interpretable, in order that a UMI value of 1 implies that defaults needs to be in keeping with long-term average expectations, a price of two means they needs to be doubled, a price of 0.5 means they needs to be halved, and so forth.

Now that we’ve got our quantification of the relative macro effects, we (and also you, as of today!) can update our perspectives monthly on how the creditworthiness of American consumers is evolving. Moreover, we will apply an adjustment to the hazard ratios for loans we’re originating today, which is informed by UMI. This forward adjustment is a strong tool that permits our lending partners to mix their unique outlook on macro with our state-of-the-art risk models to acquire a bespoke model on the innovative of credit risk.

A final query you would possibly ask is whether or not we’re capable of use this observation-month variable in prediction given we won’t have any training data for future months. That is an exciting area of lively research, and we’ve got multiple promising ideas to pursue here — so, stay tuned!

Upstart intends to release UMI monthly, including revisions to prior months when applicable. Subscribe for first access to those updates here: upstart.com/umi

The statements and data on this site are current as of March 21, 2023, unless one other date with respect to any information is indicated, and are provided for informational purposes only. Past UMI performance can provide no assurance and isn’t indicative of future UMI results. UMI relies on historical data and Upstart’s evaluation of the losses inside Upstart-powered loan portfolios and is particular to Upstart’s borrower base. UMI isn’t intended to measure the macroeconomic risks when it comes to losses of loan portfolios or asset classes that should not Upstart-powered loans, including loans held by other segments of the U.S. population. It isn’t designed to measure the present state of the general economy or to measure or predict future macroeconomic conditions, trends or risks. It is usually not designed to measure or predict the longer term performance of Upstart-powered loans or of Upstart’s other products, overall financial results of operations or stock price. We expect that our research and development efforts to enhance UMI could end in changes or revisions to current or past UMI values.

All forward-looking statements or information on this site are subject to risks and uncertainties which will cause actual results to differ materially from those who Upstart expected. Any forward-looking statements or information on this site are only as of the date hereof. Upstart undertakes no obligation to update or revise any forward-looking statements or information on this site in consequence of latest information, future events or otherwise. More details about these risks and uncertainties is provided in Upstart’s public filings with the Securities and Exchange Commission, copies of which could also be obtained by visiting Upstart’s investor relations website at www.upstart.com or the SEC’s website at www.sec.gov.