Over a whole lot of Kaggle competitions, we’ve refined a playbook that consistently lands us near the highest of the leaderboard—irrespective of if we’re working with tens of millions of rows, missing values, or test sets that behave nothing just like the training data. This isn’t just a set of modeling tricks—it’s a repeatable system for solving real-world tabular problems fast.
Below are seven of our most battle-tested techniques, every one made practical through GPU acceleration. Whether you’re climbing the leaderboard or deploying models in production, these strategies can offer you an edge.
We’ve included links to example write-ups or notebooks from past competitions for every technique.
Note: Kaggle and Google Colab notebooks include free GPUs and accelerated drop-ins just like the ones you’ll see below pre-installed.
Core principles: the foundations of a winning workflow
Before diving into techniques, it’s price pausing to cover the 2 principles that power every little thing on this playbook: fast experimentation and careful validation. These aren’t optional best practices—they’re the muse of how we approach every tabular modeling problem.
Fast experimentation
The largest lever we have now in any competition or real-world project is the variety of high-quality experiments we are able to run. The more we iterate, the more patterns we discover—and the faster we catch when a model is failing, drifting, or overfitting—so we are able to course-correct early and improve faster.
In practice, which means we optimize our entire pipeline for speed, not only our model training step.
Here’s how we make it work:
- Speed up dataframe operations using GPU drop-in replacements for pandas or Polars to remodel and engineer features at scale.
- Train models with NVIDIA cuML or GPU backends of XGBoost, LightGBM, and CatBoost.
GPU acceleration isn’t only for deep learning—it’s often the one technique to make advanced tabular techniques practical at scale.
Local Validation
Should you can’t trust your validation rating, you’re flying blind. That’s why cross-validation (CV) is a cornerstone to our workflow.
Our approach:
- Use k-fold cross-validation, where the model trains on a lot of the data and tests on the part that’s held out.
- Rotate through folds so every a part of the information is tested once.
This offers a way more reliable measure of performance than a single train/validation split.
Pro tip: Match your CV technique to how the test data is structured.
For instance:
- Use TimeSeriesSplit for time-dependent data
- Use GroupKFold for grouped data (like users or patients)
With those foundations in place—moving fast and validating rigorously—we are able to now dive into the techniques themselves. Each builds on these principles and shows how we turn raw data into world-class models.
1. Start with smarter EDA, not only the fundamentals
Most practitioners know the fundamentals: Check for missing values, outliers, correlations, and have ranges. Those steps are vital, but they’re table stakes. To construct models that delay in the true world, you must explore the information a little bit deeper—a few quick checks that we’ve found useful, but many individuals miss:
Train vs. test distribution checks: Spot when evaluation data differs from training, since distribution shift may cause models to validate well but fail in deployment.
Analyze goal variable for temporal patterns: Check for trends or seasonality, since ignoring temporal patterns can result in models that look accurate in training but break in production.
These techniques aren’t brand recent—but they’re often missed, and ignoring them can sink a project.
Why it matters: Skipping these checks can derail an otherwise solid workflow.
In motion: Within the winning solution to the Amazon KDD Cup ‘23, the team uncovered each a train—test distribution shift and temporal patterns within the goal—insights that shaped the ultimate approach. Read the complete write-up >
Made practical with GPUs: Real-world datasets are sometimes tens of millions of rows, which might slow to a crawl in pandas. By adding GPU acceleration with NVIDIA cuDF, you may run distribution comparisons and correlations at scale in seconds. Read the technical blog >
2. Construct diverse baselines, fast
Most individuals construct just a few easy baselines—possibly a mean prediction, a logistic regression, or a fast XGBoost—after which move on. The issue is that a single baseline doesn’t let you know much in regards to the landscape of your data.
Our approach is different: We spin up a various set of baselines across model types immediately. Seeing how linear models, GBDTs, and even small neural nets perform side-by-side gives us much more context to guide experimentation.
Why it matters: Baselines are your gut check—they confirm your model is doing higher than guessing, set a minimum performance bar, and act as a rapid feedback loop. Re-running baselines after data changes can reveal whether you’re making progress—or uncover problems like leakage.
Diverse baselines also show you early which model families suit your data best, so you may double-down on what works as a substitute of wasting cycles on the fallacious path.
In motion: Within the Binary Prediction with a Rainfall Dataset competition, we were tasked with forecasting rainfall amounts from weather data. Our baselines carried us far—an ensemble of gradient-boosted trees, neural nets, and Support Vector Regression (SVR) models, with none feature engineering, was enough to earn us second place. And while exploring other baselines, we found that even a single Support Vector Classifier (SVC) baseline would have placed near the highest of the leaderboard. Read the complete write-up >
Made practical with GPUs: Training a wide range of models may be painfully slow on CPUs. With GPU acceleration, it’s practical to try all of them—cuDF for quick stats, cuML for linear/logistic regression, and GPU-accelerated XGBoost, LightGBM, CatBoost, and neural nets—so you may recuperate insight in minutes, not hours.
3. Generate more features, discover more patterns
Feature engineering remains to be one of the crucial effective ways to spice up accuracy on tabular data. The challenge: generating and validating 1000’s of features with pandas on CPUs is way too slow to be practical.
Why it matters: Scaling beyond a handful of manual transformations—into a whole lot or 1000’s of engineered features—often reveals hidden signals that models alone can’t capture.
Example: Combining categorical columns
In a single Kaggle competition, the dataset had eight categorical columns. By combining pairs of them, we created 28 recent categorical features that captured interactions the unique data didn’t show. Here’s a simplified snippet of the approach:
for i,c1 in enumerate(CATS[:-1]):
for j,c2 in enumerate(CATS[i+1:]):
n = f"{c1}_{c2}"
train[n] = train[c1].astype('str')+"_"+train[c2].astype('str')
In motion: Large-scale feature engineering powered first-place finishes within the Kaggle Backpack and Insurance competitions, where 1000’s of recent features made the difference.
Made practical with GPUs: With cuDF, pandas operations like groupby, aggregation, and encoding run orders of magnitude faster, making it possible to generate and test 1000’s of recent features in days as a substitute of months.
Take a look at the technical blog and training course below for hands-on examples:
Combing diverse models (ensembling) boosts performance
We found that combining the strengths of various models often pushes performance beyond what anybody model can achieve. Two techniques which can be particularly useful are hill climbing and model stacking.
4. Hill climbing
Hill climbing is an easy, but powerful technique to ensemble models. Start together with your strongest single model, then systematically add others with different weights, keeping only the combos that improve validation. Repeat until no further gains.
Why it matters: Ensembling captures complementary strengths across models, but finding the best mix is difficult. Hill climbing automates the search, often squeezing out accuracy and outperforming single model solutions.
In motion: Within the Predict Calorie Expenditure competition, we used a hill climbing ensemble of XGBoost, CatBoost, neural nets, and linear models to secure first place. Read the write-up >
Made practical with GPUs: Hill climbing itself isn’t recent—it’s a typical ensemble technique in competitions—but it surely normally becomes too slow to use at large-scale. With CuPy on GPUs, we are able to vectorize metric calculations (like RMSE or AUC) and evaluate 1000’s of weight combos in parallel. That speedup makes it practical to check much more ensembles than could be feasible on CPUs, often uncovering stronger blends.
Here’s a simplified version of the code used to guage Hill Climbing ensembles on GPU:
import cupy as cp
def multiple_rmse_scores(actual, predicted):
if len(actual.shape)==1:
actual = actual[:,cp.newaxis]
rmses = cp.sqrt(cp.mean((actual-predicted)**2.0,axis=0))
return rmses
def multiple_roc_auc_scores(actual, predicted):
n_pos = cp.sum(actual)
n_neg = len(actual) - n_pos
ranked = cp.argsort(cp.argsort(predicted, axis=0), axis=0)+1
aucs = (cp.sum(ranked[actual == 1, :], axis=0)- n_pos
*(n_pos + 1)/2) / (n_pos*n_neg)
return aucs
5. Stacking
Stacking takes ensembling a step further by training one model on the outputs of others. As a substitute of averaging predictions with weights (like hill climbing), stacking builds a second-level model that learns how best to mix the outputs of other models.
Why it matters: Stacking is particularly effective when the dataset has complex patterns that different models capture in alternative ways – like linear trends vs nonlinear interactions.
Pro tip: Two-ways to stack:
- Residuals: Train a Stage 2 model on what Stage 1 got fallacious (the residuals).
- OOF Features: Use Stage 1 predictions as recent input features for Stage 2.
Each approaches help squeeze more signal out of the information by capturing patterns that base models miss.
In motion: Stacking was used to win first place within the Podcast Listening Time competition, where a three-level stack of diverse models (linear, GBDT, neural nets, and AutoML) was used. Read the technical blog >
Made practical with GPUs: Stacking is a widely known ensembling technique—but deep stacks quickly change into computationally expensive, requiring a whole lot of model suits across folds and levels. With cuML and GPU-accelerated GBDTs, we are able to train an evaluate stacks an order of magnitude faster, making it realistic to explore multi-level ensembles in hours as a substitute of days.
6. Turn unlabeled data into training signal with pseudo-labeling
Pseudo-labeling turns unlabeled data into training signal. You employ your best model to infer labels on data that lacks them (for instance, test data or external datasets), then fold those “pseudo-labels” back into training to spice up model performance.
Why it matters: More data = more signal. Pseudo-labeling improves robustness, acts like knowledge distillation (student models learn from a powerful teacher’s predictions), and might even help denoise labeled data by filtering out samples where models disagree. Using soft labels (probabilities as a substitute of hard 0/1s) adds regularization and reduces noise.
Pro suggestions for effective pseudo-labeling:
- The stronger the model, the higher the pseudo-labels. Ensembles, or multi-round pseudo-labeling normally outperform single-pass approaches
- Pseudo-labels will also be used for pretraining. High quality-tune on the initial data as a final step to cut back noise introduced earlier.
- Use soft pseudo-labels. They add more signal, reduce noise, and allow you to filter out low-confidence samples.
- Pseudo-labels may be used on labeled data—useful for removing noisy samples.
- Avoid information leakage. When using k-fold, you should compute k sets of pseudo-labels in order that validation data never sees labels from models trained on itself.
In motion: Within the BirdCLEF 2024 competition, the duty was species classification from bird audio recordings. Pseudo-labeling expanded the training set with soft labels on unlabeled clips, which helped our model generalize higher to recent species and recording conditions. Read the complete write-up >
Made practical with GPUs: Pseudo-labeling normally requires retraining pipelines multiple times (baseline > pseudo-labeled > improved pseudo-labels). This may take days on a CPU, making iteration impractical. With GPU acceleration (via cuML, XGBoost or CatBoost GPU backends), you may run several pseudo-labeling cycles in hours.
Even after optimizing our models and ensembles, we found two final tweaks that may squeeze out extra performance:
- Train with different random seeds. Changing initialization and training paths, then averaging predictions, often improves performance.
- Retrain on 100% of the information. After finding optimal hyperparameters, fitting your final model on all training data squeezes out extra accuracy.
Why it matters: These steps don’t require recent architectures—just more runs of the models you already trust. Together, they boost robustness and make sure you’re making full use of your data.
In motion: Within the Predicting Optimal Fertilizers challenge, ensembling XGBoost models across 100 different seeds clearly outperformed single-seed training. Retraining on the complete dataset provided one other leaderboard bump. Read the complete write-up >
Note: MAP@3 (Mean Average Precision at 3) measures how often the right label appears within the model’s top three ranked predictions.
Made practical with GPUs: Faster training and inference on GPUs make it feasible to rerun models repeatedly. What might take days on CPU becomes hours on GPU—turning “extra” training into a practical step in every project.
Wrapping up: the Grandmasters’ playbook
This playbook is battle-tested, forged through years of competitions and countless experiments. It’s grounded in two principles—fast experimentation and careful validation—that we apply to each project. With GPU acceleration, these advanced techniques change into practical at scale, making them just as effective for real-world tabular problems as they’re for climbing leaderboards.
If you need to put these ideas into practice, listed here are some resources to start with GPU acceleration within the tools you already use:
