Why Neural Networks Still Underperform Gradient Boosting on Tabular Data

Deep learning displaced every previous approach on images, audio and text within a few years. On tables it has been trying for a decade and gradient-boosted trees are still the default. The reason is not effort, tuning budget or academic fashion; it is that the assumptions baked into a neural network are the wrong assumptions for this data format, in three ways that have been isolated and named.

What is actually being claimed

Be precise about the claim, because the loose version of it is false. Nobody serious argues that a neural network cannot fit tabular data; a sufficiently wide network is a universal approximator and can fit anything. The claim is about sample efficiency at the dataset sizes real tabular problems come in — thousands to low millions of rows, not the billions that vision and language pretraining assume. At that scale the model’s inductive bias, meaning which hypotheses it reaches for first, dominates its expressive capacity.

The systematic version of this argument is Léo Grinsztajn, Edouard Oyallon and Gaël Varoquaux’s “Why do tree-based models still outperform deep learning on tabular data?”, published in the NeurIPS 2022 Datasets and Benchmarks track. Rather than reporting that trees won, they ran ablations that break the typical tabular dataset in specific ways and watched which model family degraded. Three properties came out of that, and each is a separate mechanism.

Target functions on tables are not smooth

A decision tree partitions the input space with axis-aligned thresholds and predicts a constant inside each region. The function it represents is a step function: piecewise constant, discontinuous at every split. A neural network with smooth activations represents a smooth function, and gradient descent on it is biased toward the smoothest function that fits — a bias that is enormously helpful when the truth is smooth.

On a table, the truth frequently is not. “Approved if credit score is at least 680” is a genuine discontinuity in the data-generating process, not an artefact. So is a tax bracket, a shipping band, a regulatory threshold, an SLA tier. These are not approximations of an underlying smooth curve; the curve was created by somebody writing a rule with a number in it. Grinsztajn et al. tested this directly by smoothing the target function on real datasets and found that smoothing closed the gap — tree performance fell toward the networks’, which is the signature of the mechanism being the right one. An axis-aligned step function is precisely what a tree represents natively and what a smooth network has to spend capacity approximating.

The asymmetry compounds. To approximate one sharp threshold, a network must place a steep sigmoid exactly there, which means learning large weights, which is what weight decay and standard initialisation are designed to discourage. The tree gets it in one split.

Uninformative features hurt networks more

The second finding is about the columns that do not matter. A real tabular dataset is a warehouse export: it has an internal record id, a field that was deprecated in 2019, three near-duplicate address columns and a flag nobody remembers setting. Images do not have this problem, because every pixel carries some signal and the neighbourhood structure is uniform.

Grinsztajn et al. removed uninformative features and added synthetic ones, and the two model families responded asymmetrically: adding uninformative features degraded the networks substantially more than the trees. The mechanism is that a tree’s splitting criterion is a feature-selection step performed at every node — a column that never improves the criterion is simply never chosen, and costs nothing beyond the search. A dense layer has a weight on every input from initialisation onward, so noise columns contribute gradient noise throughout training and must be actively suppressed rather than passively ignored.

This is the finding with the most direct practical consequence, and it is why feature selection is more valuable in front of a network than in front of a boosted tree.

Rotation invariance is the wrong prior

The third mechanism is the subtlest and the most interesting. An MLP trained on rotated data learns essentially the same function — it is close to rotationally invariant, in the sense that rotating the input space and retraining costs little. Tree-based models are not: their splits are axis-aligned, so a rotation destroys them.

Rotational invariance sounds like a virtue and on images it is close to one. On a table it is a defect, because the axes are not arbitrary directions in a vector space. Column 7 is age in years. It means something on its own, it was recorded by a specific process, and a linear combination of age and postcode is not a quantity anybody measured. A model whose bias says “the coordinate system is meaningless” has thrown away the single strongest piece of prior knowledge the format offers: that each column is individually meaningful. Grinsztajn et al. demonstrate this by rotating the features and observing that tree performance collapses while MLP performance is largely unchanged — the trees were exploiting something real that the network was structurally ignoring.

The gap is narrower than the folklore

Two things have to be said against the strong version of this argument, and a page that omits them is selling a conclusion.

First, the size of the gap is contested. Duncan McElfresh and co-authors’ “When Do Neural Nets Outperform Boosted Trees on Tabular Data?”, also a NeurIPS Datasets and Benchmarks paper, ran 19 algorithms over 176 datasets and concluded that the debate is overemphasised: on a large fraction of datasets the difference between the two families is negligible, and the hyperparameter budget frequently matters more than the algorithm. Their abstract does single out the case where the gap is real — gradient-boosted trees are much better at handling skewed or heavy-tailed feature distributions and other dataset irregularities — which is the same mechanism as the first section above, arrived at from the other direction.

Second, Ravid Shwartz-Ziv and Amitai Armon’s “Tabular Data: Deep Learning is Not All You Need” makes a point that is about engineering rather than statistics: the tree ensemble needed far less tuning to reach its result. In any setting where the search budget is finite — which is all of them — a model that is good at its defaults beats a model that is better at its optimum you did not find.

What follows in practice

The useful conclusion is not “never use a network on a table”. It is that the three mechanisms tell you when the default is likely to be wrong.

• Start with a boosted tree. It is close to correct at its defaults, tolerates unscaled and skewed columns, ignores noise columns for free, and gives you a baseline in minutes. Everything else is measured against it.
• Reach for a network when the table has structure a tree cannot see. Very high-cardinality categoricals with genuine similarity between levels, free-text or image columns alongside the numbers, or a multi-task setup where a shared representation is the point. That is where learned embeddings pay, as in tabular row embeddings.
• If you use a network, fix the three mismatches explicitly. Select features aggressively, since the network will not do it for you. Scale and de-skew the numeric columns, which a tree does not need — see numeric feature scaling. And budget real tuning time, because the evidence says that is where the network’s remaining disadvantage mostly lives.