Evidence Pack: Tree-Based Models Versus Deep Learning for Tabular Data

In July 2022, researchers at Inria in Paris downloaded 45 mid-sized tabular datasets and initiated a 20,000-compute-hour hyperparameter search. They were looking for the boundary line of a revolution. Deep learning had already conquered image recognition, natural language processing, and audio generation, establishing neural networks as the default architecture for complex artificial intelligence. But the researchers wanted to know if that dominance extended to the rows and columns of standard spreadsheets—the format that houses the vast majority of the world's corporate, medical, and government data.[2]

The results of that massive compute run confirmed a stubborn reality in data science: on typical tabular data, neural networks still lose to decision trees. Specifically, gradient-boosted decision trees (GBDTs)—algorithms that build sequential ensembles of trees to correct each other's errors—remained the state-of-the-art for medium-sized datasets of around 10,000 samples. The performance gap did not vanish even after the neural networks were subjected to exhaustive hyperparameter tuning.[2]

The Inria study identified three specific inductive biases that explain why tree-based models hold their ground. First, neural networks are highly sensitive to uninformative features; adding useless columns to a dataset degrades their performance significantly, whereas decision trees simply ignore them. Second, neural networks are rotationally invariant, meaning they treat the feature space geometrically. In tabular data, where one column might represent age and another income, rotating the data destroys the independent meaning of those features. Finally, decision trees are naturally adept at learning irregular, non-smooth functions, which are common in tabular datasets.[2]

These structural disadvantages were corroborated by a separate benchmark published by Ravid Shwartz-Ziv and Amitai Armon. They tested several novel deep learning architectures that had claimed to outperform XGBoost, the most widely used tree-boosting system. When applied to a rigorous, independent set of datasets, the deep models failed to consistently beat the XGBoost baseline. Furthermore, the tree-based model required significantly less computational time and hyperparameter tuning to reach its peak accuracy. "Still, our systematic study demonstrates that deep learning is currently not all we need for tabular data, despite the recent significant progress," the authors concluded.[1]

The dominance of XGBoost is not a new phenomenon. Introduced by Tianqi Chen and Carlos Guestrin in March 2016, the system was designed with a sparsity-aware algorithm and cache-aware access patterns that allowed it to scale to billions of examples using minimal computing resources. It quickly became the default tool for competitive data science, winning 17 of the 29 Kaggle challenges published on the platform's blog in 2015. A decade later, its underlying architecture remains the benchmark against which all deep tabular models are measured.[5]

However, framing the landscape as a binary contest obscures the nuance of when each architecture actually excels. In May 2023, a massive empirical study by Duncan McElfresh and colleagues at New York University and Abacus.AI sought to map that exact boundary. They conducted the largest tabular data analysis to date, evaluating 19 different algorithms across 176 datasets, resulting in over 500,000 individual model evaluations.[3]

The McElfresh benchmark revealed that the rivalry is often a distraction. As the authors wrote, "the 'NN vs. GBDT' debate is overemphasized: for a surprisingly high number of datasets, either the performance difference between GBDTs and NNs is negligible, or light hyperparameter tuning on a GBDT is more important than choosing between NNs and GBDTs." But when a clear winner did emerge, the deciding factor was dataset regularity.[3]

Gradient-boosted trees consistently dominated when the data was irregular—characterized by skewed distributions, heavy tails, and high variance. They also maintained a distinct advantage on larger datasets. Conversely, neural networks pulled ahead when the feature distributions were highly regular and smooth. This meta-feature analysis provided the first quantitative guide for practitioners, proving that the shape of the data, rather than the sophistication of the algorithm, dictates the winner.[3]

The most notable exception to the tree-model dominance in the McElfresh study was TabPFN, a prior-data fitted network that operates differently from standard neural architectures. Rather than training on the target dataset via gradient descent, TabPFN is a foundation model pre-trained entirely on synthetic data. It uses in-context learning to make predictions on new datasets in a single forward pass, mimicking the way large language models process text prompts.[3]

Initially, TabPFN was strictly limited to small datasets of up to 3,000 training examples. But the architecture has scaled rapidly. In November 2025, researchers introduced TabPFN-2.5, expanding its capacity to handle up to 50,000 data points and 2,000 features. On small-to-medium classification datasets of under 10,000 samples, the default TabPFN-2.5 achieved a 100 percent win rate against default XGBoost, signaling that foundation models are beginning to conquer the tabular domain from the bottom up.[4]

Despite these advances, the most pragmatic approach for high-stakes tabular prediction often involves abandoning the rivalry entirely. Shwartz-Ziv and Armon demonstrated that an ensemble combining both deep learning models and XGBoost achieved higher accuracy than either method operating alone. Because neural networks and decision trees rely on fundamentally different mathematical representations of the data, they tend to make different types of errors, allowing an ensemble to average out their respective blind spots.[1][6]

The evidence pack makes one thing clear: the spreadsheet is a fundamentally different terrain than a photograph or a paragraph of text. While foundation models are making rapid inroads on small datasets, the jagged, uninformative, and irregular reality of most enterprise data means that gradient-boosted trees are not obsolete. The next verifiable milestone for deep tabular learning will not be another benchmark on clean, 3,000-row datasets, but a neural architecture that can natively ingest a 10-million-row, highly skewed financial database and beat XGBoost without requiring a 20,000-hour hyperparameter search.[6]