Jargonautica
Classical Machine Learning

Random Forest and Gradient Boosting

Also called: Random Forest, Gradient Boosting, XGBoost

Two ways of combining many decision trees into something far better than any one of them. A random forest grows hundreds of trees on different slices of the data and takes a vote. Gradient boosting grows them one after another, each correcting the mistakes of the last. For ordinary business data in rows and columns these still routinely beat neural networks, and it is worth knowing that before anyone reaches for deep learning.

Technical view

Random forests reduce variance through bootstrap aggregation and random feature subsetting at each split, which decorrelates the trees. Boosting reduces bias by fitting each new tree to the residual errors of the current ensemble, with implementations such as XGBoost, LightGBM and CatBoost dominating practical use on tabular data. Boosted models are more accurate but more sensitive to hyperparameters and more prone to overfitting without early stopping. Both report feature importance, which should be read carefully, since correlated features share credit in unpredictable ways.


Part of AI Terminology Index, section: Classical Machine Learning.

How it connects

Random Forest and Gradient Boosting links to 11 other terms. The faint lines are links between those terms. See it in the full graph.

Feature EngineeringFeature EngineeringDecision TreeDecision TreeOverfittingOverfittingEnsembleEnsembleDeep LearningDeep LearningTime Series ForecastingTime Series Forecas…Feature ImportanceFeature ImportanceTime Series Foundation ModelTime Series Foundat…BaselineBaselineHyperparametersHyperparametersLearning to RankLearning to RankRandom Forest and Gradient Boosting