Ensemble machine learning methods

Ensemble machine learning methods are created by combining a set of models and then using a weighted vote to classify new data points (Dietterich, 2000).  In order for an ensemble of classifiers to be more effective than their individual elements, they must include diverse and accurate models.

The basic idea is that by leveraging the results of multiple classifiers the deficiencies of a given classifier can be overcame.  For example, neural networks using a gradient decent approach are subject to local optima. They can essentially “get stuck” on what they think is the best answer, when a better value exists. By combining a neural network with a different approach, such as a k-nearest neighbor or Bayesian based approach, that situation can be avoided.  The essential characteristic of an ensemble approach is that the results of the combination of the approaches is better than any given approach in isolation.

Ensemble methods often include bagging, boosting, and random forests.  Bagging (also known as bootstrap aggregation) is used to reduce variance by splitting training data randomly (James, Witten, & Hastie, 2014).  Boosting is similar to bagging, with the inclusion of the assignment of a weight (Ledolter, 2013).

 A random forest is created by creating multiple random samples from a given dataset and creating a decision tree from each (Ahlemeyer-Stubbe & Coleman, 2014).  This combination of trees is the forest (in graph terminology).  The motivation behind the creation of a random forest is that while a decision tree will always produce a result, that result might be weak. 

References

Ahlemeyer-Stubbe, A., & Coleman, S. (2014). A practical guide to data mining for business and industry: John Wiley & Sons.

Dietterich, T. G. (2000). Ensemble methods in machine learning. Paper presented at the International workshop on multiple classifier systems.

James, G., Witten, D., & Hastie, T. (2014). An introduction to statistical learning: With applications in R.

Ledolter, J. (2013). Data mining and business analytics with R: John Wiley & Sons.