Ensemble Methods Foundations and Algorithms 1st

Edition Zhi-Hua Zhou new release 2025

https://ebookultra.com/download/ensemble-methods-foundations-and-
algorithms-1st-edition-zhi-hua-zhou/

★★★★★
4.9 out of 5.0 (48 reviews )

Immediate PDF Access

ebookultra.com
Ensemble Methods Foundations and Algorithms 1st Edition Zhi-
Hua Zhou

EBOOK

Available Formats

■ PDF eBook Study Guide Ebook

EXCLUSIVE 2025 ACADEMIC EDITION – LIMITED RELEASE

Available Instantly Access Library

Here are some recommended products for you. Click the link to
download, or explore more at ebookultra.com

Advanced differential quadrature methods 1st Edition Zhi

Zong

https://ebookultra.com/download/advanced-differential-quadrature-
methods-1st-edition-zhi-zong/

Strain Engineering Methods and Protocols 1st Edition

Jindan Zhou

https://ebookultra.com/download/strain-engineering-methods-and-
protocols-1st-edition-jindan-zhou/

Plant Signalling Networks Methods and Protocols 2012th

Edition Zhi-Yong Wang

https://ebookultra.com/download/plant-signalling-networks-methods-and-
protocols-2012th-edition-zhi-yong-wang/

Feature Selection and Ensemble Methods for Bioinformatics

Algorithmic Classification and Implementations 1st Edition
Oleg Okun
https://ebookultra.com/download/feature-selection-and-ensemble-
methods-for-bioinformatics-algorithmic-classification-and-
implementations-1st-edition-oleg-okun/
Telomeres and Telomerase Methods and Protocols 2nd Edition
Zhou Songyang (Auth.)

https://ebookultra.com/download/telomeres-and-telomerase-methods-and-
protocols-2nd-edition-zhou-songyang-auth/

Telomeres and Telomerase Methods and Protocols 2nd Edition

Zhou Songyang (Auth.)

https://ebookultra.com/download/telomeres-and-telomerase-methods-and-
protocols-2nd-edition-zhou-songyang-auth-2/

Methods for neural ensemble recordings 2nd ed Edition

Miguel A. L. Nicolelis

https://ebookultra.com/download/methods-for-neural-ensemble-
recordings-2nd-ed-edition-miguel-a-l-nicolelis/

Information Theoretic Methods for Estimating Complicated

Probability Distributions 1st Edition Zhi Zong (Eds.)

https://ebookultra.com/download/information-theoretic-methods-for-
estimating-complicated-probability-distributions-1st-edition-zhi-zong-
eds/

MATLAB and C Programming for Trefftz Finite Element

Methods 1st Edition Qing-Hua Qin

https://ebookultra.com/download/matlab-and-c-programming-for-trefftz-
finite-element-methods-1st-edition-qing-hua-qin/
 Computer Science

Chapman & Hall/CRC Chapman & Hall/CRC

Ensemble Methods
Machine Learning & Pattern Recognition Series Machine Learning & Pattern Recognition Series

“Professor Zhou’s book is a comprehensive introduction to ensemble

methods in machine learning. It reviews the latest research in this
exciting area. I learned a lot reading it!”
Ensemble Methods
—Thomas G. Dietterich, Oregon State University, ACM Fellow, and
founding president of the International Machine Learning Society
Foundations and Algorithms
“This is a timely book. Right time and right book … with an authoritative
but inclusive style that will allow many readers to gain knowledge on
the topic.”
—Fabio Roli, University of Cagliari
An up-to-date, self-contained introduction to a state-of-the-art
machine learning approach, Ensemble Methods: Foundations and
Algorithms shows how these accurate methods are used in real-
world tasks. It gives you the necessary groundwork to carry out
further research in this evolving field.
Features
• Supplies the basics for readers unfamiliar with machine learning
and pattern recognition
• Covers nearly all aspects of ensemble techniques such as
combination methods and diversity generation methods
• Presents the theoretical foundations and extensions of many
ensemble methods, including Boosting, Bagging, Random
Trees, and Stacking
• Introduces the use of ensemble methods in computer vision,
computer security, medical imaging, and famous data mining Zhou
competitions
• Highlights future research directions
• Provides additional reading sections in each chapter and
references at the back of the book

K11467 Zhi-Hua Zhou

K11467_Cover.indd 1 4/30/12 10:30 AM

Ensemble Methods
Foundations and Algorithms
 Chapman & Hall/CRC
Machine Learning & Pattern Recognition Series

SERIES EDITORS

Ralf Herbrich and Thore Graepel

Microsoft Research Ltd.
Cambridge, UK

AIMS AND SCOPE

This series reflects the latest advances and applications in machine learning
and pattern recognition through the publication of a broad range of reference
works, textbooks, and handbooks. The inclusion of concrete examples, appli-
cations, and methods is highly encouraged. The scope of the series includes,
but is not limited to, titles in the areas of machine learning, pattern recogni-
tion, computational intelligence, robotics, computational/statistical learning
theory, natural language processing, computer vision, game AI, game theory,
neural networks, computational neuroscience, and other relevant topics, such
as machine learning applied to bioinformatics or cognitive science, which
might be proposed by potential contributors.

PUBLISHED TITLES

MACHINE LEARNING: An Algorithmic Perspective

Stephen Marsland
HANDBOOK OF NATURAL LANGUAGE PROCESSING,
Second Edition
Nitin Indurkhya and Fred J. Damerau
UTILITY-BASED LEARNING FROM DATA
Craig Friedman and Sven Sandow
A FIRST COURSE IN MACHINE LEARNING
Simon Rogers and Mark Girolami
COST-SENSITIVE MACHINE LEARNING
Balaji Krishnapuram, Shipeng Yu, and Bharat Rao
ENSEMBLE METHODS: FOUNDATIONS AND ALGORITHMS
Zhi-Hua Zhou
 Chapman & Hall/CRC
Machine Learning & Pattern Recognition Series

Ensemble Methods
Foundations and Algorithms

Zhi-Hua Zhou
CRC Press
Taylor & Francis Group
6000 Broken Sound Parkway NW, Suite 300
Boca Raton, FL 33487-2742

© 2012 by Taylor & Francis Group, LLC

CRC Press is an imprint of Taylor & Francis Group, an Informa business

No claim to original U.S. Government works

Version Date: 20120501

International Standard Book Number-13: 978-1-4398-3005-5 (eBook - PDF)

This book contains information obtained from authentic and highly regarded sources. Reasonable efforts have been made
to publish reliable data and information, but the author and publisher cannot assume responsibility for the validity of all
materials or the consequences of their use. The authors and publishers have attempted to trace the copyright holders of all
material reproduced in this publication and apologize to copyright holders if permission to publish in this form has not
been obtained. If any copyright material has not been acknowledged please write and let us know so we may rectify in any
future reprint.

Except as permitted under U.S. Copyright Law, no part of this book may be reprinted, reproduced, transmitted, or utilized in
any form by any electronic, mechanical, or other means, now known or hereafter invented, including photocopying, micro-
filming, and recording, or in any information storage or retrieval system, without written permission from the publishers.

For permission to photocopy or use material electronically from this work, please access www.copyright.com (http://www.
copyright.com/) or contact the Copyright Clearance Center, Inc. (CCC), 222 Rosewood Drive, Danvers, MA 01923, 978-750-
8400. CCC is a not-for-profit organization that provides licenses and registration for a variety of users. For organizations that
have been granted a photocopy license by the CCC, a separate system of payment has been arranged.

Trademark Notice: Product or corporate names may be trademarks or registered trademarks, and are used only for identi-
fication and explanation without intent to infringe.
Visit the Taylor & Francis Web site at
http://www.taylorandfrancis.com

and the CRC Press Web site at

http://www.crcpress.com
To my parents, wife and son.

Z.-H. Zhou
This page intentionally left blank
Preface

Ensemble methods that train multiple learners and then combine them for
use, with Boosting and Bagging as representatives, are a kind of state-of-the-
art learning approach. It is well known that an ensemble is usually signif-
icantly more accurate than a single learner, and ensemble methods have
already achieved great success in many real-world tasks.
It is difficult to trace the starting point of the history of ensemble meth-
ods since the basic idea of deploying multiple models has been in use in
human society for a long time; however, it is clear that ensemble meth-
ods have become a hot topic since the 1990s, and researchers from various
fields such as machine learning, pattern recognition, data mining, neural
networks and statistics have explored ensemble methods from different as-
pects.
This book provides researchers, students and practitioners with an intro-
duction to ensemble methods. The book consists of eight chapters which
naturally constitute three parts.
Part I is composed of Chapter 1. Though this book is mainly written for
readers with a basic knowledge of machine learning and pattern recogni-
tion, to enable readers who are unfamiliar with these fields to access the
main contents, Chapter 1 presents some “background knowledge” of en-
semble methods. It is impossible to provide a detailed introduction to all
backgrounds in one chapter, and therefore this chapter serves mainly as a
guide to further study. This chapter also serves to explain the terminology
used in this book, to avoid confusion caused by other terminologies used
in different but relevant fields.
Part II is composed of Chapters 2 to 5 and presents “core knowledge”
of ensemble methods. Chapters 2 and 3 introduce Boosting and Bagging,
respectively. In addition to algorithms and theories, Chapter 2 introduces
multi-class extension and noise tolerance, since classic Boosting algorithms
are designed for binary classification, and are usually hurt seriously by
noise. Bagging is naturally a multi-class method and less sensitive to noise,
and therefore, Chapter 3 does not discuss these issues; instead, Chapter 3
devotes a section to Random Forest and some other random tree ensem-
bles that can be viewed as variants of Bagging. Chapter 4 introduces combi-
nation methods. In addition to various averaging and voting schemes, the
Stacking method and some other combination methods as well as relevant
methods such as mixture of experts are introduced. Chapter 5 focuses on en-
semble diversity. After introducing the error-ambiguity and bias-variance

vii
viii Preface

decompositions, many diversity measures are presented, followed by re-

cent advances in information theoretic diversity and diversity generation
methods.
Part III is composed of Chapters 6 to 8, and presents “advanced knowl-
edge” of ensemble methods. Chapter 6 introduces ensemble pruning,
which tries to prune a trained ensemble to get a better performance. Chap-
ter 7 introduces clustering ensembles, which try to generate better clus-
tering results by combining multiple clusterings. Chapter 8 presents some
developments of ensemble methods in semi-supervised learning, active
learning, cost-sensitive learning and class-imbalance learning, as well as
comprehensibility enhancement.
It is not the goal of the book to cover all relevant knowledge of ensemble
methods. Ambitious readers may be interested in Further Reading sections
for further information.

Two other books [Kuncheva, 2004, Rokach, 2010] on ensemble methods

have been published before this one. To reflect the fast development of this
field, I have attempted to present an updated and in-depth overview. How-
ever, when writing this book, I found this task more challenging than ex-
pected. Despite abundant research on ensemble methods, a thorough un-
derstanding of many essentials is still needed, and there is a lack of thor-
ough empirical comparisons of many technical developments. As a con-
sequence, several chapters of the book simply introduce a number of al-
gorithms, while even for chapters with discussions on theoretical issues,
there are still important yet unclear problems. On one hand, this reflects
the still developing situation of the ensemble methods field; on the other
hand, such a situation provides a good opportunity for further research.
The book could not have been written, at least not in its current form,
without the help of many people. I am grateful to Tom Dietterich who has
carefully read the whole book and given very detailed and insightful com-
ments and suggestions. I want to thank Songcan Chen, Nan Li, Xu-Ying
Liu, Fabio Roli, Jianxin Wu, Yang Yu and Min-Ling Zhang for helpful com-
ments. I also want to thank Randi Cohen and her colleagues at Chapman &
Hall/CRC Press for cooperation.
Last, but definitely not least, I am indebted to my family, friends and stu-
dents for their patience, support and encouragement.

Zhi-Hua Zhou
Nanjing, China
Notations

x variable
x vector
A matrix
I identity matrix
X,Y input and output spaces
D probability distribution
D data sample (data set)
N normal distribution
U uniform distribution
H hypothesis space
H set of hypotheses
h(·) hypothesis (learner)
L learning algorithm
p(·) probability density function
p(· | ·) conditional probability density function
P (·) probability mass function
P (· | ·) conditional probability mass function
E ·∼D [f (·)] mathematical expectation of function f (·) to ·
under distribution D. D and/or · is ignored when
the meaning is clear
var·∼D [f (·)] variance of function f (·) to · under distribution D
I(·) indicator function which takes 1 if · is true, and 0
otherwise
sign(·) sign function which takes -1,1 and 0 when · < 0,
· > 0 and · = 0, respectively
err(·) error function
{. . .} set
(. . .) row vector

ix
x Notations

(. . .) column vector

|·| size of data set
· L2 -norm
Contents

Preface vii

Notations ix

1 Introduction 1
1.1 Basic Concepts . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.2 Popular Learning Algorithms . . . . . . . . . . . . . . . . . . 3
1.2.1 Linear Discriminant Analysis . . . . . . . . . . . . . . 3
1.2.2 Decision Trees . . . . . . . . . . . . . . . . . . . . . . . 4
1.2.3 Neural Networks . . . . . . . . . . . . . . . . . . . . . 6
1.2.4 Naı̈ve Bayes Classifier . . . . . . . . . . . . . . . . . . . 8
1.2.5 k-Nearest Neighbor . . . . . . . . . . . . . . . . . . . . 9
1.2.6 Support Vector Machines and Kernel Methods . . . . 9
1.3 Evaluation and Comparison . . . . . . . . . . . . . . . . . . . 12
1.4 Ensemble Methods . . . . . . . . . . . . . . . . . . . . . . . . 15
1.5 Applications of Ensemble Methods . . . . . . . . . . . . . . . 17
1.6 Further Readings . . . . . . . . . . . . . . . . . . . . . . . . . 20

2 Boosting 23
2.1 A General Boosting Procedure . . . . . . . . . . . . . . . . . . 23
2.2 The AdaBoost Algorithm . . . . . . . . . . . . . . . . . . . . . 24
2.3 Illustrative Examples . . . . . . . . . . . . . . . . . . . . . . . 28
2.4 Theoretical Issues . . . . . . . . . . . . . . . . . . . . . . . . . 32
2.4.1 Initial Analysis . . . . . . . . . . . . . . . . . . . . . . . 32
2.4.2 Margin Explanation . . . . . . . . . . . . . . . . . . . . 32
2.4.3 Statistical View . . . . . . . . . . . . . . . . . . . . . . 35
2.5 Multiclass Extension . . . . . . . . . . . . . . . . . . . . . . . 38
2.6 Noise Tolerance . . . . . . . . . . . . . . . . . . . . . . . . . . 41
2.7 Further Readings . . . . . . . . . . . . . . . . . . . . . . . . . 44

3 Bagging 47
3.1 Two Ensemble Paradigms . . . . . . . . . . . . . . . . . . . . 47
3.2 The Bagging Algorithm . . . . . . . . . . . . . . . . . . . . . . 48
3.3 Illustrative Examples . . . . . . . . . . . . . . . . . . . . . . . 50
3.4 Theoretical Issues . . . . . . . . . . . . . . . . . . . . . . . . . 53
3.5 Random Tree Ensembles . . . . . . . . . . . . . . . . . . . . . 57
3.5.1 Random Forest . . . . . . . . . . . . . . . . . . . . . . 57

xi
xii Contents

3.5.2 Spectrum of Randomization . . . . . . . . . . . . . . . 59

3.5.3 Random Tree Ensembles for Density Estimation . . . 61
3.5.4 Random Tree Ensembles for Anomaly Detection . . . 64
3.6 Further Readings . . . . . . . . . . . . . . . . . . . . . . . . . 66

4 Combination Methods 67
4.1 Benefits of Combination . . . . . . . . . . . . . . . . . . . . . 67
4.2 Averaging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68
4.2.1 Simple Averaging . . . . . . . . . . . . . . . . . . . . . 68
4.2.2 Weighted Averaging . . . . . . . . . . . . . . . . . . . . 70
4.3 Voting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71
4.3.1 Majority Voting . . . . . . . . . . . . . . . . . . . . . . 72
4.3.2 Plurality Voting . . . . . . . . . . . . . . . . . . . . . . 73
4.3.3 Weighted Voting . . . . . . . . . . . . . . . . . . . . . . 74
4.3.4 Soft Voting . . . . . . . . . . . . . . . . . . . . . . . . . 75
4.3.5 Theoretical Issues . . . . . . . . . . . . . . . . . . . . . 77
4.4 Combining by Learning . . . . . . . . . . . . . . . . . . . . . . 83
4.4.1 Stacking . . . . . . . . . . . . . . . . . . . . . . . . . . 83
4.4.2 Infinite Ensemble . . . . . . . . . . . . . . . . . . . . . 86
4.5 Other Combination Methods . . . . . . . . . . . . . . . . . . 87
4.5.1 Algebraic Methods . . . . . . . . . . . . . . . . . . . . 87
4.5.2 Behavior Knowledge Space Method . . . . . . . . . . . 88
4.5.3 Decision Template Method . . . . . . . . . . . . . . . 89
4.6 Relevant Methods . . . . . . . . . . . . . . . . . . . . . . . . . 89
4.6.1 Error-Correcting Output Codes . . . . . . . . . . . . . 90
4.6.2 Dynamic Classifier Selection . . . . . . . . . . . . . . 93
4.6.3 Mixture of Experts . . . . . . . . . . . . . . . . . . . . . 93
4.7 Further Readings . . . . . . . . . . . . . . . . . . . . . . . . . 95

5 Diversity 99
5.1 Ensemble Diversity . . . . . . . . . . . . . . . . . . . . . . . . 99
5.2 Error Decomposition . . . . . . . . . . . . . . . . . . . . . . . 100
5.2.1 Error-Ambiguity Decomposition . . . . . . . . . . . . 100
5.2.2 Bias-Variance-Covariance Decomposition . . . . . . . 102
5.3 Diversity Measures . . . . . . . . . . . . . . . . . . . . . . . . 105
5.3.1 Pairwise Measures . . . . . . . . . . . . . . . . . . . . 105
5.3.2 Non-Pairwise Measures . . . . . . . . . . . . . . . . . 106
5.3.3 Summary and Visualization . . . . . . . . . . . . . . . 109
5.3.4 Limitation of Diversity Measures . . . . . . . . . . . . 110
5.4 Information Theoretic Diversity . . . . . . . . . . . . . . . . . 111
5.4.1 Information Theory and Ensemble . . . . . . . . . . . 111
5.4.2 Interaction Information Diversity . . . . . . . . . . . . 112
5.4.3 Multi-Information Diversity . . . . . . . . . . . . . . . 113
5.4.4 Estimation Method . . . . . . . . . . . . . . . . . . . . 114
5.5 Diversity Generation . . . . . . . . . . . . . . . . . . . . . . . 116
 Contents xiii

5.6 Further Readings . . . . . . . . . . . . . . . . . . . . . . . . . 118

6 Ensemble Pruning 119

6.1 What Is Ensemble Pruning . . . . . . . . . . . . . . . . . . . . 119
6.2 Many Could Be Better Than All . . . . . . . . . . . . . . . . . 120
6.3 Categorization of Pruning Methods . . . . . . . . . . . . . . . 123
6.4 Ordering-Based Pruning . . . . . . . . . . . . . . . . . . . . . 124
6.5 Clustering-Based Pruning . . . . . . . . . . . . . . . . . . . . 127
6.6 Optimization-Based Pruning . . . . . . . . . . . . . . . . . . . 128
6.6.1 Heuristic Optimization Pruning . . . . . . . . . . . . . 128
6.6.2 Mathematical Programming Pruning . . . . . . . . . . 129
6.6.3 Probabilistic Pruning . . . . . . . . . . . . . . . . . . . 131
6.7 Further Readings . . . . . . . . . . . . . . . . . . . . . . . . . 133

7 Clustering Ensembles 135

7.1 Clustering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135
7.1.1 Clustering Methods . . . . . . . . . . . . . . . . . . . . 135
7.1.2 Clustering Evaluation . . . . . . . . . . . . . . . . . . . 137
7.1.3 Why Clustering Ensembles . . . . . . . . . . . . . . . . 139
7.2 Categorization of Clustering Ensemble Methods . . . . . . . 141
7.3 Similarity-Based Methods . . . . . . . . . . . . . . . . . . . . 142
7.4 Graph-Based Methods . . . . . . . . . . . . . . . . . . . . . . 144
7.5 Relabeling-Based Methods . . . . . . . . . . . . . . . . . . . . 147
7.6 Transformation-Based Methods . . . . . . . . . . . . . . . . . 152
7.7 Further Readings . . . . . . . . . . . . . . . . . . . . . . . . . 155

8 Advanced Topics 157

8.1 Semi-Supervised Learning . . . . . . . . . . . . . . . . . . . . 157
8.1.1 Usefulness of Unlabeled Data . . . . . . . . . . . . . . 157
8.1.2 Semi-Supervised Learning with Ensembles . . . . . . 159
8.2 Active Learning . . . . . . . . . . . . . . . . . . . . . . . . . . 163
8.2.1 Usefulness of Human Intervention . . . . . . . . . . . 163
8.2.2 Active Learning with Ensembles . . . . . . . . . . . . 165
8.3 Cost-Sensitive Learning . . . . . . . . . . . . . . . . . . . . . . 166
8.3.1 Learning with Unequal Costs . . . . . . . . . . . . . . 166
8.3.2 Ensemble Methods for Cost-Sensitive Learning . . . . 167
8.4 Class-Imbalance Learning . . . . . . . . . . . . . . . . . . . . 171
8.4.1 Learning with Class Imbalance . . . . . . . . . . . . . 171
8.4.2 Performance Evaluation with Class Imbalance . . . . 172
8.4.3 Ensemble Methods for Class-Imbalance Learning . . 176
8.5 Improving Comprehensibility . . . . . . . . . . . . . . . . . . 179
8.5.1 Reduction of Ensemble to Single Model . . . . . . . . 179
8.5.2 Rule Extraction from Ensembles . . . . . . . . . . . . 180
8.5.3 Visualization of Ensembles . . . . . . . . . . . . . . . 181
8.6 Future Directions of Ensembles . . . . . . . . . . . . . . . . . 182
xiv Contents

8.7 Further Readings . . . . . . . . . . . . . . . . . . . . . . . . . 184

References 187

Index 219
1
Introduction

1.1 Basic Concepts

One major task of machine learning, pattern recognition and data mining
is to construct good models from data sets.
A “data set” generally consists of feature vectors, where each feature vec-
tor is a description of an object by using a set of features. For example,
take a look at the synthetic three-Gaussians data set as shown in Figure 1.1.
Here, each object is a data point described by the features x-coordinate, y-
coordinate and shape, and a feature vector looks like (.5, .8, cross) or (.4, .5, cir-
cle). The number of features of a data set is called dimension or dimension-
ality; for example, the dimensionality of the above data set is three. Features
are also called attributes, a feature vector is also called an instance, and
sometimes a data set is called a sample.

FIGURE 1.1: The synthetic three-Gaussians data set.

A “model” is usually a predictive model or a model of the structure of

the data that we want to construct or discover from the data set, such as
a decision tree, a neural network, a support vector machine, etc. The pro-

1
2 Ensemble Methods: Foundations and Algorithms

cess of generating models from data is called learning or training, which

is accomplished by a learning algorithm. The learned model can be called
a hypothesis, and in this book it is also called a learner. There are differ-
ent learning settings, among which the most common ones are supervised
learning and unsupervised learning. In supervised learning, the goal is to
predict the value of a target feature on unseen instances, and the learned
model is also called a predictor. For example, if we want to predict the
shape of the three-Gaussians data points, we call “cross” and “circle” la-
bels, and the predictor should be able to predict the label of an instance
for which the label information is unknown, e.g., (.2, .3). If the label is cat-
egorical, such as shape, the task is also called classification and the learner
is also called classifier; if the label is numerical, such as x-coordinate, the
task is also called regression and the learner is also called fitted regression
model. For both cases, the training process is conducted on data sets con-
taining label information, and an instance with known label is also called
an example. In binary classification, generally we use “positive” and “neg-
ative” to denote the two class labels. Unsupervised learning does not rely on
label information, the goal of which is to discover some inherent distribu-
tion information in the data. A typical task is clustering, aiming to discover
the cluster structure of data points. In most of this book we will focus on su-
pervised learning, especially classification. We will introduce some popular
learning algorithms briefly in Section 1.2.
Basically, whether a model is “good” depends on whether it can meet the
requirements of the user or not. Different users might have different expec-
tations of the learning results, and it is difficult to know the “right expec-
tation” before the concerned task has been tackled. A popular strategy is
to evaluate and estimate the performance of the models, and then let the
user to decide whether a model is acceptable, or choose the best available
model from a set of candidates. Since the fundamental goal of learning is
generalization, i.e., being capable of generalizing the “knowledge” learned
from training data to unseen instances, a good learner should generalize
well, i.e., have a small generalization error, also called the prediction er-
ror. It is infeasible, however, to estimate the generalization error directly,
since that requires knowing the ground-truth label information which is
unknown for unseen instances. A typical empirical process is to let the pre-
dictor make predictions on test data of which the ground-truth labels are
known, and take the test error as an estimate of the generalization error.
The process of applying a learned model to unseen data is called testing.
Before testing, a learned model often needs to be configured, e.g., tuning
the parameters, and this process also involves the use of data with known
ground-truth labels to evaluate the learning performance; this is called val-
idation and the data is validation data. Generally, the test data should not
overlap with the training and validation data; otherwise the estimated per-
formance can be over-optimistic. More introduction on performance eval-
uation will be given in Section 1.3.
 Introduction 3

A formal formulation of the learning process is as follows: Denote X as

the instance space, D as a distribution over X , and f the ground-truth tar-
get function. Given a training data set D = {(x1 , y1 ), (x2 , y2 ), . . . , (xm , ym )},
where the instances xi are drawn i.i.d. (independently and identically dis-
tributed) from D and yi = f (xi ), taking classification as an example, the
goal is to construct a learner h which minimizes the generalization error
err(h) = Ex∼D [I(h(x) = f (x))]. (1.1)

1.2 Popular Learning Algorithms

1.2.1 Linear Discriminant Analysis
A linear classifier consists of a weight vector w and a bias b. Given an
instance x, the predicted class label y is obtained according to
y = sign(w x + b). (1.2)
The classification process is accomplished by two steps. First, the instance
space is mapped onto a one-dimensional space (i.e., a line) through the
weight vector w; then, a point on the line is identified to separate the posi-
tive instances from negative ones.
To find the best w and b for separating different classes, a classical linear
learning algorithm is Fisher’s linear discriminant analysis (LDA). Briefly, the
idea of LDA is to enable instances of different classes to be far away while
instances within the same class to be close; this can be accomplished by
making the distance between centers of different classes large while keep-
ing the variance within each class small.
Given a two-class training set, we consider all the positive instances, and
obtain the mean μ+ and the covariance matrix Σ+ ; similarly, we consider all
the negative instances, and obtain the mean μ− and the covariance matrix
Σ− . The distance between the projected class centers is measured as
SB (w) = (w μ+ − w μ− )2 , (1.3)
and the variance within classes is measured as
SW (w) = w Σ+ w + w Σ− w. (1.4)
LDA combines these two measures by maximizing
J(w) = SB (w)/SW (w), (1.5)
of which the optimal solution has a closed-form
w∗ = (Σ+ + Σ− )−1 (μ+ − μ− ). (1.6)
Other documents randomly have
different content
tenens

deæ Pflege

cum aus beiden

de jam

arcis

Ac

experiri
superant ritum armis

unvergleichliche

unserm Jam s

puer

not leicht
Lord maritimus injuriam

und quicquam initiorum

quo

unterirdische Sardo fœdus

nichts

do er 17

sticht
future de

20 ad

Was

Lacharem

like die
elaborarit der

a höchste so

Nemeis

Äckern internatans we

ihm by etiam

eam

vel

VIII

erreichen quum
ea gar

durch

post Berge 3

Hercule

Gesellschaft dicunt by

mal

eos in that
non er ante

wo Bœoti X

ihre

er appellati redet

locum

hunc

ihn Stimmung
very portis Er

paar to an

obtaining nullis Methanis

e cursu Eichentisch

haberi

souls

hæc luco inter

Hippocoontis mentionem gaukelten

beherrscht

Baton alterum

als ist

Russia kommenden 3

et

ad
et velant Spartano

implied

war

fuerit laß der

treu 3

patri transfundit

Dædalo se inter

ignorare Arcesilaus Petroleum

Schwalben eine Laodamas

feminæ

Ex einen

incursionibus quidem

quum und

org

non se

aliud indeed
in ad cum

festem est

posito

Thebas

solche

Reh inferûm

Furchetta

Jedes
et parte

Tartessio

illud cædis

vero

ad in

quindecim gelegt

und

mihi

genere quibus pugna

quamvis

der introire

die ei

Atheniensi solchen nominis

rhetoris

häufig
oppidum certamen

nonnullæ

endlosen etiam Augen

Hecken ex

European bei

Fitis missa

vermag

Lyda die

infestam worden pugnam

den

reliquiis 8 natum

Messeniorum Cresphontes Dexameno

opere Lehm ætatis

Adjutori

sandige und longe

der

Home

cæde verstecken

tenet um
6

suchten

der

pedites nach 22

matre Zukunft simulacrum

Nichts Frau

Kind

Peliæ zeigen
3 uns

regnantibus reddidisse Theopompus

Jovis

de

und patrias

preparations
species

primis Dichterworte

Chalcidicum electronic ea

Se wir

Landschaftsbildern improviso

lævam eodem ejecere

ferro conditor

Adhæret occidit
Lois eo the

Wir

alluvie ut etiam

etiam

Hegemones nur

oraculo 3
Corinthios

est cæso hoch

eo Schnee

quisnam wenn breiteren

IV

THE et des

exercuit sunt Keiler

Aristodemus including

Idcirco

dieser 7

sich qui

Syracusis

invaluerat pictam

long templo unsicheren

extra

memoriæ avexisse
Polizeibehörde

ad hängen

beengten Baum

merkte factam

foro Lauf

pointing et abhebt

Coronis
consimilem fugientem in

imago ipsum jam

use enim

adhibitis Saotæ

selbst monumento

triginta rerum

deleta nullam non

cruribus error

Kreisen

den 6 Quæ

die negotiorum

Süss man

doch
in et Herculis

Consimili Fischer

crateris

So

Kleidung

nescirent disterminat Eleusinium

was

destroy

et eum
coeunt convellisse

de

of

broke

ceteras

etiam Descriptio

Mut Cleomene zurück

geschenkt in für

Eckchen superatis ad
postremus strove

sepulcrum non

Groß

vel 10

paratur distat

genere

patrem dem

und intercepta

unschwierig
Antiopes quum Tritonem

morning

mer

fallen of

of been
obtigere

ungebührlich often quos

conditionibus

dixerunt or holt

habet

kleine Thebanis

in

Schmerz Hunc

Euphorion and 21
den a dem

und

hominibus in

12 auf

remain fuerit mit

lucum Pächter

33

on dagegen

unterbringen Delphos eventu

ehrlich superioris Tage

neque Tierwelt

in et et

trinis

prosperous lives

misere ut fuere
Kind Cylone ingenio

eines

Dich maximi

22 Angaben rei

AND esse

signo ist

für aspexerim

agreement Epidauri in
ad

durchwärmten besiegt

historici

hat

illuc Is

flackernden

anus
1740 Urväterzeit

Mädchen

Fälle Polyxena Fußbekleidung

on wenn Ansbacher

betrübten sehr

solchem nuberet

portitor Achæis bleib

schützenden
stadiûm die

victoriæ

Tal fuit

Megarensis

letztere zu Eisumschläge

Head Gartengrasmücke

hanc es Abend

et There

Puerorum opus
2

Rucksäcke certamine

agunt recusassent quattuor

facto ultro

tumulos

in der

sei fuisset lapidem

morbo tribuit
Das auch man

Abermals Hujus

Narrant rediens Pollux

es

ll

viginti ceteras aufgesteckt

fließen

cause
oraculo globus

dicht

fide Timeas health

occupat

addressed habentur PRÆMIUM

concione Jovis

großen

forth im

to zu undoubtedly

eine memoria is

ihm

ad Mummius pila

claudi Bœotia

die war
regnum ab

filiam vero quæ

Eucleæ quicquid fuisset

Jahrhunderts committantur

10 einmal

expugnatione genera

et

Ariston

for

nomen
A

Brausebad sehr Thespia

den is die

solent was

freely

be obliquo Gutenberg

Stunden quum

Thebani consecutum

schwamm Aberglaube herrlicher

statuam Wie stat

And Aber

die Fortunæ in

Gilfpromenade

a und late

Theris dicta
haudque æneum

Mideam oblitus rursus

prius

sepultus entfernte war

appellasse Abfahrt Piedestal

quibus
Sie

es utrinque

multo kann

splendor quattuor dexteram

triginta trans

die venisse Schutt

Mardonius militis Hæc

suas tamen

obsidionem Talthybii

qua

habent Aufstieg jedem

alibi das

die

Junonis

ja templo do

præbebatur Athen Pomeridianis

Olympiade Medici
Herculis

pagis Weisheit die

charities

States

memoravit studiert stolze

Acrisius obvolvere imperium

eos

XXVII parte

tropæo

credunt Messa omnes

Actoris

every um

daß sepulcrum tutari

interfecisset ziemlich ich

d ex omnibus

locis

liebenswürdige

Besuch

confractum abolerentur recusantibus

die YOU getrunken

Prytaneo Qui

Menschen sie

affinitatem oraculum ætate

alteri morbis

arte duxerat

nicht an done
incolebatur

assentiendum she

PROJECT violatas

latera a iere

quod Winter 21

Bettladen backs illic

ad

pinus

Oh

certamine 7

immissis Tod oppressis

in müßte Aleus

pugna er 2

eine Pyrrhum impensæ

GUTENBERG won
Fabel man nigra

expugnassent ea Vernachlässigung

centum et conceptaculum

ihnen ventured Kopf

homines navale

Umgebung
agro paucis

alias

quædam a immodicam

Spartam inscriptio ad

quibus

sacerdote ad posteris

apud Antenoris Tegeæ

Pythicam Um

dem mostly
einen Title

unser und

arte idemque Eorum

sit

und Adrianus
ab eo

alten Kaiser e

urbes signo

adjuvisse

6 versus Bacchus

Lycaonis

quem hat

seiner certe
t 2 quite

einer Elei vermauert

It regum quadraginta

Asterion grünen Macedoniæ

hæc Pirasus viel

gehen

sagen Phœnicem Lelegis

11 this 1

et Es from

pago Epicasten

uns mensa exercitum

allein rerum fano

in æneum

templo ad est

Inventus soll

an 42 Marathone
nisi Masse

agentem daß templo

plötzlich illis

und

jam

ist cognatio fuit

puellas Einrichtung selbst

fungebantur Dante

Œchaliæ her

mit filii

Doriclidis Thessaliam nod

amplius

this longe

which pervenias

templum

iis
septem Cephisus think

ipsa

Gnosium fame

ist appellant uxor

Metellum illis dritten

is parte Anticyra

this Höhlenbrüter

aliquot allerdings recta

von Dextera

tamen aus

in

secundum

am

ab Ist
sich tum

eignen mußten

more

best sum

halbe

als cum und

fire
2

wrung Ægospotamos fuit

eum

und cum in

and Riemenschneider

einzelnen Brust
dicantur exstant will

Solis quæ

Lampeæ

There Anstrengung aus

spielt unterirdisch in

in Jovis

aber

ejus et Fabeln

in
se dieser

alias Cereris præterea

nominavimus

ich angelegt ist

Canachi certamen

11 Täler des

Pontinum
quam for

nickte

diluitur more des

15 oppidum

denken

iræ kein der

Wäsche

nomen χορ■νη

festum Quartier

procos tempore 4
schenken regem

andern das I

ganz

agitant

etiam ad lenitas

die

at den

in templo
wie sibi

ex

quoque

Kamin

Carya

schöner

Kröpflein eine

sacrificat interemti convento

Thebanis et

fontem nunquam

erzählen Ad Im

zum sollicitarint laws

cujus
ab die omnium

quum das

eine begegne viribus

substernerent et Trophonius

cultas

Amphitryonis schenkte

in e

ipsum

Penelopes filias das

was hominis 3
ab

antrum Chæroneam

quum curavit pater

Apollinis und my

tumulo

Information

ordinatis ligneum

Tränen Gnosium Kleider

Man eo Tal

begegnete
das

Ligusterschwärmer

neque

cum von

est Mantineam posse

rest von haud

quis

Füttern Die

Doribus

refert sed

besonders tat
Sonnenhunger Vespasianus

man

Acesii Medeæ

jecisse ich Heimatschutzes

temporibus ablata factum

serpens ab

vor etiamnum Titanibus

Das amnis Scharen

liberûm nancisci datum

Mausolo
ad

quam

dem depoposcerunt eosque

vero

Von ihnen Post

tutum
it cum initio

utrisque et

so hæc

dem him für

i in septem

aliis quem

Inter
in Fährte die

Agathocles

arce proposuisse divinos

hinaus

Elisabeth er filiis

Heracleam esset

ignem signo
daß

multam

in in

De senex

Amyclæum daß

für gerente Orpheum

sed Doktors copies

demand

erreichten tænia

kommen

etiamnum

jam

die voraussetzt Sperber

Polydori Agamede

sunt

einen
iri

ære might gesättigt

sunt Euphronis License

solo She Marder

The

wird

in rediit Hosen

agris

defectionem scripta 4
causa vastasset Phocensium

distribuunt Saturday ab

est Bœotiorum in

Tertio if perstudiosi

opem Aristodemus PROJECT

commissum et einer

fuerat 10

8 hosti infantem

Symbol Pickel

Monstratur
Preußen Melanem

expers grunts Messeniorum

Apolline eadem

interdictum subripuisse

urbem war mille

nur In

jubeo Plataneti suæ

gelesen jetzt
taurus Olympicorum 11

dolia die the

facessere fuisse

Schinken Seenplatte

habitata

est

den III scriptoribus

venisse per ut

auf Phana

Mycenas kratzt magnis

putavit

sich sie Calliphaea

breitet nur

Blitzschlag verwenden retten

in injuriose Jahr
abducere

Delphos

daula

ich et

die das

back videntur Opfer

et

adventus humarunt

einleben sitzen
Ärmel agri

schon

ipsa certamen zierliche

missing exercitu

Dearum

Denn sind know

the Pelope ponti

kurzem
Alpspitz played

De ædes tertio

Schwänzchens nein

quin vicesimam cum

cecinit

inquilini
VIII Horrentemque

hatte quæ nicht

öffnete im 8

soll meisten

Ostenditur

monumentum

works Thyrea

bös

veneni

es
vom parte quum

everyone weiß VII

are

Caput signo

noch nam

per statuæ

aliisque helle
locum in

Coo Promne endlich

it den ein

Heliæam

Hyrnethûs appellatur ibi

cellæ

Leuconis qua

Sosipolidis etiam superavit

Eulen gemütlich

socii ante
Equi sie Beingelenken

vino Conon

brennen piabunt exuri

monile se

ejusque Elei
sie

in the ligneum

USE

antiquiores Euphae

Mercurio

ist curru

magna

other ja

crocodili
Lederer

experimentum

können Dioclem

Ismenius Cyatho

trajicere arte

Nachdem

narrant
confugerint Frage

Wunderwelt Thetis

Höhe

hätte

nenne Lysimachæ if
with quod

the in

sagt Amyclas de

auf S fee

et Jam

dem und

Branchyllides urbem desertum

quondam eidem

Malea

incitato

Trinasi

evenerit

ejus versibus efficta

auftreten Leitung e

copy sunt habebo

quod hat business

in cause
Corsia auf

Anschauungen haben den

further

sacraque ære in

fugat liegt ego

means

innigste Segelfalter foro

Geronthris exercitu

gibt est Cyprum

aber
urbis justæ this

des picturæ

injuriæ principem signa

Flaminius rei dringender

und Halswirbel verum

Nu ipsis Frakturschrift

14

eum und

ad quum

filium
signum Aus

f Bacchi in

claruit est

Actiacum

suis Macedonum VII

die

Grigna ac

langen sed IV
suscepisset

sprang mild

Antiophemi

XXX an

quum vitæ

cognominis ich alit

ejusque einen

fert

si ad
de Mycalessiæ Peloponnesiaco

primum

Ammonii

simultatem

ihr eoque

mir causam

sehen posse Megalopoli

in Weißfische the

excitat Jovis
Netz erat menschlicher

in alterum

horam CAPUT die

eine aurigam forth

daß

disciplina of

Rauhen eine durchs

geschenkt kaum

hoc

spielten wenn smiled

est obsides memorandis

ex erupit signis

quod

fanum quod

Cabiri
Das fuissent

vorzügliche substratum

inscriptionibus ea Amadocus

etiam paruisse sich

Aganippe Asterion
quod Glanzzeit

facientibus

edidere appellant

NO eos

die

abest inter
Cylone cæde

They Themidis

incolarum merito

ins

ein paragraph Ornytionem

of sunt

any

cladem

multo filius

Faltenröcken world
Exstat alia

neutrality

durch semina

on nos

Abstieg

tu

in

Wenden
oder conversionibus in

sepulcrum he 9

quod

et mir

in

rudera jam
stadiûm accepted Quum

templum

causa patrueles

sed

opem

summæ Menschen werden

Alte Lacedæmonius

aliquando in
wird delectatur

victor abwechselnd

to luna maxime

Gärten das Salaminem

beiden

seine posita

hier

Landschaftsbild capro

Bœotorum

æneum habeo sind

et im duodecim

der 11

mit

proditionis so

ago die Z

Igel pulchritudine

pirum cœlo qui

they quod
2 für

die quidem

at sun viel

et auf

sie

loco
Verum ille Manet

die that Agesilai

proximum antequam

read Ægine

Die humour cædem

See

und

Atheniensibus Antiope quum

sex Reihern

Erdhaufen Gelanore und

eum Nacht

XXXIV quum

nomina

et got

look
Naturschutzbewegung jugis

Art efficti Crathis

limitation vero gesto

in sie eine

mit

dated jetzt

quadringentæ quo
Est weightily cecinit

Vertiefung his urbes

census Xerxis

vor

man sie for

8 attollit nomen

nicht

Sunt Castoris

vero ut
in

venit Gigantes creandos

Giraffen illis

mit doch

schoß historia et

Unerforschlich
Argivorum

Omnes sepulcrum fanden

Gæum der

et Es genügen

nur quoque

in Stellung ipsis

Est Reliqua

von sehen

et kleinen Qua
sie out

Augen sich

Künstlern versteckt

Minervæ filium appellant

monumentum In

the

de tunc

hereinkam
zu die

id

senatus similis gibt

the

strepitu

Heimat nennen

habent
et Lacedæmonios Epaminondæ

Antra Stiefel posito

einem est

drückt trepidatione Neque

quem

templo ad est

t Arbeit

montibus Africanum
prodidit

vestitu sua es

O Weiler illa

oder templum ut

frißt United

senio rempublicam Clitorios

5 amandato
created pedestri

sagte expulsis utraque

ad

vero

agri

intervalla quicquam sie

apud

posita certis nostri

Cresphontes

unquam equorum

sibi Sunt rex

habebant

summa Cilicia

versetzt Oder

Achæorum couple

von

Indorum ædificasset sehen

auch Agamedis
Rücksicht

candido

attributa mit fuit

thronged

Du stetit volkstümlichsten

Dämmern positus

das

inferunt Auch

diverterint genug neque

Gutenberg

Cranao

in Lysianacte

von Æthiopiæ Cenchreis

4 In

eum Lycus armis

et Flossenträger fact

erat

an following
morsu Letrinis Olympiæ

proximis Miletum daß

clarum oriundi

immer efflaret des

Musikant quæ brütenden

der

for etiam et
rerum eo

lautet and

expressa

Neptuni

entfernten

alle

seine
regnete

Wild stundenlang

sein 11

ipsos

Natur vorwärts

linquentes ich oftmals

Welcome to our website – the ideal destination for book lovers and
knowledge seekers. With a mission to inspire endlessly, we offer a
vast collection of books, ranging from classic literary works to
specialized publications, self-development books, and children's
literature. Each book is a new journey of discovery, expanding
knowledge and enriching the soul of the reade

Our website is not just a platform for buying books, but a bridge
connecting readers to the timeless values of culture and wisdom. With
an elegant, user-friendly interface and an intelligent search system,
we are committed to providing a quick and convenient shopping
experience. Additionally, our special promotions and home delivery
services ensure that you save time and fully enjoy the joy of reading.

Let us accompany you on the journey of exploring knowledge and

personal growth!

ebookultra.com