Chapter 1 Introduction
Three important parts: Probability distribution, decision theory and information theory.
From decision theory, loss function is provided; from information theory, entropy and KL is provided. From probability, conditional, joint probability; bayesian formalism and frequentist formalism.
Other topics about high dimension situation: for naive methods, requirement on data size grow exponentially with the number of dimensions and high-dimension is counter-intuitive. But we also have other insight on high dimension data: real data exist in a manifold of high dimension and local smoothness is guaranteed.
For model selection: we have cross validation from frequentist, other method combine model complexity and training performance from bayesian.
Chapter 2 Probability distribution
Focus on the probability distribution related to machine learning. Specially focus on Gaussian distribution. One important thing I learned from this chapter is how to derive the conditional and marginal distribution from a joint Gaussian distribution: Gaussian distribution has two very important components, first is the quadratic term involving precision matrix, second is the mean term; we can find the corresponding distribution by completing this quadratic terms.
Another important stuff about Gaussian distribution is the precision matrix, this is very helpful when derive the conditional and marginal distribution.
Other stuff in chapter 2 about Exponential Family, which is a generalized concepts with density function form where is sufficient statistics, is a normalizing constant depend on parameter . or called Partition Function.
For probability density, despite parametric form there is Nonparametric form. For Nonparametric probability density, we can have Nearest Neighbour Method and Kernel Method two different approaches. But these two ideas all come from one basic principal of estimating probability.
Chapter 3 Linear Regression
This book focus on bayesian approach to every model ( or hypothesis ). For linear regression, there are several different prospect for derivation:
1. MLE: assuming a gaussian distribution of noisy.
2. Geometry point: Projecting target value into the range of columns space of data samples.
For regularization part, assuming a gaussian prior distribution on parameter .
The ultimate purpose of learning model is predicting target value for new input data point , how to expression the uncertainly of predicted value ? Frequentist and Bayesian have different method:
1. Bayesian expression the uncertainly through of posterior distribution of parameter .
2. Frequentist will make a point estimate of at first, then through a series of though experiment to determine the uncertainty.
One important stuff: Bias-Variance Decompositioin is used for Frequentist, because the interpretation of Bias and Variance depend on the following ideas:
We have a set of different data sets, each data set comprised of N data points. From each set, learning algorithm will get an point estimate of , based on this parameter, prediction made on new data point . Since there are multiple data set from some unknown distribution , then can take expectation and variance of all the predicted values on new data point. This is the origin of Bias and Variance. Different model have different bias and variance depend on the model complexity. Thus the control of model complexity is vital for machine learning.
But what is the Bayesian approach to Linear Regress Estimate & Model Selection ?
Start with a prior distribution over parameter (mostly choose gaussian), then updating this distribution when new data point observed.
Frequentist start model selection with cross validation. But Bayesian will do it based on model evidence.
Another question would be the variant based on linear regression?
There are lots of variation of simple linear regression.
1. In original linear regression, original data point is used. But can use Basis Function to transform the data point at first : from -> , can have many basis function, then get the linear representation of data point. Some of the well-known basis function is: gaussian function, wavelet function and sigmoid function.
2. Another extension would be the norm of regularization. From to and . If a purely linear regression and norm regularization, this is called LASSO.
Other stuffs ?
1. Hypothesis complexity of hypothesis for linear regression. Used to derive the generalization bound and sample complexity.
Chapter 4 Linear Classification
For classification, there are three different approaches:
1. Discriminant Function: From training instance to class label directly.
2. Probabilistic Generative Model: model the joint probability of instance and class label.
3. Probabilistic Discriminative Model: model the conditional probability of class label give training instance.
In this chapter, it’s about how to use linear model to realize three different approaches.
For discriminant function, Linear Regression, Finsher Discriminant Analysis and Perceptron algorithm. Linear Regression and Finsher Discriminant function with different objective function. For perceptron, it’s quiet unusual. It’s hard to find a appropriate category for this algorithm.
For probabilistic generative model, it’s modeling as follow:
where represent the class conditional probability distribution. For binary and multi-class classification, if class conditional probability is gaussian and share the same covariance matrix, then the posterior of class label has the following form: . The function is called activation function and function classed link function in statistics. So one question would be this: which specific form of class conditional probability distribution will lead to a linear model? Answer is exponential family distribution with shared scaling parameters.
For discriminative function, it’s modeling the conditional probability directly. Linear discriminative model has the following form:
activation function is sigmoid function for binary classification, soft-max function for multi-class classification. Since nonlinear activation function, there is no closed form solution for this problem, only be solved through iterative approach. **I**terative **R**egularized **L**east **S**quare (IRLS) is apply the newton method to linear discriminative model. Another point need attention: when using 1-of-K coding schema for class label, optimize the negative loglikelihood function of training data is the same as optimize the negative cross entropy function of training data. this is true when binary using 0 and 1 to represent different class. (But in normal case, everyone is using 1 and -1, interesting).
According to the spirit of this book, There is a bayesian version of logistic regression. But the posterior of parameter given training data is intractable. So laplace approximation is used to approximate the posterior distribution.
How does Laplace Approximation works? It approximate the target distribution with a Gaussian. And this gaussian sit on the model, its precision matrix is the negative of the hessian of the target probability density at the mode.
Summary: Approaches to classification, cross-entropy, probit regression, laplace approximation, BIC.
Chapter 5 Neural Network
Most important concepts: Neural Network is adaptive linear model. Or can be understand as hierarchical linear model. Because each layer of neural network is just perform linear model operation plus some nonlinear activation. So neural network is itself nonlinear but composed of linear models. The most important motivation is the the input of linear model can be the output of other linear model. When thinking in this way, it is basis expansion but with adaptive basis. Much more interesting than aspects from neuron inspiration.
When recognized as adaptive linear model, neural network need some objective function: cross-entropy for classification, least square for regression. These concepts are all from the previous chapters.
But adaptive linear model is hard to calculate the gradient and hessian. So the back-propagation comes into help.
Using back propagation, gradient and hessian can be calculated easily.
As a new function mapping different from linear model, it need some ways to perform model selection. The old regularization on all the parameters still works well. However, as a adaptive linear model, itself is hierarchical!
For neural network, some other approaches can be used: consistent prior, tangent propagation, convolution and soft weigh sharing. I think all these techniques are too much complicated, don’t know the real application in real open-source tools.
Neural network is a function space and it can be used to do anything. Mixture density network is using neural network to predict the mixing coefficients, mean parameters and covariance matrix. Too much parameters, i don’t think this is good.
Still, this book is for Bayesian Method. Bayesian neural network for regression and classification. Using lapalace approximation, posterior distribution can be approximated to give predictive distribution and so on. Using so many approximation, what’s the meaning of getting a distribution rather than a single parameter. I doubt the effectiveness of the bayesion for neural network.
Chapter 6 Kernel Method
Previous chapters focus on linear method and its extension (i mean Neural Network). But kernel method is very different from kernel method, it involves nonlinear mapping in the model directly.
All the linear method can get a dual representation. In this representation, model is represented with a kernel function involved.
For kernel function, possible kernel should have positive definite gram matrix. And there are multiple ways to construct new kernel function:
1. Composite new kernel function according to a set of rules.
2. Composite new kernel function with probabilistic generative model, i.e. combine kernel function with a mixture model way.
This is the keypoint about the kernel function, others are skipped.
Another important knowledge is Gaussian Process, it seems that i can understand it now. From prior distribution, any existed data point has a distribution of output . Gaussian Process means the distribution of is gaussian. Well, most interesting point is we do not need to worry about selecting a proper prior over the parameter .
Chapter 7 Sparse Kernel Machine
Start with SVM algorithm, more interesting is Relevance Vector Machine. This is a Bayesian version of Support Vector Machine. The only different from Bayesian linear model, is the prior distribution for parameter is composed element-wise way:
So with this variant, sparse effect is achieved. The bayesian version is existed for classification & regression model at the same time.
Chapter 8 Graphical Model
Graphical model has different representation: directed & undirected. Each graphical representation specify the factorization of the joint probability distribution and conditional independence of the joint distribution. The factorization and conditional independence decide the efficiency of inference and learning algorithm.
For directed graphical model, there is d-separation to decide the conditional independence. For undirected graphical model, this is much simpler.
Inference on chain model & tree model is very simple. One important information: message passing used to passing message from other nodes about current node.
For general graphical model, loopy belief propagation, variational inference and sampling is the solution. Junction tree need complicated steps, i don’t think is used widely.
Some thing like factor graph and clique tree is another different representation of same graphical model. Different representation do not change the underlying distribution, just for the computation convenience.
So for graphical model, information is about structure of the joint distribution and how to use the structure to accelerate the computation.
Chapter 9 EM & Mixture Models
EM is used to get maximum likelihood estimator of some models with latent variable. The introduce of latent variable is used to simplify the computation of likelihood of observed data, even though the latent variable do not have any physical interpretation.
EM algorithm will try to get posterior distribution of hidden variable given observed data at first. Then it will calculate the expected complete data log-likelihood under the posterior distribution of hidden variables. At last, it will maximize the expectation with respect to model parameters.
For a long time, I can’t understand EM because don’t know how to get the distribution, then calculate the expectation under this distribution. Finally, i get the point. The key is not the separate the distribution and expectation, but to find the expectation of complete likelihood. With this information, the only thing required is the expectation of the posterior distribution.
If the posterior distribution is easy, EM can be very simple. Just maximize the complete likelihood of data, but replace the value of hidden variable with expected value of hidden variable. When posterior distribution is complex, approximate inference method is required. We can get the expected with different ways.
Chapter 10 Variational Inference
Variational inference is used for inference problem: marginal problem, posterior problem and MAP inference. If the distribution is very complex, we can make it simple by adding more extra conditional independence. Mean field approximation works by assuming some group of variables are independent even though not. Variatioanl inference minimized the following KL divergence:
and is factorized with different group of variables, i.e. . Apply this to the KL function then can get a function for minimization.
For a long time, i don’t understand this algorithm. Just because do not understand how to evaluate the expectation of complete data log-likelihood under some distribution.
Chapter 11 Sampling Method
The fantastic name “Monte Carlo Method” do not reveal the real content of this subject. Numerical Sampling method is using computer generated pseudo random number generator to get the real sample from target distribution, then perform all kind of inference operations.
So generally, there are specific sampling method, MCMC sampling method. Using MCMC method, there are several conditions:
1. aperiodic: This means there is no circle in the state traveling space.
2. irreducible: This means all the space can be explored.
3. reversible: also called detailed balance.
4. ergodicity: starting with any possible point, the convergence distribution is the same.
Among all the methods, Gibbs Sampling is the most important one. Another interesting method is Hamilton Monte Carlo Method.
From my understanding, the most important part for MCMC like method:
1. How to propose new point in the whole domain of the distribution.
2. How to use detailed balance to determine the acceptance of new point.
There are other general idea related with sampling: Data Augmentation. Data Augmentation works in the following way: If you want to sample from distribution , but you construct another distribution which satisfy at first. The variable called auxiliary variable. With new distribution , it’s much simpler to sample from it. So we get samples from then just drop the part. Slice sampling, Hamilton Monte Carlo method belong to this kind of general idea.
Chapter 12 Continuous Latent Variable
In this chapter, author gives some very important idea on dimension reduction task. In dimension reduction, there are some continuous latent variable, then after some kind of transformation become a high dimension space. But the latent variable only exist in a small manifold of high dimension. But in real word, data do not exist in a small manifold. How to interpret this? Well, data point do not exist in the small manifold, these data points will be interpreted as real data point plus some noise. PCA is interpreted as model with continuous latent variable, plus a Gaussian noise. Start with PCA, there are many other dimension reduction algorithm can be derived. Amazing!
Chapter 13 Sequential Data
Sequential data has the basic model: HMM for discrete latent variable, it’s the extension of mixture of Gaussians; LDS for continuous latent variable, it’s extension of PCA like models. It’s just the application of Graphical Model.
Combing Models
For combing models, it’s called Ensemble Method. Widely used method: Random Forest, AdaBoost, GBDT. For the rest, I don’t know the real applications.
Summary of Reading
For PRML, it covers the Linear Method for Classification/Regression, Neural Network, Kernel Method, SVM, Graphical Model, Variational Inference, Numerical Sampling, Ensemble Learning. I think the basic understand of machine learning techniques is acquired.
Written with StackEdit.
