Using locale information

Kernel Method (or Non-parametric Method)

Seems there are two different definition of “kernel methods” in machine learning. One definition related with RKHS and so on. The other one, just like some non-parametric methods. And the latter one is the topic in post, the main idea is about how to use localized information to get a model.

Unlike linear model, which construct a global function over all the sample spaces; kernel methods works by construct a localized function for each new sample point $x_0$. We can see how this method can be applied to different tasks.

When apply this method to regression task, for each new sample point $x_0$, it will construct a weight matrix $W_{x_0}$ based on some kernel function $k(x, x_0)$. This kernel function will assign higher weights to closer training points based on some norm. Then a weighted regression will be performed, getting a brand new predicting function $\hat{f_{x_0}}$ and return the predicted value.

For regression task, there is also another kernel method. Which will weight samples within the neighborhood of new sample point $x_0$ and return a weighted average of response variable value.

When apply this method to density estimation, it will also construct a weight kernel decaying with the distance from the point $x_0$. And then perform classification according to bayes rule. We can also use mixture of Gaussian to estimate the density more clearly for each classes.

For all the methods mentioned above, there will be a issue of bias and variance trade off.

Written with StackEdit.