Statistical Models

Statistical models are broken down into two main catagories, generative and discriminative. Discriminative models are more straight forward to understand but are also more limited in application. These models allow a machine learning componnet to discriminate or classify between input elements. These models contain data that relates conditional probablities of input to the unknown decision parameters.

Generative models contain more information on the system's dynanaics. This makes the model more complex but allows the user to ask more questions. It can generate instances of system with specified characterstics.

Generative Models

In statistics, a generative model is a model for randomly generating observed data, typically given some hidden parameters. It specifies a joint probability distribution over observation and label sequences. Generative models are used in machine learning for either modeling data directly (i.e., modeling observed draws from a probability density function), or as an intermediate step to forming a conditional probability density function. A conditional distribution can be formed from a generative model through the use of Bayes' rule.

Generative models contrast with discriminative models, in that a generative model is a full probability model of all variables, whereas a discriminative model provides a model only of the target variable(s) conditional on the observed variables. Thus a generative model can be used, for example, to simulate (i.e. generate) values of any variable in the model, whereas a discriminative model allows only sampling of the target variables conditional on the observed quantities.

Examples of generative models include:

• Gaussian distribution
• Gaussian mixture model
• Multinomial distribution
• Hidden Markov model
• Generative grammar
• Naive Bayes
• Latent Dirichlet allocation

If the observed data are truly sampled from the generative model, then fitting the parameters of the generative model to maximize the data likelihood is a common method. However, since most statistical models are only approximates to the true distribution, if the model's application is to inference about a subset of variables conditional on known values of others, then it can be argued that the approximation makes more assumptions than are necessary to solve the problem at hand. In such cases, it is often more accurate to model the conditional density functions directly: i.e., performing classification or regression analysis.

Discriminative Models

Discriminative models are a class of models used in machine learning for modeling the dependence of an unobserved variable y on an observed variable x. Within a statistical framework, this is done by modeling the conditional probability distribution P(y|x), which can be used for predicting y from x.

Discriminative models differ from generative models in that they do not allow one to generate samples from the joint distribution of x and y.

Examples of discriminative models used in machine learning include:

• Linear discriminant analysis
• Support vector machines
• Boosting
• Conditional Random Fields
• Logistic Regression