User:Jimmy Novik/Cost Function

The cost function is a function that describes how well a machine learning algorithm approximates some function $$f^{*}$$, or, in other words, how close our algorithm is at achieving its main purpose of predicting the output based on a given input.

Most of the time the cost function being used is the cross-entropy between the training data and the model’s predictions.

The cost function is given by:

$$J(\theta)=-\mathbb{E}_{x,y~p_{data}}\log{p}_{model}(y | x)$$,

Which gets expanded to:

$$J(\theta)=\frac{1}{2}\mathbb{E}_{x,y~p_{data}}\|y-f(x;\theta)\|^2 + const$$

Extra Mean square distance function


 * $${\rm MSD}\equiv\langle |\mathbf{x}(t)-\mathbf{x_0}|^2\rangle=\frac{1}{N}\sum_{i=1}^N |\mathbf{x^{(i)}}(t) - \mathbf{x^{(i)}}(0)|^2$$