User:Pixtonc/sandbox

In machine learning, sample complexity is the number of examples needed for the estimate of a target function to be within a given size of error. The sample complexity of a machine learning algorithm characterizes its rate of consistency.

Mathematical Setup
Given a set of samples $$S_n = \{(x_1,y_1),\ldots,(x_n,y_n)\}$$ drawn i.i.d. according to a distribution $$\rho$$ from some input space $$\mathcal X \times \mathcal Y$$, a supervised learning algorithm chooses a function $$f:\mathcal X \to \mathcal Y$$ from some hypothesis class $$\mathcal H$$. A desirable property of the algorithm is that it chooses functions with small expected prediction error with respect to $$\rho$$ and some loss function $$V:\mathcal Y \times \mathcal Y \to \mathbb R_+$$. Specifically, it is desirable to have a consistent algorithm, or an algorithm that generates functions whose expected risks converge to the best possible expected risk. Formally, let

$$ \mathcal R^*_\mathcal{H} = \underset{f \in \mathcal H}{\inf}\mathbb E_\rho[V(f(x),y)], $$

and let $$f_n$$ be the functions generated by an algorithm as the number of data points $$n$$ grows. The algorithm is consistent if

$$ \underset{n \to \infty}{\lim}\mathbb P_{S_n}(\mathbb E_\rho[V(f(x),y)] - \mathcal R^*_\mathcal{H} > \epsilon), $$

for all $$\epsilon > 0$$, where $$\mathbb P_{S_n}$$ denotes the probability measure $$\rho^n$$.

The consistency property is nice, but it says nothing about how fast the expected risks converge. Since in practice one always deals with finite data, it is important to answer the question of how many samples are needed to achieve a risk that is close, in the $$\epsilon$$ sense, to the best possible for the function class. The notion of sample complexity answers this question. The sample complexity of a learning algorithm is a function $$n(\rho,\epsilon,\delta)$$ such that for all $$n \ge n(\rho,\epsilon,\delta)$$,

$$ \mathbb P_{S_n}(\mathbb E_\rho[V(f(x),y)] - \mathcal R^*_\mathcal{H} \le \epsilon) \ge 1- \delta. $$

In words, the sample complexity $$n(\rho,\epsilon,\delta)$$ defines the rate of consistency of the algorithm. Given a desired accuracy $$\epsilon$$ and confidence $$\delta$$, one needs at most $$n(\rho,\epsilon,\delta)$$ samples to guarantee that the expected risk of the output function is within $$\epsilon$$ of the best possible expected risk with probability at least $$1-\delta$$.

No Free Lunch Theorem (Machine Learning)
Optimistically one could hope for a stronger notion of sample complexity that is independent of the distribution $$\rho$$ on the input and output spaces. However, it has been shown that without restrictions on the hypothesis class $$\mathcal H$$, there always exists "bad" distributions for which the sample complexity is arbitrarily large. Thus in order to make statements about the rate of convergence of the quantity

$$ \underset{\rho}{\sup}\ \mathbb P_{S_n}(\mathbb E_\rho[V(f(x),y)] - \mathcal R^*_\mathcal{H} > \epsilon), $$

one must either The latter approach leads to concepts such as VC dimension and Rademacher complexity which control the complexity of the space $$\mathcal H$$. A smaller hypothesis space introduces more bias into the inference process, meaning that $$\mathcal R^*_\mathcal{H}$$ may be larger than the best possible expected risk in a larger space. However, by restricting the complexity of the hypothesis space it becomes possible for an algorithm to produce functions converging in expected risk to $$\mathcal R^*_\mathcal{H}$$. This trade-off leads to the concept of regularization.
 * Constrain the set of probability distributions $$\rho$$, i.e. via a parametric approach, or
 * Constrain the set $$\mathcal H$$ to be small, as in distribution free approaches.

Other Settings
In addition to the supervised learning setting, sample complexity is relevant to semi-supervised learning problems including active learning, where the algorithm can ask for labels to specifically chosen inputs in order to reduce the cost of obtaining many labels. The concept of sample complexity also shows up in reinforcement learning, online learning, and unsupervised algorithms, e.g. for dictionary learning.