User:ByPhoebe/sandbox

Functional data analysis (FDA) is a branch of statistics that analyzes data providing information about curves, surfaces or anything else varying over a continuum. In its most general form, under an FDA framework, each sample element of functional data is considered to be a function. The physical continuum over which these functions are defined is often time, but may also be spatial location, wavelength, probability, etc. Intrinsically, functional data are infinite dimensional. The high intrinsic dimensionality of these data brings challenges for theory as well as computation, where these challenges vary with how the functional data were sampled. However, the high or infinite dimensional structure of the data is a rich source of information and there are many interesting challenges for research and data analysis.

History
Functional data analysis has roots going back to work by Grenander and Karhunen in the 1940's and 1950's. They considered the decomposition of square-integrable continuous time stochastic process into eigencomponents, now known as the Karhunen-Loève decomposition. A rigorous analysis of functional principal components analysis was done in the 1970's by Kleffe, Dauxois and Pousse including results about the asymptotic distribution of the eigenvalues. More recently in the 1990's and 2000's the field has focused more on application and understanding the effects of dense and sparse observations schemes. The term "Functional Data Analysis" was coined by James O. Ramsay.

Mathematical formalism
Random functions can be viewed as random elements taking values in a Hilbert space, or as a stochastic process. The former is mathematically convenient, whereas the latter is somewhat more suitable from an applied perspective. These two approaches coincide if the random functions are continuous and a condition called mean-squared continuity is satisfied.

Hilbertian random variables
In the Hilbert space viewpoint, one considers an $$H$$-valued random element $$X$$, where $$H$$ is a separable Hilbert space such as the space of square-integrable functions $$L^2[0,1]$$. Under the integrability condition that $$\mathbb E\|X\|_{L^2}^2 = \mathbb{E} (\int^1_0 |X(t)|^2 dt) < \infty$$, one can define the mean of $$X$$ as the unique element $$\mu\in H$$ satisfying

\mathbb E\langle X, h \rangle = \langle \mu,h\rangle ,\qquad h\in H. $$ This formulation is the Pettis integral but the mean can also be defined as $$\mu=\mathbb EX$$ the Bochner sense. Under the integrability condition that $$\mathbb E\|X\|^2_{L^2}$$ is finite, the covariance operator of $$X$$ is a linear operator $$\mathcal{C}:H\to H$$ that is uniquely defined by the relation

\mathcal{C} h = \mathbb E[\langle h,X-\mu\rangle (X-\mu)],\qquad h\in H, $$ or, in tensor form, $$\mathcal{C} = \mathbb E[(X-\mu)\otimes(X-\mu)]$$. The spectral theorem allows to decompose $$X$$ as the Karhunen-Loève decomposition

X=\mu+\sum_{i=1}^\infty \langle X,\varphi_i\rangle \varphi_i, $$ where $$\varphi_i$$ are eigenvectors of $$\mathcal S$$, corresponding to the nonnegative eigenvalues of $$\mathcal S$$, in a non-increasing order. Truncating this infinite series to a finite order underpins functional principal component analysis.

Stochastic processes
The Hilbertian point of view is mathematically convenient, but abstract; the above considerations do not necessarily even view $$X$$ as a function at all, since common choices of $$H$$ like $$L^2[0,1]$$ and Sobolev spaces consist of equivalence classes, not functions. The stochastic process perspective views $$X$$ as a collection of random variables

\{X(t)\}_{t\in [0,1]} $$ indexed by the unit interval (or more generally some compact metric space $$K$$). The mean and covariance functions are defined in a pointwise manner as

\mu(t) = \mathbb EX(t),\qquad \Sigma(s,t) = \textrm{Cov}(X(s), X(t)),\qquad s,t\in [0, 1] $$ (if $$\mathbb E[X(t)^2]<\infty$$ for all $$t\in [0,1]$$). We can hope to view $$X$$ as a random element on the Hilbert function space $$H=L^2[0,1]$$. However, additional conditions are required for such a pursuit to be fruitful, since if we let $$X(t)$$ be Gaussian white noise, i.e. $$X(t)$$ is standard Gaussian and independent from $$X(s)$$ for any $$s, t \in [0, 1]$$, it is clear that we have no hope of viewing this as a square integrable function.

A convenient sufficient condition is mean square continuity, stipulating that $$\mu$$ and $$c$$ are continuous functions. In this case $$c$$ defines a covariance operator $$\mathcal{C}:H\to H$$ by

(\mathcal{C} f)(t) =\int_K \Sigma(s,t)f(s) \, \mathrm{d}s. $$ The spectral theorem applies to $$\mathcal{C}$$, yielding eigenpairs $$(\lambda_j,\varphi_j)$$, so that in tensor product notation $$\mathcal{C}$$ writes

\mathcal{C}=\sum_{j=1}^\infty \lambda_j \varphi_j\otimes\varphi_j. $$ Moreover, since $$\mathcal Sf$$ is continuous for all $$f\in H$$, all the $$\varphi_j$$'s are continuous. Mercer's theorem then states that the covariance function $$c$$ admits an analogous decomposition

\sup_{s,t\in [0,1]} \left|c(s,t) - \sum_{j=1}^K \lambda_j \varphi_j(s)\varphi_j(t)\right|\to0,\qquad K\to\infty. $$ Finally, under the extra assumption that $$X$$ has continuous sample paths, namely that with probability one, the random function $$X:[0,1]\to\mathbb R$$ is continuous, the Karhunen-Loève expansion above holds for $$X$$ and the Hilbert space machinery can be subsequently applied. Continuity of sample paths can be shown using Kolmogorov continuity theorem.

Functional data designs
Functional data are considered as realizations of a stochastic process $$X(t), t\in[0,1]$$ that is an $$L^2$$ process on a bounded and closed interval $$[0,1]$$ with mean function $$\mu(t)=\mathbb{E}(X(t))$$ and covariance function $$\Sigma(s,t)=\textrm{Cov}(X(s),X(t))$$. The realizations of the process for the i-th subject is $$X_i(\cdot)$$, and the sample is assumed to consist of $$n$$ independent subjects. The sampling schedule may vary across subjects, denoted as $$T_{i1},..., T_{iN_i}$$ for the i-th subject. The corresponding i-th observation is denoted as $$\textbf{X}_i = (X_{i1},..., X_{iN_i}) $$, where $$X_{ij} = X_i(t_{ij})$$. In addition, the measurement of $$X_{ij} $$ is assumed to have random noise $$\epsilon_{ij}$$ with $$\mathbb{E}(\epsilon_{ij})=0$$ and $$\textrm{Var}(\epsilon_{ij})=\sigma^2_{ij}$$, which are independent across $$i$$ and $$j$$.

1. Fully observed functions without noise at arbitrarily dense grid
Measurements $$ Y_{it}=X_i(t) $$ available for all $$ t\in \mathcal{I},\, i=1,\ldots,n $$

Often unrealistic but mathematically convenient.

Real life example: Tecator spectral data.

2. Dense design with noisy measurements
Measurements $$ Y_{ij}=X_i(T_{ij}) + \varepsilon_{ij} $$, where $$ T_{ij} $$ are recorded on a regular grid,

$$ T_{i1},\ldots,T_{iN_i} $$, and $$ N_{i}\rarr\infty $$ applies to typical functional data.

Real life example: Berkeley Growth Study Data and Stock data

3. Sparse design with noisy measurements (longitudinal data)
Measurements $$ Y_{ij}=X_i(T_{ij}) + \varepsilon_{ij} $$, where $$ T_{ij} $$ are random times and their number $$ N_{i} $$ per subject is random and finite.

Real life example: CD4 count data for AIDS patients.

Functional principal component analysis
Functional principal component analysis (FPCA) is the most prevalent tool in FDA, partly because FPCA facilitates dimension reduction of the inherently infinite-dimensional functional data to finite-dimensional random vector of $$scores$$. More specifically, dimension reduction is achieved by expansion the underlying observed random trajectories $$X_i(t)$$ in a functional basis that consists of the eigenfunctions of the (auto)-covariance operator on $$X$$. Consider the (auto)-covariance operator $$\mathcal{C}: L^2[0,1] \rightarrow L^2[0,1]$$, $$(\mathcal{C} X)(t)=\int_0^1\Sigma(s,t)X(s)ds$$, which is a compact operator on Hilbert space. By Mercer's theorem, the kernel of $$\mathcal{C}$$, i.e., the covariance function $$\Sigma (\cdot, \cdot)$$, has spectral decomposition $$\Sigma(s,t)=\sum^{\infty}_{k=1}\lambda_k\varphi_k(s)\varphi_k(t)$$, where the series convergence is absolute and uniform, and $$\lambda_k$$ are real-valued nonnegative eigenvalues in descending order with the corresponding orthonormal eigenfunctions $$\varphi_k(t)$$. By the Karhunen–Loève theorem, the FPCA expansion of an underlying random trajectory is $$X_i(t) =\mu(t)+\sum^{\infty}_{k=1} A_{ik}\varphi_k(t)$$, where $$A_{ik}=\int_0^1(X_i(t)-\mu(t))\varphi_k(t)dt$$ are the functional principal components (FPCs), sometimes referred to as $$scores$$. The Karhunen–Loève expansion facilitates dimension reduction in the sense that the partial sum converges uniformly, i.e., $$\sup_{t\in[0,1]} \mathbb{E}[X_i(t)-\mu(t)-\sum^{K}_{k=1}A_{ik}\varphi_k(t)]^2 \rightarrow 0$$ as $$K \rightarrow \infty$$ and thus the partial sum with a large enough $$K$$ yields a good approximation to the infinite sum. Thereby, the information in $$X_i$$ is reduced from infinite dimensional to a $$K$$-dimensional vector $$A_i=(A_{i1},...,A_{iK})$$ with the approximated process： Other popular bases include spline, Fourier series and wavelet bases. Important applications of FPCA include the modes of variation and functional principal component regression.

Functional linear regression models
Functional linear models can be viewed as an extension of the traditional multivariate linear models that associates vector responses with vector covariates. The traditional linear model with scalar response $$Y\in\mathbb{R}$$ and vector covariate $$X\in\mathbb{R}^p$$ can be expressed aswhere $$\langle\cdot,\cdot\rangle$$ denotes the inner product in Euclidean space, $$\beta_0\in\mathbb{R}$$ and $$\beta\in\mathbb{R}^p$$ denote the regression coefficients, and $$\varepsilon$$ is a zero mean finite variance random error (noise). Functional linear models can be divided into two types based on the responses.

Functional regression models with scalar response
Replacing the vector covariate $$X$$ and the coefficient vector $$\beta$$ in model ($$) by a centered functional covariate $$X^c(t) = X(t) - \mu(t)$$ and coefficient function $$\beta = \beta(t)$$ for $$t \in [0,1]$$ and replacing the inner product in Euclidean space by that in Hilbert space $L^2$, one arrives at the functional linear modelThe simple functional linear model ($$) can be extended to multiple functional covariates, $$\{X_j\}_{j=1}^p$$, also including additional vector covariates $$Z = (Z_1, \cdots,Z_q)$$, where $$Z_1=1$$, bywhere $$\theta\in \mathbb{R^q}$$ is regression coefficient for $$Z$$, the domain of $$X_j$$ is $$[0,1]$$, $$X^c_j$$ is the centered functional covariate given by $$X_j^c(t) = X_j(t) - \mu_j(t)$$, and $$\beta_j$$ is regression coefficient function for $$X_j^c $$, for $$j=1,\ldots,p$$. Model ($$) and ($$) have been studied extensively.

Functional regression models with functional response
For a functional response $$Y(s)$$ on $$[0,1]$$ and multiple functional covariates $$X_j(t)$$, $$t\in [0,1]$$, two major models have been considered. One of these two models is generally referred to as functional linear model (FLM) which can be written as:where $$\alpha_0(s)$$ is the functional intercept, for $$j=1,\ldots,p$$, $$X_j^c(t)=X_j(t) - \mu_j(t)$$ is a centered functional covariate on $$[0,1]$$, $$\alpha_j(s,t)$$ is the corresponding functional slopes with same domain, respectively, and $$\varepsilon(s)$$ is usually a random process with mean zero and finite variance. In this case, at any given time $$s\in[0,1]$$, the value of $$Y$$, i.e., $$Y(s)$$, depends on the entire trajectories of $$\{X_j(t)\}^p_{j=1}$$. Model ($$) are also studied extensively.

Function-on-scalar regression
In particular, taking $$X_j(\cdot)$$ as a constant function yields a special case of model ($$)$$Y(s) = \alpha_0(s)+\sum_{j=1}^p X_j \alpha_j(s) + \varepsilon(s),\ \text{for}\ s\in[0,1],$$which is a functional linear model with functional responses and scalar covariates.

Concurrent regression models
This model is given by,where $$X_1,\ldots,X_p$$ are multiple functional covariates on $$[0,1]$$, $$\beta_0,\beta_1,\ldots,\beta_p$$ are the coefficient functions defined on the same interval and $$\varepsilon(s)$$ is usually assumed to be a random process with mean zero and finite variance. This model assumes that the value of $$Y(s)$$ depends on the current value of $$\{X_j(s)\}^p_{j=1}$$ only and not the history $$\{X_j(t):t\le s\}^p_{j=1}$$ or future value, hence it is a "concurrent regression model", which also has been referred to as "varying-coefficient" model. Various estimation methods have been proposed.

Functional nonlinear regression models
Direct nonlinear extensions of the classical functional linear regression models (FLMs) still involve a linear predictor, but combine it with a nonlinear link function, analogous to the idea of generalized linear model from the conventional linear model. Developments towards fully nonparametric regression models for functional data encounter problems such as curse of dimensionality. In order to bypass the “curse” and the metric selection problem, we are motivated to consider nonlinear functional regression models, which are subject to some structural constraints but do not overly infringe flexibility. One desires models that retain polynomial rates of convergence, while being more flexible than, say, functional linear models. Such models are particularly useful when diagnostics for the functional linear model indicate lack of fit, which is often encountered in real life situations. In particular, functional polynomial models, functional single and multiple index models and functional additive models are three special cases of functional nonlinear regression models.

Functional polynomial regression models
Functional polynomial regression models may be viewed as a natural extension of the Functional Linear Models (FLMs) with scalar responses, analogous to extending linear regression model to polynomial regression model. For a scalar response $$Y$$ and a functional covariate $$X(\cdot)$$ with domain $$[0,1]$$ and the corresponding centered predictor processes $$X^c$$, the simplest and the most prominent member in the family of functional polynomial regression models is the quadratic functional regression given as follows,$$\mathbb{E}(Y|X) = \alpha + \int_0^1\beta(t)X^c(t)\,dt + \int_0^1 \int_0^1 \gamma(s,t) X^c(s)X^c(t) \,ds\,dt $$where $$X^c(\cdot) = X(\cdot) - \mathbb{E}(X(\cdot))$$ is the centered functional covariate, $$\alpha$$ is a scalar coefficient, $$\beta(\cdot)$$ and $$\gamma(\cdot,\cdot)$$ are coefficient functions with domains $$[0,1]$$ and $$[0,1]\times[0,1]$$, respectively. In addition to the parameter function β that the above functional quadratic regression model shares with the FLM, it also features a parameter surface γ. By analogy to FLMs with scalar responses, estimation of functional polynomial models can be obtained through expanding both the centered covariate $$X^c$$ and the coefficient functions $$\beta$$ and $$\gamma$$ in an orthonormal basis.

Functional single and multiple index models
A functional multiple index model is given as below, with symbols having their usual meanings as formerly described,$$ \mathbb{E}(Y|X) = g\left(\int_0^1X^c(t) \beta_1(t)\,dt, \ldots, \int_0^1X^c(t) \beta_p(t)\,dt \right) $$Here g represents an (unknown) general smooth function defined on a p-dimensional domain. The case $$p=1$$ yields a functional single index model while multiple index models correspond to the case  $$p>1$$. However, for $$p>1$$, this model is problematic due to curse of dimensionality. With $$p>1$$ and relatively small sample sizes, the estimator given by this model often has large variance.

Functional additive models (FAMs)
Let $$X(t) = \sum_{k=1}^\infty x_k \phi_k(t)$$ denote an expansion of a functional covariate $$X$$ with domain $$[0,1]$$ in an orthonormal basis $$\{\phi_k\}_{k=1}^\infty$$.

A functional linear model with scalar responses [as shown in model ($$) of FLM] can be thus be written as follows,$$\mathbb{E}(Y|X)=\mathbb{E}(Y) + \sum_{k=1}^\infty \beta_k x_k.$$One form of FAMs is obtained by replacing the linear function of $$x_k$$ in the above expression ( i.e., $$\beta_k x_k$$) by a general smooth function $$f_k$$, analogous to the extension of multiple linear regression models to additive models and is expressed as,$$\mathbb{E}(Y|X)=\mathbb{E}(Y) + \sum_{k=1}^\infty f_k(x_k),$$where $$f_k$$ satisfies $$\mathbb{E}(f_k(x_k))=0$$ for $$k\in\mathbb{N}$$. This constraint on the general smooth functions $$f_k$$ ensures identifiability in the sense that the estimates of these additive component functions do not interfere with that of the intercept term $$\mathbb{E}(Y)$$.

Another form of FAM is the continuously additive model, expressed as,$$\mathbb{E}(Y|X) = \mathbb{E}(Y) + \left(\int_0^1 g(t,X(t)) dt\right)$$for a bivariate smooth (i.e. twice differentiable) additive surface g : $$[0,1]\times\mathbb{R} \longrightarrow \mathbb{R}$$ which is required to satisfy $$\mathbb{E}[g(t,X(t))]=0$$ for all t in $$[0,1]$$ in order to ensure identifiability.

Generalized functional linear model
An obvious and direct extension of FLMs with scalar responses [shown in model ($$)] is to add a link function so as to create a generalized functional linear model (GFLM) by analogy to extending linear model to generalized linear model (GLM), of which the three components are:


 * 1) Linear predictor $$\eta = \beta_0 + \int_0^1 X^c(t)\beta(t)\,dt$$;  [systematic component]
 * 2) Variance function $$\text{Var}(Y|X) = V(\mu)$$, where $$\mu = \mathbb{E}(Y|X)$$ is the conditional mean;  [random component]
 * 3) Link function $$g$$ connecting the conditional mean $$\mu$$ and the linear predictor $$\eta$$ through $$\mu=g(\eta)$$.  [systematic component]

Clustering and classification of functional data
For vector-valued multivariate data, k-means partitioning methods and hierarchical clustering are two main approaches. These classical clustering concepts for vector-valued multivariate data have been extended to functional data. For clustering of functional data, k-means clustering methods are more popular than hierarchical clustering methods. For k-means clustering on functional data, mean functions are usually regarded as the cluster centers. Covariance structures have also been taken into consideration. Besides k-means type clustering, functional clustering based on mixture models is also widely used in clustering vector-valued multivariate data and has been extended to functional data clustering. Furthermore, Bayesian hierarchical clustering also plays an important role in the development of model-based functional clustering. Functional classification assigns a group membership to a new data object either based on functional regression or functional discriminant analysis. Functional data classification methods based on functional regression models use class levels as responses and the observed functional data and other covariates as predictors. For regression based functional classification models, functional generalized linear models or more specifically, functional binary regression, such as functional logistic regression for binary response, are commonly used classification approaches. More generally, the generalized functional linear regression model based on the FPCA approach is used. Functional Linear Discriminant Analysis (FLDA) has also been considered as a classification method for functional data. Functional data classification involving density ratios has also been proposed. A study of the asymptotic behavior of the proposed classifiers in the large sample limit shows that under certain conditions the misclassification rate converges to zero, a phenomenon that has been referred to as "perfect classification".

Motivations


In addition to amplitude variation, time variation may also be assumed to present in functional data. Time variation occurs when the subject-specific timing of certain events of interest varies among subjects. One classical example is Berkeley Growth Study Data, where the amplitude variation is the growth rate and the time variation explains the difference in children's biological age at which the pubertal and the pre-pubertal growth spurt occurred. In the presence of time variation, cross-sectional mean function may be an efficient estimate as peaks and troughs are located randomly and thus meaningful signals may be distorted or hidden.

Time warping, also known as curve registration, curve alignment or time synchronization, aims to identify and separate amplitude variation and time variation. If assume both components exist, then the functional data are viewed as a result of an underlying template function "warped" non-linearly in time by a smooth random process, known as time warping function. Let $$Y_i$$ denote the observed functions with both phase and amplitude variability and $$X_i$$ to be the underlying functions where only the amplitude variation is present. The time warping function $$h_i$$ removes the time variation such that $$Y_i(t)=X_i[h_i^{-1}(t_j)], t\in [0,1]$$, where $$h_i$$ are the realizations of an underlying time warping function that transforms the subject-specific time to the time scale of the template. The time warping functions $$h_i$$ are assumed to be invertible and have average identity $$\mathbb{E}(h(t))=t$$. In most cases, the time is assumed to flow forward, so $$h_i$$ are assumed to be strictly monotonic increasing. One exception where the time can flow backward has been presented in the context of modeling the declines in house price as time reversals.

The simplest case of a family of warping functions to specify phase variation is linear transformation, that is $$h(t)=\delta+\gamma t$$, which warps the time of an underlying template function by subjected-specific shift and scale. More general class of warping functions includes diffeomorphisms of the domain to itself, that is, loosely speaking, a class of invertible functions that maps the compact domain to itself such that both the function and its inverse are smooth. The set of linear transformation is contained in the set of diffeomorphisms. One challenge in time warping is identifiability of amplitude and phase variation. To break the non-identifiability requires specific assumptions.

Methods
Earlier approaches include dynamic time warping (DTW) for applications such as speech recognition. Another traditional method for time warping is landmark registration, which aligns special features such as peak locations to an average location. Other relevant warping methods include pairwise warping, registration using $$\mathcal{L}^2$$ distance and elastic warping.

Dynamic time warping
The template function is determined through an iteration process, starting from cross-sectional mean, performing registration and recalculating the cross-sectional mean for the warped curves, expecting convergence after a few iterations. DTW minimizes a cost function through dynamic programming algorithm. Problems of non-smooth differentiable warps or greedy computation in DTW can be resolved by adding a regularization term to the cost function.

Landmark registration
Landmark registration (or feature alignment) assumes well-expressed features are present in all sample curves and uses the location of such features as a gold-standard. Special features such as peak or valley locations or derivatives on the observed sample functions are aligned to their average locations on the template function. Then the warping function is introduced through a smooth transformation from the average location to the subject-specific locations. A problem of landmark registration is that the features may be missing or hard to identify due to the noise in the data.

R packages
While the presentation of the models above assumes fully-observed functions, software is available for fitting the models with discretely-observed functions in software such as R.


 * fda
 * refund
 * fdapace
 * FDboost
 * classiFunc
 * fda.usc
 * dtw
 * fdasrvf