User talk:Dswers

Problem 3
Suppose we have a system of equations in variables $$x_1,x_2,\ldots,x_n$$:
 * $$f_1(x_1,x_2,\ldots x_n)=0$$
 * $$f_2(x_1,x_2,\ldots x_n)=0\ \ (eq.5)$$
 * $$\vdots$$
 * $$f_m(x_1,x_2,\ldots x_n)=0$$

We may rewrite this equation in vector form:
 * $$\overrightarrow{F}(\overrightarrow{x})=\overrightarrow{0}$$

where $$\overrightarrow{x}$$ is the row vector $$x_1,x_2,\ldots x_n$$ and $$\overrightarrow{F}(\overrightarrow{x})$$ is the column vector $$(f_1(\overrightarrow{x}),f_2(\overrightarrow{x}),\ldots,f_m(\overrightarrow{x}))^T$$. We wish to find a close approximation to an actual root of this system, denoted $$\overrightarrow{p}$$, starting with an initial guess, $$\overrightarrow{x_0}$$. We will generate a recursive sequence of points $$\{\overrightarrow{x_k}\}$$ such that $$\lim_{k\rightarrow\infty}\overrightarrow{x_k}=\overrightarrow{p}$$. One method for generating the sequence $$\{\overrightarrow{x_k}\}$$ is to use the "Generalized Newton's Method", which extends the single variable Newton's Method technique to the multivariable case. The derivation of this method presented here is due to section 10.2 of Burden & Faires, "Numerical Analysis (8ed.)".

Given a point $$\overrightarrow{x_k}$$, we generate linear approximations to each $$f_i(x)$$ at the point $$\overrightarrow{x}=\overrightarrow{x_k}$$. We then use these linearizations to obtain a (hopefully) closer approximation to $$\overrightarrow{p}$$, which we then label as the point $$\overrightarrow{x_{k+1}}$$ in the recursive sequence. To obtain $$\overrightarrow{x_{k+1}}$$, we must therefore solve the following system of equations:
 * $$f_1(\overrightarrow{x_k})+\Delta x_1\frac{\partial f_1}{\partial x_1}+\Delta x_2\frac{\partial f_1}{\partial x_2}+\cdots+\Delta x_n\frac{\partial f_1}{\partial x_n}=0$$
 * $$f_2(\overrightarrow{x_k})+\Delta x_1\frac{\partial f_2}{\partial x_1}+\Delta x_2\frac{\partial f_2}{\partial x_2}+\cdots+\Delta x_n\frac{\partial f_2}{\partial x_n}=0$$
 * $$\vdots$$
 * $$f_m(\overrightarrow{x_k})+\Delta x_1\frac{\partial f_m}{\partial x_1}+\Delta x_2\frac{\partial f_m}{\partial x_2}+\cdots+\Delta x_n\frac{\partial f_m}{\partial x_n}=0$$
 * where $$\Delta x_i=(\overrightarrow{x_{k+1}})_i-(\overrightarrow{x_k})_i$$.

In matrix form, we may rewrite this system of equations as
 * $$\overrightarrow{F}(\overrightarrow{x_k})+\mathbf{J}(\overrightarrow{x_k})\overrightarrow{\Delta x}=0$$,

where $$\overrightarrow{\Delta x}$$ is the n-dimensional column vector with entries $$(\overrightarrow{\Delta x})_i=\Delta x_i$$, and $$\mathbf{J}(\overrightarrow{x})$$ is the Jacobian matrix of $$\overrightarrow{F}$$ defined by $$\mathbf{J}_{i,j}(\overrightarrow{x})=\frac{\partial f_i(\overrightarrow{x})}{\partial x_j}$$. We thus solve for vector $$\overrightarrow{\Delta x}$$ as:
 * $$\overrightarrow{\Delta x}=-\mathbf{J}(\overrightarrow{x_k})^{-1}\overrightarrow{F}(\overrightarrow{x_k})$$

The vector $$\overrightarrow{x_{k+1}}$$ is then given by $$\overrightarrow{x_{k+1}}=\overrightarrow{x_k}+\overrightarrow{\Delta x}$$, or upon substituting for $$\overrightarrow{\Delta x}$$, $$\overrightarrow{x_{k+1}}=\overrightarrow{x_k}-\mathbf{J}(\overrightarrow{x_k})^{-1}\overrightarrow{F}(\overrightarrow{x_k})$$. So if $$g(\overrightarrow{x})$$ is the recursion function for this algorithm, we define $$g(\overrightarrow{x})=\overrightarrow{x}-\mathbf{J}(\overrightarrow{x})^{-1}\overrightarrow{F}(\overrightarrow{x})$$, so that $$\overrightarrow{x_{k+1}}=g(\overrightarrow{x_k})$$. We shall now prove a theorem concerning the convergence of the sequence $$\{\overrightarrow{x_k}\}$$, but we require a special lemma before we can proceed.

Lemma (Multivariable Mean Value Theorem)
Let $$g(x_1,x_2,\ldots,x_n)$$ be a function of n variables and let $$g(\overrightarrow{x})$$ have continuous first partial derivatives with respect to all variables in some closed ball, $$\mathbb{B}$$ about the point $$\overrightarrow{x}=\overrightarrow{p}$$. Then
 * $$g(\overrightarrow{x})-g(\overrightarrow{p})=g_{x_1}(\overrightarrow{\xi_1})(x_1-p_1)+g_{x_2}(\overrightarrow{\xi_2})(x_2-p_2)+\cdots+g_{x_n}(\overrightarrow{\xi_n})(x_n-p_n)$$,

where $$g_{x_i}=\frac{\partial g}{\partial x_i}$$ and $$\overrightarrow{\xi_i}$$ is a point in $$\mathbb{B}$$ for $$i=1,2,\ldots,n$$.

Proof
We proceed by induction. The case $$\ n=1$$ is just the usual single variable Mean Value Theorem. Suppose the theorem holds for $$\ n=k$$. Let $$\overrightarrow{x}=(x_1,x_2,\ldots,x_k)$$, and $$\overrightarrow{p}=(p_1,p_2,\ldots,p_k)$$. Then, applying the induction principle,
 * $$g(\overrightarrow{x},x_{k+1})-g(\overrightarrow{p},p_{k+1})=g(\overrightarrow{x},x_{k+1})-g(\overrightarrow{p},x_{k+1})+g(\overrightarrow{p},x_{k+1})-g(\overrightarrow{p},p_{k+1})$$
 * $$=g_{x_1}(\overrightarrow{\xi_1},x_{k+1})(x_1-p_1)+g_{x_2}(\overrightarrow{\xi_2},x_{k+1})(x_2-p_2)+\cdots+g_{x_k}(\overrightarrow{\xi_k},x_{k+1})(x_k-p_k)+g_{x_{k+1}}(\overrightarrow{p},\xi_{k+1})(x_{k+1}-p_{k+1})$$,

where $$\ \xi_{k+1}$$ is a real number between $$\ x_{k+1}$$ and $$\ p_{k+1}$$. Each of the points $$(\overrightarrow{\xi},x_{k+1})$$ may be labelled by a new symbol (this time, a (k+1)-vector) $$\overrightarrow{\eta_i}\in\mathbb{B}$$, since these points are all contained in $$\mathbb{B}$$. Similarly, we may write $$(\overrightarrow{p},\xi_{k+1})=\eta_{k+1}\in\mathbb{B}$$. We have thus expressed the difference $$g(\overrightarrow{x},x_{k+1})-g(\overrightarrow{p},p_{k+1})$$ in the desired form, so by induction the theorem is proved.