Engel's theorem

In representation theory, a branch of mathematics, Engel's theorem states that a finite-dimensional Lie algebra $$\mathfrak g$$ is a nilpotent Lie algebra if and only if for each $$X \in \mathfrak g$$, the adjoint map


 * $$\operatorname{ad}(X)\colon \mathfrak{g} \to \mathfrak{g},$$

given by $$\operatorname{ad}(X)(Y) = [X, Y]$$, is a nilpotent endomorphism on $$\mathfrak{g}$$; i.e., $$\operatorname{ad}(X)^k = 0$$ for some k. It is a consequence of the theorem, also called Engel's theorem, which says that if a Lie algebra of matrices consists of nilpotent matrices, then the matrices can all be simultaneously brought to a strictly upper triangular form. Note that if we merely have a Lie algebra of matrices which is nilpotent as a Lie algebra, then this conclusion does not follow (i.e. the naïve replacement in Lie's theorem of "solvable" with "nilpotent", and "upper triangular" with "strictly upper triangular", is false; this already fails for the one-dimensional Lie subalgebra of scalar matrices).

The theorem is named after the mathematician Friedrich Engel, who sketched a proof of it in a letter to Wilhelm Killing dated 20 July 1890. Engel's student K.A. Umlauf gave a complete proof in his 1891 dissertation, reprinted as.

Statements
Let $$\mathfrak{gl}(V)$$ be the Lie algebra of the endomorphisms of a finite-dimensional vector space V and $$\mathfrak g \subset \mathfrak{gl}(V)$$ a subalgebra. Then Engel's theorem states the following are equivalent: 0, \, \operatorname{codim} V_i = i$$ such that $$\mathfrak g \cdot V_i \subset V_{i+1}$$; i.e., the elements of $$\mathfrak g$$ are simultaneously strictly upper-triangulizable.
 * 1) Each $$X \in \mathfrak{g}$$ is a nilpotent endomorphism on V.
 * 2) There exists a flag $$V = V_0 \supset V_1 \supset \cdots \supset V_n =

Note that no assumption on the underlying base field is required.

We note that Statement 2. for various $$\mathfrak g$$ and V is equivalent to the statement
 * For each nonzero finite-dimensional vector space V and a subalgebra $$\mathfrak g \subset \mathfrak{gl}(V)$$, there exists a nonzero vector v in V such that $$X(v) = 0$$ for every $$X \in \mathfrak g.$$

This is the form of the theorem proven in. (This statement is trivially equivalent to Statement 2 since it allows one to inductively construct a flag with the required property.)

In general, a Lie algebra $$\mathfrak g$$ is said to be nilpotent if the lower central series of it vanishes in a finite step; i.e., for $$C^0 \mathfrak g = \mathfrak g, C^i \mathfrak g = [\mathfrak g, C^{i-1} \mathfrak g]$$ = (i+1)-th power of $$\mathfrak g$$, there is some k such that $$C^k \mathfrak g = 0$$. Then Engel's theorem implies the following theorem (also called Engel's theorem): when $$\mathfrak g$$ has finite dimension, Indeed, if $$\operatorname{ad}(\mathfrak g)$$ consists of nilpotent operators, then by 1. $$\Leftrightarrow$$ 2. applied to the algebra $$\operatorname{ad}(\mathfrak g) \subset \mathfrak{gl}(\mathfrak g)$$, there exists a flag $$\mathfrak g = \mathfrak{g}_0 \supset \mathfrak{g}_1 \supset \cdots \supset \mathfrak{g}_n = 0$$ such that $$[\mathfrak g, \mathfrak g_i] \subset \mathfrak g_{i+1}$$. Since $$C^i \mathfrak g\subset \mathfrak g_i$$, this implies $$\mathfrak g$$ is nilpotent. (The converse follows straightforwardly from the definition.)
 * $$\mathfrak g$$ is nilpotent if and only if $$\operatorname{ad}(X)$$ is nilpotent for each $$X \in \mathfrak g$$.

Proof
We prove the following form of the theorem: if $$\mathfrak{g} \subset \mathfrak{gl}(V)$$ is a Lie subalgebra such that every $$X \in \mathfrak{g}$$ is a nilpotent endomorphism and if V has positive dimension, then there exists a nonzero vector v in V such that $$X(v) = 0$$ for each X in $$\mathfrak{g}$$.

The proof is by induction on the dimension of $$\mathfrak{g}$$ and consists of a few steps. (Note the structure of the proof is very similar to that for Lie's theorem, which concerns a solvable algebra.) The basic case is trivial and we assume the dimension of $$\mathfrak{g}$$ is positive.

Step 1: Find an ideal $$\mathfrak{h}$$ of codimension one in $$\mathfrak{g}$$.


 * This is the most difficult step. Let $$\mathfrak{h}$$ be a maximal (proper) subalgebra of $$\mathfrak{g}$$, which exists by finite-dimensionality. We claim it is an ideal of codimension one. For each $$X \in \mathfrak h$$, it is easy to check that (1) $$\operatorname{ad}(X)$$ induces a linear endomorphism $$\mathfrak{g}/\mathfrak{h} \to \mathfrak{g}/\mathfrak{h}$$ and (2) this induced map is nilpotent (in fact, $$\operatorname{ad}(X)$$ is nilpotent as $$X$$ is nilpotent; see Jordan decomposition in Lie algebras). Thus, by inductive hypothesis applied to the Lie subalgebra of $$\mathfrak{gl}(\mathfrak{g}/\mathfrak{h})$$ generated by $$\operatorname{ad}(\mathfrak{h})$$, there exists a nonzero vector v in $$\mathfrak{g}/\mathfrak{h}$$ such that $$\operatorname{ad}(X)(v) = 0$$ for each $$X \in \mathfrak{h}$$. That is to say, if $$v = [Y]$$ for some Y in $$\mathfrak{g}$$ but not in $$\mathfrak h$$, then $$[X, Y] = \operatorname{ad}(X)(Y) \in \mathfrak{h}$$ for every $$X \in \mathfrak{h}$$. But then the subspace $$\mathfrak{h}' \subset \mathfrak{g}$$ spanned by $$\mathfrak{h}$$ and Y is a Lie subalgebra in which $$\mathfrak{h}$$ is an ideal of codimension one. Hence, by maximality, $$\mathfrak{h}' = \mathfrak g$$. This proves the claim.

Step 2: Let $$W = \{ v \in V | X(v) = 0, X \in \mathfrak{h} \}$$. Then $$\mathfrak{g}$$ stabilizes W; i.e., $$X (v) \in W$$ for each $$X \in \mathfrak{g}, v \in W$$.


 * Indeed, for $$Y$$ in $$\mathfrak{g}$$ and $$X$$ in $$\mathfrak{h}$$, we have: $$X(Y(v)) = Y(X(v)) + [X, Y](v) = 0$$ since $$\mathfrak{h}$$ is an ideal and so $$[X, Y] \in \mathfrak{h}$$. Thus, $$Y(v)$$ is in W.

Step 3: Finish up the proof by finding a nonzero vector that gets killed by $$\mathfrak{g}$$.


 * Write $$\mathfrak{g} = \mathfrak{h} + L$$ where L is a one-dimensional vector subspace. Let Y be a nonzero vector in L and v a nonzero vector in W. Now, $$Y$$ is a nilpotent endomorphism (by hypothesis) and so $$Y^k(v) \ne 0, Y^{k+1}(v) = 0$$ for some k. Then $$Y^k(v)$$ is a required vector as the vector lies in W by Step 2. $$\square$$