Inclination instability

An inclination instability is a dynamical instability that can occur in a disk of objects with eccentric orbits, causing it to form into a conical shape. The gravity of the objects causes an exponential growth of their inclinations while reducing their eccentricities. The inclination instability also results in a clustering of the arguments of perihelion of the objects orbits, similar to what has been observed among the extreme trans-Neptunian objects with semi-major axes greater than 150 AU, it does not produce an alignment of the longitudes of perihelion, however. For an inclination instability to be responsible for the observed clustering, a disk with a mass of 1-10 Earth masses must have existed for over a billion years. This is more than is estimated from current observations, and longer than the timescale of the depletion of the planetesimal disk in models of the early Solar System.

Dynamics of the inclination instability
In a flat disk of objects with eccentric orbits a small initial vertical perturbation is amplified by the inclination instability. The initial perturbation exerts a vertical force. On very long timescales relative to the period of an object's orbit this force produces a net torque on the orbit due to the object spending more time near aphelion. This torque causes the plane of the orbit to roll on its major axis. In a disk this results in the orbits rolling with respect to each other so that the orbits are no longer co-planar. The gravity of the objects now exerts forces on each other that are out of planes of their orbits. Unlike the force due to the initial perturbation these forces are in opposite directions, up and down respectively, on the inbound and outbound portions of their orbits. The resulting torque causes their orbits to rotate about their minor axes, lifting their aphelia, causing the disk to form a cone. The angular momentum of the orbit is also increased due to this torque resulting in reduction of the eccentricity of the orbits. The inclination instability requires an initial eccentricity of 0.6 or larger, and saturates when inclinations reach ~1 radian, after which orbits precess due to the gravity toward the cone's axis of symmetry.