Shanks's square forms factorization

Shanks' square forms factorization is a method for integer factorization devised by Daniel Shanks as an improvement on Fermat's factorization method.

The success of Fermat's method depends on finding integers $$x$$ and $$y$$ such that $$x^2-y^2=N$$, where $$N$$ is the integer to be factored. An improvement (noticed by Kraitchik) is to look for integers $$x$$ and $$y$$ such that $$x^2\equiv y^2\pmod{N}$$. Finding a suitable pair $$(x, y)$$ does not guarantee a factorization of $$N$$, but it implies that $$N$$ is a factor of $$x^2-y^2=(x-y)(x+y)$$, and there is a good chance that the prime divisors of $$N$$ are distributed between these two factors, so that calculation of the greatest common divisor of $$N$$ and $$x-y$$ will give a non-trivial factor of $$N$$.

A practical algorithm for finding pairs $$(x,y)$$ which satisfy $$x^2\equiv y^2\pmod{N}$$ was developed by Shanks, who named it Square Forms Factorization or SQUFOF. The algorithm can be expressed in terms of continued fractions or in terms of quadratic forms. Although there are now much more efficient factorization methods available, SQUFOF has the advantage that it is small enough to be implemented on a programmable calculator. Shanks programmed it on an HP-65, made in 1974, which has storage for only nine digit numbers and allows only 100 steps/keystrokes of programming. There are versions of the algorithm that use little memory and versions that store a list of values that run more quickly.

In 1858, the Czech mathematician Václav Šimerka used a method similar to SQUFOF to factor $$(10^{17} - 1)/9$$ $$=$$ $$11111111111111111$$ $$=$$ $$2071723 \cdot 5363222357$$.

Algorithm
Note This version of the algorithm works on some examples but often gets stuck in a loop.

This version does not use a list.

Input: $$N$$, the integer to be factored, which must be neither a prime number nor a perfect square, and a small positive integer, $$k$$.

Output: a non-trivial factor of $$N$$.

The algorithm:

Initialize $$i=0,P_0=\lfloor\sqrt{kN}\rfloor,Q_{-1}=1,Q_0=kN-P_0^2.$$

Repeat

$$i=i+1,b_i=\left\lfloor\frac{P_0+P_{i-1}}{Q_{i-1}}\right\rfloor,P_i=b_iQ_{i-1}-P_{i-1},Q_i=Q_{i-2}+b_i(P_{i-1}-P_i)$$

until $$Q_i$$ is a perfect square at some odd value of $$i$$.

Start the second phase (reverse cycle).

Initialize $$b_0=\left\lfloor\frac{P_0-P_i}{\sqrt{Q_i}}\right\rfloor$$, $$Q_{-1}=\sqrt{Q_i}$$, and $$P_0=b_0\sqrt{Q_i}+P_i$$, where $$P_0, P_i$$, and $$Q_{i}$$ are from the previous phase. The $$b_0$$ used in the calculation of $$P_0$$ is the recently calculated value of $$b_0$$.

Set $$ i=0$$ and $$ Q_0=\frac{kN-P_0^2}{Q_{-1}}$$, where $$P_0$$ is the recently calculated value of $$P_0$$.

Repeat

$$i=i+1,b_i=\left\lfloor\frac{P_0+P_{i-1}}{Q_{i-1}}\right\rfloor,P_i=b_iQ_{i-1}-P_{i-1},Q_i=Q_{i-2}+b_i(P_{i-1}-P_i)$$

until $$P_i=P_{i-1}.$$

Then if $$f=\gcd(N,P_i)$$ is not equal to $$1$$ and not equal to $$N$$, then $$f$$ is a non-trivial factor of $$N$$. Otherwise try another value of $$k$$.

Shanks' method has time complexity $$O(\sqrt[4]{N})$$.

Stephen S. McMath wrote a more detailed discussion of the mathematics of Shanks' method, together with a proof of its correctness.

Example
Let $$N = 11111$$

$$Q_{-1}=1$$

Here $$Q_{5} = 25$$ is a perfect square, so the first phase ends.

For the second phase, set $$Q_{-1} = \sqrt{25} = 5$$. Then:

Here $$P_{3} = P_{4} = 82$$, so the second phase ends. Now calculate $$gcd(11111, 82) = 41$$, which is a factor of $$11111$$.

Thus, $$N = 11111 = 41 \cdot 271$$.

Example implementation
Below is an example of C function for performing SQUFOF factorization on unsigned integer not larger than 64 bits, without overflow of the transient operations.