User:Ahazred8/dupelist

COMPARISON of duplicate file utilities. This is a list of duplicate file management utilities.

Draft List Of Utils

 * clink - replaces duplicates with symlinks
 * duff - uses fast O(n log n) comparison
 * dupedit - Compares many files at once without checksumming. Avoids comparing files against themselves when multiple paths point to the same file.
 * dupefinder -
 * duper -
 * dupmerge - runs across most platforms (Win32/64 with Cygwin, *nix, Linux etc.)
 * dupseek - Perl with algorithm optimized to reduce reads
 * epac - Erik's Partial Archive Collator - replaces duplicates with hardlinks
 * fdf - Perl/c based and runs across most platforms (Win32, *nix and probably others). Uses MD5, SHA1 and other checksum algorithms
 * fdupe - a small script written in Perl. Doing its job fast and efficiently.
 * fdupes -
 * findrepe - Java-based command-line tool; can search within zips and jars. (Linux, Mac OS X, *nix, Windows)
 * freedup - POSIX C compliant and runs across platforms (Windows with Cygwin, Linux, AIX, etc.)
 * freedups - shell script
 * fslint - has command line interface and GUI.
 * ftwin -
 * liten - Pure Python deduplication command line tool, and library, using md5 checksums and a novel byte comparison algorithm. (Linux, Mac OS X, *nix, Windows)
 * liten2 - A rewrite of the original Liten, still a command line tool but with a faster interactive mode using SHA-1 checksums (Linux, Mac OS X, *nix)
 * rdfind - One of the few which rank duplicates based on the order of input parameters (directories to scan) in order not to delete in "original/well known" sources (if multiple directories are given). Uses MD5 or SHA1. uses fast O(n log n) comparison
 * rmlint - Fast finder with command line interface and many options to find other lint too (uses MD5)
 * ssdeep - identify almost identical files using Context Triggered Piecewise Hashing
 * ua - Unix/Linux command line tool, designed to work with find (and the like).

Related: duplicate image match finders Related: data deduplication