Open Source BSD Duplicate File Finders

Create zip files, extract zip files, replace text in files, search in files using expressions, stream text editor, instant command line ftp and http server, send folder via network, copy folder excluding sub folders and files, find duplicate files, run a command on all files of a folder, split and join large files, make md5 checksum lists of files, remove tab characters, convert CR/LF, list newest or biggest files of a folder, compare folders, treesize, show first or last lines of a file, find filenames fast using index files, rename many files using expressions, copy part of a file, change times of a file, set file time from filename, print colored text to terminal, convert csv to tab separated, download files from web, send http or udp requests, print tcp or udp traffic, create hexdump of files, join many text files into one, list nested .zip .tar .tar.gz .tar.bz2 archive contents. Fully portable, for USB stick, without installation. For Windows, Linux and Mac OS/X.

FDUPES is a lightweight command-line utility that helps users find and optionally delete duplicate files within specified directories by comparing file contents, which can be extremely useful for cleaning up storage clutter or organizing large collections of files. It works by scanning directories and subdirectories, identifying sets of files with identical content through size and hash comparisons, and then listing them together so users can examine duplicates. Once duplicates are identified, the tool offers interactive deletion options where users can choose which copy to keep and which to remove, or apply flags to automate deletion while preserving a single instance. Because it operates directly on file content rather than just filenames, fdupes can accurately detect true copies and guide cleaning operations in data cleanup or migration tasks. It’s a simple, efficient, and widely used utility on Unix-like systems, appreciated by administrators, developers, and power users.

Duplicate Files Finder is a cross-platform application for finding and removing duplicate files by deleting, creating hardlinks or creating symbolic links. A special algorithm minimizes the amount of data read from disk, so the program is very fast.

The littleutils include duplicate file finders (repeats, repeats.pl, repeats.py), image optimizers (opt-jpg, opt-png, opt-gif, recomp-jpg), file rename tools (lowercase, uppercase, pren), archive recompressors (to-gzip, to-bzip2, to-bzip3, to-7zip, to-lzma, to-lzip, to-xz), a tempfile utility (tempname), file property tools (filedate, filemode, filenode, fileown, filesize, and lrealpath), and others. See the README file for more details.

(AYDUFF)a yet duplicate file finder.

This software finds duplicate files across all directory levels inside a given folder.

HashCatalog is a program that can find duplicate files in one or more folders, or between 2 lists of one or more folders. HashCatalog can also create an xml database file containing a listing of files with hashes.

JustOneFile help you to find duplicate files and managed them. It's a Free and Open Source software under GNU GPL licence.

Synclair is a tool for diff'ing and synchronizing directories. It also includes a utility to find duplicate files (by name, or md5 hash). It is a command line tool, written in Java.

An easy to use UI which will recursively search a specified location for duplicate files

find duplicate files on your harddisk very fast by comparing only needed files

A Linux/UNIX command line tool to find duplicate files. The tool supports multiple ways to consider a set of files as duplicates to each other (e.g. content, size, name ... etc.).

Simple GUI based lightweight duplicate file finder and remover. Configurable file search and preview (via external tools) option for file data. No command-line interface available! This is basically a reimplementation of dupfinder as: it doesn't compile on my computer, the executable won't work either, the projects seems dead and I didn't wanna work through the original code. So Look'n'Feel should be somewhat the same, though some changes have been made. Algorithm is simple and suitable for large file sets with few differences: basic set of duplicate candidates is determined by file size, then candidates are compared byte-wise. So for files which don't start to differ by growing at the tail it reduces read operations greatly compared to hash-based comparision (i.e. test case: 40GB of files, 16GB RAM, many differences: initial compare (uncached by OS) around 300sec, subsequent compare (relevant file parts cached by OS) ca. 2sec) Note: this program is not related to lsdup

dlFindDuplicates finds duplicate files in selected directories, based on filelength and MD5 hashes. Duplicate files then can be selected via a lua-based scripting interface. Selected duplicates can be deleted, converted to a hardlink or moved away. dlFindDuplicates explores multithreading in order to be fast.