Remove Duplicate Lines Online

When duplicate lines sneak in

Merged spreadsheets, combined mailing lists, log files tailed twice, and copy-pasted data from multiple sources all tend to accumulate exact repeats. A quick visual scan rarely catches every one, especially once a list runs past a screen's worth of lines. This tool compares every line against every other line and keeps just one copy of each.

By default it preserves your original ordering — the first time a line dedupes it stays where it was, rather than the whole list being resorted — so a deduplicated list is a strict subset of what you pasted in, easy to diff against the original if you need to check what was removed.

Example

Input:

apple
banana
Apple
cherry
banana
date

With Ignore letter case on, the result is:

apple
banana
cherry
date

“Apple” was treated as a duplicate of “apple” and dropped; without that option, both would survive as distinct lines.

Doing the same thing on the command line

sort -u file.txt                 # sorted, unique lines
awk '!seen[$0]++' file.txt       # unique lines, original order preserved

Remove Duplicate Lines FAQ

Is this the same as “sort unique” on the command line?

It does the same job as sort -u, but keeps your original order by default instead of always sorting, and adds options Unix tools don’t: ignore-case matching, trimming before comparing, and choosing whether the first or last copy of a repeated line survives.

What does “Keep only lines that had duplicates” do?

Instead of removing duplicates, it flips the filter around and shows you only the lines that appeared more than once — one copy of each — which is useful for spotting exactly what was duplicated in a large export.

Does trimming change the lines in the output, or just the comparison?

Both — when “Trim spaces before comparing” is on, the output lines are trimmed too, so “apple” and “ apple ” are treated as the same line and the surviving copy has no stray spaces.