Remove Duplicate Lines
Delete repeated lines from a list and see what was removed. Runs in your browser.
About this remove duplicate lines
This tool removes duplicate lines from any list you paste and keeps one copy of each. It's for anyone cleaning up a list: email addresses, keywords, URLs, product codes, log lines or names copied out of a spreadsheet.
Paste the list and the clean version appears next to it as you type. You decide what counts as a duplicate. Should Apple and apple match? Should a line with a stray space at the end match one without? You also choose whether to keep the first or the last copy, and what happens to blank lines.
Most removers only tell you a number. This one shows you exactly which lines it took out, with their line numbers, and how often each repeated line appeared. Your text stays in your browser the whole time.
Features
See every removed line
The Removed tab lists each deleted line with its original line number and the reason: duplicate, empty or filtered. You can check the result before you trust it.
Count how often lines repeat
The Repeats tab ranks repeated lines by how many times they appear. Handy when the duplicates themselves are the finding, such as the same error in a log.
Keep the first or the last copy
Keep first leaves each line where it first appeared. Keep last keeps the final copy instead, which is what you want when later entries are newer.
Unique only or repeated only
Show every line once, only the lines that appear exactly once, or one copy of each line that repeats.
Matching rules you control
Ignore case, trim spaces at the ends of lines, and treat runs of spaces or tabs as one space. Accented letters always match however they were typed.
Blank lines and sorting
Remove all blank lines, keep them all so paragraphs stay intact, or keep just one. Optionally sort the result A to Z with numbers in natural order (2 before 10).
Big lists, original line endings
A list of a million lines takes about a second. Windows line endings (CRLF) and a final newline are kept, so the file you download matches the one you opened.
How to use the remove duplicate lines
- 1Paste your list, one item per line, or open a text file.
- 2Choose how lines match (case, spaces), which copy to keep and what to do with empty lines.
- 3Copy or download the clean list, or check the Removed and Repeats tabs.
When to use it
Cleaning an email list
Merge two signup exports and remove repeats before an import. Turn on Ignore case, because [email protected] and [email protected] reach the same inbox with nearly every mail provider.
Keyword research
Combine keyword lists from several sources into one clean set. Then check how often those keywords appear on your page.
Reading log files
Paste a noisy log and use the Repeats tab to see which message fires most often. The counts tell you where to look first.
Spreadsheet columns
Copy one column from Excel or Google Sheets, dedupe it here and paste it back. No formulas, no extra sheet.
What counts as a duplicate line?
Two lines are duplicates when they are identical after the rules you pick. With the defaults, "cherry" and " cherry" match, because spaces at the ends of lines are ignored. "Banana" and "banana" do not match until you turn on Ignore case.
Some lines look the same on screen but are not. These are the usual suspects:
- Trailing spaces. Invisible, and very common in text copied from spreadsheets. Trim handles them.
- Non-breaking spaces. Text copied from web pages often has U+00A0 instead of a normal space. Turn on "Treat repeated spaces as one" to match them.
- Double spaces inside a line, such as "New York" and "New York". Same fix.
- Accents typed two ways. é can be one character or e plus a combining accent. The tool normalizes both to the same form (Unicode NFC), so they always match.
Does removing duplicates change the order of my list?
No, not unless you ask it to. With Keep first, every surviving line stays at the position where it first appeared. With Keep last, it moves to where its last copy was. Lines that never repeat stay exactly where they were.
If you would rather have the list in order, pick A to Z or Z to A. For more ways to order a list, such as by length or at random, sort or shuffle the cleaned lines afterwards.
How do I remove duplicate lines in Excel, Sheets or the command line?
In Excel, select the column and use Data, then Remove Duplicates. In Google Sheets, use Data, then Data cleanup, then Remove duplicates. Excel's version ignores letter case. Neither one trims spaces as part of that step, so a cell with a trailing space survives as a separate row.
On Linux or macOS, sort -u file.txt removes duplicates but also sorts the lines. To keep the original order, awk '!seen[$0]++' file.txt prints each line only the first time it appears. Neither one tells you what it removed.
Why use Mavros
| Mavros | Typical upload sites | |
|---|---|---|
| Where your list goes | Stays in your browser | Often sent to a server |
| Which lines were removed | Listed with line numbers and reasons | Only a count, or nothing |
| Repeat counts | Every repeated line, ranked | Rarely offered |
| Keep last copy | Yes | Usually first copy only |
| Unique only / repeated only | Both | Rare |
| Lookalike spaces and accents | Non-breaking spaces and Unicode forms handled | Treated as different lines |
Tips
- Check the Removed tab before you copy. If a line you need is there, change the matching rules.
- For email lists, turn on Ignore case and Trim. Mail servers treat the part after @ as case-insensitive.
- Keep blank lines set to Keep all when you clean paragraphs of prose rather than a list.
- Want to see what changed? Compare the original and the cleaned list side by side.
- Need to fix a pattern rather than whole lines, like a domain typo? Replace it across the list first, then dedupe.
Questions
How do I remove duplicate lines from text?
Paste your text into the box. The duplicates are removed as you type and the clean list appears on the right. Pick your matching options, then copy the result or download it as a .txt file.
Does it keep the original order of my lines?
Yes. Each kept line stays where its first copy was, unless you choose Keep last or a sort order. Lines that appear only once never move.
Is the comparison case-sensitive?
Yes, by default. "Apple" and "apple" are kept as two lines. Turn on Ignore case to treat them as one. The copy you keep, first or last, decides which spelling stays.
Why are some duplicates not removed?
The lines are probably not identical. Hidden differences such as non-breaking spaces, double spaces or a tab are the usual cause. Turn on "Treat repeated spaces as one" and Trim, and check the Repeats tab to see what now matches.
Can I see which lines were removed?
Yes. Open the Removed tab. It lists every deleted line with its line number and why it went: duplicate, empty, or filtered out by Unique only or Repeated only.
How do I find the duplicates instead of removing them?
Choose Repeated only to get one copy of each line that appears more than once. The Repeats tab adds how many times each one appears, most frequent first.
What happens to empty lines?
By default they are removed. You can also keep all of them, which preserves paragraph breaks, or keep only the first blank line and drop the rest.
How many lines can it handle?
Hundreds of thousands of lines work smoothly, and a million lines take about a second on a typical laptop. Very long results are previewed in part, but Copy and Download always get every line.