How to dedupe a list (emails, names, strings)

The short answer. Paste your list into the list alphabetizer with Dedupe turned on — one click, deduped and sorted. In Excel, Data → Remove Duplicates, or =UNIQUE(A2:A100) on Excel 365. In Google Sheets, identical: =UNIQUE(A2:A100). On the command line, sort -u file.txt for dedupe-with-sort, or awk '!seen[$0]++' file.txt for dedupe that keeps the original order. The catch most people hit: deciding what counts as a duplicate — the table at the bottom maps your situation to the right method, and the next section unpacks the comparison rules that quietly differ between tools.

What counts as a duplicate? (three quiet differences)

"Remove duplicates" sounds obvious until two tools give you different counts on the same input. Three rules vary tool to tool, and the answer changes depending on which combination you get:

• Case-sensitivity. Does Alice match alice? Excel's Remove Duplicates is case-insensitive by default; sort -u is case-sensitive; the joinlines alphabetizer follows its Case-sensitive toggle.
• Whitespace trimming. Does "alice " (trailing space) match "alice"? Most tools don't trim before comparison — making whitespace one of the biggest sources of "I deduped but still have duplicates" complaints. The browser tool trims by default; sort -u does not.
• Diacritics. Does café match cafe? Most tools say no; some (with Unicode normalisation) say yes. The compare tool normalises diacritics in case-insensitive mode; most spreadsheets and CLI tools don't.

Pick a tool whose rules match your intent. When in doubt, normalise the data first (lowercase + trim) and the comparison rules stop mattering.

How do you dedupe a list online, instantly?

Open the list alphabetizer. Paste your list. Dedupe is on by default; pick A→Z or another sort mode. Copy the result.

• Dedupe + sort in one pass. If you also want the output alphabetised, this is the fastest path — one tool, one click. We use the same single Intl.Collator pass behind the scenes.
• Honest limitation: this tool always sorts. There's no "dedupe only, keep original order" mode — the alphabetizer is built around the sort step. For order-preserving dedupe, use the awk recipe in the command-line section below.
• Trim and case toggles. Both toggles control comparison: with Trim on (default), trailing whitespace is ignored. With Case-sensitive off (default), Alice and alice collapse to one row.
• Nothing uploads. Open DevTools → Network and run a dedupe; zero outbound requests carrying your list.

How do you dedupe a list in Excel?

Two clean paths, depending on whether you want to destroy or preserve the source column.

Remove Duplicates (destructive, in place).

1. Select the column.
2. Data → Remove Duplicates.
3. Confirm the column scope; click OK.

Excel keeps the first occurrence of each value and removes later ones. Comparison is case-insensitive by default and there's no toggle to change it — if case matters, the formula route is better.

UNIQUE formula (non-destructive, Excel 365 / 2021).

=UNIQUE(A2:A100)

The result spills into a fresh range; the source column is untouched. For case-sensitive comparison, pass TRUE as the third argument:

=UNIQUE(A2:A100)                  // all distinct values (default)
=UNIQUE(A2:A100, FALSE, TRUE)     // values that appear exactly once only

Heads up: the third argument controls exactly-once filtering, not case. UNIQUE itself compares case-insensitively, like most Excel text operations — Alice and alice collapse to one row. For forced case-sensitive dedupe, pre-process with a helper column like =A1 turned into something case-aware (Excel has no clean built-in), or use the command-line route. UNIQUE is unavailable in Excel 2019 and older — use Remove Duplicates or Advanced Filter instead.

Google Sheets: identical syntax — =UNIQUE(A2:A100). Works on every Sheets version.

How do you dedupe in Notepad++ or VS Code?

Text editors handle one-off dedupes faster than a spreadsheet for plain-text lists.

Notepad++

1. Select the lines (Ctrl+A for the whole file).
2. Edit → Line Operations → Remove Consecutive Duplicate Lines — removes duplicates that are next to each other only.
3. For non-consecutive duplicates, first Edit → Line Operations → Sort Lines Lexicographically Ascending, then run the dedupe.

If you have the TextFX plugin installed (no longer bundled with Notepad++ by default, but still available via Plugins Admin on older builds), TextFX Tools → Sort lines case-insensitive and remove duplicates does the same in one step.

VS Code

1. Install the popular Sort Lines extension by Tyriar (most VS Code users already have it).
2. Select the lines (Ctrl+A).
3. Open the Command Palette (Ctrl+Shift+P) and run Sort Lines (Unique).

Without the extension, VS Code has no built-in dedupe command — you'd have to fall back to a regex Find & Replace, which is fiddly for this task.

How do you dedupe on the command line?

Two recipes cover almost every case, and the choice between them is about whether you want sorted output.

sort -u — dedupe with sort.

sort -u file.txt

Combines a sort and a unique-filter in one pass. The output is alphabetically sorted and deduped. Add -f for case-insensitive: sort -fu file.txt. The shortest path when you don't care about original order.

awk '!seen[$0]++' — dedupe keeping original order.

awk '!seen[$0]++' file.txt

The idiomatic Unix "dedupe in original order" one-liner. seen[$0] is an associative-array lookup keyed by the line; ++ increments after the check, so the first time a line is seen the test returns true (and the line prints), every subsequent time it returns false (and the line is skipped). Eight characters of expression. The Unix recipe that everyone eventually memorises.

For case-insensitive order-preserving dedupe:

awk '!seen[tolower($0)]++' file.txt

On Windows in PowerShell:

Get-Content file.txt | Select-Object -Unique | Set-Content out.txt

Select-Object -Unique preserves the input order, unlike Sort-Object -Unique which sorts as a side effect.

Which method should you use?

Frequently asked questions

Why does my dedupe leave some duplicates behind?

Almost always whitespace or case differences. "alice" and "alice " (with a trailing space) are different strings to most tools. Normalise first: awk '{gsub(/[[:space:]]+$/,""); print}' file.txt | awk '!seen[$0]++' trims trailing whitespace before deduping. In Excel, wrap with =TRIM(LOWER(A2)) in a helper column, then dedupe that. In the browser tool, both Trim and Case-sensitive toggles are wired into the comparison — defaults sidestep the gotcha.

How do I dedupe but keep the first occurrence?

Most order-preserving methods keep the first occurrence by design. awk '!seen[$0]++' does. Excel's Remove Duplicates does. Select-Object -Unique in PowerShell does. The exception is anything sort-based (sort -u, the alphabetizer with sort modes) — the "first" disappears because the order changes. If "keep first" matters, use a preserve-order method.

How do I dedupe by one column when the data has multiple columns?

This is no longer a list-dedupe; it's a row-dedupe by key. In Excel, Data → Remove Duplicates lets you pick which columns to compare. In SQL, SELECT DISTINCT column_name FROM .... In pandas, df.drop_duplicates(subset=["email"]). On the command line for a CSV: awk -F, '!seen[$3]++' file.csv (dedupes by the 3rd column). Plain-line dedupers don't help — they compare whole lines.

What if I want to know how many duplicates there were?

wc -l file.txt gives the original count; sort -u file.txt | wc -l gives the deduped count; subtract for the duplicate count. For a per-value count of repeats: sort file.txt | uniq -c | sort -rn — prints "count value" for every unique line, sorted by frequency descending. In the line counter, paste the deduped vs the original and compare.

Can I dedupe across multiple files at once?

On the command line: cat *.txt | sort -u — concatenates all the files, then sort-dedupes the combined stream. To keep original order across the union: cat *.txt | awk '!seen[$0]++'. For Excel or a browser tool, copy the columns together first.

Is it safe to paste a sensitive list into an online deduper?

Only if the tool runs entirely in the browser. Open DevTools, switch to the Network tab, paste your list, run the dedupe, watch for outbound requests. If any carry your data, the page sent it to a server. The joinlines tools make zero requests after page load — verifiable in DevTools.

— the joinlines team, 2026-05-06