How to use the Remove Duplicate CSV Rows
- 1 Drop the file on the studio, or try the example.
- 2 In the step, tick the columns that define a duplicate – for a contact list, the email column. Nothing ticked compares whole rows.
- 3 Choose to keep the first, the last or none of each group, and whether to ignore case and spaces.
- 4 Check the count of removed rows and download the result.
What counts as a duplicate
Two rows are duplicates when the values in the chosen columns are the same. Comparing whole rows finds exact copies; comparing one column – an email address, an order number – finds the same entity entered twice with other details that differ. Choose which copy survives: the first or the last occurrence, or none, which removes every row that has a twin and leaves only the rows that were unique.
Near-duplicates
“Anna@Example.nl” and “anna@example.nl ” are the same address. Two switches catch this: ignore upper and lower case, and trim spaces at the start and end. They are on by default for trimming. The studio does not do fuzzy matching of misspelt names; that needs a judgement a program should not make silently.
Careful with numbers
Values are compared as text, exactly as written, so 007 and 7 are different – which is right for postcodes and ids, and the reason leading zeros survive. If you do want them equal, pad or convert the column first.
Limits
The order of the remaining rows is unchanged. Files up to 25 MB; very wide files use more memory. If the duplicates are in a list of lines rather than a table, the duplicate lines tool is simpler.
Frequently asked questions
Does it change the order of my rows?
No. The rows that remain keep the order they had.
Can I see which rows were removed?
The step shows how many rows and groups were affected. To list them, compare the original with the result using the compare tables step.
Does it work on Excel files?
Yes. Open the .xlsx (its first sheet or the one you choose) and export as Excel or CSV.
Last updated: