
Duplicate Check: Finding and Preventing Duplicate Records
By Matthias Mut in Data Management — October 15, 2026
CEO & Datenstrategie - Matthias Mut
Dublettenprüfung
Datenqualität
Stammdaten
CRM
What Is a Duplicate Check?
A duplicate is a redundant record: two or more entries that describe the same real-world object – the same customer, the same supplier, the same invoice. A duplicate check looks for such multiple entries, assesses whether they really refer to the same object, and merges them or prevents them from arising in the first place. Cleaning up the duplicates found is also called deduplication.
The hard part: real duplicates are rarely exactly the same. "Müller GmbH, Hauptstr. 5" and "Mueller GmbH, Hauptstraße 5" mean the same company, but to a simple database query they are two different customers. A good duplicate check therefore works with fuzzy matching instead of exact matches. It is an effective lever for improving data quality.
How Do Duplicates Arise?
- Manual entry: a customer is created anew because nobody finds the existing record under the usual spelling.
- Imports: trade fair leads, purchased address lists or web forms land in the CRM unchecked.
- Several systems: CRM, ERP and accounting maintain the same customers separately. Every transfer without matching creates new duplicates.
- Changes: companies rename, move or are acquired – the old record stays, a new one is added.
- Migrations: when moving to a new system, datasets are combined without matching them first.
What Duplicates Cost
Duplicates seem minor but have far-reaching effects:
- Double work and mail: customers receive offers, mailings and catalogs twice, and sales approaches the same company twice.
- Wrong figures: revenue per customer, customer count and credit limits are wrong when one customer is spread across several records.
- Double payments: an incoming invoice recorded twice quickly leads to a double transfer.
- Data protection risks: if a person objects to direct marketing, their data may no longer be used for it [1]. If the objection is attached to only one of two duplicates, the next mailing goes out anyway. A complete access request under Art. 15 GDPR [2] also becomes difficult when a person's data is scattered across several records.
How a Duplicate Check Works
A duplicate check first standardizes the data, then assesses the similarity of possible pairs and finally classifies the matches. Several methods exist for the comparison, and they are usually combined:
| Method | What it detects | Example |
|---|---|---|
| Exact matching | identical values | the same email address twice |
| Normalization | spelling variants based on fixed rules | "Hauptstr." becomes "Hauptstraße", legal forms are written uniformly |
| Phonetic matching | names that sound alike | Meier, Maier, Mayer and Meyer produce the same code with Cologne phonetics [3] |
| Similarity measures | typos and transposed letters | "Schmidt" and "Schmitd" differ in only two characters |
| Identifiers | very strong indicators | same VAT ID, IBAN or commercial register number including the register court |
| AI-assisted matching | similarity across many fields at once | the same company despite a new name and different contacts |
Even matching identifiers do not prove a duplicate on their own: branches of the same company usually share a VAT ID and commercial register number but are often deliberately kept separate, and with factoring or a central group payment office, several partners have the same IBAN.
The individual comparisons produce a score. If it is above an upper threshold, the pair counts as a definite duplicate. If it falls into a gray zone, a person decides. Below that, it is not a match. The thresholds determine the result: if they are too strict, duplicates remain undetected. If they are too loose, different customers are wrongly merged. That is why every duplicate check should start with a sample on which the rules are fine-tuned.
Checking at Entry and in the Existing Data
A duplicate check can run at two points:
- At entry: when a record is created, the system checks whether it already exists and suggests the existing one. This prevents new duplicates at the source.
- In the existing data: a comparison across all records finds the duplicates that have accumulated over the years. It makes sense before migrations, after imports and as a regular check.
Both belong together: without a check at entry, the cleaned data fills up again. Without cleaning the existing data, the old clutter remains.

Deduplication: Merge Instead of Delete
Anyone who simply deletes duplicates loses information: orders, invoices, notes and contacts are often attached to both records. The better way is merging:
- Determine the leading record: which entry stays? Usually the most complete one or the one from the leading system.
- Merge values: for each field, define which value wins – for example the most recent address or the verified bank details. This creates the golden record.
- Move links: orders, invoices, activities and contacts move to the leading record.
- Retire and log the duplicate: the surplus record is archived or marked as a duplicate, and the merge is documented. If it contains personal data, the retention periods of your deletion policy apply to the archive.
Our article on master data management describes how the golden record emerges. For large datasets, our data consolidation for master data takes care of this.
Duplicate Checks in CRM, ERP and Accounting
- CRM – customers and contacts: in our experience the most common problem area. It is usually about checking addresses for duplicates: company name, street, postal code and city are standardized and matched, and for people also name and email address.
- ERP – suppliers and items: duplicate suppliers distort purchasing volumes and conditions, duplicate items lead to double stock and orders.
- Accounting – incoming invoices: same supplier, same invoice number, same amount. A check before payment approval prevents double transfers.
Many CRM and ERP systems already include a duplicate check. Check whether it is active and which fields it compares. Our guide to improving data quality shows which other software groups help.
Five Steps to a Duplicate-Free Database
- Choose a data type: start with the data type whose duplicates cause the most trouble – usually customer data.
- Define rules: what counts as a duplicate? Which fields are compared, which thresholds apply?
- Check the existing data: merge definite matches automatically, decide borderline cases by hand.
- Merge: build the golden record, move links, retire duplicates.
- Secure entry and measure: activate the check on creation and measure the duplicate rate regularly – this shows whether new duplicates are growing back.
The duplicate rate is one of the metrics that data quality management works with.
How many duplicates are hiding in your data? Our data hygiene audit checks your databases for duplicates and inconsistencies and delivers a prioritized cleanup guide. The fastest way is a direct conversation: book a 30-minute intro call.
Frequently Asked Questions About Duplicate Checks
What Is the Difference Between a Duplicate and an Exact Copy?
In everyday use, the terms are often used interchangeably. Where they are distinguished, an exact copy means an identical record, while a duplicate means any multiple entry of the same object – even if the spelling or individual details differ.
What Is a Duplicate Check for Addresses?
It checks addresses for multiple entries. Because the same address can be written in different ways, street, house number, postal code, city and company name are first standardized and then compared with fuzzy matching. It is typical in CRM systems and before mailings.
Can You Simply Delete Duplicates?
Better not. Orders, invoices and notes are often attached to duplicates. Merge the records and archive the surplus one – this way nothing is lost and the history remains traceable. The retention periods of your deletion policy apply to archived personal data.
How Often Should You Run a Duplicate Check?
At entry continuously; in the existing data before migrations, after larger imports and as a regular check via the duplicate rate.
Conclusion
Duplicates arise in almost every company – through data entry, imports, renamings and system changes. What matters is finding them systematically, merging them cleanly and preventing them from growing back with a check at entry. Anyone who starts with customer data, sets clear rules and measures the duplicate rate regularly can often improve the situation noticeably within a few weeks – and creates a reliable basis for sales, accounting and AI.
References
Let's talk
Stay in touch with us
Whether you have a specific project or just want to explore options — we look forward to hearing from you.