dedupe
dedupe is an open-source Python library for fuzzy matching, deduplication, and entity resolution on structured data.
What dedupe is
dedupe is an open-source Python library for fuzzy matching, deduplication, and entity resolution on structured data. It learns to identify similar records from human-labeled examples, including in very large databases. The project describes uses such as removing duplicate spreadsheet entries, linking customer and order records without unique IDs, and identifying people across campaign contribution records. Developers can run the library on local machines or servers and integrate it into applications or import scripts to build self-hosted entity-resolution pipelines. Its documented features include laptop execution, extensible data types and comparators, and learning from labeled data. Installation is documented with `pip install dedupe`; the separate csvdedupe command-line tool supports CSV deduplication. The project directs bug reports to GitHub Issues and lists a mailing list for the library. Dedupe.io, the related web service, shut down on January 31, 2023, and its team now focuses on consulting through DataMade. Its former service and API workflows are no longer current options.
Good
- Learns matching rules from human-labeled examples.
- Can run locally or on servers.
- Integrates into applications and import scripts.
- Open-source library with documented installation.
Watch out
- Dedupe.io shut down on January 31, 2023.
- The former web service and API are no longer current options.
Who it suits
dedupe suits Python developers working with structured records who need fuzzy matching or entity resolution. It may also suit teams that want to build self-hosted pipelines or integrate matching into applications and import scripts.
Verdict
dedupe is a library for developers who want to build fuzzy matching and entity-resolution workflows around structured data. The related Dedupe.io service has shut down, so the current option described here is the open-source library.
The facts
Each one read from dedupe's own pages, with the date it was checked.
What it does
- API
- The former Dedupe.io web API supported matching and training against projects through REST calls, including incremental checks, matches and additions after deduplication.checked 4 Oct 2026
- Compliance
- The former service's FAQ states that Dedupe.io was certified FERPA compliant as of July 2019.checked 10 Oct 2026
- Data formats
- The former service’s FAQ says uploads accepted Excel spreadsheets and CSV files, with one record per row and column names in the first row.checked 4 Oct 2026
- Data privacy
- The former service's FAQ says it would not rent or sell submitted data and would not allow third-party access without signed consent.checked 10 Oct 2026
- Features
- Documented features include learned weights and blocking rules, laptop execution, library integration, and support for custom data types, comparators, and blocking rules.checked 10 Oct 2026
- How it learns
- It uses human-labeled training data to learn rules for finding similar records, including in large databases.checked 10 Oct 2026
- Input formats
- The former Dedupe.io service accepted spreadsheet files (.xls, .xslx) and comma-separated text (.csv).checked 10 Oct 2026
- Installation
- The documentation gives `pip install dedupe` as the installation command.checked 10 Oct 2026
- Integrations
- The project describes dedupe as a library that can be integrated into applications or import scripts; it also documents CSV deduplication through the separate csvdedupe command-line tool.checked 4 Oct 2026
- Intended users
- The maintainers describe the library as suitable for integration into applications or import scripts, while the former Dedupe.io service was designed as an online tool for non-developers.checked 10 Oct 2026
- License
- The GitHub repository identifies the project as MIT licensed.checked 10 Oct 2026
- Open source library
- dedupe is a Python library that uses machine learning for fuzzy matching, deduplication and entity resolution on structured data.checked 4 Oct 2026
- Privacy
- The former service’s FAQ says its privacy policy prohibited renting or selling user data and third-party access without signed consent, and limited company access to debugging and service improvement.checked 4 Oct 2026
- Product status
- Dedupe.io, the hosted service, shut down on January 31, 2023.checked 10 Oct 2026
- Self-hosting
- The developer page says Python developers can run the library on local machines or servers and build self-hosted entity-resolution pipelines.checked 4 Oct 2026
- Support
- The library documentation directs bug reports to GitHub Issues and lists a mailing list for users.checked 10 Oct 2026
- Training
- The library uses human-labeled examples to learn rules for finding similar records, including in very large databases.checked 4 Oct 2026
- Training limit
- The former service required at least 10 positive and 10 negative record-pair examples during training.checked 4 Oct 2026
- Training requirement
- The former Dedupe.io service required at least 10 positive and 10 negative record-pair examples during model training.checked 10 Oct 2026
- Use cases
- The library can remove duplicate spreadsheet entries and link customer, order history, or campaign contribution records without relying on unique IDs.checked 10 Oct 2026
- Web service workflow
- The former Dedupe.io service described a workflow to upload data, train a model, review automatically found matches, and download results.checked 4 Oct 2026
- What it does
- The dedupe Python library uses machine learning for fuzzy matching, deduplication, and entity resolution on structured data.checked 10 Oct 2026
The company
- Headquarters
- Chicago, Illinois, United Stateschecked 28 Sep 2026
Alternatives to dedupe
- 1
Zingg
Free planOpen sourceAPILinuxmacOSSelf-hosted7.4Free plan - 2
Tilores
Free planWebAPILinuxmacOS7.2Free plan - 3
Workbooks
AndroidiOSWeb6.9From $47 a month - 4
DataMatch Enterprise
WebAPILinuxSelf-hosted6.8 - 5
Senzing Entity Resolution
WebAPILinuxmacOS6.7From $58,560 a year - 6
DupeCatcher
Free planAndroidiOSWeb6.6Free plan