Dedupe

dedupe groups duplicate rows in a DataFrame based on a natural-language equivalence relation, assigns cluster IDs, and selects a canonical row per cluster. The duplicate criterion is semantic and LLM-powered. This handles abbreviations, name variations, job changes, and entity relationships that no string similarity threshold can capture.

Examples

Every operation is a coroutine. Run it with asyncio.run() as below, or await it directly in a Jupyter notebook. Later snippets on this page omit the wrapper for brevity.

import asyncio

from pandas import DataFrame
from futuresearch.ops import dedupe

crm_data = DataFrame([
    {"name": "Acme Corp"},
    {"name": "ACME Corporation"},
    {"name": "Globex"},
])


async def main():
    result = await dedupe(
        input=crm_data,
        equivalence_relation="Two entries are duplicates if they represent the same legal entity",
    )
    print(result.data.head())


asyncio.run(main())

The equivalence_relation is natural language. Be as specific as you need:

result = await dedupe(
    input=researchers,
    equivalence_relation="""
        Two rows are duplicates if they're the same person, even if:
        - They changed jobs (different org/email)
        - Name is abbreviated (A. Smith vs Alex Smith)
        - There are typos (Naomi vs Namoi)
        - They use a nickname (Bob vs Robert)
    """,
)
print(result.data.head())

Strategies

Control what happens after clusters are identified using the strategy parameter:

select (default)

Picks the best representative from each cluster. Three columns are added:

• equivalence_class_id: rows with the same ID are duplicates of each other
• equivalence_class_name: human-readable label for the cluster
• selected: True for the canonical record in each cluster

result = await dedupe(
    input=crm_data,
    equivalence_relation="Same legal entity",
    strategy="select",
    strategy_prompt="Prefer the record with the most complete contact information",
)
deduped = result.data[result.data["selected"] == True]

identify

Cluster only, with no selection or combining. Useful when you want to review clusters before deciding what to do.

• equivalence_class_id: rows with the same ID are duplicates of each other
• equivalence_class_name: human-readable label for the cluster

result = await dedupe(
    input=crm_data,
    equivalence_relation="Same legal entity",
    strategy="identify",
)

combine

Synthesizes a single combined row per cluster, merging the best information from all duplicates. Original rows are marked selected=False, and new combined rows are added with selected=True.

result = await dedupe(
    input=crm_data,
    equivalence_relation="Same legal entity",
    strategy="combine",
    strategy_prompt="For each field, keep the most recent and complete value",
)
combined = result.data[result.data["selected"] == True]

What you get back

Three columns added to your data (when using select or combine strategy):

• equivalence_class_id: rows with the same ID are duplicates of each other
• equivalence_class_name: human-readable label for the cluster ("Alexandra Butoi", "Naomi Saphra", etc.)
• selected: True for the canonical record in each cluster (usually the most complete one)

To get just the deduplicated rows:

deduped = result.data[result.data["selected"] == True]

Example

Input:

Output (selected rows only):

Parameters

Performance

Via MCP

MCP tool: futuresearch_dedupe

Related docs

Blog posts

• CRM Deduplication
• Researcher Deduplication