Resolve Duplicate Entities

September 2026: the operations on this page (classify, dedupe, merge, rank) are deprecated and will be removed. They still work for now.

Consolidating records where the same entity appears under different spellings, abbreviations, and nicknames. Here, 500 messy CRM company records are resolved down to their unique entities -- handling typos, ticker symbols, and brand variants that string similarity misses entirely.

claude mcp add futuresearch --scope project --transport http https://mcp.futuresearch.ai/mcp

Deduplicate this CRM dataset. Two entries are duplicates if they represent
the same company.

Tool: futuresearch_dedupe
├─ equivalence_relation: "Two entries are duplicates if they represent the same company."
└─ input_csv: "/Users/you/case_01_crm_data.csv"

→ Submitted: 500 rows for deduplication.
  Task ID: d5d1...

Tool: futuresearch_progress
├─ task_id: "d5d1..."
→ Running: 0/500 complete (30s elapsed)

...

Tool: futuresearch_progress
→ Completed: 500/500 (0 failed) in 920s.

Tool: futuresearch_results
├─ task_id: "d5d1..."
├─ output_path: "/Users/you/crm_deduplicated.csv"
→ Saved 500 rows to /Users/you/crm_deduplicated.csv

pip install futuresearch
export FUTURESEARCH_API_KEY=your_key_here  # Get one at futuresearch.ai

import asyncio
import pandas as pd
from futuresearch.ops import dedupe

data = pd.read_csv("case_01_crm_data.csv").fillna("")

async def main():
    result = await dedupe(
        input=data,
        equivalence_relation="Two entries are duplicates if they represent the same company.",
    )

    # Filter to keep only the best record per entity
    unique = result.data[result.data["selected"] == True]
    print(f"Reduced {len(data)} records to {len(unique)} unique entities")

asyncio.run(main())

Results

The input data contains variations like these, all representing the same company:

The system clusters these into a single entity and selects the most complete record. The output includes equivalence_class_id and selected columns. Filter to selected == True to get one record per entity.

The system handles cases that string similarity misses entirely. "AAPL" matches to "Apple Inc." because the model knows the ticker symbol. "Big Blue" matches to "IBM Corporation" because that's IBM's nickname. "W-Mart" and "Wallmart" match to "Walmart Inc." despite having different typos.

The equivalence relation is flexible. For matching people: "Two entries are duplicates if they refer to the same person, accounting for name variations and nicknames." For products: "Two entries represent the same product if they're the same item sold under different names or SKUs."

See the full notebook for additional examples including how to merge the clustered records into consolidated entries.

Built with FutureSearch. See the dedupe documentation for more options.