AI Matching Engine

Challenge

Technical and C-level stakeholders were dealing with the consequences of 200,000 duplicate and incomplete records across several key units of the company, making this a critical issue that had to be dealt with from the ground up.

Design

A Spark-based matching engine combined with human-in-the-loop review. The system normalizes messy fields, uses blocking keys to avoid brute-force comparisons, applies weighted scoring across name, address, phone, and geo signals, and then buckets outcomes into Correct, Manual Review, or Incorrect for operational follow-through.

This solution was successfully implemented in partnership with Databris and their top-notch data engineers, who made data readiness possible. See more results in this post.

Implementation Highlights

Blocking keys reduce comparisons by grouping records on shared phone and ZIP fragments.
Weighted scoring and validation rules surface high-confidence matches and isolate ambiguous ones for review.
Configuration-driven mappings allow source schema changes without rewriting core matching logic.

Interactive Architecture

Entity Matching Flow

Click each stage to inspect its role in the production pipeline.

Data Inputs

Multi-source account data

Raw account records from multiple business units are brought into a common staging pipeline so matching can run across the full customer footprint.

Key signal: 200k problematic records

Result

The system was able to restore CRM single source of truth, reducing duplicate and incomplete records by 70% and saving +1 FTE worth of manual effort.

Takeaway

Workshops with key stakeholders who understood the business context were crucial to building this solution. Knowing where to separate AI tasks from human tasks was critical to success.