The reading room / EasyJobs
How to de-duplicate job postings across boards
Watch multiple job boards and you will see the same opening again and again. De-duplication is what turns that noisy overlap into a clean shortlist where every role appears once. Here is how to do it well.
Why duplicates happen
Companies syndicate a single opening to many channels: their own ATS, one or more aggregators, and the big networks. Each channel assigns its own listing ID, so naively the "same" job looks like five different postings. Without de-duplication, a multi-board search multiplies your reading and your alert volume.
Choose a stable merge key
De-duplication is only as good as the key you merge on. A per-listing ID is not stable across boards; you need something that identifies the underlying role.
- Company plus normalized title works for most cases (lowercase, trim seniority noise, collapse whitespace).
- The canonical apply URL is an even stronger signal when available, since cross-posts often point to the same ATS link.
- Combine both where you can: agree on either signal to treat two postings as one.
Make the seen-list durable
De-duplicating within a single search run is not enough — you also must not be re-shown a role you saw yesterday, or have a dismissed role reappear after a restart. That requires a durable, per-user record of what you have already been shown, stored so it survives restarts and redeploys.
EasyJobs keys its seen-list per user by source and listing, so a role that is cross-posted appears once, a dismissed role stays gone, and no opening emails you twice.
Prune so it stays fast
A seen-list that grows forever eventually slows everything down and keeps stale roles around. Pruning records older than a sensible retention window keeps the system fast and your shortlist current without losing recent de-duplication history.
Frequently asked questions
- Why do I keep getting the same job in different alerts?
- Because each board lists it separately and nothing is merging them. De-duplicating on a stable key like company plus normalized title collapses those into one entry so you are notified once.
- Will de-duplication hide roles I want to see?
- No — it merges genuine duplicates of the same role, not distinct openings. Two different positions at the same company still show separately because their titles differ.