We 7x'd Our Job-Board Coverage Overnight. 13,000 of the New Postings Were Duplicates.

Thirty-eight percent of the new postings were the same job

Of the roughly 34,000 raw postings ingested that run, about 13,000, or 38%, were byte-for-byte identical to another posting ingested in the same batch. Same title, same body text, same requirements, just filed under a different location or a slightly different req ID. Left alone, every one of those would have shown up in a job feed as a separate opening.

One company, one job description, about 800 locations

The single biggest contributor was a security-staffing company, Securitas, which had reposted one job description across roughly 800 different locations. Same guard-post listing, same requirements paragraph, filed once per site. It is not hard to see why: an ATS lets a company push one requisition to every branch with a form field, and each copy reads as a separate "opening" to anyone counting postings, us included, before we started deduplicating.

This is the mechanic behind a complaint job seekers already have a name for: ghost jobs, or more precisely here, phantom volume. The role count on a board looks large because one real requisition got multiplied, not because a company has hundreds of genuinely distinct openings.

Byte-identical, not 'looks similar': why the stricter signal won

Our dedupe pass runs after every scan and keeps one row per identical job-description body (favoring the copy with extracted skill data, then the one with a salary, then the newest), deactivating the rest so duplicates get swept from the live feed several times a day.

We tried a looser version first: normalize the text (lowercase, strip bullets and extra whitespace) before comparing, so two postings that differ only in formatting still count as the same job. That was too aggressive. On a later run it collapsed about 17,000 postings that were only superficially similar, shared boilerplate or a template across different companies, not the same job at all. We narrowed the match back to raw, byte-identical text. Slightly less aggressive, but it never merges two postings that only look alike.

The spam never stops. Neither does the cleanup.

Here is the part we did not expect: the fix is not a one-time cleanup, it is a treadmill. The source boards never stop listing those duplicates, so every fresh crawl re-activates them, and the pass at the end of the crawl collapses them again. Our scan logs show the dedupe pass deactivating roughly 23,000 duplicate postings on every single scan (the last six scans: between 22,918 and 23,484 each), several times a day, every day.

That means the duplicate share of the active feed breathes with the crawl cycle. We measured it twice on the same day while writing this post: a few hours after a dedupe pass it was 2.8%; mid-crawl, while a scan's writes were re-activating postings the boards still list, it peaked at 11.9%. The pass at the end of that crawl knocks it back down, and the cycle repeats. Right after a pass the active US feed holds about 170,000 postings.

And Securitas is not special. A crawl running while we wrote this showed fresh clusters: a personal-training company (Svetness) with one job description active in 383 copies, a boutique-fitness chain (Solidcore) at 163, and a delivery company (Gopuff) at 158. New names, same mechanic, every day.

Methodology and honest caveats

The surge numbers. ~530 to ~3,700 last-24h US postings, ~34,000 raw ingested, ~13,000 (38%) exact duplicates, and the Securitas figure are all from the single provider-activation run on 2026-07-02. They describe that event, not a permanent daily rate.

The current numbers. All queried live against the production database on 2026-07-14. The active count and duplicate rate oscillate with the crawl cycle, so we quote both ends: ~170,000 active US postings and a 2.8% duplicate rate measured a few hours after a dedupe pass; an 11.9% duplicate rate measured mid-crawl the same day, before that crawl's pass ran. The ~23,000-per-scan collapse figure is from the pipeline's own scan logs across the six most recent scans (range 22,918 to 23,484).

What "duplicate" means here. An exact match on job-description text (MD5 hash of the raw body), not a fuzzy or semantic match. Two postings with the same requirements worded differently are not counted as duplicates by this measure, so the true phantom-volume rate, including near-duplicates, is higher than what we report.

Source. Postings pulled directly from each company's public ATS or career-site API. Deduping runs server-side after every scan, several times a day.

FAQ