Job scraping explained: how it works, what you get by board, and when it is legal
What job scraping involves in production (rendering, sessions, structure drift, dedup, liveness), the share of postings with a stated salary, a labelled seniority, an employment type and a thin description on every collected board, a plain summary of the legal position with a link to the full treatment, and the honest math on scraping versus using an API.
Dvir Atias
Founder, JobsPipe
Job scraping is the automated collection of job postings from career sites, job boards and ATS pages. It powers job boards, sourcing tools, sales-signal engines and labour-market research, and it is misunderstood on two fronts: how much production machinery it takes, and what the law does and does not allow. Here is the honest version of both, with the board-level data that shows what scraped postings actually contain.
How production job scraping actually works
The tutorial version, fetch a page and parse the HTML, survives contact with about three sites. A production pipeline deals with:
- Rendering. Many career sites are JavaScript applications; the postings exist only after a headless browser executes the page.
- Sessions and rate shaping. Large boards challenge unfamiliar traffic. Sustainable collection means behaving like a polite client: realistic sessions, conservative request rates, and backing off when asked.
- Structure drift. Every source changes markup on its own schedule. A 30-source pipeline without monitoring decays into a 12-source pipeline within months.
- Dedup and normalisation. The same job appears on the company site, LinkedIn and two boards, with three salary formats and four location spellings. Collapsing that into one record is where most of the engineering goes.
- Liveness. Postings close silently. Knowing a job is gone is as valuable as knowing it exists, and requires revisiting everything you have already collected.
Where scraped job postings come from
Four kinds of source, with very different economics. Applicant tracking system feeds (Greenhouse, Lever, Ashby, Workable, SmartRecruiters, Workday) publish structured JSON per employer and are the cheapest to read and the hardest to discover, because nobody lists their customers. Aggregators (LinkedIn, Indeed) hold the most postings and the most duplicates, republish from the feeds, and defend themselves against crawlers. Company career pages on smaller or custom systems need rendering and per-site parsers. Niche boards such as Y Combinator’s cover one segment well. A production pipeline reads all four and tags every record with where it came from; the platforms JobsPipe collects, and how each is read, are documented under sources.
What scraped postings actually contain, by board
Not every board gives you the same posting. Aggregators strip pay and structure; ATS feeds carry the employer’s own fields, but each feed carries a different set. Four shares tell you most of what you need before choosing sources: how many postings state a salary, how many carry a labelled seniority, how many state an employment type, and how many have a description too thin to extract skills from.
| Board | Salary stated | Seniority labelled | Employment type stated | Thin description |
|---|---|---|---|---|
| Y Combinator | 67.7% | 36.8% | 100.0% | 1.1% |
| Ashby | 56.1% | 39.0% | 100.0% | <0.5% |
| Greenhouse | 46.8% | 38.2% | 0.0% | <0.5% |
| Indeed | 36.8% | 17.9% | 71.6% | 1.0% |
| Workday | 23.3% | 32.9% | 95.1% | <0.5% |
| Lever | 22.7% | 29.6% | 55.3% | 6.7% |
| 22.1% | 48.6% | 99.1% | 1.7% | |
| Workable | 19.6% | 22.3% | 79.0% | <0.5% |
| Paylocity | 10.7% | 7.4% | 0.0% | 2.1% |
| SmartRecruiters | 5.3% | 26.3% | 99.9% | 0.7% |
Two things to notice. Thin descriptions are rare on every board, so description length is not where boards differ. The structured fields are: the boards at the top on salary are not the boards at the top on seniority or employment type, and no board is complete on all three. A scraper that reads only one board inherits that board’s gaps, and a dataset built from several has to reconcile fields that mean slightly different things on each.
Is job scraping legal?
In summary, and not as legal advice: scraping publicly accessible pages is generally not a computer-misuse offence in the US after hiQ v. LinkedIn and Van Buren, but terms of service bind anyone who holds an account, description text is the employer’s copyright, and personal data carries privacy obligations however it was collected. The practical norms are public postings only, no logins, polite rates, and attribution if you republish text. The full treatment, including what LinkedIn and Indeed allow and why ATS feeds sidestep the question, is in is web scraping legal; the Workday case is in the Workday-specific guide.
Scrape or buy: the honest math
A single-source scraper is a weekend project. The production pipeline above is an ongoing engineering commitment; the build-vs-buy guide prices a multi-source operation once rendering infrastructure, proxies and maintenance time are counted. The buy side starts at free: JobsPipe’s free tier is 1,000 jobs a month across every collected source, and paid plans start at $49 a month for 25,000. If scraping is your product, the tools comparison ranks the serious options. If job data merely feeds your product, buy the pipeline and spend the engineering on what differentiates you. Every collected platform is documented under sources.
Skip the pipeline - every collected source, one schema, free tier first.
Get a free API key
Frequently Asked Questions
What is job scraping?
Job scraping is the automated collection of job postings from career sites, boards and ATS pages. In production it involves far more than fetching HTML: JavaScript rendering, session management, per-source structure monitoring, cross-source deduplication and liveness tracking so closed postings are detected. It powers job boards, sourcing tools, sales-signal engines and labour-market research.
Is job scraping legal?
In the US, hiQ v. LinkedIn and Van Buren established that scraping publicly accessible pages is generally not a Computer Fraud and Abuse Act violation, but that is not blanket permission: terms of service bind anyone with an account, description text is the employer's copyright, and personal data carries privacy obligations however it is collected. The practical norms are public postings only, no logins, polite request rates and attribution if you republish text. This is a summary, not legal advice.
Do all job boards give you the same posting?
No. Aggregators strip pay and structure, while ATS feeds carry the employer's own fields, but each feed carries a different set. JobsPipe publishes four shares per collected board: postings that state a salary, postings with a labelled seniority, postings that state an employment type, and postings the ghost scorer flags for a thin description. Thin descriptions are rare everywhere; the structured fields are where boards differ, and no board is complete on all of them.
Should I scrape jobs or use an API?
Scrape when the pipeline itself is your product or you need one source no vendor covers. Buy when job data feeds your product: a multi-source production scraper is an ongoing engineering commitment in rendering infrastructure, proxies and maintenance, while JobsPipe starts free at 1,000 jobs a month across every collected source and paid plans start at $49 a month for 25,000, in one schema across every board.