[NewSearch millions of jobs from your AI agent with MCP→](/blog/jobspipe-mcp-server) [All posts](/blog) ![](/listly/card-field.png) GuideScraping Guide·Aug 7, 2026·7 min read # Job scraping explained: how it works and when it is legal What job scraping actually involves in production - rendering, sessions, dedup - what US courts have said about scraping public postings, and the honest math on scraping versus using an API. ![Dvir Atias](/authors/dvir-atias.jpg) Dvir Atias Founder, JobsPipe Job scraping is the automated collection of job postings from career sites, job boards, and ATS pages. It powers job boards, sourcing tools, sales-signal engines, and labour-market research - and it is widely misunderstood on two fronts: how much production machinery it actually takes, and what the law does and does not allow. Here is the honest version of both. ## How production job scraping actually works The tutorial version - fetch a page, parse the HTML - survives contact with about three sites. A production pipeline deals with: - **Rendering** - many career sites are JavaScript applications; the postings exist only after a headless browser executes the page. - **Sessions and rate shaping** - large boards challenge unfamiliar traffic. Sustainable collection means behaving like a polite client: realistic sessions, conservative request rates, and backing off when asked. - **Structure drift** - every source changes markup on its own schedule. A 30-source pipeline without monitoring decays into a 12-source pipeline within months. - **Dedup and normalization** - the same job appears on the company site, LinkedIn, and two boards, with three salary formats and four location spellings. Collapsing that into one record is where most of the engineering actually goes; across JobsPipe’s 1.55M active postings, cross-source duplicates measure 1.3% _after_ canonicalization. - **Liveness** - postings close silently. Knowing a job is gone is as valuable as knowing it exists, and requires revisiting everything you have already collected. ## Is job scraping legal? Not legal advice, but the shape of US law is settled enough to summarize. The _hiQ v. LinkedIn_ litigation established that scraping publicly accessible data does not violate the Computer Fraud and Abuse Act - public pages are not “unauthorized access.” That is narrower than a green light: terms of service still create contract exposure, logged-in scraping is a different regime entirely, and personal data carries privacy-law obligations regardless of how it was collected. The practical norms: collect only public postings, respect technical signals, keep volume polite, and treat people-data with more care than job-data. Our [Workday-specific legal guide](/blog/scrape-workday-jobs-legally) goes deeper on the ATS case. ## Scrape or buy: the honest math A single-source scraper is a weekend project. The production pipeline above is an ongoing engineering commitment - our [build-vs-buy guide](/blog/job-scraper-build-vs-buy) prices a multi-source operation in the thousands per month once rendering infrastructure, proxies, and maintenance time are counted. The buy side starts at free: JobsPipe’s free tier is 1,000 jobs monthly from all 30+ sources, and $49/mo covers 25,000. If scraping _is_ your product, our [tools comparison](/blog/best-job-scrapers) ranks the 8 serious options. If job data merely feeds your product, buy the pipeline and spend the engineering on what differentiates you. Skip the pipeline - 30+ sources, deduped, free tier first. ![](/listly/shape-heart.png) FAQs ## Frequently Asked Questions ### What is job scraping? Job scraping is the automated collection of job postings from career sites, boards, and ATS pages. In production it involves far more than fetching HTML: JavaScript rendering, session management, per-source structure monitoring, cross-source deduplication, and liveness tracking so closed postings are detected. It powers job boards, sourcing tools, sales-signal engines, and labour-market research. ### Is job scraping legal? In the US, the hiQ v. LinkedIn litigation established that scraping publicly accessible pages does not violate the CFAA, but that is not blanket permission: terms of service create contract exposure, logged-in scraping is a different regime, and personal data carries privacy obligations however it is collected. The practical norms are public postings only, polite request rates, respect for technical signals, and extra care with people-data versus job-data. This is a summary, not legal advice. ### Should I scrape jobs or use an API? Scrape when the pipeline itself is your product or you need one source a vendor does not cover. Buy when job data feeds your product: a multi-source production scraper costs thousands per month in infrastructure and maintenance, while JobsPipe starts free at 1,000 jobs monthly across 30+ sources and $49/mo covers 25,000, already deduplicated to a measured 1.3% duplicate rate. [ ← Previous Google for Jobs API: indexing in, and how to get data out ](/blog/google-for-jobs-api)[ Next → Jobs API in Python: pip install to live postings in 5 minutes ](/blog/python-jobs-api) --- Canonical URL: https://jobspipe.dev/blog/what-is-job-scraping Title: Job scraping explained: how it works and when it is legal Description: What job scraping actually involves in production - rendering, sessions, dedup - what US courts have said about scraping public postings, and the honest math on scraping versus using an API. --- AI agents: the full machine-readable index of this site is https://jobspipe.dev/llms.txt?src=md-twin - API quickstart, no-key sandbox (POST https://api.jobspipe.dev/v1/sandbox/jobs/search), MCP server, pricing. Free key: https://jobspipe.dev/signup