Is web scraping legal? Public data, terms of service, hiQ v. LinkedIn, and the job-data answer

This is general information written by people who collect job postings for a living, not legal advice. The law on scraping is a patchwork of computer-misuse statutes, contract law, copyright, database rights and privacy law, and it differs by country and by what you collect. The sections below give the shape of it, the cases people cite, what LinkedIn and Indeed actually allow, and the practical route most job-data teams end up taking.

Is web scraping legal?

Usually yes for publicly accessible pages and often no for content behind a login, but the answer depends on three things: what you collect, how you collect it, and where you and the site are. Public factual data, collected politely, for a purpose the site’s terms do not forbid, is the safe end. Personal data, bypassed access controls and republished text are the risky end.

The four bodies of law that come up, in the order they usually matter:

• Computer-misuse statutes. In the US that is the Computer Fraud and Abuse Act (CFAA), which criminalises access “without authorization”. The Supreme Court’s 2021 decision in Van Buren narrowed it to a gates-up-or-down question: if no gate stops you, you are not accessing without authorization. Scraping a page anyone can load is on the gates-up side. Logging in, using stolen credentials or defeating a block moves you to the other side. The UK’s Computer Misuse Act and similar laws elsewhere draw the line in the same place.
• Terms of service. A site’s terms are a contract. Whether you are bound depends on whether you agreed: creating an account and clicking accept binds you; a link in a footer you never saw may not. Nearly every major job site’s terms prohibit scraping, so an account-based scraper is in breach from its first request even where no statute is broken.
• Copyright and database rights. Facts are not copyrightable: a title, an employer, a location and a salary are facts. The prose of a job description is expressive text that the employer owns. In the EU a separate database right protects substantial extraction from a protected database, with text-and-data-mining exceptions that sites can opt out of for commercial use.
• Privacy law. GDPR in the EU and UK, and state laws in the US, attach to personal data however it was collected. A job posting is mostly not personal data; a recruiter’s name and email inside it is. Scraping people profiles is a different activity from scraping postings, with its own obligations.

hiQ v. LinkedIn: what it decided and what it did not

hiQ v. LinkedIn decided that scraping publicly accessible LinkedIn pages likely does not violate the CFAA, because public pages have no gate. It did not decide that scraping is legal in general, and hiQ ultimately lost the case on contract grounds and agreed to stop scraping.

The sequence matters because the case is cited for more than it held. hiQ scraped public LinkedIn profiles to sell workforce analytics; LinkedIn sent a cease-and-desist and blocked it; hiQ sued. The Ninth Circuit sided with hiQ on the CFAA point in 2019, the Supreme Court sent the case back after Van Buren in 2021, and the Ninth Circuit reached the same conclusion again in 2022. Then the trial court took up LinkedIn’s other claims and found that hiQ had breached LinkedIn’s User Agreement, including by using fake accounts to collect data. The parties settled late in 2022 with hiQ barred from scraping LinkedIn. Read together: the CFAA is not the tool sites use against public-data scrapers, contract law is, and it works when the scraper agreed to the terms.

A later case pointed the other way on contract: in 2024 a court found that Meta’s terms did not bar a data vendor from collecting public pages while logged out, because a party that is not logged in is not using the service under those terms. The lesson from both is the same. Whether you hold an account on the site you scrape is the single most consequential fact.

Does LinkedIn allow scraping?

No. LinkedIn’s User Agreement lists scraping, crawling and using bots or browser extensions to extract data among the things members agree not to do, and LinkedIn enforces it with rate limits, account restrictions and litigation. Logged-out job listings are publicly accessible pages, which is what hiQ was about, but the User Agreement still binds anyone with an account.

In practice that splits into three activities with different risk. Scraping member profiles while logged in breaches the agreement and collects personal data. Browser extensions that read pages inside your own session are covered by the same clause, and LinkedIn has sued makers of them. Reading public job listings without an account is the activity the CFAA ruling covers, and it is how JobsPipe collects LinkedIn postings: public listing pages, no account, no session, rate-limited. The LinkedIn jobs page covers what that yields and what it does not.

Is scraping Indeed allowed?

Not under Indeed’s terms. Indeed’s Terms of Service prohibit crawling and scraping its pages, and Indeed protects the site with bot management. The sanctioned routes are Indeed’s partner programs, which are contractual and not open to everyone.

Job seekers rarely notice, but Indeed is itself an aggregator: many of its listings are republished from employer career sites and ATS feeds. For a data team that means the posting you want from Indeed usually exists upstream on the employer’s Greenhouse, Workday or Lever page, where it is published in a public feed with no terms problem. JobsPipe collects Indeed’s public search results alongside those upstream feeds; every row names its board in sources[0].provider, so you can prefer the employer’s feed copy when both exist. The Indeed API guide covers the official programs.

Is job scraping legal?

Job scraping sits at the low-risk end of web scraping: postings are advertisements an employer wants distributed, they contain little personal data, and the fields most products need are facts. The risks that remain are terms of service on the board you scrape, copyright in the description text if you republish it, and the technical line of not defeating access controls.

A checklist that keeps job scraping on the right side of each body of law:

• Collect only pages that load without an account. Never scrape through a login.
• Prefer the employer’s own ATS feed to the aggregator that copied it.
• Store the facts; if you republish description text, attribute and link to the source.
• Strip or protect recruiter names and emails, and honour deletion requests.
• Rate-limit, identify your client, and stop when a site asks you to.
• Do not circumvent blocks, captchas or bot management. That is the gate.

How to scrape data from a website (the lawful way)

Start by not scraping: look for an official API, a public JSON feed or a sitemap, because most sites that want their data used publish one. If you must fetch HTML, read robots.txt, send an identifying user agent, request slowly, cache what you fetch, and parse only public pages.

1. Check for a feed. Open the network tab on the page you want. If the page loads its data from a JSON endpoint, use the endpoint. It is faster, structured and usually intended for exactly this.
2. Read robots.txt and the terms. robots.txt is not law, but it is the site’s stated preference and courts and juries hear it that way. Terms tell you whether an account would bind you.
3. Fetch politely. One request at a time per host, a real user agent with a contact address, exponential back-off on errors, and honour Retry-After.
4. Parse and store the minimum. Keep the fields your product needs. Drop personal data you did not need to collect.
5. Re-check liveness. Postings close silently; a scraped record without a close check becomes a ghost job in your product within weeks.

The tooling is covered in the job scraping tools comparison, and the production reality of rendering, sessions and dedup in what job scraping actually involves.

The practical answer: most ATS platforms publish public feeds

The job-data teams that never see a lawyer are the ones that read applicant tracking system feeds instead of scraping boards. Greenhouse, Lever, Ashby, Workable and SmartRecruiters each publish a public, unauthenticated JSON endpoint per customer, and Workday tenants expose a JSON search feed on each careers site. The employer opted into publishing that feed; the data is structured; no account is involved.

# Greenhouse job board API (per board token)
curl "https://boards-api.greenhouse.io/v1/boards/{board_token}/jobs?content=true"

# Lever postings API (per site slug)
curl "https://api.lever.co/v0/postings/{site}?mode=json"

# Ashby job board API (per organization slug)
curl "https://api.ashbyhq.com/posting-api/job-board/{organization}"

# Workable widget API (per account subdomain)
curl "https://apply.workable.com/api/v1/widget/accounts/{subdomain}"

# SmartRecruiters postings API (per company id, when enabled)
curl "https://api.smartrecruiters.com/v1/companies/{companyId}/postings"

The feed route shows up in the data, and so do the differences between feeds. Each platform publishes its own set of structured fields: some carry an employment type on nearly every posting and some none at all, and salary presence tracks the employer’s pay-transparency exposure more than the platform. Here is the completeness of the fields that matter on the feed-backed boards JobsPipe collects.

The catch with feeds is discovery, not law: there is no list of Greenhouse or Lever customers, so a feed reader has to find and maintain thousands of slugs. That is the part JobsPipe does. Every collected platform, with its feed shape and coverage, is documented under sources, and the whole set is one endpoint:

curl -X POST https://api.jobspipe.dev/v1/jobs/search \
  -H "Authorization: Bearer $JOBSPIPE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "source_or": ["greenhouse", "lever", "ashby", "workable", "smartrecruiters"],
    "posted_at_max_age_days": 7,
    "limit": 50
  }'