How to Prevent Web Scraping?

How to prevent web scraping? You cannot stop it outright: anything a browser can read, a program can read. What you can do is make bulk collection slow, expensive and visible, with rate limits, bot challenges, fingerprinting, honeypots and enforceable terms, and then give the people who want your data a sanctioned way to get it so they stop scraping.

How to prevent web scraping, layer by layer

1. Decide what you are protecting. Capacity, exclusivity or both. Search results and detail pages are where scrapers spend their requests, so those get the controls; the home page does not need them. Decide too which bots you want to keep: search engines, partners and aggregators that send you traffic.
2. Rate-limit per client. Count requests per IP, per session and per token over a sliding window, answer excess with HTTP 429 and a Retry-After header, and give expensive endpoints a smaller budget than cheap ones. This alone ends casual scraping and costs real users nothing.
3. Challenge suspicious traffic. Bot management from a CDN or a dedicated vendor issues a JavaScript challenge, a proof-of-work puzzle or a CAPTCHA before serving the page. Apply it to the expensive pages and to clients with a poor score, not to every visitor, or you will pay in bounce rate.
4. Fingerprint and score. The TLS handshake, header order, JavaScript properties and IP reputation together separate a browser from a library and a person from a headless session. Treat the result as a score that tightens the limits and triggers the challenges above, not as a hard block, because every signal has false positives.
5. Lay honeypots and watermarks. Links people cannot see catch crawlers that follow everything; a few decoy records prove provenance when your data turns up elsewhere. Both are cheap and neither touches real visitors.
6. Write terms you can enforce. State in the terms of service that automated collection needs permission, keep robots.txt consistent with that, and publish a contact for licensing requests. The terms turn a technical nuisance into a contractual breach, which is what a lawyer can act on.

How each of these looks from the scraper’s side is in can websites detect scraping, and the legal footing for the terms is in is web scraping legal. The layers raise the cost; determined operators with residential proxies and browser farms still get through, so the aim is to be the most expensive site to scrape, not an impossible one.

For a job board there is a better answer than any of the six layers. Postings exist to be found, and the aggregators and niche boards that would scrape you are the same sites that send you applicants. Publish a feed or an API with attribution and a rate limit, the way the applicant tracking systems do, and most of the scraping stops on its own because reading the feed is easier. If your board is the one that needs postings, the same logic runs in reverse: take a licensed feed rather than scraping one, as described in job board backfill.

Where JobsPipe fits

JobsPipe is a jobs data API that collects live postings from LinkedIn, Indeed, Y Combinator, Naukri, Workday, Greenhouse, Workable, SmartRecruiters, Ashby, Lever and Paylocity, returns them as one schema with closure tracking and a ghost score, and includes a free tier of 1,000 jobs a month at jobspipe.dev. For board operators it is the licensed feed on the receiving end of the argument above: a backfill query with source_or, job_country_code_or and max_ghost_score, refreshed with discovered_at_gte, fills a board without anyone scraping anyone.