Which Websites Allow Web Scraping?

Which websites allow web scraping? Fewer than the tutorials suggest. Does Amazon allow web scraping? No: its Conditions of Use prohibit data mining and extraction tools without written consent, and Reddit and YouTube say much the same in their terms. Wikipedia is the rare large site that welcomes reuse, through an API and full database dumps.

How to find out whether a website allows web scraping

1. Read the terms of service. Search the page for automated, robot, scrape, crawl and data mining. Does Amazon allow web scraping? Its Conditions of Use answer in one clause: no data mining, robots or similar gathering tools without express written consent. Most large consumer sites carry an equivalent sentence.
2. Read robots.txt. The file at /robots.txt lists the paths the site asks crawlers to stay out of, sometimes per user agent. It is a request rather than a contract, but a site that disallows everything has told you its position, and ignoring it weighs against you if the matter is ever argued.
3. Look for an official API. A public API with a free tier is the site telling you how it wants to be read. Its quotas tell you the volume the site tolerates, and its terms usually say what you may store and republish.
4. Look for a licensing programme. Sites that guard their data often sell it: Wikimedia Enterprise for high-volume Wikipedia access, Reddit’s data licensing for commercial use. If a programme exists, unlicensed scraping is the thing it was created to replace.
5. Check whether the data is public or behind a login. Pages any visitor can read without an account are one category; anything that needed you to accept terms and sign in is another, with more exposure under both contract and computer-misuse law.
6. Weigh volume and purpose. A few pages for a personal project and a continuous commercial crawl of a whole catalogue are treated differently by sites and by courts, even when the terms read the same.

The case law behind all of this, including hiQ v. LinkedIn and what it did not decide, is in is web scraping legal. For job data the answer is unusually clear: the applicant tracking systems publish a feed per customer board, Greenhouse, Lever, Ashby, Workable and SmartRecruiters among them, so their postings can be read the way they were published. What each board exposes, and what JobsPipe returns for it, is on the sources pages; the basics are in what is web scraping.

Where JobsPipe fits

JobsPipe is a jobs data API that collects live postings from LinkedIn, Indeed, Y Combinator, Naukri, Workday, Greenhouse, Workable, SmartRecruiters, Ashby, Lever and Paylocity, returns them as one schema with closure tracking and a ghost score, and includes a free tier of 1,000 jobs a month at jobspipe.dev. It answers the question for job postings the way an official API answers it for any other site: one endpoint, documented terms, a quota you can see. Filter by board with source_or and every row carries sources[0].provider so you know where it was published.