---
title: "What does web scraping mean?"
description: "**Canonical URL:** https://jobspipe.dev/answers/what-does-web-scraping-mean"
canonical: https://jobspipe.dev/answers/what-does-web-scraping-mean
last_updated: 2026-09-10
---

# What does web scraping mean?

> Web scraping means using a program to fetch web pages and extract structured data from them, such as a list of job titles, prices or addresses, instead of reading the pages by hand. The program requests the page like a browser does, parses the returned HTML or JSON, and stores the fields it finds in a database or file.

**Canonical URL:** https://jobspipe.dev/answers/what-does-web-scraping-mean
**Topic:** Scraping and legality (https://jobspipe.dev/blog/is-web-scraping-legal)

Scraping differs from an API or a feed in who designed the data for you. An API is an interface the site owner publishes for programs, with a documented shape and usually a key, so the data arrives clean and the owner can meter it. A feed is the same idea pushed on a schedule, as XML or JSON. Scraping reads pages built for humans, so the scraper has to infer structure from layout, survive redesigns, and handle anti-bot measures the site never intended for it. That is why scraped data breaks more often and costs more to maintain.

For job postings specifically, scraping is mostly unnecessary because the applicant tracking systems that host employer career pages publish public, structured endpoints. Greenhouse, Lever, Ashby and Workable expose a public job board feed per company, and every Workday career site serves its listings from a JSON endpoint under the tenant's myworkdayjobs domain. Those feeds are the source the big boards copy from. The genuinely scraped surfaces are the boards themselves, chiefly LinkedIn and Indeed, whose public guest pages carry postings that never appear on an employer feed.

JobsPipe collects both kinds of surface, the public applicant tracking feeds and the public pages of LinkedIn, Indeed and Y Combinator, and normalizes them into one record shape. A developer who needs job data calls the jobs search endpoint with filters such as job_title_or, job_country_code_or and posted_at_max_age_days and gets back the same fields whichever board a posting came from, with the provider named on each record's sources entry. The free tier returns 1,000 jobs a month, and the sandbox endpoint needs no key at all.

## Related questions

### Is web scraping the same as web crawling?

Not quite. Crawling is discovering and fetching pages by following links, the way a search engine does. Scraping is extracting specific fields from pages once you have them. A job data pipeline does both: it crawls a career site to find each posting, then scrapes the title, location, salary and description from each one.

### Is web scraping legal?

Accessing public pages is generally lawful in the United States after hiQ v. LinkedIn, but terms of service, copyright on the content, personal data rules and anti-circumvention law all still apply, and other countries differ. Reading public job postings without logging in sits at the low-risk end. JobsPipe's guide to whether web scraping is legal goes through the cases in detail.

### What is a web scraper written in?

Any language that can make HTTP requests and parse HTML or JSON. Python with a requests library and a parser is the most common starting point, JavaScript with a headless browser is common where pages render client-side, and production systems add queues, proxies and schedulers around the core. The language matters less than the maintenance plan.

### Why use an API instead of scraping job sites?

Because the hard part is not fetching one page but keeping thousands of sources current every day, deduplicating across boards, and noticing when a posting closes. An API such as JobsPipe absorbs that work and returns fields a scraper cannot easily produce, such as discovered_at, last_seen_at, status and ghost_score.

## More answers

- [How do scrapers work?](https://jobspipe.dev/answers/how-do-web-scrapers-work)
- [Does LinkedIn have an API?](https://jobspipe.dev/answers/does-linkedin-have-an-api)

Data behind these answers is published as shares only; every figure can be reproduced against the JobsPipe API (https://jobspipe.dev/jobs-api).

---
AI agents: the full machine-readable index of this site is https://jobspipe.dev/llms.txt?src=md-twin - API quickstart, no-key sandbox (POST https://api.jobspipe.dev/v1/sandbox/jobs/search), MCP server, pricing. Free key: https://jobspipe.dev/signup