--- title: "What Is Web Scraping?" description: "[NewSearch millions of jobs from your AI agent with MCP→](/blog/jobspipe-mcp-server)" canonical: https://jobspipe.dev/blog/what-is-web-scraping last_updated: 2026-09-10 --- [NewSearch millions of jobs from your AI agent with MCP→](/blog/jobspipe-mcp-server) [All posts](/blog) ![](/listly/card-field.png) GuideScraping Guide·Sep 9, 2026·5 min read # What Is Web Scraping? What is web scraping? Software that fetches web pages and pulls structured data out of them. How it works, what it is used for, and when a feed beats it. ![Dvir Atias](/authors/dvir-atias.jpg) Dvir Atias Founder, JobsPipe What is web scraping? It is the use of software to fetch web pages and pull structured data out of them. A script requests a URL, reads the HTML or JSON that comes back, extracts the fields it wants and stores them as rows. People do it because a website shows data to humans that it does not offer as a download. ## How is web scraping done? 1. **Choose the target and look for a feed first.** Before writing a scraper, check whether the site publishes an API, an RSS feed, a sitemap or a JSON endpoint that its own front end calls. A documented feed is faster, more stable and usually explicitly permitted, which removes most of the work and most of the risk. 2. **Fetch.** The scraper sends an HTTP request for the page, the same request a browser sends, and receives HTML or JSON. Sites that build the page with JavaScript in the browser need a headless browser to run those scripts first; sites that render on the server do not. 3. **Parse.** The response is turned into a structure a program can walk: a DOM tree for HTML, an object for JSON. Libraries such as BeautifulSoup, lxml and Cheerio do this for HTML; every language parses JSON natively. 4. **Extract and normalise.** Selectors pick out the fields, then the values are cleaned: whitespace trimmed, dates and prices parsed, relative links resolved, the site’s labels mapped onto your own schema. 5. **Store.** Rows go into a CSV, a database or a queue for the next system. Good pipelines also record when each row was first and last seen, which is what makes change detection possible later. 6. **Repeat and monitor.** A scraper is only useful if it runs again. That means pagination, a schedule, retries, and an alert for the day the site changes its markup and the extraction silently returns nothing. A worked example of the whole loop in Python, against a feed that is public by design, is in [how to web scrape with Python](/blog/how-to-web-scrape-with-python). ## What is web scraping used for? Retailers monitor competitors’ prices and stock. Researchers collect posts, listings and public records for studies. Search engines and archives fetch pages to index or preserve them. Sales teams build lead lists from directories and company sites. SEO tools track rankings and page changes. Analysts collect job postings to read hiring demand, skills and pay. Machine learning teams assemble training corpora. The common thread is data that is visible on a website but not offered in bulk. ## Scraping, APIs and feeds An API is an interface the site owner designed for programs: stable fields, documented rate limits and terms that say what you may do with the output. A feed is the read-only version of that idea, such as RSS or a public JSON listing. Scraping is reading the interface built for humans with a program instead. Scraping wins on coverage, because it works on any page; APIs and feeds win on stability, because they do not break when a designer moves a button. Is web scraping the same as crawling? No, though the words travel together. Crawling is discovering and fetching pages by following links, the way a search engine does. Scraping is extracting data from pages once you have them. A price monitor scrapes a known list of URLs; a search engine crawls the web and extracts little; most production systems do both. ## Is web scraping legal? It depends on what you take, how you take it and what you do with it. Public pages read without a login sit at the permissive end; content behind a login, personal data and copyrighted material sit at the other, and every site’s terms add a contractual layer on top. The case law and the practical rules are in [is web scraping legal](/blog/is-web-scraping-legal), so this post will not restate them. ## Web scraping for job postings For job data, most scraping is unnecessary. The applicant tracking systems that employers publish through expose public feeds for every customer board: Greenhouse has a boards API, Lever a postings API, Ashby a posting API, Workable a widget endpoint and SmartRecruiters a postings endpoint. Reading them takes an HTTP request and a JSON parser, and the data is published to be read. The hard part is the aggregators, LinkedIn and Indeed, which offer no feed and defend their pages. What a production job pipeline looks like is in [what is job scraping](/blog/what-is-job-scraping), and the tools that do it for you are compared in [best job scrapers](/blog/best-job-scrapers). ## Where JobsPipe fits JobsPipe is a jobs data API that collects live postings from LinkedIn, Indeed, Y Combinator, Naukri, Workday, Greenhouse, Workable, SmartRecruiters, Ashby, Lever and Paylocity, returns them as one schema with closure tracking and a ghost score, and includes a free tier of 1,000 jobs a month at [jobspipe.dev](https://jobspipe.dev). It is what you use instead of the loop above when the target is job postings: one request replaces the fetch, parse, normalise and monitor steps for every board it collects. ``` curl -X POST https://api.jobspipe.dev/v1/jobs/search \ -H "Authorization: Bearer $JOBSPIPE_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "job_title_or": ["data analyst"], "source_or": ["greenhouse", "lever", "ashby"], "posted_at_max_age_days": 7, "limit": 25 }' ``` Each row carries `date_posted`, `discovered_at`, `status` and `sources[0].provider`, so you know which board it came from and whether it is still open. Skip the scraper for job postings - 1,000 jobs a month free. ![](/listly/shape-heart.png) FAQs ## Frequently Asked Questions ### What is web scraping used for? Price and stock monitoring, market and academic research, search indexing and archiving, lead lists, SEO and content monitoring, job posting and labour-market analysis, and assembling training data. In every case the data is visible on a website but not offered in bulk, so a program reads the pages instead. ### How is web scraping done? A script requests a URL, receives HTML or JSON, parses it into a structure it can walk, extracts the fields with selectors, normalises the values and stores them as rows. Production scrapers add pagination, a schedule, retries and monitoring, and check first whether the site publishes a feed or API that makes the scraping unnecessary. ### Is web scraping the same as crawling? No. Crawling is discovering and fetching pages by following links, which is what search engines do. Scraping is extracting structured data from pages you already have. A price monitor scrapes a fixed list of URLs without crawling; a search engine crawls widely and extracts little; most data pipelines do some of each. ### Is web scraping legal? It depends on what is taken, how and for what. Public pages read without a login are the most defensible; content behind a login, personal data and copyrighted material are not, and a site's terms of service add a contractual layer. The case law and practical rules are covered in the separate post on whether web scraping is legal. [ ← Previous Are Data Jobs in Demand? ](/blog/are-data-jobs-in-demand)[ Next → How to Web Scrape With Python? ](/blog/how-to-web-scrape-with-python) --- Canonical URL: https://jobspipe.dev/blog/what-is-web-scraping Title: What Is Web Scraping? Description: What is web scraping? Software that fetches web pages and pulls structured data out of them. How it works, what it is used for, and when a feed beats it. --- AI agents: the full machine-readable index of this site is https://jobspipe.dev/llms.txt?src=md-twin - API quickstart, no-key sandbox (POST https://api.jobspipe.dev/v1/sandbox/jobs/search), MCP server, pricing. Free key: https://jobspipe.dev/signup