How do scrapers work?
Scrapers work in a loop: fetch a page over HTTP, parse the HTML or JSON it returns, pull out the fields you want, follow links or pagination to the next page, and store the results. Around that loop sit scheduling, so the data stays current, and deduplication, so the same record seen twice is stored once.
The fetch step is a normal web request with headers that look like a browser. Server-rendered pages come back as HTML that a parser walks with CSS selectors or XPath to find the title, company and description. Pages rendered in the browser come back nearly empty, so the scraper either runs a headless browser to execute the JavaScript or, more efficiently, calls the JSON endpoint the page itself calls. Most job boards and career sites fall into the second group, which is why a good scraper spends more time reading network traffic than reading markup.
Pagination and scheduling turn a one-off script into a pipeline. The scraper walks result pages until they run out, records each item with a stable identifier from the source, and compares it with what it already holds so a re-seen posting updates a last-seen timestamp instead of creating a duplicate. A scheduler re-runs each source on a cadence, and the absence of a record that was present last time is itself a signal: that is how a job posting is marked closed. Sighting logs of this kind are what make first-seen dates and lifetimes measurable.
The realities are less tidy. Sites rate-limit, fingerprint clients, serve challenges and change layouts without notice, so production scrapers need rotating proxies, retry logic that stops rather than hammers, and monitoring for the day a parser silently returns nothing. JobsPipe runs that machinery for ten collected boards and exposes the output through the jobs search endpoint, with discovered_at, last_seen_at and status on every record. For most job-data use cases that is the cheaper end of the build-versus-buy math, and the free tier returns 1,000 jobs a month.
The full treatment is on Scraping and legality.
Related questions
How does a scraper work?
It sends a request for a page, receives HTML or JSON, parses that response into fields, and saves them. A single scraper is a function from a URL to a record. The rest of a scraping system, meaning pagination, scheduling, deduplication and error handling, exists to run that function reliably across many pages and many days.
How does scraping work?
Scraping works by treating a web page as a data source. The program requests the page, locates the data in the response using selectors or by reading the JSON the page loads, extracts the fields it needs and stores them. Repeating that across a site's pages on a schedule produces a dataset that stays current.
Do scrapers need a headless browser?
Only when the data exists solely after JavaScript runs and there is no underlying JSON endpoint to call. Headless browsers are slow and expensive per page. For job boards and applicant tracking systems the listing data usually arrives from a JSON call the page makes, so the efficient scraper calls that directly.
How do scrapers avoid duplicates?
By keying every record on a stable identifier from the source, such as a job id or canonical URL, and updating the existing row when it is seen again. Cross-source duplicates, the same role on a company career site and on an aggregator, need a separate matching step on company, title and location, which is why aggregated feeds often show one role several times.
How do scrapers detect that a job closed?
By noticing absence. When a scheduled re-crawl of a source no longer returns a posting that was present before, the pipeline records a closure. Boards read from an employer feed report that promptly; boards with a listing expiry report it when the listing lapses. JobsPipe stores the result as status, closed_at and closed_reason.
More answers
Query live postings yourself, 1,000 jobs a month free.
Get a free API key