How to Web Scrape With Python?

How to web scrape with Python? Install requests, send a GET or POST to the URL you want, parse the JSON or HTML that comes back, and write the rows somewhere. The best first project is a feed that is public by design, so this tutorial reads a Greenhouse job board and then the JobsPipe sandbox, with no browser and no API key.

How to web scrape with Python, step by step

1. Set up a project. Python 3.9 or newer, a virtual environment and one dependency. Everything else in this tutorial is the standard library.

   python3 -m venv .venv
   source .venv/bin/activate
   pip install requests

2. Fetch a public feed and parse the JSON. Greenhouse publishes every public board at boards-api.greenhouse.io/v1/boards/{board_token}/jobs. GitLab hires through Greenhouse and its token is gitlab. The script below fetches the feed, keeps the fields worth keeping and writes them to a file.

   import json
   import requests
   
   BOARD = "gitlab"
   URL = f"https://boards-api.greenhouse.io/v1/boards/{BOARD}/jobs"
   HEADERS = {"User-Agent": "jobs-tutorial/1.0 (contact: you@example.com)"}
   
   resp = requests.get(URL, headers=HEADERS, timeout=30)
   resp.raise_for_status()
   jobs = resp.json()["jobs"]
   
   rows = []
   for job in jobs:
       rows.append(
           {
               "id": job["id"],
               "title": job["title"],
               "location": job["location"]["name"],
               "url": job["absolute_url"],
               "first_published": job.get("first_published"),
               "updated_at": job["updated_at"],
           }
       )
   
   print(len(rows), "published jobs on the", BOARD, "board")
   for row in rows[:5]:
       print(row["title"], "|", row["location"], "|", row["url"])
   
   with open(f"{BOARD}_jobs.json", "w", encoding="utf-8") as f:
       json.dump(rows, f, indent=2)

   raise_for_status() turns a 4xx or 5xx into an exception instead of a confusing parse error later, and resp.json() is the whole parsing step. The User-Agent with a contact address is a courtesy that also helps if the site’s operators ever need to reach you.
3. Pull descriptions, politely. Adding content=true returns the description HTML, departments and offices for each job in the same response, which makes the payload much larger. Ask for it once, then pause between any further requests.

   import time
   import requests
   
   BOARD = "gitlab"
   URL = f"https://boards-api.greenhouse.io/v1/boards/{BOARD}/jobs"
   HEADERS = {"User-Agent": "jobs-tutorial/1.0 (contact: you@example.com)"}
   
   resp = requests.get(URL, params={"content": "true"}, headers=HEADERS, timeout=30)
   resp.raise_for_status()
   for job in resp.json()["jobs"][:3]:
       departments = [d["name"] for d in job.get("departments", [])]
       print(job["title"], "|", ", ".join(departments), "|", len(job["content"]), "chars of HTML")
       time.sleep(1)

4. Query the JobsPipe sandbox. The same technique works against a search API. The sandbox accepts the live filters and needs no key; it returns sample rows shaped like the live response, with a reduced field set.

   import requests
   
   URL = "https://api.jobspipe.dev/v1/sandbox/jobs/search"
   body = {"job_title_or": ["backend engineer"], "remote": True, "limit": 5}
   
   resp = requests.post(URL, json=body, timeout=30)
   resp.raise_for_status()
   payload = resp.json()
   
   for job in payload["data"]:
       print(job["job_title"], "|", job["company"], "|", job["date_posted"], "|", job["final_url"])
   print(payload["metadata"])

   Swap the URL for https://api.jobspipe.dev/v1/jobs/search and add an Authorization: Bearer header with a free key, and the same body returns live postings with status, ghost_score and sources[0].provider on every row.
5. Store and dedupe. Key each row on the source’s own id, the Greenhouse job id or the JobsPipe id, and upsert. SQLite from the standard library is enough for a first pipeline; keep first_seen and last_seen columns so a run that no longer returns a row tells you the posting closed.
6. Schedule and respect limits. Run the script from cron or launchd, back off on HTTP 429 and honour the Retry-After header, and never fetch faster than a person would browse. Public feeds are rate-limited per IP, and a script that hammers one gets the whole office blocked.

When you need a browser, and when you do not

You need a browser only when the data is assembled by JavaScript after the page loads and there is no underlying JSON call you can make directly. Selenium and Playwright drive a real Chrome, wait for the scripts to finish and hand you the rendered HTML. Selenium 4 downloads a matching driver itself when Chrome is installed, so the minimal version is short:

from selenium import webdriver
from selenium.webdriver.chrome.options import Options

options = Options()
options.add_argument("--headless=new")
driver = webdriver.Chrome(options=options)
try:
    driver.get("https://job-boards.greenhouse.io/gitlab")
    print(driver.title)
    print(len(driver.page_source), "bytes of rendered HTML")
finally:
    driver.quit()

Notice that this fetches the same GitLab board the feed already gave us as JSON, at many times the cost. That is the general rule for ATS boards: Greenhouse, Lever, Ashby, Workable and SmartRecruiters all publish machine-readable feeds, so a browser is the wrong tool for them. It is the right tool for pages with no feed at all, which for job data means the aggregators, and there it comes with bot challenges, proxies and constant maintenance. The definitions behind all of this are in what is web scraping.

Where JobsPipe fits

JobsPipe is a jobs data API that collects live postings from LinkedIn, Indeed, Y Combinator, Naukri, Workday, Greenhouse, Workable, SmartRecruiters, Ashby, Lever and Paylocity, returns them as one schema with closure tracking and a ghost score, and includes a free tier of 1,000 jobs a month at jobspipe.dev. Step 2 above works for one board whose token you know; JobsPipe collects every public Greenhouse board it can identify, plus the other ten sources, and returns them through one endpoint. What the Greenhouse rows look like is on the Greenhouse source page, and the keyless endpoints are described on the sandbox page.