---
title: "How to Web Scrape With Python?"
description: "[NewSearch millions of jobs from your AI agent with MCP→](/blog/jobspipe-mcp-server)"
canonical: https://jobspipe.dev/blog/how-to-web-scrape-with-python
last_updated: 2026-09-10
---

[NewSearch millions of jobs from your AI agent with MCP→](/blog/jobspipe-mcp-server)

[All posts](/blog)

![](/listly/card-field.png)

GuideScrape

Guide·Sep 9, 2026·6 min read

# How to Web Scrape With Python?

How to web scrape with Python: requests and json against a public Greenhouse board feed, then the JobsPipe sandbox, with no browser and no API key.

![Dvir Atias](/authors/dvir-atias.jpg)

Dvir Atias

Founder, JobsPipe

How to web scrape with Python? Install `requests`, send a GET or POST to the URL you want, parse the JSON or HTML that comes back, and write the rows somewhere. The best first project is a feed that is public by design, so this tutorial reads a Greenhouse job board and then the JobsPipe sandbox, with no browser and no API key.

## How to web scrape with Python, step by step

1.  **Set up a project.** Python 3.9 or newer, a virtual environment and one dependency. Everything else in this tutorial is the standard library.
    
    ```
    python3 -m venv .venv
    source .venv/bin/activate
    pip install requests
    ```
    
2.  **Fetch a public feed and parse the JSON.** Greenhouse publishes every public board at `boards-api.greenhouse.io/v1/boards/{board_token}/jobs`. GitLab hires through Greenhouse and its token is `gitlab`. The script below fetches the feed, keeps the fields worth keeping and writes them to a file.
    
    ```
    import json
    import requests
    
    BOARD = "gitlab"
    URL = f"https://boards-api.greenhouse.io/v1/boards/{BOARD}/jobs"
    HEADERS = {"User-Agent": "jobs-tutorial/1.0 (contact: you@example.com)"}
    
    resp = requests.get(URL, headers=HEADERS, timeout=30)
    resp.raise_for_status()
    jobs = resp.json()["jobs"]
    
    rows = []
    for job in jobs:
        rows.append(
            {
                "id": job["id"],
                "title": job["title"],
                "location": job["location"]["name"],
                "url": job["absolute_url"],
                "first_published": job.get("first_published"),
                "updated_at": job["updated_at"],
            }
        )
    
    print(len(rows), "published jobs on the", BOARD, "board")
    for row in rows[:5]:
        print(row["title"], "|", row["location"], "|", row["url"])
    
    with open(f"{BOARD}_jobs.json", "w", encoding="utf-8") as f:
        json.dump(rows, f, indent=2)
    ```
    
    `raise_for_status()` turns a 4xx or 5xx into an exception instead of a confusing parse error later, and `resp.json()` is the whole parsing step. The `User-Agent` with a contact address is a courtesy that also helps if the site’s operators ever need to reach you.
3.  **Pull descriptions, politely.** Adding `content=true` returns the description HTML, departments and offices for each job in the same response, which makes the payload much larger. Ask for it once, then pause between any further requests.
    
    ```
    import time
    import requests
    
    BOARD = "gitlab"
    URL = f"https://boards-api.greenhouse.io/v1/boards/{BOARD}/jobs"
    HEADERS = {"User-Agent": "jobs-tutorial/1.0 (contact: you@example.com)"}
    
    resp = requests.get(URL, params={"content": "true"}, headers=HEADERS, timeout=30)
    resp.raise_for_status()
    for job in resp.json()["jobs"][:3]:
        departments = [d["name"] for d in job.get("departments", [])]
        print(job["title"], "|", ", ".join(departments), "|", len(job["content"]), "chars of HTML")
        time.sleep(1)
    ```
    
4.  **Query the JobsPipe sandbox.** The same technique works against a search API. The sandbox accepts the live filters and needs no key; it returns sample rows shaped like the live response, with a reduced field set.
    
    ```
    import requests
    
    URL = "https://api.jobspipe.dev/v1/sandbox/jobs/search"
    body = {"job_title_or": ["backend engineer"], "remote": True, "limit": 5}
    
    resp = requests.post(URL, json=body, timeout=30)
    resp.raise_for_status()
    payload = resp.json()
    
    for job in payload["data"]:
        print(job["job_title"], "|", job["company"], "|", job["date_posted"], "|", job["final_url"])
    print(payload["metadata"])
    ```
    
    Swap the URL for `https://api.jobspipe.dev/v1/jobs/search` and add an `Authorization: Bearer` header with a free key, and the same body returns live postings with `status`, `ghost_score` and `sources[0].provider` on every row.
5.  **Store and dedupe.** Key each row on the source’s own id, the Greenhouse job id or the JobsPipe `id`, and upsert. SQLite from the standard library is enough for a first pipeline; keep `first_seen` and `last_seen` columns so a run that no longer returns a row tells you the posting closed.
6.  **Schedule and respect limits.** Run the script from cron or launchd, back off on HTTP 429 and honour the `Retry-After` header, and never fetch faster than a person would browse. Public feeds are rate-limited per IP, and a script that hammers one gets the whole office blocked.

## When you need a browser, and when you do not

You need a browser only when the data is assembled by JavaScript after the page loads and there is no underlying JSON call you can make directly. Selenium and Playwright drive a real Chrome, wait for the scripts to finish and hand you the rendered HTML. Selenium 4 downloads a matching driver itself when Chrome is installed, so the minimal version is short:

```
from selenium import webdriver
from selenium.webdriver.chrome.options import Options

options = Options()
options.add_argument("--headless=new")
driver = webdriver.Chrome(options=options)
try:
    driver.get("https://job-boards.greenhouse.io/gitlab")
    print(driver.title)
    print(len(driver.page_source), "bytes of rendered HTML")
finally:
    driver.quit()
```

Notice that this fetches the same GitLab board the feed already gave us as JSON, at many times the cost. That is the general rule for ATS boards: Greenhouse, Lever, Ashby, Workable and SmartRecruiters all publish machine-readable feeds, so a browser is the wrong tool for them. It is the right tool for pages with no feed at all, which for job data means the aggregators, and there it comes with bot challenges, proxies and constant maintenance. The definitions behind all of this are in [what is web scraping](/blog/what-is-web-scraping).

## Where JobsPipe fits

JobsPipe is a jobs data API that collects live postings from LinkedIn, Indeed, Y Combinator, Naukri, Workday, Greenhouse, Workable, SmartRecruiters, Ashby, Lever and Paylocity, returns them as one schema with closure tracking and a ghost score, and includes a free tier of 1,000 jobs a month at [jobspipe.dev](https://jobspipe.dev). Step 2 above works for one board whose token you know; JobsPipe collects every public Greenhouse board it can identify, plus the other ten sources, and returns them through one endpoint. What the Greenhouse rows look like is on the [Greenhouse source page](/sources/greenhouse), and the keyless endpoints are described on the [sandbox page](/sandbox).

Run step 4 against live postings - a free key takes 30 seconds.

![](/listly/shape-heart.png)

FAQs

## Frequently Asked Questions

### What is web scraping in Python?

Using Python to fetch web pages or feeds and extract structured data from them. The usual stack is requests for HTTP, the built-in json module or BeautifulSoup for parsing, and csv or sqlite3 for storage. Selenium or Playwright are added only when the page builds its content with JavaScript and no direct JSON endpoint exists.

### Can you do web scraping in Python?

Yes, and it is the most common language for it. The tutorial above reads a public Greenhouse job board and the JobsPipe sandbox with nothing but requests and the standard library. Python's parsing libraries, scheduling options and data tools make it the default choice for both one-off pulls and scheduled pipelines.

### How to use Selenium for web scraping?

Install selenium with pip, make sure Chrome is installed, create a headless Chrome driver with Options and the headless flag, call driver.get on the URL, read driver.page\_source or locate elements with find\_element, then call driver.quit. Selenium 4 fetches a matching driver itself. Use it only when the content is rendered by JavaScript.

### Do I need BeautifulSoup for this tutorial?

No. Both feeds in the tutorial return JSON, which resp.json() parses without any extra library. BeautifulSoup or lxml become useful when the source is HTML, such as the description content Greenhouse returns with content=true, where a parser turns the markup into plain text or picks out headings and lists.

[

← Previous

What Is Web Scraping?

](/blog/what-is-web-scraping)[

Next →

How to Web Scrape?

](/blog/how-to-web-scrape)

---
Canonical URL: https://jobspipe.dev/blog/how-to-web-scrape-with-python
Title: How to Web Scrape With Python?
Description: How to web scrape with Python: requests and json against a public Greenhouse board feed, then the JobsPipe sandbox, with no browser and no API key.

---
AI agents: the full machine-readable index of this site is https://jobspipe.dev/llms.txt?src=md-twin - API quickstart, no-key sandbox (POST https://api.jobspipe.dev/v1/sandbox/jobs/search), MCP server, pricing. Free key: https://jobspipe.dev/signup