---
title: "Job description dataset: what a posting record contains, the schema, and how to pull a live slice"
description: "[NewSearch millions of jobs from your AI agent with MCP→](/blog/jobspipe-mcp-server)"
canonical: https://jobspipe.dev/blog/job-description-dataset
last_updated: 2026-09-10
---

[NewSearch millions of jobs from your AI agent with MCP→](/blog/jobspipe-mcp-server)

[All posts](/blog)

![](/listly/card-field.png)

ReferenceDescription

Reference·Sep 9, 2026·8 min read

# Job description dataset: what a posting record contains, the schema, and how to pull a live slice

Every field in a collected job posting record, the schema as a table, a sample row, the API loop that pulls a dataset slice with skills\_or and cursor, how complete each field is on every collected board, a frozen-snapshot versus live comparison for Kaggle-style downloads, and what the licence lets you do with the text.

![Dvir Atias](/authors/dvir-atias.jpg)

Dvir Atias

Founder, JobsPipe

A job description dataset is what you reach for when the question needs the text: what employers ask for, how they describe a role, which tools and skills appear together, and how that differs by seniority, industry or country. This post describes the record a posting becomes once collected, the schema, how to pull a slice through the API, how complete each field is on each board, and where a frozen download beats a live feed.

## What is a job description dataset?

A job description dataset is a structured collection of job postings where each record holds the full description text together with parsed fields: title, employer, location, pay, seniority, occupation code and extracted skills. It differs from a job postings list in keeping the text and from a résumé dataset in describing demand rather than supply.

Three groups use them. Machine learning teams train and evaluate title classifiers, skill extractors and salary models. Analysts measure demand by occupation, industry and skill. Product teams build matching, search and alerting on top. Each cares about a different part of the record, which is why the schema below is wide.

## What is in a job posting record?

Every record JobsPipe returns from `POST /v1/jobs/search` carries the same fields whichever board it came from. Grouped by what they are for:

-   **Identity and timing.** `id`, `job_title`, `company`, `company_domain`, `date_posted`, `discovered_at`, `last_seen_at`, `verified_at`.
-   **Liveness.** `status`, `closed_at`, `closed_reason`, `ghost_score`.
-   **Pay.** `min_annual_salary_usd`, `max_annual_salary_usd`, `salary_currency`, `salary_source`.
-   **Place.** `location`, `country_code`, `remote`, `work_arrangement`.
-   **Classification.** `normalized_title`, `occupation_code` (ISCO-08), `seniority`, `job_function`, `isic_division`.
-   **Skills.** `keyword_slugs`, `technology_slugs`, `esco_skills`.
-   **Text and provenance.** `description` and `sources`, where `sources[0].provider` names the board.

## The schema

| Field | Type | Notes |
| --- | --- | --- |
| id | string | Stable per source; use job_ids to fetch one record again |
| job_title | string | Title exactly as the employer wrote it |
| normalized_title | string \| null | Cleaned title with level, location and noise removed |
| company | string \| null | Employer name; company_domain carries the resolved domain |
| description | string \| null | Full posting text as published |
| date_posted | date \| null | Date the employer states; discovered_at is when JobsPipe first saw it |
| discovered_at / last_seen_at / verified_at | timestamp | Crawl-log timestamps: first sighting, last sighting, last liveness check |
| status / closed_at / closed_reason | string \| timestamp | active or closed, when, and how it left the board |
| ghost_score | integer 0-100 \| null | Daily score of how likely the posting is not being filled |
| min_annual_salary_usd / max_annual_salary_usd | number \| null | Parsed and normalised to annual USD; salary_source says where it came from |
| location / country_code / remote / work_arrangement | string \| boolean | Raw location plus ISO country and remote or hybrid flags |
| occupation_code / occupation_label | string \| null | ISCO-08 four-digit code and its label |
| seniority / job_function | string \| null | entry_level, mid_level, senior, director, executive; and the function |
| isic_division / isic_division_label | string \| null | Employer industry, ISIC Rev 4 division |
| keyword_slugs / technology_slugs | string[] | Skills extracted from title and description; technologies are the subset |
| esco_skills | { id, label }[] | ESCO skill concepts matched in the text |
| sources | { provider, url, seen_at }[] | Where the posting was collected; sources[0].provider is the board |

## How to pull a slice through the API

A dataset slice is a filter plus a cursor loop. This pulls active postings naming Python or SQL, 100 at a time, and follows `metadata.next_cursor` until it is null:

```
curl -X POST https://api.jobspipe.dev/v1/jobs/search \
  -H "Authorization: Bearer $JOBSPIPE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "skills_or": ["python", "sql"],
    "status": "active",
    "posted_at_max_age_days": 30,
    "limit": 100
  }'

# next page: pass the cursor from metadata.next_cursor
curl -X POST https://api.jobspipe.dev/v1/jobs/search \
  -H "Authorization: Bearer $JOBSPIPE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "skills_or": ["python", "sql"],
    "status": "active",
    "posted_at_max_age_days": 30,
    "limit": 100,
    "cursor": "eyJ..."
  }'
```

Billing is one credit per job returned, so size the slice first: the sandbox endpoint at `POST https://api.jobspipe.dev/v1/sandbox/jobs/search` takes the same filters with no key and returns a field subset, and the free tier covers 1,000 jobs a month. Narrow with `occupation_code_or`, `job_country_code_or`, `job_seniority_or` or `source_or` to keep the slice to the rows you will use. Add `status: "closed"` when the dataset needs the closed cohort too.

## A sample row

Trimmed to the fields above, with a placeholder employer:

```
{
  "id": "4123456789",
  "job_title": "Senior Data Engineer (Remote, US)",
  "normalized_title": "data engineer",
  "company": "Example Corp",
  "company_domain": "example.com",
  "date_posted": "2026-09-01",
  "discovered_at": "2026-09-01T14:22:10Z",
  "last_seen_at": "2026-09-08T03:10:44Z",
  "verified_at": "2026-09-08T03:10:44Z",
  "status": "active",
  "closed_at": null,
  "closed_reason": null,
  "ghost_score": 12,
  "min_annual_salary_usd": 150000,
  "max_annual_salary_usd": 185000,
  "salary_currency": "USD",
  "location": "Austin, TX (Remote)",
  "country_code": "US",
  "remote": true,
  "work_arrangement": "remote",
  "occupation_code": "2521",
  "occupation_label": "Database designers and administrators",
  "seniority": "senior",
  "job_function": "Engineering",
  "isic_division": "62",
  "keyword_slugs": ["python", "sql", "airflow", "aws", "communication"],
  "technology_slugs": ["python", "airflow", "aws"],
  "esco_skills": [{ "id": "00000000-0000-4000-8000-000000000000", "label": "use databases" }],
  "sources": [{ "provider": "greenhouse", "url": "https://boards.greenhouse.io/example/jobs/1", "seen_at": "2026-09-01T14:22:10Z" }],
  "description": "Example Corp is hiring a Senior Data Engineer to own our ..."
}
```

## How complete is each field, by board?

A dataset is only as good as its emptiest column. The shares below are the fraction of active postings on each collected board where the field is populated, which is the first thing to check before you train on one board or merge several.

| Board | Salary | Seniority | Employment type | Remote flag | Thin description |
| --- | --- | --- | --- | --- | --- |
| Ashby | 56.1% | 39.0% | 100.0% | 24.9% | <0.5% |
| Greenhouse | 46.8% | 38.2% | 0.0% | 15.4% | <0.5% |
| Indeed | 36.8% | 17.9% | 71.6% | 3.9% | 1.0% |
| Lever | 22.7% | 29.6% | 55.3% | 45.5% | 6.7% |
| LinkedIn | 22.1% | 48.6% | 99.1% | 4.3% | 1.7% |
| Paylocity | 10.7% | 7.4% | 0.0% | 2.8% | 2.1% |
| SmartRecruiters | 5.3% | 26.3% | 99.9% | 3.8% | 0.7% |
| Workable | 19.6% | 22.3% | 79.0% | 16.7% | <0.5% |
| Workday | 23.3% | 32.9% | 95.1% | 5.6% | <0.5% |
| Y Combinator | 67.7% | 36.8% | 100.0% | 36.5% | 1.1% |

Two consequences for dataset builders. Salary is present on a minority of postings on most boards, so a salary model trained without a `min_salary_usd` filter learns mostly nulls. And no board is complete on every field, so a model trained on one board should be evaluated on the others before it is trusted on the merged corpus. Thin descriptions, by contrast, are rare on every board, so the text column is the one you can rely on.

How this was measured

Measured from active postings per collected board. Generated 2026-09-09. Shares only: JobsPipe publishes percentages, not counts.

Each share is computed within one board’s active postings. Salary counts a parsed minimum or maximum; seniority and employment type count a non-empty normalised value; remote counts the remote flag. Thin description comes from the ghost scorer and is measured over scored postings, seen at least twice or older than 14 days.

```
curl -X POST https://api.jobspipe.dev/v1/jobs/search \
  -H "Authorization: Bearer $JOBSPIPE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"source_or":["greenhouse"],"min_salary_usd":1,"limit":50}'
```

## Building a training set from the slice

A few habits make the slice usable as training data rather than a pile of JSON.

-   **Pull both cohorts.** Run the query with `status: "active"` and again with `status: "closed"`. The closed cohort is the only place to learn what a posting that was actually filled or withdrawn looks like.
-   **Keep the raw title and the board.** `job_title` and `sources[0].provider` are the two fields you will need when a label looks wrong, and the board is the right unit for a held-out split.
-   **Use the classification fields as weak labels.** `occupation_code`, `seniority` and `job_function` are machine-assigned. They are good enough to bootstrap a classifier and should be audited on a sample before they are treated as ground truth.
-   **Filter label noise with the liveness fields.** A posting with a high `ghost_score` or a `discovered_at` months old is a stale example of employer demand; drop or down-weight it.
-   **Split by time, not at random.** `discovered_at` gives you a clean temporal split, which is the split a production model will face.

## Kaggle job description datasets vs a live dataset

The Kaggle job description datasets are frozen snapshots: one crawl of one or two boards, published once, with no close dates and no refresh. They are free and fine for a classroom or a first model. A live dataset is a query against postings that are still being collected, with liveness, the board named on every row and the same schema every day.

|  | Frozen snapshot (Kaggle and similar) | Live dataset (API) |
| --- | --- | --- |
| Freshness | Fixed at crawl date, often years old | Postings first sighted today are queryable today |
| Liveness | No close dates; every row looks open | status, closed_at, closed_reason, ghost_score on every row |
| Coverage | One or two boards, one country | Ten boards, worldwide, the board named on every row |
| Schema | Whatever the crawler kept | Same fields on every row, null when unknown |
| Licence | Varies per upload, often unclear | Governed by the API terms |
| Cost | Free | Free tier, then per job returned |

The wider survey of where to get postings data, from USAJOBS to bulk vendors, is in [job market datasets: 7 real sources](/blog/job-postings-dataset).

## Licensing: what you can do with the data

The parsed fields are facts, and facts are yours to analyse, aggregate and publish. The description text remains the employer’s; JobsPipe’s [terms](/terms) govern how you may store and use what the API returns, and republishing full descriptions as a dataset of your own is the case to read them for. Postings contain little personal data, but recruiter names and emails inside a description are personal data where privacy law applies, so strip them unless you need them. The legal background is in [is web scraping legal](/blog/is-web-scraping-legal).

For the classification side of the same records, the [job title taxonomy](/blog/job-title-taxonomy) post explains how `normalized_title`, `occupation_code` and `seniority` are produced, and the [skills taxonomy](/blog/skills-taxonomy) post covers the slugs.

Pull a job description dataset slice - same schema on every board, free tier first.

![](/listly/shape-heart.png)

FAQs

## Frequently Asked Questions

### What is a job description dataset?

A structured collection of job postings where each record holds the full description text together with parsed fields: title, employer, location, pay, seniority, ISCO-08 occupation code and extracted skills. It differs from a plain job postings list by keeping the text, and from a résumé dataset by describing demand rather than supply. ML teams, analysts and product teams each use a different part of the record.

### What fields are in a job posting record?

Identity and timing (id, job\_title, company, company\_domain, date\_posted, discovered\_at, last\_seen\_at, verified\_at), liveness (status, closed\_at, closed\_reason, ghost\_score), pay (min\_annual\_salary\_usd, max\_annual\_salary\_usd, salary\_currency, salary\_source), place (location, country\_code, remote, work\_arrangement), classification (normalized\_title, occupation\_code, seniority, job\_function, isic\_division), skills (keyword\_slugs, technology\_slugs, esco\_skills), the description text and the sources array naming the board.

### How do I pull a job description dataset through the API?

POST your filters to https://api.jobspipe.dev/v1/jobs/search with a bearer key, for example skills\_or, status, posted\_at\_max\_age\_days and limit, then pass metadata.next\_cursor back as cursor until it is null. Billing is one credit per job returned, the free tier covers 1,000 jobs a month, and the sandbox endpoint at /v1/sandbox/jobs/search takes the same filters with no key and returns a field subset for sizing the slice.

### How complete are the fields on each job board?

It varies by board and the post publishes the shares: the fraction of active postings on each collected board with a stated salary, a labelled seniority, a stated employment type and a remote flag, plus the share the ghost scorer flags for a thin description. Salary is present on a minority of postings on most boards and no board is complete on every structured field, while thin descriptions are rare everywhere, which changes how you should train and evaluate models on the data.

### How does a Kaggle job description dataset compare to a live one?

Kaggle datasets are frozen snapshots of one or two boards, published once, with no close dates, no refresh and often an unclear licence; they are free and fine for a classroom or a first model. A live dataset is a query against postings still being collected, with status and close dates on every row, ten boards with the board named on every row, and the same schema every day, on a free tier and then per job returned.

### Can I republish job descriptions from the dataset?

The parsed fields are facts and are yours to analyse, aggregate and publish. The description text remains the employer's, and JobsPipe's terms govern how you may store and use what the API returns; republishing full descriptions as a dataset of your own is the case to read them for. Recruiter names and emails inside descriptions are personal data where privacy law applies and should be stripped unless needed.

[

← Previous

Is web scraping legal? Public data, terms of service, hiQ v. LinkedIn, and the job-data answer

](/blog/is-web-scraping-legal)[

Next →

Job title taxonomy: how raw titles become normalized titles, ISCO-08 codes, seniority and function

](/blog/job-title-taxonomy)

---
Canonical URL: https://jobspipe.dev/blog/job-description-dataset
Title: Job description dataset: what a posting record contains, the schema, and how to pull a live slice
Description: Every field in a collected job posting record, the schema as a table, a sample row, the API loop that pulls a dataset slice with skills_or and cursor, how complete each field is on every collected board, a frozen-snapshot versus live comparison for Kaggle-style downloads, and what the licence lets you do with the text.

---
AI agents: the full machine-readable index of this site is https://jobspipe.dev/llms.txt?src=md-twin - API quickstart, no-key sandbox (POST https://api.jobspipe.dev/v1/sandbox/jobs/search), MCP server, pricing. Free key: https://jobspipe.dev/signup