---
title: "How to create a dataset?"
description: "**Canonical URL:** https://jobspipe.dev/answers/how-to-create-a-dataset"
canonical: https://jobspipe.dev/answers/how-to-create-a-dataset
last_updated: 2026-09-10
---

# How to create a dataset?

> To create a dataset, define the unit of observation and the fields you need, collect records from a source that can supply them, deduplicate on a stable key, label or enrich the fields your analysis depends on, and document the license, collection date and method. The documentation is what turns a pile of rows into a dataset.

**Canonical URL:** https://jobspipe.dev/answers/how-to-create-a-dataset
**Topic:** Jobs API (https://jobspipe.dev/jobs-api)

Take a job postings dataset as the worked example. The unit is one posting on one board, and the fields are title, company, location, remote flag, salary where stated, posted date and status. Collect with POST /v1/jobs/search: send the filters that define the slice, such as job_country_code_or, occupation_code_or or job_title_or, with a limit, and read the page back. The response includes a cursor. Send it in the next request and repeat until no cursor is returned. That loop is the whole extraction. Store the raw JSON as you go so the dataset can be rebuilt if the schema changes.

Keep the dataset current with discovered_at_gte set to the timestamp of the last run, which returns only postings JobsPipe first saw since then, so a nightly job reads new rows instead of the whole corpus. Deduplicate on the id field, which is stable per posting and per board. The same role on two boards is two rows by design, and sources tells you which board each came from. To capture closures, run a second pass with status set to closed and record closed_at and closed_reason, which turns a snapshot into a lifecycle dataset.

Labeling is often already done: each row carries a ghost_score for the likelihood that the posting is not a real opening, technology_slugs for the tools it names, and min_annual_salary_usd where a salary was parsed. Add your own labels on top, then write the data card: the filters used, the dates of each run, the boards included, the dedup rule and the terms of use. Rebuilding from the same card should produce the same dataset.

## Related questions

### How do I create a dataset for machine learning?

The same steps, plus labels and splits. Define the unit and target, collect rows, deduplicate, label the target consistently, then split into training, validation and test sets in a way that prevents leakage, such as by company or by date. For posting data, existing fields like ghost_score or a seniority bucket can serve as targets.

### How big does a dataset need to be?

As large as the question demands and no larger than you can document. A market snapshot for one occupation in one country can be small and still be sound. A model that must generalize across industries needs breadth. Quality, provenance and a stable dedup key matter more than the row count.

### How do I keep a dataset up to date?

Run the same query on a schedule with discovered_at_gte set to the last run's timestamp, so only newly discovered postings come back, and run a second pass with status set to closed to record closures. Append each run with its date rather than overwriting, so the dataset carries its own history.

### Can I build a job postings dataset without an API key?

You can build and test the pipeline against the sandbox endpoint, which returns sample rows in the live schema with no key, but the rows are samples, not the corpus. A live key from the dashboard is needed for real postings. The free tier covers 1,000 jobs a month.

## More answers

- [Where to find datasets?](https://jobspipe.dev/answers/where-to-find-datasets)
- [What is a data API?](https://jobspipe.dev/answers/what-is-a-data-api)
- [Where to find API key?](https://jobspipe.dev/answers/where-to-find-your-api-key)

Data behind these answers is published as shares only; every figure can be reproduced against the JobsPipe API (https://jobspipe.dev/jobs-api).

---
AI agents: the full machine-readable index of this site is https://jobspipe.dev/llms.txt?src=md-twin - API quickstart, no-key sandbox (POST https://api.jobspipe.dev/v1/sandbox/jobs/search), MCP server, pricing. Free key: https://jobspipe.dev/signup