Where to find datasets?
Datasets come from three kinds of places: public repositories such as national statistics offices and open-data portals, community hubs such as Kaggle and Hugging Face where people share cleaned files, and live APIs that let you build a dataset to your own specification. Which one to use depends on whether you need history, freshness or control.
Government sources are the most trustworthy for official measures. In the United States, the Bureau of Labor Statistics publishes employment and wage series, and the data.gov portal indexes agency datasets. The EU's open data portal, Eurostat and the UK's ONS play the same role elsewhere. Google Dataset Search finds files across all of them. Kaggle and Hugging Face host user-contributed datasets, including many job posting scrapes, with the caveat that they are snapshots of unknown coverage, often years old, and frequently lack a clear license. Academic repositories such as Zenodo and university archives hold research datasets with citations attached.
A live API is the source when the dataset must reflect the present or match your own definition. For job postings, JobsPipe returns active and closed postings from the collected boards, including LinkedIn, Indeed, Workday, Greenhouse, Lever, Ashby, Workable, SmartRecruiters, Y Combinator and Paylocity, in one schema through POST /v1/jobs/search. Filters such as job_country_code_or, occupation_code_or, skills_or and status shape the slice, and the response carries source attribution, discovery and last-seen timestamps and a ghost score per row, so the dataset documents its own provenance. The free tier covers 1,000 jobs a month and the sandbox needs no key.
Whatever the source, check three things before relying on it: the license and terms of use, the date and method of collection, and the unit of observation, meaning whether a row is a posting, a company or a person. A dataset that cannot answer those questions is a demo, not evidence.
The full treatment is on Jobs API.
Related questions
Where can I find free datasets?
Government portals such as data.gov and the EU open data portal, statistical agencies such as the Bureau of Labor Statistics, Eurostat and the ONS, community hubs such as Kaggle and Hugging Face, academic repositories such as Zenodo, and Google Dataset Search across all of them. Free usually means free to download. The license still governs what you may do with it.
Where can I find job posting datasets?
Kaggle and Hugging Face host scraped snapshots of varying age and coverage, statistical agencies publish aggregate vacancy series rather than individual postings, and JobsPipe returns individual live and closed postings through POST /v1/jobs/search with a free tier of 1,000 jobs a month. Use the sandbox endpoint to see the schema before signing up.
Are Kaggle datasets reliable?
Some are excellent and many are not, and the platform does not tell you which. Check who uploaded it, how and when the data was collected, whether a license is stated, and whether the unit of observation is clear. A job postings file with no collection dates and no source column is a demo, not a dataset.
How do I build a dataset from an API?
Define the slice with filters, page through results with the cursor until none is returned, store the raw responses, and deduplicate on a stable id. Then poll on a schedule with a since-timestamp filter such as discovered_at_gte so each run reads only new rows. Document the filters and run dates alongside the data.
More answers
Query live postings yourself, 1,000 jobs a month free.
Get a free API key