r/datasets • • 1h ago

resource [self-promotion] SDWIS water-system files: keep the quarter with the PWSID

1 Upvotes

EPA's SDWIS download is a useful source for public-water-system records, but PWSID alone is not the snapshot key. The field documentation identifies SUBMISSIONYEARQUARTER plus PWSID for the public water system table.

Source and dictionary: https://echo.epa.gov/tools/data-downloads/sdwis-download-summary

If you combine quarters, retain the quarter field. Otherwise the same system can look like a duplicate or overwrite a previous snapshot. When joining the geographic-area table, expect more than one geography row for a system. The mailing ZIP field and the served-ZIP field also describe different things.

A simple validation is to count geography rows per system and quarter before joining them to system-level population. Summing a whole-system population once for every ZIP would count the same people repeatedly. Missing served geography should stay unknown, rather than becoming zero service. These are proposed checks, not a claim that I have audited the full national extract.

I publish GOV.HOUSE. Its ZIP-based lookup offers a more accessible way to find reported systems and their source context: https://gov.house/drinking-water/?utm_source=reddit&utm_medium=social . Reported service areas can be incomplete, so confirm the utility on a water bill. A utility record is not a test of an individual home's tap water.


r/datasets • • 9h ago

resource I wrote some weird prompts for my drawing game and ended up with a pretty interesting dataset

3 Upvotes

I’ve been looking through what people have drawn for some of the prompts I added to Artbitrator. One was “plot twist”. Someone drew an actual strip of film twisting round on itself, with little people in the frames. I thought that one was mental. Someone else drew a smiling mask with a sly face peeking out from behind it. Same prompt.

“Robot having an existential crisis” is another one I added. One person drew a robot clutching its head over an open book. Someone else got the idea across with a boxy robot, a frown and a load of question marks.

That’s the bit I find interesting. I’ve got all these different ways of drawing the same oddly specific things, just from people playing a game. I don’t know how I’d have easily collected something like this before AI.

Anyway, I’m curious what people here make of it. If I put together a sample with the prompts, drawings and replays, what would you want to look at first?

Plot twist drawings · Robot having an existential crisis drawings
or just browser the whole dataset. -> https://artbitrator.com/gallery


r/datasets • • 17h ago

resource Poverty tables: keep the measure before joining the dollars

1 Upvotes

For U.S. poverty data, Census thresholds and HHS poverty guidelines should be identified as different measures before joining dollar values by year or family size.

Census publishes historical threshold tables by family size and number of children: https://www.census.gov/data/tables/time-series/demo/income-poverty/historical-poverty-thresholds.html

The Census FAQ explains the boundary: thresholds are used for statistical poverty measurement and are the same nationwide. HHS guidelines have administrative uses and separate schedules for the contiguous 48 states/DC, Alaska and Hawaii. The FAQ is here: https://www.census.gov/help/topics/faq.difference-between-poverty-thresholds-and-poverty-guidelines.html

A proposed extraction key would retain issuing agency, measure name, year, family size, family composition where applicable, geographic schedule where applicable and unit. This is a schema suggestion, not a new cleaned dataset of my own.

A table used to count statistical poverty is not automatically an eligibility table for a benefit program. Program rules need their own source. Neither measure is a full local cost-of-living budget, so a county-cost estimate should not silently replace the published threshold in a poverty classification.


r/datasets • • 1d ago

dataset [self-promotion] Nasdaq/NYSE reverse stock splits and listing deficiency notices, parsed from SEC filings (CSV, CC BY 4.0)

1 Upvotes

Disclosure: I run Signal8, the site that publishes these files.

We parse SEC filings for two events that are scattered across thousands of 8-Ks and hard to collect:

  • Reverse stock splits (977 rows from 539 companies since Dec 2025): every filing in which a Nasdaq or NYSE company fixed a reverse split ratio or announced its effective date. Columns include the 1-for-N ratio, the effective date as the filing states it, form type, CIK and a link to the filing.
  • Listing deficiency notices (1,123 rows from 652 companies since Nov 2025): 8-K Item 3.01 and 6-K notices (bid price under $1, stockholders' equity, late filings, delisting determinations), with the rules cited and the compliance deadline the company states.

CSV downloads, stable URLs, data dictionary and method: https://signal8.ai/datasets

Mirrors: Hugging Face (huggingface.co/datasets/signal8/reverse-splits and signal8/deficiency-notices) and Zenodo (DOI 10.5281/zenodo.24394950 and 10.5281/zenodo.24394958).

One thing the two files show together: of 171 companies that got a minimum bid price notice before May 2026, 96 (56%) later filed a reverse split, a median of 147 days after the notice.

Limits, up front: values are only what the filings state (a blank means the filing did not say, not zero); the ticker is the issuer's current SEC ticker, which can differ from the ticker at filing time; there are no prices in the files. Corrections welcome; every row links to its source filing so you can check it.


r/datasets • • 1d ago

dataset [self-promotion] Snowchive - every published Snowden document in one place

Thumbnail snowchive.org
2 Upvotes

r/datasets • • 1d ago

dataset PhiloText: an open corpus of philosophy and psychology texts

2 Upvotes

Releasing PhiloText, an open corpus of 0.5B tokens of philosophy and psychology texts (disclosure: my own dataset).

It gathers philosophy encyclopedias (SEP, IEP), public-domain books, reviews, textbooks and educational web pages.

I used it to train PhiloLM, a 112M base model trained from scratch on (only!) 1.5B tokens in 15 hours on a $350 GPU (Intel Arc A770).

Try it in your browser: https://gustavecortal.com/philosophy-lm/

Data: https://huggingface.co/datasets/gustavecortal/PhiloText

Model: https://huggingface.co/gustavecortal/PhiloLM

Happy to answer questions about everything :)


r/datasets • • 1d ago

resource [self-promotion] IRS ZIP income data: keep the unit of observation

6 Upvotes

The IRS publishes downloadable 2023 ZIP Code tax data as CSV files, including an all-states file with AGI. The official page and documentation are here:

https://www.irs.gov/statistics/soi-tax-stats-individual-income-tax-statistics-2023-zip-code-data-soi

Before calling an extract “household income,” preserve its actual unit: these are tax-return data classified by state, ZIP Code and adjusted-gross-income size. The IRS says the return count approximates the number of households. An approximation is not an identity, and a return is not a person.

For a reproducible comparison, I would retain tax year, geography, AGI bracket, return count, the selected amount field and its documented units. Check whether a row is a total or a component before summing brackets, and distinguish a mean derived from aggregate amounts from a median. A mean cannot be relabeled as the income of the middle filer.

I publish GOV.HOUSE. Its income browser provides a way to explore the IRS-based figures by place, with the mean/median and historical/estimated distinctions shown on the page: https://gov.house/income/?utm_source=reddit&utm_medium=social

The IRS files and record layout remain the source for a research extract. The browser is a convenience, not a replacement for checking the unit and year.


r/datasets • • 1d ago

resource [self-promotion] H.P. Lovecraft's Comprehensive Bibliography

2 Upvotes

I could not find any truly comprehensive datasets / archives containing H.P Lovecraft's entire bibliography so I decided to make one myself by scraping from wikisource.org, hplovecraft.com and Lovecraft-Letters. It is not complete but bigger than everything else I have found.

I share it here in the chance it might be useful to someone else.

Dataset:

The GitHub repository also includes the scraper used to collect the data.


r/datasets • • 1d ago

request [self promotion] datasets for public bodies in Ireland

3 Upvotes

Hi,

I’ve pulled together a bunch of datasets indexed around public bodies in Ireland, which might be useful to someone.

https://publicinformation.ie

Caveat 1: I used AI to build pretty much all the software and pipelines that pull the data, transform it, and so on. I’m a former software engineer, and started out thinking I’d write most of it, but I’m rusty and it’s just way faster with AI tools

Caveat 2: The datasets contain errors. I’ve tried to publish the error information so at least you can see the errors I’m aware of. There may be unknown errors too.

Caveat 3: I’m not a data engineer. Be gentle.

If you find it useful, I’d appreciate a star for visibility https://github.com/publicinformation-ie/publicinformation-data

If not, tell me why and what I should do better 🙏

Cheers

Dave


r/datasets • • 1d ago

resource Free Dataset: U.S. state reform laws, 2015-2026, state-year panel with effective dates and statute citations

1 Upvotes

I put together a state-by-state panel of pharmacy benefit manager (PBM) reform laws for all 50 states+D.C. , 2015-2026. Each state-year records whether specific provisions were in effect (spread-pricing bans, clawback bans, gag-clause bans, and more), and every row links to the statute or session law it came from, with separate and enacted dates. I built it because most public trackers list when a bill is passed, not when it actually took effect, and that difference matters a difference-in-differences or event study. Free download(CSV in wide and long format, plus a codebook): https://lawpanels.com/#sample If you spot a wrong date or missing law, please tell me. Corrections are the most useful thing I can get right now. Disclosure: I also build custom state-law panels like this for other topics. That's on the same site, but the PBM data is free to use with attribution.


r/datasets • • 1d ago

dataset [Dataset] 964 Coursera courses scored from their most-helpful reviews

1 Upvotes

I scored 964 Coursera courses from the reviews learners voted most helpful, 12,566

reviews in total, about 20 per course, weighted by √(1 + upvotes) so a review with 40 votes counts for more than one with 1, without letting a single review dominate.

Data (JSON, CORS-open, no login):

https://starisk.matiasagelvis.com/data/scores.json

Method:

https://starisk.matiasagelvis.com/method

Per course it has Coursera's headline rating and review count, the score, how many reviews went in, the star spread of those reviews, and each review's contribution, so anything can be recomputed from the file.

On the 798 courses that also carry a headline rating: the median score sits 0.30 stars below Coursera's, 194 of them (24%) a full star or more below, and 103 (13%) 1.5 or more below.

Plots: https://starisk.matiasagelvis.com/findings

Two caveats. The reviews are the ones learners voted helpful, so they're self-selected and skew toward strong opinions, the score is what those reviews say, not a quality grade, and courses with few reviews are noisier. Worth noting anyway: 69% of the reviews that went in are five-star, so the gap isn't a pile of complaints.

Personal project, nothing for sale. Requests for specific courses go through the /wanted page.


r/datasets • • 2d ago

resource [self-promotion] Historical HTS data: preserve the edition and hierarchy

2 Upvotes

For a historical U.S. tariff dataset, the edition is part of the observation. The official USITC archive is the primary starting point for retrieving a schedule associated with a particular publication date:

https://www.usitc.gov/harmonized_tariff_information/hts/archive/list

A reproducible extract can keep edition/revision, source URL, retrieval date, the code as text, its parent heading, description, units and applicable notes. Keeping codes as text avoids losing leading zeroes. Those are suggested fields, not a separate official dataset.

The hierarchy matters as much as the digits. USITC's FAQ explains why a keyword hit is not necessarily the legal classification, and why the eight-digit duty provision and full ten-digit statistical reporting number serve different purposes. Additional chapter 98/99 provisions can also matter; a general-rate column alone is not an all-in import bill.

Official interpretation guide: https://www.usitc.gov/harmonized_tariff_information/frequently_asked_questions

I publish GOV.HOUSE. Its current tariff lookup is a searchable companion for exploring descriptions and codes, rather than an archive or a binding classification service: https://gov.house/tariff-lookup/?utm_source=reddit&utm_medium=social

Use the dated USITC source for a historical extract, and keep the legal notes with it.


r/datasets • • 2d ago

request Looking for pre-2020 IMDB Datasets (ratings)

1 Upvotes

Hello there,

I am trying to find a few pre-2020 title.ratings.tsv.gz datasets from IMDB. 2019 is missing from web archive and I am wondering if anyone got a copy by any chance.

Thanks!


r/datasets • • 2d ago

resource Free Dataset: U.S. state reform laws, 2015-2026, state-year panel with effective dates and statute citations

5 Upvotes

I put together a state-by-state panel of pharmacy benefit manager (PBM) reform laws for all 50 states+D.C. , 2015-2026. Each state-year records whether specific provisions were in effect (spread-pricing bans, clawback bans, gag-clause bans, and more), and every row links to the statute or session law it came from, with separate and enacted dates. I built it because most public trackers list when a bill is passed, not when it actually took effect, and that difference matters a difference-in-differences or event study. Free download(CSV in wide and long format, plus a codebook): https://lawpanels.com/#sample If you spot a wrong date or missing law, please tell me. Corrections are the most useful thing I can get right now. Disclosure: I also build custom state-law panels like this for other topics. That's on the same site, but the PBM data is free to use with attribution.


r/datasets • • 2d ago

resource [self-promotion] US Walmart supercenter parking lot fullness from free Sentinel-1 radar

5 Upvotes

Hi again.

I wanted to see if free Sentinel-1 radar could tell how busy a parking lot is, since the optical images that can actually count cars are expensive. It turns out you don't need to count them at all. Cars are metal and bounce the radar back much harder than asphalt, so a full lot comes out brighter than an empty one. I measure each lot's brightness against the ground around it and compare it to the store's own baseline, so each satellite pass gives a fullness reading for every store.

It covers 3.5k US Walmart supercenters, about 1.9k of which have a mapped lot that gets measured, and it goes back to November 2023 with about 100k rows so far. It's rebuilt every week, so it keeps updating.

Use this link to download it for free: https://mostlyright.md/datasets/walmart-supercenter-parking-lot-fullness-us-since-2023?pass=walmart

Happy to answer questions on how it works.


r/datasets • • 2d ago

discussion [self-promotion] Share counts from 10-K and 10-Q cover pages: four traps we hit comparing 3,692 companies a year apart

3 Upvotes

Disclosure: I work on Equibles, which serves SEC filings, financials and earnings call transcripts to ChatGPT and Claude.

We compared every Nasdaq and NYSE company's share count a year apart, using the number on the cover of its 10-K or 10-Q (the XBRL tag dei:EntityCommonStockSharesOutstanding). It's the most comparable share count in SEC data, but four things nearly broke the comparison:

  • Counts that halve with no split on record. Between two consecutive covers, that's a reverse split our records missed or a tagging error, and we couldn't tell which, so we left those 23 companies out. Splits we did apply had to be backed by the company's own counts or a filing stating the ratio.
  • Scale errors. Three companies' counts jumped more than 500-fold between two covers. That's a scale error in the tagging, not issuance.
  • Splits and discounted offerings look alike. When the count jumps by the same factor that the price fell in a single day, it can be an unrecorded split or a deeply discounted offering. We dropped the returns of the 18 companies where we couldn't tell.
  • Share classes that change. For class A and class B companies we summed the common classes only when every cover in the window listed the same classes. 129 issuers whose classes changed during the year were left out.

What survived: 278 of the 3,692 companies at least doubled their count, 234 of them in the smallest third by trading volume.

If you've built share-count histories from XBRL, how did you handle splits: a corporate-actions feed, or inferring them from the counts themselves?

The filings behind this are searchable from ChatGPT or Claude here: https://equibles.com/mcp


r/datasets • • 2d ago

dataset California Death Row Commutation Data Set

1 Upvotes

Governor Gavin Newsom has announced plans to commute the death sentences of all 548 people on California's death row.

Until now, there was no single public list of the crimes behind those 548 death sentences. So I built one.

The California Death Row Commutation Dataset documents every case, including the capital crimes, the victims, the sentencing county, and the sources. Los Angeles, Riverside, and Orange counties account for 53% of the cases, and the sentences span from 1981 to 2026.

The data was compiled from court documents available online. Where court records could not be located, local news reports were used, so accuracy cannot be guaranteed at 100%. Corrections backed by a source are welcome.

See every case for yourself:
https://docs.google.com/spreadsheets/d/1iKVTYFtsmoBl4YlhUPwSnnmAWkQAizIWw-4JYryXdRw/edit


r/datasets • • 2d ago

resource EIA electricity-price data: combine revenue and sales, not price ratios

1 Upvotes

For US electricity-price work, EIA’s EIA-861M resource is useful because it keeps the underlying sales and revenue data close to the average-price measure. It is a monthly sample survey, not an individual household tariff database.

If you combine areas or utilities, calculate the combined average from total relevant revenue divided by total relevant sales, after reconciling the workbook’s units. Taking a simple average of the displayed price ratios gives each row equal weight regardless of how much electricity it represents.

Keep the sector, geography, period, units and release vintage in the extract. A residential average and an all-sector average answer different questions; neither tells you the exact marginal price on one customer’s bill. This is a method note for using the official files, not a newly collected dataset.

EIA detailed files and documentation:

https://www.eia.gov/electricity/data/eia861m/

EIA explanation of average retail prices:

https://www.eia.gov/tools/Faqs/faq.php?id=507&t=5


r/datasets • • 2d ago

resource ACS comparisons: keep the margin of error

2 Upvotes

If you’re comparing places with American Community Survey estimates, keep each estimate’s margin of error in the same extract. A ranking built only from point estimates can imply a difference the sample does not establish.

Census’s comparison guidance also distinguishes the 1-year and 5-year products. For a 5-year comparison, use non-overlapping periods and the same product rather than reading two overlapping windows as an annual change.

A reproducible comparison file can retain geography code and vintage, table/variable identifier, period, estimate, margin of error and retrieval date. Before computing a difference, check that the variable definition and universe still match. Those fields are a suggested working schema, not a new Census dataset.

The Census ACS 5-year API resource lists Detailed Tables and Comparison Profiles, with examples and geography documentation. Its 2024 Comparison Profiles compare 2020–2024 with 2015–2019 and include significance testing. Data queries require an API key; the resource also links other access routes.

https://www.census.gov/data/developers/data-sets/acs-5year.html

Official comparison guidance:

https://www.census.gov/programs-surveys/acs/guidance/comparing-acs-data.2019.html

Census explanation of margins of error:

https://www.census.gov/help/topics/faq.do-i-need-to-consider-the-margins-of-error-when-comparing-acs-estimates.html


r/datasets • • 2d ago

dataset Dataset for SARIMA-Transformer (Heeelp)

Thumbnail reddit.com
1 Upvotes

r/datasets • • 3d ago

dataset Dataset search help — any ideas welcome

2 Upvotes

Looking for cool datasets or websites to webscrape to run cool analyses or regressions that will help us learn more about crim! Taking any and all ideas. Interested in writing a paper that makes people think something new about the field of criminal law in particular!


r/datasets • • 3d ago

dataset [self promotion] Los Angeles taxes, budgets, program spending and 250,000+ City Council and Board of Supervisors roll-call votes as free CSVs

2 Upvotes

I built Tally LA (tallyla.com) to follow Los Angeles tax dollars from collection to the programs they fund. Every table on the site is a free CSV with no signup: tax rates, monthly tax receipts (Metro's 4 sales taxes, Measure ULA, Measure A), City and County budgets, County spending by department, public programs and grants, State of California payments to LA agencies, approved contracts, and one row per member for every recorded City Council and Board of Supervisors roll call.

Everything comes from official records (City and County budgets, Metro and City receipts, Council Journals, County Statements of Proceedings, Open Fi$Cal), and each row links to the original record. Each file says where its figures come from and how often it updates with most updating nightly. No personal data - only names are elected officials, public agencies, and organizations.

Downloads: https://tallyla.com/downloads

Disclosure: my company Blac Flag LLG builds and pays for it. Corrections welcome at corrections@tallyla.com


r/datasets • • 4d ago

resource Census county building-permit files: monthly, year-to-date and annual are different inputs

4 Upvotes

The Census Building Permits Survey has an official county-level download directory for new privately owned residential construction: https://www2.census.gov/econ/bps/County/

Before combining files, choose the time basis. In the county documentation, CO<YYMM>C is the current month, CO<YYMM>Y is year-to-date, and CO<YYYY>A is annual. Adding successive year-to-date files would count earlier months repeatedly. The year-to-date series can also incorporate late reports and corrections, so it need not equal a sum of previously downloaded current-month snapshots.

The record layout contains separate buildings, housing units and valuation fields. It also separates estimates with imputation from reported-only values. For a county housing-unit panel, choose one of those bases consistently and retain the reference period and download date. Read state and county FIPS fields as text so leading zeros survive the join.

A reproducible starting table would therefore identify county FIPS, reference period, time basis, construction type, measurement field and reported/imputed basis before adding the value. That is a proposed schema, not a tested extract or a new dataset of my own.

Official record layout and filename definitions: https://www.census.gov/construction/bps/sample/cntyasc.pdf

The scope is new residential authorizations. It is not a census of every local renovation or commercial permit, and it does not show completed homes.


r/datasets • • 4d ago

dataset [self-promotion] US IEEPA tariff-refund claim sales and pass-through obligations, extracted from SEC filings (324 events, 2026)

3 Upvotes

i put together a dataset of US IEEPA tariff-refund claims as they show up in 10-Q, 10-K and 8-K filings. not every tariff mention. just filings that talk about IEEPA and a refund, from April 2026 onward.

324 claim events so far: sales, loans, refund receipts, customer or supplier obligations, payments, and accruals. plus 1,189 filing candidates. each event has the company, amounts, an implied price when a sale lists both face value and proceeds, the quoted sentence, and a link back to the sec.gov filing. it rebuilds daily.

one example already in the data: Funko's participatory sale transferred $22.1 million in claims for $19.2 million proceeds. that's about 87 cents on the dollar.

a few caveats before anyone sums the column. repeated disclosures are not separate deals. amendments stay as their own rows. coverage is search-derived, so a missing company does not mean they have no claim. Court of International Trade dockets and private-company activity are out of scope. some amounts were checked by an AI review against the quote; that is not a human audit. read the original filing.

format is parquet (and a json api). first 500 rows of each table are readable with no account, which covers the whole claim_events table (324 rows). a free account is needed to query, filter or download past that, and it counts against a monthly allowance.

disclosure: i built this at mostly right, my company.

https://mostlyright.md/datasets/us-tariff-refund-claims-sales-and-pass-through-2026