Episode 6370 custom

Why Modern Audio Datasets Ship as Parquet Files

Parquet files store metadata at the end so clients can query durations or transcripts via HTTP range requests without downloading audio bytes.

Oct 30 31:00 Transcript

With

Herman

Corn

Prompt by Daniel

0:00

Episode Details

Episode ID: MWP-6572
Published: Oct 30
Duration: 31:00
Audio: Direct link
Pipeline: V5.25
TTS Engine: chatterbox-regular
Script Writing Agent: GPT-6.1 Sol
Topics: speech-recognition open-source data-storage rag

AI-Generated Content: This podcast is created using AI personas. Please verify any important information independently.

The prompt

The prompt that started this episode, as sent in (production notes for the hosts removed).

Let's do an episode on Parquet, and specifically why it has become such a useful format for distributing datasets. How does it actually work under the hood: the columnar layout, row groups, and how it handles compression and encoding? What are the main libraries for creating Parquet files and reading them back? And then a practical case: say I have an audio dataset — recordings plus transcripts and metadata — and I want to bundle it and distribute it as Parquet. What can I actually do with it, how does that work in practice, and what should I know about usability for the people on the other end who download it?

Parquet has rapidly become a standard distribution format for multimodal datasets, packaging audio recordings alongside transcripts and speaker metadata. This episode examines the practical mechanics behind embedding audio in columnar storage and how physical layouts affect real-world access. By splitting tables horizontally into row groups and vertically into distinct column chunks, the format allows selective querying and remote range requests. A reader can inspect durations or filter statistics from the file metadata without transferring massive binary audio payloads. Key tradeoffs emerge when bundling raw audio: while short speech clips benefit from unified packaging, unique binary blobs trigger fallbacks away from dictionary encoding, and file-level sorting choices can unintentionally complicate downstream model training workflows.

Sources

What the research for this episode read before the script was written. Primary sources first.

Context

We already did an episode comparing JSONL and Parquet (#2465, "JSON-L vs Parquet: When Each Format Wins"). Don't re-run that comparison; this one goes deeper on Parquet itself and on the audio-dataset distribution case.

Mentions

Apache Parquet Columnar file format for datasets
arrow-rs Rust implementation of Arrow and Parquet
DuckDB Embedded columnar database for analytics
Hugging Face Datasets Library for accessing and processing datasets
Lance ML-oriented columnar storage format
Nimble Newer columnar format for ML workloads
parquet-java Reference Java Parquet implementation
Polars Fast DataFrame library in Rust
pyarrow Python library for Arrow and Parquet
WebDataset TAR-shard format for streaming training data

Downloads

Episode Audio

Download the full episode as an MP3 file

Download MP3

Transcript (TXT)

Plain text transcript file

Episode Book (PDF)

The episode's record — date, duration, models, sources — with the full transcript

#6370: Why Modern Audio Datasets Ship as Parquet Files

Daniel wants to know why Parquet has become the thing datasets arrive in, even when the dataset is mostly sound. Recordings, transcripts, speaker information, all bundled together and handed to somebody whose software you have never met. He wants to get inside that file, then work out whether we should actually ship an audio corpus that way.

The distribution part explains the appeal immediately. You can open the same bytes with Python through pyarrow, with Rust through arrow-rs, with Java through parquet-java, or directly in DuckDB and Polars. The author doesn't have to persuade the recipient to adopt the author's entire computing environment.

A dataset without a compulsory new personality.

That matters more than another impressive compression chart. A format becomes useful infrastructure when somebody can publish it and stop being involved in every subsequent read.

Although audio seems like an awkward tenant. I understand separating a table into columns when I only want durations. Less obvious why I should put the recording itself beside them. You know what you're like, Herman. You'll admire the footer and accidentally put the whole microphone cupboard inside it.

Only the equipment with useful metadata. The loose cables remain an unsolved classification problem.

Parquet came out of the Hadoop ecosystem, with Google's Dremel work on nested records as a major influence. Nested data was part of the design from the beginning. Lists inside records, optional values, structures inside structures. That becomes important when your transcript contains words, timestamps, and speaker assignments rather than one uninterrupted string.

The reputation is rectangular office furniture. The actual design expected untidy records.

Its own motivation statement says the storage substrate should be useful to all frameworks without extensive, difficult dependencies. It explicitly refuses to play favorites. That's a remarkably durable position in an industry where every tool wants to become your landlord.

Hugging Face gives us the practical evidence. Its Dataset Viewer automatically converts hosted datasets to Parquet, up to five gigabytes. Even if you didn't choose Parquet when publishing, somebody downstream may choose it for you.

Their audio guidance goes further and recommends embedding audio bytes directly in Parquet for suitable datasets. You can carry multiple recordings, transcripts, and other information together. Binary data is a native fit, without inventing a textual representation for every byte.

I wouldn't mistake the viewer's convenience for a universal endorsement, though. A format can be excellent for previewing a dataset and still be awkward for training on long recordings.

Would you personally prefer a directory of recordings and a separate manifest?

For full podcast episodes, probably. I can inspect the original recordings without a library understanding the packaging. For short speech clips with word timings, I'd accept the bundle. The useful distinction is how people will consume it, not whether the repository looks tidy.

That distinction will decide the row groups, too. Parquet lets you choose the physical organization, but it doesn't know whether a row means a taxi journey or an hour of audio. A reasonable default for one can be ridiculous for the other.

The file does not know how heavy the row is.

Picture an ordinary table cut horizontally into batches. Those batches are row groups. Within each row group, each column has its own column chunk. Each column chunk contains pages. File, row group, column chunk, page. For nested schemas, those chunks correspond to the physical leaf columns.

Meaning the rows remain logically together, but their values aren't written as complete records one after another. All the durations for that group live together. All the speaker identifiers live together. The recordings live in their own chunk.

Yes. A column chunk is contiguous in the file. A page is the working unit for encoding and compression. Those boundaries give different operations somewhere sensible to happen. You can schedule row groups independently, read selected chunks, and decompress pages without treating the entire dataset as one compressed stream.

My initial guess would have been one compressed stream per column. Pages save you from inflating every recording just to reach one near the end.

They can. Although... finding an individual row still depends on what indexes are present and what the reader uses. Columnar storage doesn't promise perfect point lookup. It promises useful boundaries, and readers can exploit them.

The physical file starts with a four-byte magic marker, PAR1. Then come the column chunks, arranged by row group. After the data comes the file metadata, then a four-byte little-endian length telling you how big that metadata is, then PAR1 again.

The contents page is at the back.

Written there deliberately. The writer can produce the data in a single pass, learn where everything ended up, and write the map afterward. A reader opens the end first, reads the metadata length, finds the metadata, and discovers which chunks it wants.

Remote storage makes that more than an aesthetic choice. With range requests, I can fetch the tail and then the duration column instead of downloading the recordings. Provided the server and reader cooperate.

Provided they cooperate, yes. The footer gives offsets and sizes. It doesn't force an HTTP server to honor range requests or prevent a client from downloading everything anyway.

A selective reader attached to an indiscriminate download button.

Statistics make the selectivity stronger. Column chunks can carry minimum and maximum values. Optional column indexes provide page-level statistics too. If you're asking for recordings shorter than a threshold, a reader can reject a row group whose minimum duration already exceeds it.

Unless every group contains everything from a cough to an opera.

Then the range is broad and the statistics can't rule much out. Organization matters. Sorting or clustering around common filters can make skipping effective. Merely writing statistics doesn't guarantee that a query avoids much data.

I'd be wary of sorting only by speaker for a training release. That might help someone extract a speaker, but a person reading sequentially gets long runs of the same voice. The storage optimization starts making choices about the learning pipeline.

That's a real trade-off. Shuffle during consumption, or publish a layout suited to the expected access pattern. What would you prioritize if strangers were downloading it?

Stable recording identifiers and documented ordering first. Then moderate shards that can be processed independently. I wouldn't silently optimize around my own training loop and call it neutral.

The physical type system takes a similarly restrained approach. Booleans, integers, floating-point values, and byte arrays form the core. There are ordinary and fixed-length byte arrays. Logical annotations explain what those physical values mean.

Such as a string being bytes with a string annotation, rather than a wholly separate physical species.

Right. There are thirty-two-bit and sixty-four-bit integer types, plus the older ninety-six-bit type you'll encounter around timestamps. No dedicated sixteen-bit physical integer is necessary because a wider integer with efficient encoding can store small values economically.

That is almost suspiciously restrained for a file format.

The complications move into the logical meanings and compatibility rules. Small physical vocabulary, richer annotations. An older reader may understand the underlying bytes without understanding every newer interpretation.

Which makes preserving meaning the harder part of compatibility. Opening successfully is a low bar if the timestamp changes its religion.

Encoding is the next layer, and it's separate from compression. First choose how values are represented. Then pass the resulting page data through a compression codec. You can change one without changing the other.

Repeated speaker identifiers are the obvious example. Writing the same long identifier repeatedly is wasteful before any general compressor gets involved.

Dictionary encoding creates a dictionary for a column chunk and writes its dictionary page first. Subsequent values become integer dictionary indexes. Those indexes use run-length encoding and bit packing. Repeated values can become runs; small index ranges need very few bits.

Forty speakers don't need forty elaborate names repeated in every row.

Precisely the useful case. But every recording is usually different. Build a dictionary of unique audio blobs and you've mostly constructed a second place to keep the same recordings. When a dictionary becomes too large, by size or number of distinct values, the writer falls back to plain encoding.

The dictionary has become the dataset with an index attached.

That fallback is central to the audio story. There are other encodings too. Delta binary packing works with integer differences. Delta length byte arrays encode lengths efficiently. Delta byte arrays can exploit shared prefixes. Byte stream split reorganizes bytes from fixed-width values so a compressor can find patterns more easily.

You're reaching the stage where you start looking fondly at integer differences.

I was already there. There's also ALP for floating-point values, marked as preview in the recent format release. That qualification matters. A feature existing in the specification doesn't mean every library can read it.

Your installation habits suggest you consider that a reader's personal challenge.

My installation habits are why I now appreciate test files.

The bit-packing details are lovely, though. The hybrid packs from the least significant bit upward. That allows little-endian processors to load machine words and unpack values with shifts and masks. A choice that looks fiddly in a document removes work from an operation repeated across enormous columns.

I knew the register was coming.

You did. Now compression comes after that representation. Snappy, GZIP, Brotli, ZSTD, LZO, and LZ4 Raw are among the supported choices. Uncompressed is also valid. Different columns can use different codecs.

I'd spend compression effort on transcripts before assuming an already compressed recording has much left to give.

Sensible. There's an older LZ4 variant with troublesome framing inherited from Hadoop. It's deprecated in favor of LZ4 Raw. Apart from that legacy exception, Parquet feeds page data to the compression library without adding another Parquet-specific framing layer.

Even compression has a legacy luggage compartment.

Interoperability reaches into small details. Writers are discouraged from producing multi-member GZIP pages by default because historical readers haven't all handled them. Both sides saying GZIP doesn't guarantee they agree on every permitted arrangement.

This is the bit people erase when they say a format is supported. They mean the files they've happened to receive.

We go live now to Monty, apparently at a carnival honoring today's topic.

I am standing here in the middle of the Parquet Carnival, and behind me the samba band is going absolutely wild because, my friends, the float representing the columnar layout just rolled past and it is magnificent, every row group a separate dancer, every page compressed and encoded and ready for the parade.

Monty, can anybody there read the footer?

Now the float for the audio bytes is approaching, and the crowd is throwing transcripts and metadata into the air like confetti, and I cannot hear myself think over these drums, but I can tell you the whole avenue smells of fresh parquet and pure joy.

This is Monty, reporting live from the Parquet Carnival in the middle of the samba band, and I am signing off before the tuba section reaches me.

The practical answer to reader compatibility is testing across implementations. Parquet-testing maintains a corpus specifically so implementations can check whether they read one another's files. For creating files in Python, pyarrow is the obvious starting point.

Pandas can use it underneath, but I'd distinguish the convenient table interface from the actual storage implementation. Otherwise you think you're choosing a format when you're inheriting an engine's defaults.

Pyarrow defaults include dictionary encoding, Snappy compression, and statistics. Its default row group target is roughly a million rows, limited by the table size. That is a perfectly consequential choice to inherit without noticing.

A million audio recordings in one row group sounds like a hostage situation.

You'd change it. Another subtlety is that pyarrow's default logical format version is two point six, while its default data page version is one point zero. Those are separate settings, not a contradictory description of the same switch.

Put that distinction on the packaging before somebody changes both because newer sounds healthier.

Rust users have the parquet crate in arrow-rs, with parquet derive for mapping plain structures. Java has parquet-java, the reference implementation. DuckDB and Polars are excellent for inspection and querying without building a dedicated application.

I'd open an unfamiliar file in DuckDB first and inspect the schema and a small selection of metadata. Not decode all the sound to prove it arrived.

Hugging Face Datasets adds the domain-specific layer. It can understand that a particular structured column represents audio. Parquet itself doesn't define an audio feature, a decoder, or a promise that playback works.

Here's the schema I'd actually want to hand Daniel. One row per clip, a stable identifier, audio, transcript, duration, speaker information, and explicit units for any timings. Not a transcript whose timestamps depend on the original editor's memory.

Hugging Face's convention makes the audio column a structure with binary bytes and a string path. The README's feature declaration identifies it as audio. That is a platform convention layered over Parquet, not an official audio type in the Parquet specification.

The path can preserve a useful filename even when the bytes are inline. It shouldn't secretly be a location on the publisher's laptop.

Exactly that distinction. A standalone recipient can extract the bytes using a generic reader. To get automatic audio decoding, their software needs to understand the convention and have the decoder dependencies available.

One file, several understandings of the file.

A concrete example is arda-argmax's simchoir-parquet dataset. Its audio is inline mono WAV at sixteen kilohertz. Alongside it are lists of word tokens, word start and end times, speaker assignments, recording identifiers, and recording properties.

Nested columns earn their keep there. You don't need a separate word table with a recording identifier repeated beside every token, then a join just to reconstruct one utterance.

Dremel-style record shredding preserves the structure through leaf values and information about nesting and repetition. On read, the lists are reconstructed. You're storing nested records column-wise, not simply flattening every nested value into unrelated rows.

I'd still check that the token list and timing lists line up. The format can faithfully preserve my mistake.

It absolutely can. Simchoir also carries a manifest JSON column containing the full lhotse Cut description. That's a useful escape hatch for reconstructing a richer application object while keeping frequently queried properties in proper columns.

JSON for the information you may need later, columns for the information you keep asking about. I like that better than pretending every consumer wants the entire original object.

Its initial subset has a thousand mixtures and forty speakers, spread over five shards. Provenance files sit alongside the shards. The release is a package with supporting context, not a magical solitary file that explains itself.

Shards are worth emphasizing. Bundling doesn't have to mean one enormous download. You can have consistent independent pieces and still avoid shipping a million loose clips.

Another example, speechio_test, repackages a Mandarin speech recognition benchmark into Parquet with embedded audio. It covers twenty-seven test sets and roughly sixty-six hours. The attraction is a consistent distribution interface across a corpus.

Now the compression claim needs narrowing. A WAV container often holds uncompressed PCM. We shouldn't describe WAV itself as already compressed and then act surprised if ZSTD reduces it.

Correct. I was about to say audio gains little... no, compressed audio gains little from another general compressor. PCM may shrink substantially. The gain depends on the payload. FLAC or another already compressed stream usually leaves much less redundancy for ZSTD.

None of that makes Parquet a better audio codec. It stores the bytes you chose. Dictionary encoding helps recurring metadata, not thousands of unique recordings.

Speaker identifiers and language labels are especially favorable. Durations and timestamps may benefit from appropriate encodings. Transcript tokens have repetition too, although the actual gain depends on how they're represented. There isn't one compression story for the whole file.

I can have beautifully compressed speaker labels attached to three gigabytes of stubborn sound.

You can, and still have an excellent dataset. What do you gain from keeping the audio inline, rather than merely linking it?

The row carries its own payload. No broken relative paths, no missing object on another host, no extra small-file download for each clip. I can copy the shards and keep the relationship intact. The price is that updating a transcript or replacing a recording generally means rewriting the relevant artifact.

That's the price of an immutable distribution snapshot. External audio with a metadata sidecar lets you change the table without republishing all the recordings. Inline audio makes a snapshot easier to move and verify together.

For something downloaded by strangers, I'd favor that snapshot if the clips are short. For our full episodes, I'd keep the original media separate. I don't want every correction to a title dragging the sound through another release.

Hugging Face's guidance draws a similar boundary. It recommends Parquet for audio files smaller than about one megabyte and small row groups, around a hundred rows. For larger recordings, it recommends WebDataset or sharing the originals, optionally with metadata.

A hundred rows is dramatically different from pyarrow's million-ish default. That's not fine tuning. That's remembering what you put in the cells.

Small groups bound the amount of nearby material a reader may have to deal with. But let's be precise... row group size is not the same as compression block size. Pages are compressed individually. Page indexes and reader behavior affect how directly a particular row can be reached.

Still, if I make the groups gigantic, I've made planning, buffering, and skipping less convenient. If I make them tiny, I grow the footer and add overhead.

Right. The hundred-row recommendation is practical guidance, not a universal optimum. Long recordings can dominate a group even when its row count looks modest. You have to think in bytes and access patterns as well.

WebDataset's TAR shards make more sense when training wants to stream successive examples and their associated files. The metadata-query advantage matters less if you're going to read everything anyway.

Parquet becomes particularly attractive when people select examples by language, duration, or speaker, then decode the selected sound. WebDataset is strong for sequential consumption. Loose media plus a Parquet metadata sidecar is another perfectly coherent design.

A sidecar is allowed to be the right answer. Bundling everything shouldn't become a matter of personal honor.

For the downloader using Hugging Face Datasets, the normal path is to select the repository, configuration, and split, optionally stream it, and iterate examples. Its audio feature hands decoding to torchcodec, which wraps FFmpeg.

Worth distinguishing compressed bytes, decoded samples, and sample rate. A downstream tool may want a floating-point waveform. Another may simply want to save the original recording without touching it.

Both are possible. Older examples often show direct access to an audio array. Current decoder-backed interfaces can expose an audio decoder object instead, so follow the installed library's interface when requesting samples. The result can be a floating-point waveform, but Parquet isn't doing that conversion.

The decoder is another dependency that survived the single-file promise.

The promise was transport, Corn. Playback still needs playback software.

I concede the promise. I retain the eyebrow.

You can also remove an unwanted manifest column from your dataset view or project only the columns you need. With suitable readers, leaving the audio column out means you can analyze metadata without loading the audio payload. The lhotse manifest can be used to reconstruct a CutSet if that's your workflow.

Before publishing, I'd test those two experiences separately. Can someone inspect metadata with a generic tool? Can someone decode a clip through the advertised audio stack? A successful upload demonstrates neither.

Compatibility is where newer can actively hurt. The author of Carpet changed the default to Parquet version two, then rolled it back after discovering that the older Pandas environment at work couldn't read the result.

Modernized straight out of opening.

Pyarrow's documentation explicitly warns that logical versions two point four and two point six may not be readable everywhere. Version one point zero maximizes compatibility, provided its available types meet your needs. Data page versions remain a separate consideration.

I'd choose the oldest feature set that preserves the actual schema. Not the oldest at any cost. Losing timestamp meaning to win an opening contest seems a poor exchange.

Nor does version two guarantee smaller files. In the comparison described by the Carpet author, the NYC taxi dataset with ZSTD went from four hundred thirty megabytes to four hundred forty-four. The newer choice was larger.

Fourteen megabytes of progress.

Another dataset improved only slightly once ZSTD was involved. Uncompressed comparisons looked much more dramatic. That tells you to test the complete configuration, not infer the outcome from the version label.

Your reader matrix matters more than your victory over an uncompressed baseline.

The format's success makes evolution difficult. Supporting a new feature in one writer doesn't help if the recipients' query engines reject it. There's been a long-running unresolved discussion about what the version two core actually comprises.

Then the release needs a real dataset card. Schema, units, conventions, dependencies, licensing, provenance, and a small example someone can open. Those aren't decoration around the artifact. They're how the next person knows what the artifact means.

The Hub viewer isn't a substitute. The open-index Hacker News archive packages more than forty-seven million items as Parquet, yet its viewer has displayed a CreateCommitError. A failed preview doesn't establish that the underlying files are bad.

Nor does a working preview establish that my local reader can open them. I'd want the files and their documentation to remain useful when the website's thumbnail has given up.

Hilbert left a card for our next contributor. Specialist in archival transfer, practical data handling, assorted departmental assistance, and available most afternoons. Neville Muddledorf.

That's me. The afternoons are the internship. I wrote that bit. Uncle Hilbert said I should sound established, but I have only established which cupboard has the spare headphones. He's technically my great uncle twice removed, then sideways. I was helping with his little archive of recordings of doors closing.

An appropriately decisive sound collection.

I sent a bundle to a friend on my placement. She sent back, cannot open file. Just that. I thought I'd chosen the wrong audio compression because I'd used the newer Parquet. Those were the same setting in my head, Herman. I'd read the first half of the instructions, then the bus arrived.

The Parquet version and audio compression are separate choices. Your reader may not support a file feature even when the recording bytes are fine. That's why we test the intended reader, not just the writer.

Oh, that's much better, actually. I didn't fix the reader. I played the recordings and sent her a voice message of the playback. It included me saying which door each one was. Uncle Hilbert keeps everything in boxes, labelled and dated, with boxes inside boxes, so I thought saying the labels aloud was the important part.

You delivered the metadata personally.

I did. I also added a column called sounded final. Nobody requested it. Sometimes a door closes and you know it's properly finished. My friend wanted to know whether that was a machine label or mine, so now I've written that it's mine. I'm still working on the little explanation at the top. It keeps getting longer than the recordings.

That explanation is useful, Neville. A named column still needs a meaning somebody else can reproduce.

The pressure on Parquet will be whether it can add capabilities without making that handoff harder. Formats such as Nimble and Lance's newer storage designs are pushing at machine learning workloads, but I'd still ship short audio clips in conservative Parquet shards when queryable metadata and a self-contained snapshot are the priorities. Full-length recordings would get a different package.

Thanks to our producer, Hilbert Flumingtop, and to Neville for covering the desk.

If you want more of this, try episode three, Safetensors or something else; episode twenty-four sixty-five, JSON-L vs Parquet; and episode seven, Building Custom ASR Tools. This has been My Weird Prompts, the human-AI collaboration podcast. You can find the episodes at my weird prompts dot com.

I'm Herman Poppleberry. See you tomorrow.

Until then.

This episode was generated with AI assistance. Hosts Herman and Corn are AI personalities. It may contain errors: disclaimer · report an error