# voice v1 *10 November 2026* > [!cloudy] > The faint notes like this one are the parts [Cloudy](https://cloudy.so) did. I gave it the goal, and it planned the experiments, wrote the code, ran it on GPUs and wrote up every result. Last night I trained a speech-to-speech model, inspired by [Dessa](https://block.xyz/dessa). It's full duplex, so it listens and talks at the same time like a person on a call. ==No text was used anywhere==: no transcripts, no text tokens, no pretrained language model. It learned from ~11,500 hours of human speech and nothing else. It sounds incredibly human. It doesn't make much sense yet; it's fluent-sounding babble. But ==the sentences start to make slightly more sense the more data we add==, which is the exciting part. Below, the model plays both people in every conversation. Scroll sideways for more. > [!samples] > ![[voice-v1-01.mp4]] ![[voice-v1-02.mp4]] ![[voice-v1-03.mp4]] ![[voice-v1-04.mp4]] ![[voice-v1-05.mp4]] ![[voice-v1-06.mp4]] ![[voice-v1-07.mp4]] ![[voice-v1-08.mp4]] ![[voice-v1-09.mp4]] ![[voice-v1-10.mp4]] ![[voice-v1-11.mp4]] ![[voice-v1-12.mp4]] ![[voice-v1-13.mp4]] ![[voice-v1-14.mp4]] ![[voice-v1-15.mp4]] ![[voice-v1-16.mp4]] Every sample is 60 seconds and fully generated: both voices, the timing, the "mm-hm"s and the laughs. The transcripts are only there so you can follow along; the model never sees text. #### the bitter lesson, but for audio With no text, everything is audio tokens. A neural audio codec turns each speaker's 24 kHz audio into 12.5 frames a second, with 8 tokens per frame. So an hour of two-person conversation is 720k tokens, and an hour of one person talking is 360k. | | hours | unique audio tokens | | ---------------------------- | ------ | ------------------- | | two-person conversations | 5,743 | 4.1B | | single-speaker speech | 5,812 | 2.1B | | total | 11,555 | ==6.2B== | Over training the model processed about ==33B audio tokens== (~2B frames). That's tiny next to a text LLM, and the scaling already shows. With the same model size, more conversation hours and more passes over them lowered the loss and made the words clearer every time: | conversation data | passes | held-out loss | word clarity (higher is clearer) | | ----------------- | ------ | ------------- | -------------------------------- | | 1,356 h | 2.5 | 1.954 | −0.47 | | 5,743 h | 2.4 | 1.867 | | | 5,743 h | 5.65 | ==1.809== | ==−0.29== | | real human speech | | | −0.18 | Word clarity is the average log-probability a speech recogniser gives the words it hears. The held-out loss was still falling at the end of the last run. This is [the bitter lesson](http://www.incompleteideas.net/IncIdeas/BitterLesson.html) again: no clever tricks, just more data and more compute, and it keeps getting better. I'm pretty sure that's the road Dessa took too. #### the data All of it is public human speech from the internet: - ~6,500 hours of two-person conversations (podcasts, interviews and recorded conversations), with ==each speaker on their own track==. That's what lets the model learn both sides of a conversation at once, including the moments when both people talk. - ~5,800 hours of single-speaker speech (audiobooks, spoken encyclopedia articles and talks), so the model learns how speech sounds before it learns how conversations flow. No synthetic voices, no text-to-speech. Every breath, laugh and "yeah" in the samples was learned from real people. > [!cloudy] > Cloudy found the sources (66,824 conversational podcast feeds and 3,216 Creative Commons YouTube channels), wrote the downloaders and pulled audio at up to ~5,400 hours of audio per wall-clock hour. It then built the pipeline that turns a mixed recording into one clean track per speaker, checking every step against real two-track recordings. Everything landed in the chat's shared storage, so every later experiment read the same data. #### the model A 413M parameter model with two streams: its own voice and the other person's voice. - Every 80 ms frame, a **temporal transformer** (width 1024) reads everything both people have said so far, as audio tokens. - A small **depth transformer** (width 512) then writes that frame's 8 codec tokens one after another. The first token carries most of what is being said; the other 7 carry the voice, the tone and the room. - At the same moment, the other person's latest 80 ms arrives on the second stream. There is no voice activity detector, no turn-taking rule and no "wait until they stop". ==Every 80 ms the model decides whether to talk, keep talking, say "mm-hm" or stay quiet==. - Every conversation is used both ways round, so the model learns to play either speaker. In the samples it plays both. #### why it sounds so human Because it is trained only on people talking to each other, and it hears both of them at once. Nothing in the system decides when to speak, so the timing comes straight from human data: the short gaps, the overlaps, the backchannels, the interruptions. On held-out conversations its timing and voice statistics land in the same range as real people: | | model | real conversations | | --------------------------------- | ----- | ------------------ | | turns per minute | 8.3 | 10.2 | | time both people talk at once | 8.8% | 9.0% | | time nobody talks | 11.6% | 10.6% | | "mm-hm"s per minute | 2.5 | 3.5 | | voice quality (UTMOS) | 2.49 | 2.50 | | same voice start to end (ECAPA) | 0.54 | 0.57 | Both columns are measured the same way, on the same held-out conversations. What's missing is meaning: the model has learned how people sound when they talk, but not yet what they're talking about. My bet is that more hours and more compute close that gap too. > [!cloudy] > Cloudy built this measurement too: 784 held-out conversations, scorers for timing, voice quality and speaker consistency, and checks that real audio passes and deliberately broken audio (looped, shuffled, time-shifted) fails, before any model was scored. #### training The whole run took ==about 2 hours 15 minutes, ~4 GPU-hours==: | stage | data | hardware | time | | ------------------------------- | ---------------------------------------- | -------- | ------ | | 1. learn to speak | single-speaker speech, 0.83 pass | 1 GPU | 27 min | | 2. learn to converse | 5,743 h of conversations, 2.4 passes (+20% single-speaker replay) | 2× H100 | 46 min | | 3. more passes | same data, up to 5.65 passes | 2× H100 | 61 min | One cosine learning-rate schedule (peak 3e-4) ran through stages 1 and 2, then stage 3 re-warmed to 1.5e-4 and decayed again. The model is saved whenever the held-out loss improves, and it improved at every check. > [!cloudy] > Cloudy wrote the model, the training loop and the generator from scratch, and tested each piece before trusting it: an overfit test, a checkpoint that resumes bit-identically, and the streaming generator matching the training forward pass. While this ran on 2 GPUs, Cloudy used the rest of the 8-GPU box for other experiments in parallel (a larger model, ablations, transcription), all reading and writing the same storage. When the boxes restarted, it picked everything up again from the saved checkpoints. #### making it fast I wanted training to be limited by data, not by GPUs, so we spent real time on the training step. Per H100: | change | frames/s per GPU | | --------------------------------------------- | ---------------- | | eager fp32 | 29.7k | | + bf16 and flash attention | 54.3k | | + `torch.compile` | 78.1k | | + small-matrix attention for the depth transformer | 119.9k | | + custom Triton kernels, bigger batches | ==145.6k== (34.5% MFU) | The big one: the depth transformer only attends over 8 tokens, and flash attention's 128-row tiles wasted most of their work on it. ==That one layer type was 42% of GPU time==. A custom Triton kernel that packs 2 frames × 1 head into each program brought its forward + backward from 6.70 ms to ==1.17 ms per layer==, with the same numerical error as before. A fused residual-add + RMSNorm kernel went in too. Going from 1 to 2 GPUs kept 96% of the per-GPU speed. Generation is fast too. With CUDA graphs one 80 ms frame takes ==7.1 ms on one H100==, 11× faster than real time, and a batch of 32 conversations runs ~400× faster than real time. > [!cloudy] > Cloudy profiled the training step, found the 42%, and wrote and verified the kernels. It checked outputs and gradients against an fp64 reference and ran the kernels through PyTorch's custom-op checks before switching them on. #### what's next More hours of conversation and more compute. Every time we added data, the words got a bit clearer. That's the whole plan. #### built with cloudy This is a demo of what's possible with [Cloudy](https://cloudy.so) in a few hours of GPU time. One chat did everything above: found and downloaded the data, wrote the model and the kernels, trained it, built the evaluation and wrote up every experiment, so ==you can track every experiment live in one place==, with its code, numbers and write-up. I'm really excited for people to try it. If you want to train your own model, [book a 15 min call](https://calendly.com/naklecha-cloudy/book) and I'll set it up with you :)