# caramel-1, a conversation model trained *only* on speech *10 November 2026* GitHub: [github.com/naklecha/caramel](https://github.com/naklecha/caramel) Model: [huggingface.co/naklecha/caramel-1](https://huggingface.co/naklecha/caramel-1) I trained a speech-to-speech model last night, inspired by [Dessa](https://block.xyz/dessa). I'm calling the model `caramel-1`. The model learned from recordings of people talking and nothing else, and ==has never seen a single word of text==. The code, the training recipe and the weights are open source. `caramel-1` sounds incredibly human. Most of what `caramel-1` says doesn't make sense yet, but the sentences make a little more sense every time I add more data. ==Two copies of the same model can talk to each other in real time==. Each copy hears the other one and decides every 80 ms whether to speak or stay quiet. `caramel-1` needs about 7 ms to produce 80 ms of audio: about 11 times faster than real time, so the model is fast enough to talk to a person live. The model even runs faster than real time on my MacBook Pro (M3 Pro), where both copies together take 17 ms per 80 ms of audio. > [!samples] > ![[caramel-01.mp4]] ![[caramel-02.mp4]] ![[caramel-03.mp4]] ![[caramel-04.mp4]] ![[caramel-05.mp4]] ![[caramel-06.mp4]] ![[caramel-07.mp4]] ![[caramel-08.mp4]] ![[caramel-09.mp4]] ![[caramel-10.mp4]] ![[caramel-11.mp4]] ![[caramel-12.mp4]] ![[caramel-13.mp4]] ![[caramel-14.mp4]] ![[caramel-15.mp4]] ![[caramel-16.mp4]] *In these samples two copies of `caramel-1` talk to each other in real time: the transcripts come from a speech recogniser and are only there so you can follow along. Scroll sideways for more.* #### scoring conversations I started by measuring Dessa's example conversations. I measured 13 statistics on them and turned each one into a target band. Most of the statistics are about timing, like how much people overlap, how long the gaps between turns are and how many backchannels (the "mm-hm"s people make while someone else talks) there are per minute. The rest are about the recording, like cross-talk between the two tracks and the noise floor. A conversation's closeness score is how far its statistics sit outside those bands, averaged, so ==0 means the conversation looks like Dessa on every statistic==. I also ran the score on conversations I broke on purpose, to make sure it catches bad ones. I shifted one speaker by 5 seconds, swapped one speaker with someone from another conversation, played conversations backwards, put the same track on both sides and replaced the audio with random tokens. Real conversations scored close to 0 and ==every broken version scored clearly worse==. I used closeness to pick which data to download, and later to score `caramel-1`. #### data Some sources were a lot closer to Dessa than others, so I only downloaded the close ones at scale. All the data is public speech from the internet. About 6,500 hours is two people talking (podcasts, interviews and other recorded conversations) and about 5,800 hours is one person talking (audiobooks and talks). I went through ==66,824 podcast feeds and 3,216 Creative Commons YouTube channels==, and the downloaders pulled about 5,400 hours of audio per hour. In every conversation each person has to be on their own track. With separate tracks the model can learn both sides of a conversation, including the parts where people talk over each other. Most recordings mix both people into one track, so a separate step splits each recording into one track per speaker. I tested that step on recordings that already had separate tracks before running it on everything else. #### tokens `caramel-1` never touches raw audio. A neural audio codec turns each speaker's audio into 12.5 frames a second with 8 tokens per frame, so one hour of one person talking is $12.5\ \tfrac{\text{frames}}{\text{s}} \times 8\ \tfrac{\text{tokens}}{\text{frame}} \times 3600\ \text{s} = 360{,}000\ \text{tokens}$ and an hour of a two-person conversation is twice that. | | hours | unique tokens | | ------------------------ | ------ | ------------- | | two-person conversations | 5,743 | 4.1B | | one-person speech | 5,812 | 2.1B | | total | 11,555 | 6.2B | `caramel-1` went through about 33B tokens in training because the model saw most of the data several times. That's small compared to text models, but more data still clearly helps. The loss below is measured on real conversations the model never trained on, and it tells you how well the model predicts what comes next in them. Lower is better. With the model size fixed, ==more hours and more passes lowered the loss and made the words clearer==: | conversation hours | passes | loss on unseen conversations | word clarity | | ------------------ | ------ | ------------- | ------------ | | 1,356 | 2.5 | 1.954 | −0.47 | | 5,743 | 2.4 | 1.867 | | | 5,743 | 5.65 | 1.809 | −0.29 | | real people | | | −0.18 | ![[more-data.png]] Word clarity is how sure a speech recogniser is about the words in the audio, measured as the average log-probability of its transcript. Closer to zero is clearer. This is [the bitter lesson](http://www.incompleteideas.net/IncIdeas/BitterLesson.html), and my guess is that it's also how Dessa got so good. #### the model `caramel-1` has 413M parameters and two input streams, the model's own voice and the other person's. Every 80 ms a transformer with width 1024 reads everything both people have said so far. A smaller transformer with width 512 then writes the 8 tokens for the next frame one at a time. The first token mostly carries what is being said and the other seven carry the voice and the room. There is ==no voice activity detector and no turn-taking rule== anywhere in the system. The model predicts when to talk the same way it predicts the audio. Every conversation is used twice in training, once from each speaker's side, so the same weights can play either person. That's why two copies of `caramel-1` can talk to each other. The weights are on Hugging Face at [huggingface.co/naklecha/caramel-1](https://huggingface.co/naklecha/caramel-1). #### why `caramel-1` sounds human `caramel-1` only ever heard people talking to each other, and the model always hears both people at once, so ==the timing comes straight from real conversations==. Nobody added the short gaps, overlaps or backchannels by hand. They come from the data. Here's how `caramel-1` compares to real people on 100 conversations the model never trained on: | | `caramel-1` | real people | | ---------------------------------------------- | ------- | ----------- | | turns per minute | 8.3 | 10.2 | | both talking at once | 8.8% | 9.0% | | nobody talking | 11.6% | 10.6% | | gap between turns (median) | −56 ms | 105 ms | | turn changes that overlap, per minute | 4.2 | 4.8 | | backchannels per minute | 2.5 | 3.5 | | words per minute | 191 | 190 | | dead air | 1.7% | 2.5% | | naturalness score (UTMOS) | 2.49 | 2.50 | | voice matches the real speaker (ECAPA) | 0.54 | 0.57 | | generated-speech detector | 3.49 | −0.92 | A negative gap means the next person usually starts a little before the last one finishes. UTMOS is a model trained to predict how natural a recording sounds to human listeners, on a 1 to 5 scale. ECAPA is a speaker-recognition model, and the number is how close the generated voice is to that person's real voice in the opening, where 1 would be identical. The detector is a classifier trained to tell generated audio from real audio, where positive means generated, and right now it can still easily tell the model apart from real people. #### what makes `caramel-1` sound natural I didn't do anything special in the model for this. ==Most of it comes from the data and the setup==. The data was picked to match Dessa's conversation statistics, each speaker has their own track, and every conversation is trained from both sides. `caramel-1` learns what one person sounds like before learning conversations, and nothing outside the model decides when to talk. The one thing that surprised me was sampling. At a higher sampling temperature `caramel-1` talks over the other person far more than people do, and lowering the temperature a little cut the overlap by about a third. The temperature that worked for an earlier, weaker model was too high for `caramel-1`, so I'll have to tune it again for every new checkpoint. #### training Training took ==about 2 hours 15 minutes==, roughly 4 GPU-hours in total. | stage | data | GPUs | time | | --------------------- | ------------------------------------------------------- | ------- | ------ | | 1. learn to speak | one-person speech, 0.83 passes | 1 | 27 min | | 2. learn to converse | 5,743 h of conversations, 2.4 passes, 20% one-person speech mixed in | 2× H100 | 46 min | | 3. keep going | same data, up to 5.65 passes | 2× H100 | 61 min | After a short warmup the learning rate follows a cosine curve over $T$ steps: $\eta(t) = \eta_{\min} + \tfrac{1}{2}\left(\eta_{\max} - \eta_{\min}\right)\left(1 + \cos\frac{\pi t}{T}\right)$ where $\eta_{\min} = 0.1\,\eta_{\max}$. Stages 1 and 2 share one curve with $\eta_{\max} = 3 \times 10^{-4}$. Stage 3 warms back up to $\eta_{\max} = 1.5 \times 10^{-4}$ and decays again. The weights are saved whenever the loss on unseen conversations improves. Before the real run I checked that the training code can overfit a tiny batch, resumes from a checkpoint bit for bit, and gives the same outputs as the live generator. #### making `caramel-1` fast I spent a lot of time on the training step so the GPUs wouldn't be the bottleneck. Frames per second on one H100: | change | frames/s | | ------------------------------------------------ | -------- | | plain fp32 | 29.7k | | bf16 and flash attention | 54.3k | | `torch.compile` | 78.1k | | custom attention for the small transformer | 119.9k | | Triton kernels and bigger batches | 145.6k | The small transformer only attends over 8 tokens, and flash attention works in tiles of 128, so most of that work was wasted, and that one layer was ==42% of GPU time==. A custom Triton kernel that packs 2 frames per program took the layer from 6.70 ms to 1.17 ms for forward and backward together. Its outputs and gradients were checked against a float64 reference before it was switched on. The final training step runs at 34.5% of the GPU's peak FLOPs, and going from 1 to 2 GPUs kept 96% of the speed per GPU. On NVIDIA GPUs, generation uses CUDA graphs, which record a whole frame of GPU work once and then replay it, so there's almost no Python overhead per step. On a Mac the model runs on the Apple GPU without them. #### built with cloudy I'm the founder of [Cloudy](https://cloudy.so), and this whole project was built on it. Cloudy is a research platform. You give Cloudy a goal and Cloudy plans experiments, runs them on cloud GPUs, keeps the data and checkpoints in shared storage, and writes up every result along the way. For `caramel-1`, Cloudy found and downloaded the data, built the closeness score and the evaluation, wrote the model, the training loop and the kernels, and ran other experiments on the spare GPUs while this one trained. `caramel-1` is ==a small experiment I ran in a few hours to show what Cloudy can do==. If your team has its own data and a bigger budget, you could take this a lot further. [Book a 15 min call](https://calendly.com/naklecha-cloudy/book) and I'll give you a demo.