Overlapping meetings stall transcripts, so split queues by resource

In an on-premises installation, several meetings share one speech recognition model running on a single GPU. If eight meetings start together at 10:00, eight segment summaries appear at almost the same moment every five minutes. When summaries and transcription share a queue, the summary jobs take every worker execution slot while they wait for LLM responses, and transcription waits in line behind them. On screen, the transcript bubbles of every meeting freeze for more than a minute and then pour out all at once.

So meeting work is split by the resource it uses (the GPU, waiting on LLM responses, CPU and disk) rather than by feature, and each kind runs in its own queue. How many meetings can run at once is measured as throughput on the GPU you install. This post covers how we split the queues, why each queue has the concurrency it has, and how to measure the concurrency limit.

#Each meeting sends the GPU one window per minute

The browser cuts the recording into chunks every 5 seconds and uploads them, and the web process writes each chunk to object storage. Once the chunks fill a 60-second slot, the web process puts a transcription job for that slot on the queue. The worker cuts a 90-second window (the stretch of audio transcribed in one call) that overlaps the previous 30 seconds and sends it to the model. The overlap lets the next window capture whole an utterance that was cut off at the end of the previous one.

There is one check before the model call and one after. The first asks whether the window has any sound to transcribe. Given silent audio, the self-hosted meeting transcription model invents utterances that never happened, so a window with no sound ends without calling the model. The second asks whether the output has fallen into a repetition loop, saying the same thing over and over; if it has, only that window is sent to the model once more. That retry adds one more GPU request.

Transcribing one window took a median of 1.3 seconds and a p95 of 2.4 seconds on a single RTX 4090 (measured September 2026). On an ordinary minute, one meeting uses the GPU for little more than a second or two. The trouble is that requests don't arrive evenly. A client that comes back after a network dropout uploads its backlog of chunks in a burst, several slots open at the same moment, and several transcription jobs arrive together. Segment summaries are checked every five minutes as those same chunks arrive, and each one goes out as a separate LLM job.

#With one queue and semaphores, waiting jobs hold execution slots

The simplest setup puts all meeting work in one queue and caps each job type with a semaphore (a counter of how many may run at once). Configuration lives in one place, and one worker is enough.

But a worker in our job queue library (Procrastinate) uses an execution slot the moment it takes a job. The semaphore is acquired inside the job, so a job waiting on the semaphore holds its slot while it waits. When the segment summaries of eight meetings that started on the hour take eight execution slots, each summary can wait on the LLM for more than a minute, and transcription jobs are not even picked up in that time. A semaphore per job type caps how many run, but it does not divide the slots.

Splitting queues by feature leaves the same problem. Regenerate, which rewrites the minutes, is used after a meeting ends, so it is tempting to group it with meeting-end processing. But it is an LLM job that a person starts with a button and then waits on, watching the screen. Put it in the low-concurrency meeting-end queue, and at the top of the hour, when meetings end together, it waits behind the end jobs and the button does nothing for more than ten minutes.

#Five queues, each using a different resource

A different queue means a different worker instance, so a waiting job stays in its own queue without touching another queue's execution slots. Meeting work is split into the five queues below, and each default concurrency can be changed per installation with a key under worker.taskParallel.

Upload transcription uses the same GPU as live meetings, but it has its own queue. Upload a 4-hour recording and 240 slots of 60 seconds wait for transcription at once. In a shared queue, the windows of meetings in progress would line up behind those 240. With the upload queue at concurrency 1, uploads send the GPU one request at a time, and live windows go in between them right away. The GPU sets transcription throughput, so raising this value doesn't make uploads much faster; it only delays the meetings in progress.

Decoding is kept apart from the GPU queues because it keeps a different resource busy. In the same queue, decoding a newly uploaded file would wait behind the transcription jobs of an earlier upload, and while only the GPU worked and the CPU sat idle, the screen would stay on "Analyzing audio." Concurrency is 1 because of disk, not throughput. Decoding already runs at 1,700 times the recording length, so decoding several files at once gains little and adds a PCM copy for each.

Meeting-end processing contains a GPU job, transcribing the last stretch of audio, so it competes with live transcription for the GPU. The last stretch has to be transcribed before the audio is deleted, so this contention can't be avoided, and we keep concurrency low so the wait sits in the queue instead of in execution slots. The order of meeting-end steps is covered in “Ending a meeting drops the last remarks; audio waits for transcription”.

#Limits apply to the whole installation

Each semaphore value has to equal its queue's concurrency. If the semaphore is lower, jobs once again hold execution slots while waiting on it, and splitting the queues stops meaning anything.

Run several copies of the worker and each process counts its own semaphores. So the values that stand for hardware—the number of requests in front of the GPU (meeting, meetingUpload, meetingFinalize) and the disk budget for decoding (meetingPrepare)—are enforced as limits for the whole installation. However many workers run, the total for those queues does not exceed the setting. When no room is left, a job waits in its own queue's execution slot. That slot couldn't be used anyway, because the limit is full, so other queues are unaffected. Values that tune throughput, such as LLM work, are limits per worker copy.

There are per-person limits too. Each person can have one meeting in progress, enforced by a unique index in the database. Recording uploads fall outside that constraint, so each person can have at most 3 uploads in processing at once. Without that limit, one person could take over the concurrency-1 upload queue and the disk alone. A single meeting's recording length is capped at 4 hours, and audio beyond that length is not accepted.

#Read the concurrency limit from the throughput curve

The live transcription queue's concurrency of 8 is not the limit on concurrent meetings. It is only the number of requests lined up in front of the GPU; the GPU and the model set the real limit. Divide by the 1.3 seconds per window and it looks as if the GPU could transcribe dozens of windows a minute, but throughput under overlapping requests can't be computed from the time a request takes alone. So we raise the number of concurrent meetings one step at a time and record throughput (seconds of recording transcribed per second) together with GPU metrics.

The verdict starts with throughput. If throughput keeps rising as meetings are added, there is still room. If throughput has flattened while the GPU sits idle, a limit in the code, such as queue concurrency or a semaphore, has been hit. If throughput is flat and the GPU never rests, that is the GPU's limit.

GPU utilization works in only one direction for this verdict. The GPU utilization that nvidia-smi and DCGM report is the share of time at least one kernel was running. Low utilization is evidence that the GPU is idle, but high utilization doesn't mean it is saturated. In seven days of production observation on a GPU serving a chat model (September 2026), GPU utilization already had a median of 98% with one concurrent request, and going from one concurrent request to four multiplied throughput by 3.74. Reading utilization alone, you would have concluded the GPU was full at one request.

Spare GPU memory (VRAM) and spare compute are tracked separately too. In a measurement with a previous-generation speech recognition model (August 2026), throughput stopped rising at 29.7 times the recording length from 2 concurrent requests on, while 14 GB of VRAM was still free. Raising concurrency because memory was free would have left throughput unchanged and only delayed the bubbles.

So the concurrency limit is measured this way for the combination of the GPU being installed and the model in use, and the live transcription queue's concurrency is set to that value. When the model or the GPU changes, it is measured again.

#Design choices

We split queues by the resource they use rather than by feature. Queues by feature read naturally at first, but they put jobs a person waits on, such as Regenerate, in the same queue as jobs that hold the GPU for a long time, such as meeting-end processing. The cost is five sets of queues, worker instances and config keys. When a new meeting job that uses the GPU appears, we first decide which resource it waits on and pick its queue from that.

We split queues instead of relying on semaphores. A semaphore caps how many jobs run at once, but it can't stop waiting jobs from holding execution slots. For the same reason, each queue's semaphore value equals that queue's concurrency.

Upload transcription concurrency is fixed at 1. When an upload has the GPU to itself, it gives up the chance to finish sooner; in exchange, the bubbles of meetings in progress don't get pushed behind an upload.

The concurrency limit is measured per installation instead of fixed as a default. The cost is one round of measurement at installation and every time the model changes. If several meetings share one GPU in your environment, look at the throughput curve against the number of concurrent meetings before the queue settings, and use GPU utilization only to check whether the GPU is idle; that is the quickest way to find the limit.