A meeting recorder joins the call, listens for an hour and sends a summary afterwards. A real-time AI assistant has a much harder job. It has to hear a question while it's being asked and put a useful answer on the screen before the person asking it loses patience, which in a sales or support conversation means one or two seconds.
That deadline shapes every part of the system. Below is the pipeline, stage by stage, with the engineering decisions that matter at each one. Most of them are data problems more than AI problems: getting clean input fast, finding the right record, and checking the output against the source.
The pipeline at a glance
- Capture audio from both sides of the call.
- Turn it into text as it's spoken (streaming speech-to-text).
- Decide when an answer is needed.
- Find the relevant facts in the user's own documents and tables.
- Generate a short answer, streamed word by word.
- Check the answer against the source, especially numbers.
- Show it in a window that the other side of the call can't see.
Each step has its own latency budget, and the total has to stay around a second for the answer to be usable.
1. Capturing both sides of the call
The microphone gives you one side. The other side, the customer, comes through the speakers, and capturing it is where many prototypes stall.
Bots solve this by joining the meeting as a participant, which works but puts "AI notetaker" in the guest list. Desktop assistants capture system audio instead. On macOS, Core Audio process taps (available since macOS 14.4) let an app record the audio output of other apps, with the user's permission. On Windows, WASAPI loopback capture does the same. Both approaches work with any calling app, from Zoom and Teams to WhatsApp and browser dialers, because they don't integrate with the meeting at all.
Two problems show up immediately. The first is echo: if the user is on speakers instead of headphones, the microphone picks up the customer's voice too, and the transcript shows every sentence twice. You need an echo filter that drops microphone segments matching what was just played. The second is devices that change mid-call. People switch to AirPods, unplug a headset, or join from a different room, and the capture has to follow the new device without losing the stream.
Keep the two channels separate all the way through. Knowing who said what, "you" and "them", is what lets the assistant answer the customer's question instead of the user's own words.
2. Streaming speech-to-text
Batch transcription, where you upload a recording and get text back, is useless here. You need a streaming connection, usually a WebSocket, that receives audio in small chunks and returns text continuously.
Streaming engines return two kinds of results. Interim results arrive within a fraction of a second and change as more audio comes in; final results arrive when the engine decides a phrase is complete. Show interim text so the user sees the conversation live, but trigger answers from final text so you're not answering half a sentence.
The engine decides a phrase is complete through endpointing: a pause long enough to treat as the end of an utterance. Too short and you cut people off mid-thought; too long and every answer waits. This one setting moves perceived speed more than most model choices.
Three things decide transcription quality in practice. Distance to the server: the round trip from Europe to a US region adds noticeable delay, so production systems route to the nearest region. Vocabulary: product names, unit codes and surnames are exactly the words a general model gets wrong, so you add custom terms and corrections (if it keeps hearing "Quill mate", map it to "Quillmate"). And language: international calls switch languages, which needs a multilingual mode rather than a fixed language per call.
Plan for failure, too. Speech-to-text providers have rate limits and outages. A system that holds several keys or providers and fails over instantly, with a reconnect that keeps the text already recognised, is what keeps a demo from failing in the middle of a paying customer's call.
3. Deciding when to answer
There are two approaches, and most products offer both. With a hotkey, the user presses a key when they want help. It's predictable, costs nothing between questions and never interrupts. With automatic answers, the system detects a question from the other side and answers on its own. That's convenient, but it needs throttling (an answer every few seconds at most), or the screen fills with responses to small talk.
Either way, the input to the next stage isn't just the last sentence. It's the last few turns of the conversation, plus a short brief about the call: who the user is, what the call is for, and rules such as "never quote a price that isn't in the price list".
4. Retrieval: finding the right facts in your own files
Users need answers about their own pricing, products and policies, so the assistant searches their documents first and passes only the relevant pieces to the model.
For text documents, the standard approach works: split brochures, playbooks and FAQs into passages, index them, and retrieve the few that best match the question. Keep the source name with every passage so the answer can cite it.
Tables need different treatment, and this is where generic retrieval-augmented generation (RAG) breaks. A price list split into text chunks loses the link between a unit code and its price, so the model sees "12B" in one chunk and "€385,000" in another and guesses which belong together. The fix is to treat a table as a table. Parse CSV or spreadsheet exports into rows, index the identifying columns (unit, plan, SKU), and when the conversation mentions "12B", fetch exactly that row and give it to the model whole.
Data quality matters more than the model here. Two versions of the same price list in the knowledge base will produce two different answers. Keep one current file per source, and make updating it someone's job.
5. Generation: fast first, complete second
Time to the first word on screen is what users feel. Two practical patterns help.
Use a small, fast model for short answers and a larger model when the user asks for a fuller one. A hotkey for each works well: one for "give me the line to say now", one for "explain this properly".
Stream the output. Showing words as they're generated makes a two-second answer feel like half a second, because the user starts reading immediately.
Keep the prompt tight. A short system instruction, the call brief, the last few turns and the retrieved facts, nothing else. Every extra kilobyte of context adds delay and gives the model more to get confused by.
6. Checking numbers against the source
Language models are good at sounding right. On a sales or property call, a confident wrong price is worse than no answer, because the user repeats it and the customer remembers it.
So check the output. When an answer was built from a table row, extract every number in the generated text and compare it with the numbers in that row. If a figure doesn't match, flag it right under the answer instead of letting it pass. Derived numbers count too: "25 users at €39" should come to €975, and a mismatch there is a bug in the answer, not a rounding issue.
This check is cheap, runs in milliseconds and catches the error users care about most. It's a data engineering fix: the safeguard is a join between the model's output and a source table.
7. The window the other side can't see
The answer has to appear where the user can read it while looking at the camera, and it must not show up when they share their screen. Operating systems support this directly. On macOS, a window's sharing type can be set so that screen capture and screen sharing skip it. On Windows 10 (version 2004) and later, SetWindowDisplayAffinity with WDA_EXCLUDEFROMCAPTURE does the same. Most screen-sharing tools respect these flags.
Small details matter for use on real calls: a compact window that can sit right under the webcam, a way to collapse it to a few buttons, and keyboard shortcuts for everything, because reaching for the mouse mid-sentence is visible.
The latency budget
Put together, the stages share about a second. The figures below are typical ranges, not guarantees; network distance and model choice move them a lot.
| Stage | Typical share of the budget |
|---|---|
| Speech-to-text, final text after the speaker stops | 100 to 500 ms, mostly endpointing and distance to the server |
| Retrieval from local documents and tables | Tens of milliseconds |
| Model, time to the first words | A few hundred milliseconds for a small, fast model |
| Number check and display | A few milliseconds |
The biggest wins usually come from the boring parts: the nearest server region, a well-tuned endpointing pause, a short prompt, and retrieval that runs locally instead of over the network.
Privacy and compliance
Real-time assistants handle some of the most sensitive data a company has: customer conversations, internal pricing and screens. A few design decisions make that manageable.
Store nothing by default. Answering in real time doesn't require keeping audio or transcripts; saving should be a setting the user turns on. Process documents locally where possible and send the model only the passages it needs for the current question, never whole files. Choose processing regions deliberately, since European customers will ask where their data goes. Name every sub-processor in the privacy policy. And remember consent: many jurisdictions require participants to be told when a call is transcribed, so give users an easy way to disclose it.
Build or buy?
Building a real-time assistant in-house makes sense when the knowledge it needs lives in systems no off-the-shelf product can reach, or when the conversations can't leave your infrastructure. The hard parts are less the model than the pipeline: reliable capture, failover, retrieval over messy internal data, and the checks that keep numbers honest.
For most teams that just need answers on their own calls, an existing product is faster. One example is Realtimer, a real-time AI assistant that answers from a team's own price lists, playbooks and documents during sales, property and recruiting calls, and runs as a desktop app with no bot in the meeting.
If you're building something in this space and need help with the data side (pipelines, retrieval over internal documents, table quality, evaluation), that's the kind of work our data engineers do every day. Tell us what you're building.