ProCat Solutions
Abigel.ai is live: an AI phone assistant and an AI language teacher
Our DIMOP Plusz project has closed: the phone assistant and the AI language teacher are live at abigel.ai. Lessons on latency, streaming and telephony.
Yesterday, on 14 September 2026, our project with identifier DIMOP_PLUSZ-1.1.2/A-24-2025-00236 came to a close, and the result of the development is live at https://abigel.ai. This post is about two things: what we built, and what we learned over the more than a year we spent building a system that processes a phone call in real time.
Co-financed by the European Union.
What is Abigel.ai?
Abigel.ai comprises two products on shared infrastructure.
Virtual assistant. An AI phone assistant that answers calls on the business’s phone number 24 hours a day. It talks to the caller, books appointments, takes messages, answers simple questions, and produces a transcript and a summary of every call. The customer assembles the course of the call in a no-code flow editor, without programming. The assistant speaks more than 25 languages and detects the caller’s language.
Teacher. A conversational AI language teacher that runs speaking lessons in 26 languages, from A1 to B2. The learner speaks, and the system responds, corrects and gives feedback in real time. The audio processing chain built for the phone assistant serves a different purpose here, but it is the same technology.
Both products are multi-tenant SaaS: usage-based pricing, data separated per tenant, GDPR-compliant operation inside the EU, a call log and CSV export. The project was delivered with the content committed in the grant.
Latency is the only metric that really matters
The most important technical lesson of the project, in one sentence: in a phone conversation, anything that takes more than a few hundred milliseconds is audible. People expect the other party’s reply within roughly half a second; if that grows to a second and a half, the caller talks over it, hesitates, or hangs up.
Speech recognition, the language model and speech synthesis run one after another easily add up to several seconds. What we did about it:
- Streaming at every step. Speech recognition produces partial results before the sentence ends; the language model’s reply arrives token by token; speech synthesis is already speaking the first sentence while the rest is being generated. The chain works as a pipeline, not as a sequence of steps.
- End-of-speech detection. Deciding whether the caller has finished what they were saying is harder than we expected. Too short a silence threshold cuts people off, too long a threshold feels slow. We tune the threshold based on the content of the speech as well, not just the length of the silence.
- Handling interruptions. If the caller starts speaking while the assistant is saying something, we cut playback immediately and treat the part actually spoken as the context. Without this, the conversation feels unnatural.
- Geographic proximity. We use the voice and language model providers on endpoints inside the EU, and our own components run in the same data centre. Every network hop is ten to twenty milliseconds, and they add up.
We measure latency per call and per step, and track it as percentiles. That was the metric most of our architectural decisions were based on.
Telephony: the old problems have not gone away
Everything we wrote earlier about the SIP and PSTN world came up again, with a few additions:
- the 8 kHz, G.711-encoded audio of the phone network carries considerably less information than what speech recognition models were trained on; recognition accuracy is worse over a phone line, and this has to be compensated for with confirmation questions in the flow editor,
- audio is passed between the SIP side and the processing chain without buffering, in small frames; every buffer is latency,
- operators’ SIP trunks behave differently on call transfer and DTMF handling, so the bridge layer is configurable per operator,
- the network can fluctuate during a call; jitter and packet loss degrade recognition quality, so we write the RTCP data into the call log as well.
Testing could not be based on manual calls. We start automated calls that walk through the flows with pre-recorded or synthesised speech and compare the transcript against the expected one. It is slow, but without it we would have had to phone in by hand after every change.
The language model in a phone call
The large language model is the engine of the conversation, but a phone call is a particular environment. We want short, spoken answers, not paragraphs; it has to follow the steps configured in the flow editor rather than improvise; and if recognition got something wrong, it has to ask rather than guess. The model receives the description of the flow as a structured instruction, and free text only comes into the phrasing.
Appointment booking and similar operations go to the backend as tool calls, and the backend returns the result to the model. Here too, idempotent processing is the key: a retried call must not book two appointments.
What comes next?
The project has closed, the product is live, and the coming months are about operations and fine-tuning. We will carry on sharing our experience on the blog; load testing voice-based systems and call-quality-based alerting will be the next topics.
We thank the Hungarian Government and the European Union for their support, and we thank everyone who tested, gave feedback and was patient during those first, still stuttering calls.
