At a glance
In 2024, I built a basic customer-service phone agent with Twilio Voice, Go, Gin, and Cohere. It answered a call, collected speech, sent the transcript to a model, and read the response back to the caller.
Built
A phone-to-model loop: Twilio answered the call, gathered speech, a Go webhook called Cohere, and Twilio spoke the reply.
Limit
It worked as a demo, but it had no retrieval, durable memory, streaming, tools, or handoff path.
Lesson
Voice agents are mostly about latency, state, recovery, and clear boundaries. The model call is only one piece.
Prototype
I built the simplest useful phone loop
The goal was basic: can a real phone call trigger an AI answer, and can that answer be spoken back to the caller?
The app used Twilio Voice for the call, a Go server with Gin for webhooks, and Cohere for the generated response. A caller asked a question, Twilio sent the transcript to my server, the server called the model, and Twilio read the answer back.
Build
The system was small on purpose
/answerhandled the incoming call and returned the opening TwiML./handle-user-inputhandled speech turns and generated the next response.- The prompt used a seed company message from the environment.
- The loop ended each answer by asking if the caller had another question.
Limits
It was a demo, not a support agent
The prototype treated each turn like a standalone prompt. That meant no durable conversation memory, no verified company knowledge, no tools, no escalation, and no confidence checks. If the model guessed, the caller would hear the guess out loud.
Latency was also obvious. Phone calls make every pause feel longer. Waiting for speech capture, a blocking model call, and a full spoken response was fine for learning, but not good enough for production support.
Next
I would rebuild it around reliability
If I rebuilt it now, I would start with the service workflow instead of the model call.
Streaming voice
Lower latency, interruption handling, and a more natural turn-taking loop.
Grounded answers
Retrieval over approved company knowledge instead of relying on one seed prompt.
State and tools
Track the conversation and call tools for real tasks like tickets, bookings, or order checks.
Handoff
Escalate when confidence is low, the request is sensitive, or the caller asks for a human.