When performing LLM testing, we generally start with somewhere you can type. For this one though, all we had was a phone number.
In a previous post, we wrote about one of the things we found during testing: the assistant said it had changed information on a customer's file when nothing had actually changed. What made this test different was that we couldn't interact with the model directly. Everything had to go through a live phone call, so the usual way of testing an LLM didn't quite apply here.
That meant we had to approach the test a little differently. We noticed some techniques worked as expected, some became inconsistent, and others seemed to fail for reasons that had little to do with the model itself. For this engagement, a lot of the work was figuring out what was actually happening between the prompt we sent and the response that came back.
This post looks at how we built the test setup, what changed when speech was introduced, and what that meant for the techniques we were testing.
What Changes When Testing Has to Go Through Speech
With a normal LLM test, you would normally send a prompt and get a response back. But with a phone assistant, there are a few extra steps in between. For example, our test input had to be turned into speech and played into a live call, then whatever the assistant said had to be recorded and transcribed before we could do anything with it.
That extra layer meant there was more that could go wrong before a prompt even reached the model. As such, we built the test harness around being able to see what was happening at each step.
The Architecture
The loop runs the same way every turn:
- The test runner decides what to say next
- A text-to-speech engine turns that line into audio
- The audio is routed into a live phone call through a virtual audio device and a softphone
- The target system receives it like any other phone call
- Whatever comes back gets recorded, transcribed, and handed back to the harness as the next input
- Repeat step 1
For the setup, we used Piper to generate the speech and VLC to play it back. Next, VoiceMeeter was used to route the audio into Zoiper5, which handled the call. On the way back, we used OpenAI Whisper locally to transcribe what the assistant responded on the call. The LLM deciding what to send next was accessed through a private gateway rather than directly.
Once that was working, two practical choices became important.
What stayed local. Some responses appeared to contain real account information, so we kept transcription on the same machine running the test. The LLM handling our own test prompts was also routed separately through a private gateway.
How much we automated. The next line could be sent automatically once the assistant stopped speaking, and most of the time that worked well. Sometimes though, the assistant would not respond, either because the prompt had not come through properly or it had not understood it. Usually, repeating the prompt was enough. We also saw bad transcriptions, and sometimes the next prompt started before the assistant had finished speaking. Because of that, we kept a manual option so we could step in when needed.
A quick note: Since this engagement, tools such as VoiceStudio have appeared that bring more of this workflow into one place. We did not use it for this test, but something like that would have reduced some of the setup and glue work.
A Payload Isn't a Conversation
When testing, a single prompt rarely told us much. Some of the more interesting behaviour only showed up when we came back to the same question later in the call, or asked for the same information again using different wording.
That meant the harness had to keep the conversation going rather than just send one payload and log the reply. Each new line was generated with the full call history in view:
history = [
{"tester": "Who manages this account?"},
{"assistant": "Your account manager is Alex."},
...
]
next_line = generate_test_case(history)Writing those follow-up lines by hand across dozens of calls quickly became impractical, so we used an LLM to generate the tester's side as well. It would look at what the assistant had just said and decide what to ask next. We kept it to short, natural prompts and stopped it from ending the call on its own.
One instruction turned out to be especially useful: if the assistant appeared to reveal something it should not, we treated the first answer with suspicion. The next prompt would ask for the same information again later, using different wording. If the answer stayed consistent, we looked at it more closely. If it changed or contradicted itself, it was often just the assistant making something up rather than a real disclosure.
What You Say Isn't Always What Gets Heard
We noticed this most with payloads that depended on exact formatting. By the time the text had been turned into speech and transcribed back again, brackets, punctuation, capital letters and command-style syntax could easily be lost or changed.
Sometimes the opposite would happen. A request could come through slightly garbled or informal and still work because the assistant understood what we were trying to say. But anything that depended on exact structure was much less reliable.
So that made some common prompt injection techniques difficult to use over the phone. We noticed anything that depended on exact syntax was pretty unreliable. On the other hand, simpler conversational approaches worked much better, such as claiming an identity, asking for the same information in a different way, or framing something as a correction rather than a new request. Those generally appeared to hold up much better because they did not rely on punctuation or exact wording.
Language Changed the Filtering Behaviour
In addition, we saw something similar with filtering. Partway through the test, one phrasing we had been using stopped working. When we contacted the client to ask if changes were made, they confirmed that a new filter had been added.
What was interesting was that the same request could still get through when we changed the wording. We also tried it in Bahasa Indonesia, and that bypassed the word filtering we were hitting in English.
So that made language worth testing on its own. Because if a control is looking for particular words or phrases, changing the language can produce very different results. Of course, this also depends on whether the speech-to-text and text-to-speech layers support the language well enough for the request to make it through in the first place.
Voice Changes the Attack Surface
With these kinds of tests, voice changed which techniques were practical, not just how we delivered them.
Techniques that worked more reliably:
- identity claims and social engineering style framing
- repeating or rephrasing the same question
- conversational context building across turns
- attempts to trigger an action indirectly rather than by naming it
- switching languages, such as using Bahasa Indonesia, where filtering behaved differently
Techniques that were much less reliable:
- payloads that depended on exact punctuation or formatting
- bracketed or labelled command syntax
- anything relying on capitalisation or precise structure
- encoded or Unicode-based tricks
The main thing we took away was that there were really three things to keep separate: what we sent, what the speech-to-text layer heard, and what the assistant actually did. Keeping those separate made it much easier to tell whether we had found a real issue or whether something had simply changed along the way.
Where possible, it is also worth asking for access to the backend logs during testing. Being able to see the transcription the system actually received makes it much easier to work out whether something went wrong during speech-to-text or later when the model processed it.
That is also why we also logged every turn as we went. For each exchange, we kept the transcription, the assistant's response and the timestamp.
Those logs ended up being important for the finding in our previous post. What we said matched what the system transcribed, but the assistant still confirmed a change that had never actually been made. As such, that let us rule out the voice and transcription layer as the source of the problem.
Without the logs, it would have been much harder to work out where the failure had actually happened.
Further Reading: