How to evaluate thirty municipal virtual assistants.
For an observatory on the digital transformation of public administration, we are testing the virtual assistants of Italian provincial capitals, service by service.
About thirty chatbots and virtual assistants of Italian provincial capitals, asked about the same twenty municipal services, with the same questions in the same order. The goal is a comparison between cities that holds up, and a method that can be repeated in later editions to measure progress.
Context
Many cities have put a virtual assistant on their website. The client, which studies innovation in public administration, wanted to know whether these assistants actually answer citizens about services, with a comparable measure rather than impressions.
The problem
The assistants run on different technologies, from guided answer trees to generative models. Answers change with how a question is phrased and can change over time. Without a fixed protocol and saved evidence, no comparison holds.
What we did
- 01
A clear scope
Provincial capitals and a list of twenty municipal services defined with the client. Every assistant is asked about the same services.
- 02
Same questions, same order
A set of questions per service, put to every assistant in the same way. Each answer is archived as evidence, so results can be checked.
- 03
Two levels of depth
An essential and a complete set-up, with different time and cost, chosen according to how the client uses the results.
- 04
Built to be repeated
The protocol is designed for later editions at lower cost, to measure how assistants change over time.
How it went
The project is ongoing: the client chose the essential set-up and the first edition is starting.
Tools
FAQ
Do you also evaluate generative AI assistants?
Yes. The protocol is the same for all, whatever the technology: what matters is what the assistant tells the citizen.
Can it be applied to a single organisation?
Yes. The same method lets an organisation check its own assistant before and after an update, or compare it with others.