Public sector virtual assistants, put to the test with a method.
Chatbots and virtual assistants of councils and public bodies questioned on the same services, with the same questions in the same order. Comparable results, verifiable evidence and a protocol that can be repeated over time.
An assistant is judged by its answers, not its demo.
Many public bodies have put a virtual assistant online, but few know how it answers on the services that matter: certificates, local taxes, enrolments, reports. Manual tests change the question and tone every time, and the results can't be compared or repeated.
A protocol, before the questions.
- 01
A clear scope
We define with you the assistants to assess and the list of services to question them on. Every assistant is questioned on the same services.
- 02
Same questions, same order
A set of questions per service, put to every assistant in the same way. Every answer is archived as evidence, so the result can be checked.
- 03
A scoring grid
The criteria for judging an answer are written before starting and are the same for every assistant.
- 04
Two levels of depth
An essential and a complete configuration, with different timing and cost, chosen according to how you'll use the results.
- 05
Built to be repeated
The protocol is reused in later editions, at reduced cost, to measure how assistants change over time.
Results that can be checked.
Results can be published on a private site, accessible only to authorised people.
How to build a repeatable test.
Those who need to know how they answer.
Observatories and research centres
Studies on public sector digital transformation, with data comparable across bodies and editions.
Councils and public bodies
An independent check of your own assistant before or after it goes live.
Suppliers of public sector assistants
An external test bench, on the same services you'll be judged on.
We build AI assistants too.
We're assessing around thirty virtual assistants of Italian provincial capitals on twenty council services. And we build AI assistants that only answer from validated texts: we know where they go wrong, and how to write a question that shows it.
Before assessing.
Do you need access to the body's systems?
No. Assistants are questioned the way a citizen would, from the public website.
Who chooses the services and questions?
We define the services with you; we write the questions and share them before starting, so the method is clear to everyone.
Can the assessment be repeated?
Yes, it's designed for that: later editions use the same protocol, at reduced cost.
Are the results public?
That's your call. We can deliver them on a private site accessible only to authorised people.