# Public sector virtual assistants, put to the test with a method.

> Chatbots and virtual assistants of councils and public bodies questioned on the same services, with the same questions in the same order. Comparable results, verifiable evidence and a protocol that can be repeated over time.

- Service: Virtual assistant assessment
- For: Observatories, public bodies, public sector suppliers
- Method: Repeatable protocol, archived evidence
- URL: https://cloudfactory.it/en/services/public-sector-chatbot-assessment/

## An assistant is judged by its answers, not its demo.

Many public bodies have put a virtual assistant online, but few know how it answers on the services that matter: certificates, local taxes, enrolments, reports. Manual tests change the question and tone every time, and the results can't be compared or repeated.

## We build AI assistants too.

We're assessing around thirty virtual assistants of Italian provincial capitals on twenty council services. And we build AI assistants that only answer from validated texts: we know where they go wrong, and how to write a question that shows it.

## A protocol, before the questions.

1. **A clear scope.** We define with you the assistants to assess and the list of services to question them on. Every assistant is questioned on the same services.
2. **Same questions, same order.** A set of questions per service, put to every assistant in the same way. Every answer is archived as evidence, so the result can be checked.
3. **A scoring grid.** The criteria for judging an answer are written before starting and are the same for every assistant.
4. **Two levels of depth.** An essential and a complete configuration, with different timing and cost, chosen according to how you'll use the results.
5. **Built to be repeated.** The protocol is reused in later editions, at reduced cost, to measure how assistants change over time.

## Results that can be checked.

Results can be published on a private site, accessible only to authorised people.

- Assessment protocol
- Questions per service
- Answer archive
- Scoring grid
- Comparison between assistants
- Final report
- Private results site

## How to build a repeatable test.

### The questions

- **Written first, the same for everyone.** The questions for each service are fixed before starting and put to every assistant in the same order and with the same words.
- **The way a citizen would ask.** Not just the textbook question, but also the vague one, the one with a local detail, the one outside the assistant's scope.
- **Separate sessions.** Each assistant starts from a fresh conversation, so one answer doesn't influence the next.
- **Every answer archived** with date, time and full text, so anyone can reread it and check the score.

### The scoring grid

| Criterion | What we look at |
| --- | --- |
| Accuracy | Is the information correct for that authority? |
| Completeness | Are the requirements, documents, fees and times the citizen needs there? |
| Source | Does it link to the official page or the right form? |
| Limits | When it doesn't know, does it say so and point to an office, rather than making things up? |
| Clarity | Can it be understood without already knowing the answer? |

Criteria and weights are written before starting and don't change during the test. That's what makes it possible to compare assistants with each other and with the next edition.

## Those who need to know how they answer.

- **Observatories and research centres**: Studies on public sector digital transformation, with data comparable across bodies and editions.
- **Councils and public bodies**: An independent check of your own assistant before or after it goes live.
- **Suppliers of public sector assistants**: An external test bench, on the same services you'll be judged on.

## Before assessing.

### Do you need access to the body's systems?

No. Assistants are questioned the way a citizen would, from the public website.

### Who chooses the services and questions?

We define the services with you; we write the questions and share them before starting, so the method is clear to everyone.

### Can the assessment be repeated?

Yes, it's designed for that: later editions use the same protocol, at reduced cost.

### Are the results public?

That's your call. We can deliver them on a private site accessible only to authorised people.

---

Cloudfactory S.r.l., Rome. Contact: hello@cloudfactory.it
