# How to evaluate thirty municipal virtual assistants.

> For an observatory on the digital transformation of public administration, we are testing the virtual assistants of Italian provincial capitals, service by service.

- Sector: Public sector
- Practice: AI and automation
- Status: Ongoing
- URL: https://cloudfactory.it/en/work/evaluating-municipal-virtual-assistants/

## In brief

About thirty chatbots and virtual assistants of Italian provincial capitals, asked about the same twenty municipal services, with the same questions in the same order. The goal is a comparison between cities that holds up, and a method that can be repeated in later editions to measure progress.

- **~30** municipal virtual assistants
- **20** municipal services they are asked about
- **2** analysis set-ups, essential and complete

## Context

Many cities have put a virtual assistant on their website. The client, which studies innovation in public administration, wanted to know whether these assistants actually answer citizens about services, with a comparable measure rather than impressions.

## The problem

The assistants run on different technologies, from guided answer trees to generative models. Answers change with how a question is phrased and can change over time. Without a fixed protocol and saved evidence, no comparison holds.

## What we did

1. **A clear scope.** Provincial capitals and a list of twenty municipal services defined with the client. Every assistant is asked about the same services.
2. **Same questions, same order.** A set of questions per service, put to every assistant in the same way. Each answer is archived as evidence, so results can be checked.
3. **Two levels of depth.** An essential and a complete set-up, with different time and cost, chosen according to how the client uses the results.
4. **Built to be repeated.** The protocol is designed for later editions at lower cost, to measure how assistants change over time.

## How it went

The project is ongoing: the client chose the essential set-up and the first edition is starting.

## Tools

Test protocol, Evidence archive, Comparative report

## FAQ

### Do you also evaluate generative AI assistants?

Yes. The protocol is the same for all, whatever the technology: what matters is what the assistant tells the citizen.

### Can it be applied to a single organisation?

Yes. The same method lets an organisation check its own assistant before and after an update, or compare it with others.

---

Cloudfactory S.r.l., Rome. Contact: hello@cloudfactory.it