Public sector virtual assistants, put to the test with a method.

Chatbots and virtual assistants of councils and public bodies questioned on the same services, with the same questions in the same order. Comparable results, verifiable evidence and a protocol that can be repeated over time.

Service
Virtual assistant assessment
For
Observatories, public bodies, public sector suppliers
Method
Repeatable protocol, archived evidence
The problem

An assistant is judged by its answers, not its demo.

Many public bodies have put a virtual assistant online, but few know how it answers on the services that matter: certificates, local taxes, enrolments, reports. Manual tests change the question and tone every time, and the results can't be compared or repeated.

How we work

A protocol, before the questions.

  1. 01

    A clear scope

    We define with you the assistants to assess and the list of services to question them on. Every assistant is questioned on the same services.

  2. 02

    Same questions, same order

    A set of questions per service, put to every assistant in the same way. Every answer is archived as evidence, so the result can be checked.

  3. 03

    A scoring grid

    The criteria for judging an answer are written before starting and are the same for every assistant.

  4. 04

    Two levels of depth

    An essential and a complete configuration, with different timing and cost, chosen according to how you'll use the results.

  5. 05

    Built to be repeated

    The protocol is reused in later editions, at reduced cost, to measure how assistants change over time.

What we deliver

Results that can be checked.

Results can be published on a private site, accessible only to authorised people.

Assessment protocolQuestions per serviceAnswer archiveScoring gridComparison between assistantsFinal reportPrivate results site
In detail

How to build a repeatable test.

Who it's for

Those who need to know how they answer.

Observatories and research centres

Studies on public sector digital transformation, with data comparable across bodies and editions.

Councils and public bodies

An independent check of your own assistant before or after it goes live.

Suppliers of public sector assistants

An external test bench, on the same services you'll be judged on.

Why Cloudfactory

We build AI assistants too.

We're assessing around thirty virtual assistants of Italian provincial capitals on twenty council services. And we build AI assistants that only answer from validated texts: we know where they go wrong, and how to write a question that shows it.

FAQ

Before assessing.

Do you need access to the body's systems?

No. Assistants are questioned the way a citizen would, from the public website.

Who chooses the services and questions?

We define the services with you; we write the questions and share them before starting, so the method is clear to everyone.

Can the assessment be repeated?

Yes, it's designed for that: later editions use the same protocol, at reduced cost.

Are the results public?

That's your call. We can deliver them on a private site accessible only to authorised people.

Contact

How many assistants do you want to test?

Tell us which bodies and which services you're interested in. We'll reply with a proposed protocol.

Email
hello@cloudfactory.it
Address
Viale di Trastevere 114, 00153 Rome, Italy