We evaluated a GeoAI tool under real-world conditions, with 81 participants across three different sectors of Central Macedonia.

We evaluated a GeoAI tool under real-world conditions, with 81 participants across three different sectors of Central Macedonia.

---

The context

Most large language models lack spatial awareness — they can’t answer questions that require integrating geographic, demographic, environmental and socio-economic data. JackDaw is a GeoAI chatbot built to close that gap, combining natural-language interaction with geospatial intelligence.

As part of the AURA project, we were tasked with evaluating JackDaw under real-world decision-making conditions, across three distinct application areas:

• Smart Residential Decentralisation (peri-urban Thessaloniki)
• Sustainable Rural Tourism — Pink Blossom Valley (Veria)
• Data-Driven Rural Business Intelligence (Kilkis)

Central Macedonia was chosen as the validation area because it combines a major metropolitan centre with diverse rural, agricultural, tourism and industrial regions — making the findings transferable to other European regions as well.

---

How we did it — our methodology

We used a Human-Centred Design approach across three sequential phases, aiming to combine quantitative and qualitative data to build a complete picture of usability, reliability and usefulness.

Phase 1 · Usability Testing

9 participants, 3 per use case. Individual, observation-based sessions built around specific personas, measuring task completion time, errors and points of confusion, combined with questionnaires (System Usability Scale).

Phase 2 · Collaborative Stakeholder Workshops

12 participants. Group workshops per use case, working through real-world scenarios, where stakeholders tested and discussed the tool together.

Phase 3 · Community-Wide Validation

60 participants, in-person and online. Broader validation of acceptance and usefulness, using standardised measurement instruments (SUS, Extended Technology Acceptance Model — TAM2).

---

What we measured

Beyond users’ subjective impressions, we combined observational data (actual user performance) with self-reported data (questionnaires), to check whether perceived usability matched actual performance. We used established, scientifically validated measurement instruments:

• System Usability Scale (SUS) — a standardised measure of perceived usability
• Technology Acceptance Model (TAM2) — measuring intention to use and perceived usefulness
• Observational data: task completion rate, completion times, points of difficulty

---

Who it was for

The project was carried out under the PoliRuralPlus programme, through a cascade funding call, with the goal of validating the PoliRuralPlus tool ecosystem (specifically JackDaw) in real operational settings. Throughout the process we engaged public authorities, tourism organisations, business-support organisations, SMEs and citizens — producing structured feedback on the tool’s usability, reliability and usefulness, while also helping build the digital capacity of the stakeholders involved.

PoliRuralPlus has received funding from the European Union’s Horizon Europe research and innovation programme under Grant Agreement No. 101136910. Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union or the European Research Executive Agency. Neither the European Union nor the granting authority can be held responsible for them.

---

What came out of it

The evaluation surfaced specific usability issues per application area, distinguished between problems that recurred across all use cases (and therefore most likely relate to the tool itself) versus issues specific to a single domain, and led to well-documented recommendations for improvement. The findings fed into both the project’s public final report and a detailed internal report for the consortium.