AI Chatbot

The organisation was facing an issue: their staff often received email and phone queries of a similar nature. The questions could have been answered using their website content, but customers preferred to send an email enquiry or call the organisation than to search for answers. While staff appreciated the customer engagement, it meant they had less time to focus on more specialised work.

The organisation had considered the idea of a chatbot for some time, but weren't sure whether an AI model could effectively handle the queries they received. A POC was the ideal first step in answering this question, allowing us to trial the concept before committing to a full-scale implementation.

Staff feedback about the type of enquiries the organisation was receiving highlighted a major requirement for the chatbot: it needed to synthesise complex and occasionally out-of-date or incomplete information into clear, trustworthy answers.

The organisation's website contains a lot of content, intended for a number of different audiences. Some customers contacted the organisation because they were struggling to navigate it all to find the answers they needed. Trust and accuracy had to be our central principle in designing the chatbot.

For the POC, we designed a retrieval-augmented generation (RAG) chatbot. RAG models only use the data in their knowledge base — in this case, the organisation's website — to answer questions. This helps them avoid one of AI's common pitfalls: fabricating information.

We built trustworthiness into the chatbot by designing it to always cite its sources. This feature, alongside its ability to handle complex questions, made the RAG model an ideal one for the organisation to test in a controlled environment. It also meant that people who wanted could find more information in the sources for the chatbot's responses.

Finally, to protect the organisation's information in accordance with government guidelines, we turned the model's learning function off.

While the technical construction of the chatbot was relatively straightforward, most of our work focused on crafting the right prompts and improving the accuracy of responses.

We began by integrating the chatbot with the organisation's existing data, ensuring it only pulled from trusted sources so it provided accurate, verifiable information in every interaction. We also designed it to clearly cite its sources, so users could check its answers and find extra information if they needed it.

Prompt engineering was the most critical and time-intensive aspect of the project. This involved training the chatbot to understand and respond correctly to user queries, while making sure it didn't "hallucinate" incorrect answers.

We refined the prompts several times, adjusting how the chatbot interpreted queries and extracted the right data from the knowledge base. We focused on ensuring that when it was asked a question it couldn't fully answer, it wouldn't make an answer up or draw on sources outside its knowledge base - instead, it would direct users to a contact within the organisation.

During the POC, the chatbot successfully answered common customer queries, showing its potential to free up staff for more specialised tasks. Its reliable, well-sourced responses built trust within the organisation and feedback from staff provided insights into areas for improvement. These included broadening the chatbot's knowledge base to include the organisation's legislative framework and improving the website content it drew upon, as well as design improvements.

The POC allowed the organisation to explore AI's potential in a low-risk, cost-effective manner. By trialling the chatbot with internal staff, they gathered valuable feedback and avoided the high costs of a full rollout. This iterative process ensured that the chatbot was refined before any broader implementation, helping the organisation validate its usefulness and secure stakeholder buy-in for future development. For this government organisation, the POC laid the groundwork for a smarter, AI-driven approach to customer queries.