my roles
design leadership / observational studies / conversation & nlp design
Conversational user interfaces are a hot topic as of late. Although they can be relatively easy to create, they can be much harder to execute well.
Vector is a complex business with many units all with their own sources of information and processes.
As I regularly read and analyse feedback and reviews of vector.co.nz users, it became quite apparent that vector.co.nz was struggling to surface information effectively to customers.
Vector decided to partner with Ambit to explore a solution via a new channel and I was charged with leading internally.
opportunity
Improve the of experience of customers to vector.co.nz through conversational UI
“Chatbots are fast becoming a business imperative for businesses that want to engage with their customers. Online chat through chatbots has grown faster than any prior channel.”
Discovery
Pulling on data from google analytics, search terms, Qualtrics feedback and Vector service desk logs, I began to build up a picture of the types of people visiting vector.co.nz (personas) and the types of information and tasks they were trying to complete (jobs to be done).
My research quickly identified a common set of main objectives & themes which I used as basis for iteration of conversational content.
I also identified that frequently the answers customers needed were already available on vector.co.nz.
It became apparent that, in most cases , the problem wasn’t that the information wasn’t available - it was that it was hard to get to or that customer’s had additional questions once they found it.
Key stakeholders from around Vector were invited to a kick-off workshop - to engage the necessary people internally and to offer them an opportunity to solve real. day to day customer problems.
With internal “buy in” and a clear idea of the user problems we wished to solve, the next step was to decide on an identity and personality for our bot.
Should our bot be “male” or “female”?
Would it have a sense of humour or be dry and direct?
How should the bot represent Vector so that users respond positively, even during negative experiences e.g. power outages?
A second workshop was run where we worked through my research and identified, with the subject matter experts, the most valuable user content to focus on.
In this workshop, I also co-facilitated the team to shape our bot’s personality & tone of voice.
As a team, came to the conclusion that we should treat the bot as a form of
“digital concierge” - guiding users to the content they were looking for.
prototype
Once I had a set of draft up user flows and conversation content and approved with relevant SMEs, I wanted to test drive the experience with real users.
The quickest way to get our bot into user’s hands was to create a test bot within the Ambit platform and iterate based on customer feedback.
With the support of Ambit, we were able to create a concise set of conversation flows - a testament to the ease of use of their platform.
Test
I sat with 10 key stakeholders in the business for an observational study their interactions with the bot, making notes and asking follow up questions.
Once Ambit and I had iterated based on this small groups feedback, I then widened my approach and invited a much bigger group of internal SMEs & external customers.
As it was difficult observe all users of the wider group, I reviewed conversation logs and NLP (natural language processing) matching to look for gaps in the bots knowledge.
As a success metric, I decided to use a form of ‘completion rate’ - was Watt able to answer customer’s questions? A matrix was developed to rate each individual conservation as either pass, partial-pass or fail.
A feedback mechanism was also built directly into the bot where users were asked to rate their experience and explain why they chose that rating - which was also used to improve Watt’s performance.
PILOT
Once we had Watt tuned to be consistently hitting a ~65% pass rate, it was decided he was ready for a pilot with the public on the public facing vector.co.nz site.
Watt was placed on vector.co.nz as a soft launch pilot, to test engagement rates and satisfaction and allow us to quickly gather data to use to keep improving.
Watt was quickly adopted by users - a promising sign.
I worked daily in the Ambit platform to fine tune existing content and add any new content that we identified customers were looking for.
6 months later
Watt was quickly adopted by customers as another successful channel.
By 6 months, we had achieved a > 70% conversation pass rate and helped thousands of users find the information they were looking for.
Sats showed that over 40% of conversations were based around electricity outages which created a fantastic opportunity integrate Watt with the Vector outage API. This would allow Watt to deliver details of outages directly in conversation flow.
This was added to our future road map and implemented 3 months later and has further increased his CES and Completion rates.
I continue to work with Watt regularly to refine and fine tune his performance. He has become a valued Vector employee who we continue to look for new opportunities for scale - next up, Facebook messenger integration….but that is a story for another day.
