top of page
Evaluative Research for a Virtual Agent Service Line
2021.4 - 2021.7
Context
San José 311 is the city of San Jose's official reporting platform consisting of an app, website, and a service phone line. The platform launched its Virtual Agent (VA) in fall 2020. Residents reach it by calling 311 and then select the VA option after engaging with a phone tree. But the analytics reveal that less than half of callers complete their service requests through the VA, lower than anticipated. So the city wants to figure out what happened and understand how to improve the service completion rate.
Objectives
The whole project was based on the assumption that callers couldn't complete the service requests because they have difficulties interacting with the Virtual Agent. So the project lead and I came up with the objectives for the whole study:
-
Improve the experience of interacting with the Virtual Agent
-
Increase the number of successfully submitted requests
Research Questions
With these objectives in mind, I came up with two research questions:
-
What are the pain points encountered by callers when engaging with the current San José 311 Virtual Agent?
-
how can we improve the experience of the interactions and increase the service completion rate?
Methodologies
In order to answer these questions, what I need is behavioral data that help the team to identify users' frustration. So I decided to do a usability test first to find answers.
Discussion guide
Execution
What would be a good way to conduct a usability test on a phone line remotely? I suggested using Zoom for the study and ask participants to interact with the VA on a speaker mode, followed by questions from the moderator.
Participants
We recruited 5 residents who live inside or near the city of San Jose because we think they would isolate enough problems to help us decide if there's a need to revise the VA
Prepare materials
We drafted a research plan and project debrief for the stakeholder in the city. I also prepared a discussion guide for the test, which consists of four parts:
-
Introduction
-
Warm-up questions
-
3 tasks
-
Wrap-up questions
Set up tests and analyze data
We didn't conduct a pilot session for the test as we were tight on time. We used Zoom to conduct the remote tests and recorded 2 of the in-person tests. While taking notes on participant's direct answers to the questions, I used Google spreadsheets to synthesize raw data and form different themes.
Usability test notes
Analysis spreadsheets
Findings
The result is a bit surprising to us, as we found out that participants didn't have many problems using Virtual Agent or interacting with the agent. In fact, the reason they didn't complete one of the three tasks was because of the category service option in the phone tree before they even speak to the virtual agent.
Insights
The result is a bit surprising to us, as we found out that participants didn't have many problems using Virtual Agent or interacting with the agent. In fact, the reason they didn't complete one of the three tasks was because of the category service option in the phone tree before they even speak to the virtual agent.
Participants couldn't complete T1 and T2 in a pleasant way because they had to deal with the long phone tree before interacting with the virtual agent. So it's the phone tree that influences their experience with the service line. (Research question 1)
With this analysis, we decided to conduct tree testing to further validate the problem with the phone tree architect.
Stage 2 Research on 311 Phone Tree
Research questions
-
How do callers feel about the structure of the current phone tree?
-
Is there a problematic or vague service placement in the phone tree?
-
What kind of phone tree categories do callers find more intuitive?
Methodologies
I used Tree Testing and Card Sorting in the study because the subject here is the content structure of the service, and I wanted to understand how users interact with the structure. Then I also wanted to know what kind of structure would make sense to them.
Execution
I was responsible for the whole execution of the tests. Since both of them were unmoderated, I spent more time in preparation and pilot sessions.
Prepare materials
I drafted a research plan and shared it with the project lead and stakeholders in the city.
Research plan
Pilot sessions
I created a Tree Testing and an open Card Sorting study on optimalworkshop.com and conducted pilot sessions with the team before releasing to the public
Tree Testing study
Card Sorting study
Recruit participants
The team recruited 14 participants for Tree Testing and 10 for Card Sorting through the City of San Jose Facebook group and other community-based groups online
Recruiting post on Facebook group
Findings
Tree Testing:
The tasks completion rate for Tree Testing was very low. According to the follow-up questions, users find the process confusing, not clear, and difficult to use. Based on the findings of the tree testing, I conducted an open card sorting with a group of new participants.
next step
Card Sorting:
Participants created new categories for the current phone tree, which make much more sense to them:
New categories suggested by optimalworkshop.com based on participants' card sorting results
Deliverables
Based on what I discovered in the Tree Testing and Card Sorting study, I conducted a presentation for stakeholders from the city and shared my analysis as well as findings. Then I prepared a topline report with recommendations featuring a new architect for the phone line along with suggestions for the next steps.
Project presentation
Topline report
After I shared the findings and study report with the stakeholders, the project has been on hold as the project lead has left and stakeholders needed to rebuild the team.
Reflection
-
The sample size may not be representative of all people that may use the product.
-
These two studies are conducted remotely and without moderators, so questions and tasks administered varied from participant to participant.
-
The context of the studies is different from the actual context, where users interact with the San José 311 phone line on the phone instead of a computer, so users’ experiences and impressions might be different.
bottom of page