4 Minutes
Effective Questioning in Conversational Analytics
Written by Dr. Anand Ranganathan
At Unscrambl, we believe that conversational interfaces are the new frontier in how humans interact with computers.
We have spent a considerable amount of time over the past several years in building and refining our conversational analytics product, called Qbo.
During this time, we have encountered several technical and human challenges, learned a lot about humans can converse with bots for the purpose of gaining analytical insights, and developed a few strategies to smoothen the interaction.
A conversation is an interactive communication between two or more people. There are different types of conversations. Angel (Angel 2016) had a pretty interesting grouping of conversations based on direction and tone:
- Debate is a competitive, two-way conversation. The goal is to win an argument or convince someone, such as the other participant or third-party observers.
- Dialog is a cooperative, two-way conversation. The goal is for participants to exchange information and build relationships with one another.
- Discourse is a cooperative, one-way conversation. The goal to deliver information from the speaker/writer to the listeners/readers.
- Diatribe is a competitive, one-way conversation.The goal is to express emotions, browbeat those that disagree with you, and/or inspire those that share the same perspect
For the purposes of conversational analytics, we want users to discover insights from their data through a conversation. So, the best style of conversation is a mixture of Dialog and Discourse. There is a dialog between the users and the bot, where the user might pose questions and the bot might ask for clarifications and also provides answers. The answers from the bot can be considered a discourse. The key to success is that we want the user to be able to ask questions in an effective manner so as to get the information she needs.
This brings us to the topic of questioning strategies. One way of classifying questions is open vs closed. Open questions are those that can have many answers, and often elicit opinions. For example, “How were sales last month” forces the respondent to give an opinion about the topic. On the other hand, “What was the total amount of sales last month” is a closed question, and potentially has a precise, factual answer.
One of the early design decisions we made in our product was to focus on closed questions. Firstly, these are easier to answer with limited context, and also, easier to test and tune since we know the correct answer. However, a more “true” AI system (one that can potentially pass a Turing test) should be able to answer open questions.
One interesting style of questioning is called funnel questions. (“Questioning Techniques: Asking Questions Effectively” n.d.). This technique involves starting with general questions and then drilling down to a more specific point in each subsequent question. Usually, this will involve asking for more and more detail at each level. It’s often used by detectives taking a statement from a witness:
-
“How many people were involved in the fight?”
-
- “About ten.”
“Were they kids or adults?”
-
- “Mostly kids.”
“What sort of ages were they?”
-
- “About fourteen or fifteen.”
“Were any of them wearing anything distinctive?”
-
- “Yes, several of them had red baseball caps on.”
“Can you remember if there was a logo on any of the caps?”
- “Now you come to mention it, yes, I remember seeing a big letter N.”
-
- In our conversational analytics product, we have attempted to support funnel questions. We believe this is a truly effective way of discovering the needle insight into the haystack.
There are some key properties of funnel questions:
- There is an initial “base” question that starts off the chain of questioning.
- Each subsequent question builds upon the previous questions.
- At any time, the questioner has the option of either continuing the previous chain of questions or starting a new chain.
Let’s take a specific example from our product Qbo. Here is the sequence of questions that a potential analyst of bike share data in NYC might ask:
- What is the number of trips Restrict to those in Jan 2019This is equivalent to “What is the number of trips in Jan 2019”
- Restrict to those with trip duration > 50 minutesThis is equivalent to “What is the number of trips in Jan 2019 and with trip duration > 50 minutes”
- Group by user typeThis is equivalent to “What is the number of trips in Jan 2019 and with trip duration > 50 minutes by user type”
- Further group by genderThis is equivalent to “What is the number of trips in Jan 2019 and with trip duration > 50 minutes by user type and by gender”
The figures below show how such a conversation chain might look on our Qbo interface.
One of the challenges with any funnel-based conversation is deciding when the person you are conversing with is drilling down into the same conversation chain or is starting a new topic of conversation. We have experimented with several techniques for this. One approach that works well is to have a convention that any refinement question starts with or contains keywords like refine, restrict, further, etc. This is usually an unambiguous way of recognizing refinements though it does require some amount of user familiarization with the magic words for refinements.
We have also experimented with machine learning approaches to this problem where we attempt to translate any utterance into the appropriate base or refinement query as appropriate. While this works well most of the time, there are several cases that are quite ambiguous.
Consider the following sequence of questions:
What is the number of trips with duration > 30 minutes
Show trips with start station in Central Park
It is now ambiguous whether the second question was meant to build upon the first question (i.e. the user wanted to ask What is the number of trips with duration > 30 minutes and with start station in Central Park) or if the user wanted to start a new line of questioning. In these cases, we are forced to ask the user to clarify. Try out this out yourself, sign up here.
In summary, we believe that the ability to refine queries is a powerful feature in conversational analytics. It mimics common mechanisms that humans use to extract information from others and helps them proceed along a line of the investigation while maintaining the history of the queries.
Experience the power of conversational analytics with Qbo. Sign up for a free trial to get started.
