Conversational AI—Ushering In an Era of ‘Citizen Data Scientists’

Over the past decade, enterprises have embraced the path toward data-driven decision-making through investments in technology of biblical proportions, including complex enterprise architectures, data warehouses and an army of bespoke skill sets. The goal was to build capabilities to use data to achieve supernormal growth and mitigate risks. In Asia-Pacific alone, revenues for big data and analytics will reach $14.7 billion in 2018, an increase of 14.4 percent over 2017, led by banking and telecommunications investments.

Despite these investments, it often takes a long time for business users in enterprises to get the right data and insights, even if the data is well-organized and structured. This is because they may not know where the data is located, how it is modeled and how to frame the right queries for their needs. As a result, access to data and analytics has traditionally been limited to power users and specialist data scientists.

Lack of access to data and insights is a massive issue. It is estimated that 71 percent of data collected by enterprises is not being used for decision-making and that only 32 percent of employees have access to business intelligence and analytics tools. In Asia-Pacific, 29 percent of companies said that they rely on the analytics expertise of individual employees, but do not have dedicated data-analytics departments.

Many enterprises have invested significant amounts of time and resources to build data architectures, ranging from enterprise data warehouses to big data lakes. However, they have not solved the last-mile problem of how to get relevant data and insights into the hands of the key analysts and decision-makers. Hence, a lot of their data remains locked up in their systems, never getting the chance to be used. In fact, one study found that the median Fortune 1000 company could increase its revenue by more than $2 billion a year if it increased data usability by 10 percent.

The unfulfilled promise of big data has slowly sunk in, with the focus now shifting to “smart data” or, more specifically, the ability to derive meaningful insights and outcomes from data in both big and small ways.

Conversational Analytics and AI

This is the ability to unlock the value of data for organizations through personalized and relevant insights, enabling them to understand their business better, make decisions faster and be exponentially more productive.

Conversational analytics and AI allow any user to ask questions of their data via voice or text queries and receive back a natural language and, potentially, a visual analysis of the most statistically relevant and actionable insight for that user. It incorporates a combination of key AI technologies, including natural-language processing, natural-language generation/narration, chatbots, automated visualizations, augmented analytics and automated insight discovery. Business users in different parts of an enterprise can ask queries in natural language on sales, marketing, operations, supply chain and logistics, IoT and other data sets and gain access to personalized insights that can drive strong business impact.

We have started seeing tremendous interest in this technology in the banking and telecommunications industries. How often do you receive promotions only to be dismissed as “spam” that is completely irrelevant to your spending patterns and interests? If only marketing analysts in these companies had access to and leveraged the rich data that they already have on consumer spend, demographics and merchants across spatial and temporal dimensions.

They need to quickly figure out how their current campaigns are performing, which segments of their customer base they should target with new offers, which merchants they should have deeper partnerships with, and what offers they should create and for whom. They also need to collaborate with their teams to discover and refine insights on their data and work on their marketing strategy. They need to be able to do this insight discovery and collaboration on tools they may already be using like Slack or Microsoft Teams, apart from web-based interfaces. Most importantly, they need to be able to discover these insights on their own without needing help from IT or a specialized data science team. This is critical to enable them to respond to situations in the market rapidly and with agility.

Conversational AI Makes It Easier

Several banks and telecommunications companies in Asia are starting to use conversational analytics to bridge the gap between those who ask the questions and the systems that have the answers. Conversational interfaces have conversational context, which allows users to iteratively slice, dice and refine their queries and discover insights that may be hidden deep in the data.

A critical use case for these companies is to help CEOs, CFOs, department heads and sales leaders understand trends in revenue, sales and profits and understand key differences in these metrics across different periods of time. They are starting to use natural language queries with bots to figure out the key root causes of sudden spikes or drops in indicators of interest as quickly as possible. Again, they don’t want to be reliant on an IT or data analytics team to get these insights, as that can result in critical delays to decision-making.

For example, allowing them to get answers to ad-hoc analytical queries during monthly or quarterly board meetings allows them to make decisions quickly, whereas in the past, business decisions were often delayed by weeks or months because of lack of availability of critical data in time.

Conversational AI Requirements

There are several technical and architectural requirements in enabling such a conversational capability to access data and insights. It is important to have a good model of the data so as to enable reliable and accurate translation of natural language queries to structured queries like SQL. It is also important to have data architecture and data processing pipelines that produce clean and trustworthy data.

Security is another factor to take into consideration. It is important that each user only has access to the data they have access to and are allowed to query. For example, while the raw data may have sensitive personally identifiable information, such as names, phone numbers or identity card numbers, only a few users may have access to such data. For most other users, the results must be anonymized or aggregated before they are presented to them.

Conversational interfaces have one important characteristic that makes them particularly useful. This is conversational context. That is, they allow users to iteratively slice, dice and refine their queries and, thus, discover insights that may be hidden deep in the data. They also allow users to refer to previously asked queries and results obtained in subsequent queries. Such a notion of context dramatically improves user experience and productivity.

Conversation AI in Asia

We are seeing the first wave of pilots and production deployments of this technology across Asia, including in Singapore, the Philippines, Malaysia, India and Indonesia. Many of these pilots are happening in large enterprises that had already invested in putting together big data-processing infrastructures, but desired to improve access to insights for key decision-makers.

Currently, the key language for these systems is English, given it tends to be the key language used within large enterprises. As we see the penetration of these technologies into small- and medium-sized businesses, we expect to see growing demand for local languages.

One of the advantages that Asia has is the already heavy use of chat platforms, both in the public and enterprise domains. A lot of business and decision-making happens on chat platforms such as WhatsApp, Line, Microsoft Teams, Slack, Viber and Telegram. Hence, there is a demand for accessing analytics through these chat platforms. This portends well for the large-scale adoption of conversational analytics technologies. Most of these chat platforms already support bots, and hence, adding a bot to the contact list or to a group channel will be relatively easy for most users.

The world as a whole and Asia, in particular, are just starting to see the benefits and potential of conversational analytics. We are excited to be part of this process.