Consistent performance measurement is a key factor in ensuring the successful implementation of any new business initiative. But performance measurement for a solution like chatbots requires specific parameters. Traditional web analytics cannot calculate misunderstood requests or delayed responses of a chatbot. Chatbots, therefore, require different kinds of metrics and Key Performance Indicators (KPIs.)

Chatbot analytics help solution designers and developers to get into the mind of end-users and quantitatively identify (and improve) areas of a poor experience. It helps chart information like user satisfaction levels, where conversational traffic is flowing, and the rate of user drop-offs at any stage of the conversation flow. These KPIs and metrics are usually dependent on user demography and chatbot use case. However, there are several agnostic metrics that can provide valuable insight for just about any chatbot.

Types of Chatbot Metrics

We have collated these key metrics and clubbed them into three broad categories. Here’s all you need to understand the performance of a chatbot deployed across any channel, with any level of integration, and for any purpose.

• User Metrics

  a. Total - Total user interactions are a direct reflection of the impact the chatbot is creating and the data it is exposed to; it can also potentially help in calculating the market size.

  b. Active - The number of active users is a good barometer to measure the actual success and overall popularity of the chatbot. It can clearly indicate the failure/success of use cases or design.

  c. Engaged - This metric takes the analytics even further and deeper. Engaged users are the ones that actually communicate with the chatbot sending and receiving messages. It can provide conversation statistics based on this sub-sample.

  d. New - This metric captures the overall success of any newly launched chatbot campaign. New users are necessary to keep the active user number high.

• Message Metrics

  a. Conversation length - Conversation length is a hidden indicator of the experience the chatbot is creating. If the number of steps to reach the desired value is too many, an impatient user will leave the conversation. That means the shorter the conversation length the better is the impact. At the same time, however, if the conversation length is too short, the user might have already ended it without finding what he/she was looking for. Therefore, it is advisable to keep a tap on the conversation length (with respect to the use case and context) to ensure optimal user experience and non-inflation of sessions with idle intervals.

  b. Conversation starter messages - The greeting message at the start of any session is critical foranalyzing how contextual the conversation is. The ripples created by this opening message is a metric depicting chatbot database effectiveness. The ripples can be continual replies from the users, the number of engaged and active users, and better user ratings. It is particularly important for marketing or customer service use cases.

  c. User rating - User ratingis extremely valuable to realize individual feedback. It is the only metric powerful enough to show performance on a per-message basis rather than on a per chatbot basis. That way it is easier to identify weak links in the conversation flow.

• Bot Metrics

  a. Fall Back Rate (FBR) - Fallback responses occur when chatbots do not know the appropriate response to a message. Monitoring their rate of occurrence (i.e. FBR) and the user messages that invoke them can help identify the wrong placement of the chatbot (non-relevant expectations from the users) and/or faults in Natural Language Processing (NLP) engine (bot unable to understand the questions already in the database.)

  b. Goal Completion Rate (GCR) - GCR captures the percentage of successful engagements achieved by the chatbot. It is the number of times the bot successfully processes the input and reverts with a satisfying response. It is directly affected by NLP efficiency and Machine Learning (ML) capabilities.

  c. Retention Rate - This is the percentage of users that return to the chatbot in each time frame. Retention rates help extract valuable insights regarding the customer’s preferences by making them spend time on the chatbot. It can, however, be achieved only by deploying a high-quality chatbot that meets customer expectations.

Analytics Trackers

Ideally, you should use a chatbot platform that has its own built-in analytics. That eliminates the hassle of integrating and setting up analytics through a 3rd-party service. Nevertheless, there are third-party chatbot analytics solutions, like Chatbase and Dashbot, available in the market that can be utilized if your chatbot does not have in-built analytical capabilities.

Senseforth’s proprietary chatbot platform, A.ware, is one of the platforms equipped with powerful analytics that leads to better conversations. It has four specific analytics modules that track user conversations, customer profiles, agent efficiencies, and numerous other parameters. Want to know more about A.ware? Explore all the features and capabilities offered by the platform here.