Monitoring your application
Use the OpenDialog Analyse feature to make informed decisions about the past, present and future of your conversational application.
Last updated
Use the OpenDialog Analyse feature to make informed decisions about the past, present and future of your conversational application.
Last updated
This section explains how to track user interactions with your conversational application and identify trends in project usage.
The OpenDialog Analyse dashboard provides a comprehensive view of recent interactions, a log of the latest messages, and a high-level summary of usage patterns over time.
Analyse lets you visualise your application's conversational data set, which you can use to improve your conversational application where needed.
It allows you to answer the following questions:
How many conversations are happening by how many users?
What is happening in the conversations?
Which conversations had errors and where?
What was the logic that led to specific conversational paths & errors?
Analyse provides you with a visualisation of the application's interaction logs, project usage over time, message logs, detailed conversational history per user, flow visualisation and dialogue management insights.
Viewing specific conversations and their logic, you can make informed decisions and improve your application's user experience.
By viewing the number of fallbacks and filtering conversations based on them, you can check your application's health and identify improvement areas.
By viewing the number of users that started a conversation and the number of conversations per user, you can diagnose any discoverability & desirability issues and fix them.
By visualising usage trends, you can see the evident uptake or downturn in the number of users and expand or improve the application's use case accordingly.
Analyse allows you to explore how your application has been performing. It is, therefore, part of the testing suite of a given scenario.
To access the Analyse dashboard:
Navigate to your workspace dashboard
Select a scenario
Using the left-hand sidebar menu, expand the 'Test' section
Select 'Analyse'
To use the Analyse feature within OpenDialog, navigate to the 'Analyse' dashboard within the sidebar, where you can review, explore, and expand different data snapshots of your application.
Analyse has three main components: project usage, interaction logs, and message log table:
The project usage functionality allows you to visualise project usage trends over time.
You can expand or minimize the project usage graph to your needs using the Expand/Collapse icons in the top right corner of your screen, depending on how focussed you wish the view to be.
Project usage can be viewed for a specific day, week, month or other selected date range via the date picker in the top right corner. This allows you to hone in on specific date ranges and check your conversational application's usage for that time.
This can be particularly useful, for example, if you have made specific changes to where your users can find your application, the call-to-action, or the conversational experience at a given time, and you wish to visualise and communicate its impact on the number of users.
The list of interaction logs allow you to visualise a list of conversations had by specific users, at a specific time and it's related data. Available data points per interaction log are date & time , user-ID , timezone , platform , browser , number of fallbacks in the conversation, and number of messages . Each of these can be used to sort through the interaction logs. We will specify the use of a few of them below.
In addition, you can use the date picker on the top of the screen to select conversations for a specific date range, or search the interaction log for a specific user-ID thanks to the user-ID search field in the top right corner.
Date & time gives you an indication of the timestamps of users interacting with your application. Filtering by this field allows you to visualise the most recent conversations first or, on the other hand, the oldest conversations. In addition, the timestamp can help you identify a specific conversation for a user, in case you need to review it. Bearing in mind that an interaction spans over a period of time, the single value in the Date and Time column show us the date and time of the first message sent within an interaction (within the date range specified by the filters).
User-ID indicates the user's unique user-ID which can help you identify a specific user. OpenDialog automatically generates a user-ID for each start of your application, if one is not provided. However, this user-ID can also be user-specific like user e-mails if your scenario is set up with this information.
You can also sort by fallback to visualise which logs contain more fallbacks versus which ones did not. It will be interesting to dig further into those conversations with many fallbacks to identify areas of improvement for your conversational model or your language service.
The number of messages will indicate how many exchanges a user had with your application within the provided date range. This can indicate users' engagement with your application and its capacity to retain users and help them accomplish their goals.
You can click through each item in the interaction log list to visualise its detailed conversation logs.
The message log table is an essential tool for reviewing and understanding the full history of conversations within your OpenDialog scenarios. Combined with its filtering capabilities, the table offers a powerful method for monitoring and analysing the flow of messages. It acts as a detailed log of every message exchanged between users and the application, providing valuable insights into the progression of conversations.
There is an export button in the top left hand side of the table. Clicking this button will export data with the current filters applied into a CSV file for you to download.
By examining this data, you can evaluate the performance of your application, identify patterns, and uncover potential areas for improvement.
Columns in the message log table
The table presents a variety of fields that categorise each message within your application:
There are many different ways to filter the message log table to find exactly what you are looking for. You can filter by date and time by clicking on the calender and selecting the specific time frame you would like to filter the messages by.
You can also filter your messages based on User ID. To do this, you can start to manually type in a User ID into the search bar. Or, you can click on the User ID attached to a certain message in the table and this will populate the search bar with this User ID for you.
Another way you can filter the table is with the preview filter. This allows you to select what type of messages you would like to view in your message log table. There are three different options to choose from:
Preview data - this filters all the messages that have been generated using the 'Preview' application internally within OpenDialog
Live data - this filters all the messages that have been generated from a live conversation with a user
All data - this shows you all of the available data
The message log table includes a breadcrumb filter to make message analysis quicker and more effective. This feature allows you to filter the messages based on specific criteria, ensuring that you can narrow down the data to the most relevant interactions. The filter works in a hierarchical fashion, helping you to further investigate the data in a descending order:
Conversation: The first filter layer is the conversation itself, allowing you to isolate a particular session with a user. This is helpful when you want to analyse all messages associated with a single conversation from start to finish.
Scene: Once you've selected a conversation, you can further filter by scene. This allows you to examine messages within a particular phase of the interaction, providing context on how users move through the various stages of the conversation.
Turn: With scene filtering applied, you can then add an additional filter for specific turns. This level of filtering helps you focus on moments (where either the user or system sent a message) enabling you to further investigate the flow of interaction at a more granular level.
Intent: Intents display the exact message contents from either the user or the system, allowing you to review what was communicated within each turn of your conversation.
The detailed conversational logs for each item in the interaction logs are composed of four sections:
The user details
The conversational log history
The conversational flow visualisation
The conversational logic logs
User information gives you a quick summary of the interaction log item you are viewing. You also have the option to copy the User-ID from here to search for further interaction logs by this same user in the interaction logs list.
The conversation logs contain the entire conversational history for this interaction log, tracking back all user and application exchanges. You can use the 'play' button to visualize the individual exchanges and their logic in the middle panel of the screen. You can also view the intent details for each message by using the 'View intent details' toggle at the top of the Conversation Log section. If you see any issues with the displayed message, you can thus click on the linked intent and go edit the selected message.
The considered conversational paths are displayed in the middle panel for the selected user-application exchange. You can play through them via the Play button on the conversational logs or walk through them step by-step to review the conversation engine logic by using the arrows at the bottom of the screen. This helps you to understand what incoming and outgoing intents, and messages were selected by the conversation engine and why. If any virtual intents were encountered within the request you are viewing, a further cycle of frames will be displayed. These frames will display the incoming and outgoing intents that were selected as a consequence of reaching the virtual intent.
Finally, the right-hand panel displays the conversational logic behind the selected conversational paths, displaying the available, selected and rejected intents and the in-depth reasoning behind each decision made by the conversation engine. This is particularly useful to debug conversations and, for example, to better understand why certain user utterances might have caused an unexpected answer. The detailed conversational logic contains information regarding the interpretation of the intents via the interpreters set up for your scenario, the passing of on conversational components, the reasoning behind intent selection, transitions between conversational components and what are attributes are being saved during that interaction.
I need more granular data. How can I access this information?
If you need more granular data, you can contact us via support@opendialog.ai.
I do not see the conversational flow visualisation. Is this normal?
Conversational flow visualisation is currently unavailable for exceptionally large scenarios. We are working on alternative visualisations to cater for those as well.
I want to access the analytics data via an API rather than the OpenDialog application. How can I do this?
The Interaction API enables retrieving information about interactions between the user and OpenDialog scenarios. The data is requested for over a time period ( from , to ) and is split by user_id . You can check out the Interaction API documentation through this link.
I got a warning message for the conversational flow visualization indicating 'No Request ID provided'. What does this mean?
This can happen if the event storage was not active or turned off when the user made a request. In the future, you should not encounter this any more as this is related to the roll-out of the Analyse feature.
The project usage graph shows both the number of users and the number of requests for any given date or range of dates. It can also be expanded for a more focused and in-depth visualisation.
The interaction logs are listed and filtered by date, with the most recent appearing at the top. Each interaction log can be expanded to review specific conversations.
The message logs are listed and filtered by date, with the most recent appearing at the top. Messages can also be filtered by Conversation, Scene, Turn and Intent to find specific results.
log table