LLM Usage Dashboard

In LLM Usage Dashboard, you can monitor token usage on LLMs.

PREREQUISITES

  • At least one active Knowbler client.

  • At least one active LLM integration.

  • The dashboard works for Salesforce and Zendesk. An LLM Usage dashboard for Microsoft Dynamics is under development.

Access LLM Usage Dashboard

  1. Expand Large Language Model and open Usage Dashboard.

  2. You'll either see "LLM Integration is not activated." or a report and two filters.

  3. The filters are Service Desk Management Tool and Date Range. You can use the filters to generate a report.

Report Total Requests

The first report in the dashboard is Total Requests. It lists the total calls made to an LLM.

The report is collapsed in the default state. Upon expanding it, you can see five columns:

  • Serial Number is the row number.

  • Requester Email: The email ID of the support rep who clicks Create an Article in Knowbler when Knowbler is connected with an LLM. The column is searchable.

  • Case ID. The Salesforce or Zendesk ID of the case. This column is searchable.

  • Case Subject. The title used in Salesforce or Zendesk. This column is searchable.

  • Requested Time. The time when the support rep clicks on Create an Article in Knowbler.

In the default state, 10 rows are listed. The data can be emailed and downloaded.

Four Tiles

The second part consists of four (or five tiles).

  • Input Tokens counts the total number of tokens shared from Knowbler to an LLM.

  • Generated Token counts the total number of tokens shared from an LLM to Knowbler.

  • Average LLM Response Time is the time spent on generating responses divided by the total requests.

  • Error Rate is the ratio of number of requests with errors divided by the total number of requests.

NOTE.

The tile Integration can be seen only when multiple LLM integrations are in use.

API Consumption

NOTE.

This report is only for Claude.

Each Knowbler customer gets a fixed number of tokens each month. You can monitor token usage for the current month or for any of the previous six months in the API Consumption section.

When token usage in a given month is way less than the allocated number, then the bar is green.

It turns blue as the monthly usage moves near the allocated number.

Once the token consumption exceeds the quota in a month, then the bar turns red. You receive an email when the 100% of the monthly token quota has been consumed. The emails are also sent when the consumption is 50% and 90% of the quota.

To continue using the LLM after exceeding the quota, talk to your CSM.