Bitwise Knowledge Assistant

Help & Documentation

← Back to Help Center

Analytics Dashboard Guide

Overview

The Analytics Dashboard provides insights into how the Knowledge Assistant is performing. It tracks usage patterns, response quality, user engagement, and system performance. Access the dashboard from the Admin Console.

Performance Metrics

Performance metrics help you understand how fast the system responds to queries.

Time to First Token (TTFT)

The time between when a user submits a query and when the first token of the response starts streaming. This is the perceived “thinking time” before the answer appears.

Good: <2s | Acceptable: 2-5s | Slow: >5s

Streaming Rate

The rate at which tokens are delivered to the user, measured in characters per second. Higher rates mean faster-appearing responses once streaming begins.

Good: >100 chars/s | Typical: 50-100 chars/s

Stream Completion Rate

The percentage of responses that complete successfully vs. those that are aborted by users or fail due to errors. High completion rates indicate good user experience.

Good: >95% | Concerning: <90%

Retrieval Quality Metrics

These metrics measure how well the system finds relevant content from the knowledge base.

MRR (Mean Reciprocal Rank)

Measures how high the first relevant result appears in the retrieved chunks. MRR of 1.0 means the most relevant chunk is always first; lower values mean relevant content is buried further down the list.

Excellent: >0.8 | Good: 0.6-0.8 | Needs improvement: <0.6

Recall@5

The percentage of relevant chunks that appear in the top 5 results. High recall means the system is finding most of the relevant content; low recall means important information is being missed.

Excellent: >80% | Good: 60-80% | Needs improvement: <60%

Precision@5

The percentage of the top 5 results that are actually relevant. High precision means less noise in the retrieved content; low precision means the model has to filter through irrelevant chunks.

Excellent: >70% | Good: 50-70% | Needs improvement: <50%

Hit Rate

The percentage of queries where at least one relevant chunk was retrieved. A low hit rate indicates the knowledge base may be missing content or the retrieval system is failing to find existing content.

Excellent: >95% | Good: 85-95% | Needs attention: <85%

Generation Quality Metrics

These metrics evaluate the quality of the generated responses.

Faithfulness

Measures whether the generated response is supported by the retrieved context. High faithfulness means the model is grounding its answers in the source material rather than hallucinating information.

Excellent: >90% | Good: 75-90% | Concerning: <75%

Answer Relevance

Measures how well the response actually answers the user's question. High relevance means the response addresses what was asked; low relevance indicates the model may be going off-topic.

Excellent: >85% | Good: 70-85% | Needs improvement: <70%

User Engagement Metrics

Engagement metrics show how users interact with responses.

  • Source Clicks: How often users click to view source documents
  • Follow-up CTR: Click-through rate on suggested follow-up questions
  • Feedback Rate: Positive vs. negative feedback on responses
  • Export Usage: How often users export responses (Markdown, DOCX, Google Docs)
  • Share Usage: How often responses are copied/shared

Using Analytics to Improve Results

If TTFT is High

  • Consider using Turbo variant for faster retrieval
  • Reduce the number of candidates retrieved (lower top_k)
  • Disable HyDE for simple queries

If Retrieval Quality is Low

  • Enable hybrid retrieval to improve keyword matching
  • Enable Vertex AI reranking for better result ordering
  • Adjust relevance thresholds in retrieval settings
  • Review ingestion to ensure content is properly chunked

If Faithfulness is Low

  • Increase the number of context chunks passed to the model
  • Enable entity overlap filtering to reduce noise
  • Review and improve retrieval quality first

If User Engagement is Low

  • Review feedback tags to understand user concerns
  • Adjust response tone to match user preferences
  • Ensure follow-up questions are relevant and useful

Related Articles