KDDI has improved the performance of its Buffmee consumer AI application by overhauling how it evaluates answers and analyses system bottlenecks, cutting overall response latency by 38%.
Buffmee is a retrieval-augmented generation application designed to answer questions using more than 100 content sources, including books, magazines and web media. Built around proprietary material, the service needed lower latency while keeping responses closely tied to source material.
The Japanese telecoms group also improved Time to First Token by nearly 18%, a measure of how quickly a user begins to receive an answer after submitting a prompt. At the same time, it reported a 25% improvement in groundedness scores, which measure whether responses are supported by source documents.
The project focused on replacing manual testing with automated evaluation across a large and varied content library. Engineers created hundreds of automated tests and assembled a benchmark dataset to measure answer reliability across different use cases.
The process used a model-based judging system and a sampling method called the Rule of Hundreds. Rather than reviewing every document in the corpus, the team grouped content by file format and media composition, then selected representative examples from each category.
This reduced the evaluation workload by 75% while preserving coverage across text-heavy, image-heavy and mixed-media documents. Source material included web articles, EPUB files, PDFs and structured data.
Latency issues
Response times had become a problem as Buffmee approached launch. The engineering team needed to ground responses in a wide range of books and magazines, but latency issues kept the application from meeting response targets.
To investigate the delays, the team analysed production logs to see how prompt design and task division were affecting performance. One issue was prompt bloat, particularly large multi-page system prompts that slowed the path to the first generated token.
Engineers then revised the system prompt and changed how tasks were routed between sub-agents. They also moved functions into separate skills so that only the logic needed for a given request would load at runtime.
KDDI said this modular approach helped address what it described as deep-stack bottlenecks without reducing answer accuracy. It presented the effort as a way to improve speed and reliability at the same time, rather than treating them as competing goals.
Evaluation method
Another change involved the scoring system used to test outputs. For key metrics, the team shifted from a graded one-to-five scale to a binary pass-or-fail model to reduce inconsistency in automated reviews.
Product judgement also remained part of the process. Rather than relying entirely on default thresholds, a product owner reviewed random answer samples alongside automated scores to decide what standard was acceptable for release.
This mix of automation and human review was used to set practical quality thresholds across a diverse body of content. The aim was to make testing more repeatable while still accounting for edge cases and user expectations.
Buffmee is positioned as an interactive learning and discovery tool that lets users search for information, summarise key points and explore hobbies or personal interests. By citing underlying sources, the application is intended to help users verify where information comes from and reduce concerns about fabricated answers.
The work also reflects a broader challenge in consumer-facing generative AI services, as companies try to maintain answer quality while delivering acceptable response times across different media types. Applications built on retrieval-based systems often depend on large document stores, which can increase complexity as teams expand the range of supported content.
In KDDI's case, the operational lesson was that large-scale testing and real-time log analysis were both necessary to make the service workable. The result was a launch-ready system built to handle grounded answers across a broad mix of proprietary media.
"Our vision hinged on a platform where content, once ingested, would instantly function as a working RAG system. Google's careful, hands-on guidance made that a reality - we're sincerely grateful for their support," said Shunya Onoda, AI Product Department, KDDI.