Stop Digging Through Tabs: Meet the ONEKEY Chat Agent

Why conversational access matters in a security platform

Security teams rarely suffer from a shortage of data. The problem is the opposite: too many places to look, too many things to compare, and too many ways to lose hours answering one seemingly simple question.

Modern security platforms generate and connect enormous amounts of information. SBOMs, firmware versions, dependencies, and a growing number of CVEs all have to be understood in context. Collecting the data is no longer enough. The real challenge is helping people make sense of it quickly.

Traditional ways of navigating this information works well, when the data is smaller and the question is simple. But in most cases, users are not just looking for a dashboard or a list of vulnerabilities to be fixed. They want answers. They ask questions like “What has changed?” or “Which findings should I care about first?”.

This matters especially in security, where speed is important. But speed without accuracy is just a faster way to be wrong. Teams need quick access to insight, but they also need confidence that the answer is accurate, well supported, and grounded in actual platform data. And with the number of published vulnerabilities continuing to rise, no serious team can hope to review everything manually.

This is where conversational access helps most, by turning complexity into answers people can actually use. It is becoming even more relevant in Europe with the Cyber Resilience Act (CRA), which introduces mandatory cybersecurity requirements for products with digital elements across their lifecycle, including vulnerability handling and security updates . As these requirements move from policy into implementation, teams need faster ways to understand findings, prioritize remediation, and work with evidence they can actually use in security and compliance processes. Our participation in the CRACoWi project reflects that same need for practical support, helping turn CRA requirements into usable tools and workflows for companies.

What the ONEKEY chat agent can do

The ONEKEY Chat Agent is built for a very specific job: helping product-security and compliance teams work with firmware that has already been analyzed in the ONEKEY platform. That narrow focus is deliberate. It is what makes the agent useful in practice. Instead of trying to be a general security assistant with opinions on everything, it stays close to the platform, the available data, and the questions users actually ask during daily work. The agent is designed to stay read-only, rely on platform data, and avoid pretending to know more than it does. If a user wants to change something, it will explain how to do that in the UI or via the API, rather than acting on its own.

Within that scope, the agent is built to help with a set of practical, high-value tasks. One of the most important is summarizing an analysis. When a user asks, “What’s wrong with my firmware?”, they usually do not want a long list of findings read back to them one by one. They want a clear overview of severity and status, with the most relevant issues brought to the front. That makes the answer faster to absorb and more useful for triage.

Another core task is comparing firmware versions. Users often want to understand what changed, whether the security situation improved, and where new risks may have appeared. The chat agent can support this by looking at both firmware versions, reviewing their analysis results, and highlighting the differences that matter rather than just presenting raw output.

It can also explain a single finding in a way that is easier to understand. That may mean clarifying why an issue exists, what a CVE match means for a detected component, or how the available evidence supports the result. This is especially valuable when technical findings need to be discussed across teams with different levels of expertise.

The agent can also support triage. It can explain the current triage state, point to the evidence behind a recommendation, and suggest the most appropriate next step and rank findings so users can focus on what is most important. But it does not make decisions for the user, and it does not silently change anything in the system. It helps users move faster without hiding responsibility or reducing traceability.

The chat agent can also help users when the results needs to be presented differently. It can explain findings for a specific audience, for example by turning a technical result set into a management-level summary.

Finally, the ONEKEY Chat Agent can help users with the platform itself: how to find something or how to run an analysis. In other words, it is more than an analysis assistant: it is also a guide for navigating and using the platform more effectively.

What makes these capabilities valuable is that they reflect real tasks users do, such as summarize, compare, explain, prioritize, triage, locate, and communicate and make them faster, clearer, and easier to handle.

Why we chose a multi-agent architecture

The ONEKEY Chat Agent is built as a multi-agent system because the tasks it needs to handle are too different to fit well into one general assistant. Summarizing an analysis, comparing firmware versions, answering platform usage questions, and enforcing scope or policy all require different context and different instructions. Splitting these responsibilities makes the system easier to control and more reliable in practice.

This design has some clear advantages. Each agent works with a cleaner context instead of carrying every possible rule and responsibility at once. Prompts can be specialized and focused on one job. This improves consistency, reduces noise, and makes it easier to keep the system aligned with the platform’s real use cases. It also makes the system easier to extend over time, because new capabilities can be added without turning one agent into an overloaded generalist.

In practice, the system is organized around three roles. A Moderator Agent checks whether a request is in scope. The Central Agent handles the conversation, interprets the request, and uses platform tools to retrieve the relevant data. The OQL Agent translates the user’s analytical intent into OQL queries for structured data access. Its specialized prompt and focused context help it generate queries that match the platform’s data model and the user’s request.

The tools are an important part of why this works. The agent does not answer from guesswork. It works through a focused set of read-only tools that are tied to analyzed firmware: resolving firmware, retrieving overviews, listing and inspecting Issues and CVE matches, searching extracted content, and reading relevant files. For support related questions, the agent can also search and read product documentation. This keeps answers grounded in actual platform data and documentation, while also limiting the system to the actions and information it is meant to use.

That combination of specialized agents and focused tools is what makes the architecture valuable. It allows the ONEKEY Chat Agent to stay focused, grounded, and trustworthy, while covering a wide range of real tasks inside the platform.

How we evaluated model quality, tool use, and system reliability

Building a chat agent for security work requires balancing accuracy, tool use, responsiveness, and cost. A model may sound impressive and still be the wrong fit for a product that needs accurate answers, correct tool use, reasonable response times, and predictable cost. That is why we treated model selection as an evaluation problem from the start.

This evaluation is clearly a data science task. We have to compare the quality of the output of several models, just like we do in case of classical machine learning problem. So the same basic discipline was used: prepare test data, measure what matters, compare the outputs, and select the model that performs best for the intended use case.

We began with collecting questions users are likely to ask. Some were simple, such as explaining what a CVE is. Others were much closer to day-to-day security work, with more technical depth and more room for error. For these questions, we prepared curated reference answers based on wikipedia pages and blog posts and used them to compare how different models performed. The goal was not just to see whether a model could produce fluent text, but whether it could answer IT security questions clearly, correctly, and in a way that was actually useful. Each response was evaluated independently by two large language models from different providers, which compared it with the curated reference answer and scored its coverage, accuracy, coherence, and style. Although the evaluators used slightly different scoring ranges, they produced nearly the same overall model ranking, giving us greater confidence in the results.

In addition to answer quality, we also had to assess how well each model calls tools. A chat agent like this is only reliable if it can retrieve the right data before answering, so we tested that separately in controlled setups with static tools. We asked questions that required tool calls in order to be answered properly, and then checked whether each model chose the right tool and used the right parameters. We ran these evaluations across around 50 models, including variants with different reasoning levels. Quality was important, but it was not the only factor. Response time and cost also mattered, because a model that is slightly better on paper may still be the wrong choice if it is too slow or too expensive for a real product workflow.

That process led us to the model that best fit the needs of the ONEKEY Chat Agent. Not the one with the most hype, but the one that performed best for the kind of work the platform actually needs to support.

And the process does not stop there. New models appear constantly, and we will continue to evaluate them in the future and might replace the model behind the chatbot whenever there is a realistic chance to improve quality, efficiency and reliability.

Looking ahead, deployment flexibility will matter as well. Some customers may prefer not to send sensitive data to certain model providers at all. For that reason, we plan to support customer-provided models in the future, giving organizations more control over how the chat agent fits into their own security and compliance requirements.

Conclusion

Security work will probably never become simple. There will always be large result sets, firmware comparisons, triage decisions, and CVE lists nobody should have to open before their first coffee.

What can be improved is the way analysts work with all of that information. The ONEKEY Chat Agent is meant to make those everyday tasks faster, clearer, and easier to manage without losing sight of what matters most in this domain: accuracy, traceability, and trust.

That trust does not come from the fact that the system uses AI. It comes from how the agent is built: its answers are grounded in platform data, not guesswork, the model behind it was selected through careful evaluation, its scope is controlled, its tools are read-only, and when the available information is not enough, it is designed to ask for clarification instead of pretending certainty.

This becomes even more important as the Cyber Resilience Act raises the bar for secure and maintainable digital products in the EU. Through our involvement in CRACoWi, we also contribute to the broader effort to make these new compliance expectations more practical and accessible for manufacturers and other market actors

If it helps users spend less time digging through tabs and more time acting on the right information, then it is doing its job.