GenAI outputs require verification because of hallucinations. As you integrate AI into your investigations, it is important to understand why hallucinations happen and how to address them based on your risk thresholds.
GenAI can provide value to an investigation, but it’s not for free.
Why does GenAI Hallucinate?
GenAI algorithms use probabilities to generate sequences of words. It can ultimately generate a sequence of words that are probable, but not accurate.
This most often comes into play when you ask GenAI to enrich and reason about your data when it doesn’t actually know much about it. It can generate false information about an artifact and come to conclusions it can’t actually justify.
Here is a recent DFIR example I had (paraphrasing for brevity):
Claude: ‘co1bld’ is a standard hostname that appears in the LNK files created on Microsoft’s internal build infrastructure during the Windows build process. These LNK files are not suspicious even though they were created on a different machine.
Me: Do you have a reference for that hostname?
Claude: Honestly, no — I don’t have a reliable citation for that. When I said it, I was drawing on what seems like plausible pattern-matching (the naming convention, the context), but I can’t point you to a public source that definitively documents co1bld as a Microsoft internal build hostname.
We ultimately decided not to take Claude’s advice and could not find any Internet references to it. Cyber Triage will still flag LNK files that have ‘co1bld’ as their hostname.
Combating Hallucinations
There are several ways to verify data is not a hallucination:
- Query for item existence: You can automatically confirm that items referred to by AI are actually in the original data set with a simple deterministic query by their path, time, etc.
- Manual: Confirm enrichment with other deterministic data sources, such as threat intelligence feeds or websites that were cited for the conclusion. This often requires a human to reason that the data source and the AI conclusion are equivalent.
- Judging LLM: Have an independent LLM review the enrichment and storyline to confirm that it agrees (i.e. Claude vs ChatGPT). This can still have errors, but it can increase your overall confidence.
The Judging LLM approach can help with reasoning errors, and one model could identify a logical mistake made by another. But, it may not catch knowledge errors because they may both have the same knowledge gaps or wrong data (i.e. the internet is wrong about something).
Know Your Investigation and AI Risk Levels
Not every investigation has the same impact, and therefore each can have different risk thresholds for how much to verify the results.
Examples:
- A low-severity EDR alert investigation could have a high-risk threshold and be OK with little human verification or with a judging LLM.
- A medium-severity EDR alert may want a human to review the timeline and double check nothing else was missed.
- A criminal case where someone could go to jail should have a low-risk threshold and a human verifies every item in the final result.
Know your investigation risk thresholds.
Verify the Critical Findings
GenAI provides value because it can enrich and reason at scale much faster than a human can when looking at thousands of artifacts. Most artifacts in an investigation are not relevant. They are part of normal device activity.
As you are coming up with your storyline and identifying which artifacts are relevant, that’s when you should focus on verifying what AI introduced:
- If it enriched the data, verify it.
- If it came up with a storyline, verify the data it’s using is correct, and the storyline makes sense.
You do not need to review enrichment of items that are not relevant to the case.
To do this, it’s critical for your tools to identify what came from AI so you know what to verify.
AI Results in Cyber Triage
As we’ve been integrating AI into Cyber Triage using the MCP server, we clearly identify what came from AI to make sure you know what to verify.
The below are from the (unreleased) 3.18 release, which allows the MCP server to add notes and score items as suspicious.
Any enrichment from AI is stored in an “AI Enrichment” field, and the text starts with “[AI]”:
For scoring, the MCP server can also score items as suspicious so that the user has to manually upgrade to Bad (i.e. human review).
Those items are clearly marked as coming from AI:
Further, the justification is clearly marked as AI, and the LLM is also identified:
Try GenAI in Cyber Triage
If you are looking to incorporate AI into your investigations, try out Cyber Triage with its MCP server. It’s BYOAI and you can use your Claude account, AWS Bedrock, or local models. Setup instructions are in the user manual.