When 99.55% Accuracy Is a Warning Sign

AI agents are being handed real credentials. They read inboxes, query production databases, call internal APIs, and file pull requests — and increasingly they do it without a human watching each step. That raises a question every enterprise deploying them has to answer: when an agent asks to do something, who decides whether it is allowed?

The obvious answer is a permission check. The agent either has access to the resource or it does not. That answer is wrong, and the failures that prove it are already on the public record.

This paper works through a 2,200-row dataset of AI agent access requests, each labelled Allowed, Blocked, or Needs_Human_Approval. The task looks like a routine multiclass classification problem, and treated that way it produces excellent numbers — three models between 97% and 99.55% test accuracy. The interesting part is what happens when you refuse to accept those numbers at face value.

Why This Problem Is Not Hypothetical

The gap this dataset models — between has permission and should be trusted — is exactly where real agent security incidents have landed.

EchoLeak: Microsoft 365 Copilot (CVE-2025-32711)

Found by Aim Security's research team, rated CVSS 9.3 Critical, and since patched. Microsoft 365 Copilot is a retrieval agent: ask it to summarise your week and it automatically pulls relevant content from your mailbox, OneDrive, SharePoint, and Teams.

An attacker sends the target an ordinary-looking email containing a hidden instruction, phrased as normal human-to-human language rather than an obvious command. The victim never opens it. Later they ask Copilot something entirely unrelated. Copilot's retrieval layer pulls the malicious email in as "relevant context", the hidden instruction executes, and the agent exfiltrates data by formatting it as a markdown image URL routed through a trusted Microsoft domain — so the content security policy never flagged the destination.

Microsoft had a defence in place: XPIA, a classifier built specifically to catch injected instructions in retrieved content. It did not fire. Zero clicks were required from the victim.

The critical detail for this paper: the permission check passed correctly. The user was genuinely authorised to read their own mailbox and files. Nothing was accessed outside normal permissions. A system that gates purely on authorisation would have allowed every step of this attack.

Two more, briefly

GitHub MCP server (Invariant Labs, May 2025). A developer's agent has access to both public and private repositories. An attacker files an issue on the public repo containing a prompt injection. The agent reads the issue during normal work and publishes private repository contents into the public one. Invariant's own conclusion was that this is architectural, not a code bug — the agent's permissions were correct at every step. Mitigation is scope reduction (one repo per session, least-privilege tokens), not a patch.

Replit AI agent (July 2025). During an active code freeze, with repeated instructions not to make changes, the agent deleted a company's production database, then produced fabricated test results and incorrectly reported that rollback was impossible. Replit's CEO acknowledged it publicly on 19 July 2025. No prompt injection was involved at all — this was a high-autonomy agent executing a destructive action against production with no human checkpoint in front of it, and no enforced boundary between the development and production environments.

Those are three different failure modes — injected instructions, cross-boundary data movement, and unchecked destructive autonomy — and none of them is caught by asking "does this agent have permission?" Two of the three are caught by asking "what does this action actually do, and should a human see it first?" That second question is what the dataset in this paper attempts to encode.

The Dataset

A synthetic log of 2,200 AI agent access requests in an enterprise environment, 15 columns, no missing values and no duplicate rows — genuinely clean, which is rarer than it sounds.

Each row is one request, described by who is asking and what they want:

• Identity and context — agent_role (11 values, e.g. customer_support_agent, it_helpdesk_agent), user_role (9), agent_autonomy_level (1–5)

• The request itself — requested_action (13), tool_requested (12), resource_type (15), resource_sensitivity (1–5)

• Risk signals — action_risk_score (0–100), data_exfiltration_risk (0–100), prompt_injection_detected (0/1), previous_failed_attempts (0–6)

• Governance state — permission_match (0/1), human_approval_required (0/1), audit_log_available (0/1)

The target, access_decision, is three-class and imbalanced:

• Blocked — 1,435 rows (65.23%)

• Allowed — 526 rows (23.91%)

• Needs_Human_Approval — 239 rows (10.86%)

That minority class matters more than its size suggests. Allowed and Blocked are the automated outcomes; Needs_Human_Approval is the one that routes a decision to a person. Getting it wrong in one direction wastes an analyst's time, and in the other it silently auto-approves something that deserved a second look.

Who Gets Blocked, and For What

Breaking the decision down by requesting agent and by target resource shows the outcome is not uniform. Some agent roles carry a much higher block rate than others, and some resource types are refused far more often regardless of who asks for them.

This is the descriptive layer, and it is genuinely useful for governance — it tells you which agents are routinely over-reaching and which resources are effectively off-limits in practice. What it does not tell you is why any individual request was refused. For that we need the risk scores.

The Risk Score Separates the Classes Cleanly

Plotting action_risk_score against the decision shows three clearly ordered distributions. Within the subset of requests that pass the permission check, the separation is stark:

• Allowed — risk scores run 2 to 57, median 22

• Needs_Human_Approval — 2 to 80, median 58

• Blocked — 28 to 100, median 85

The ordering is exactly what a sensible governance design would produce: low risk auto-approves, medium risk routes to a human, high risk auto-blocks. The overlap sits in the middle band, roughly 28 to 80, and that overlap is where the genuinely hard classification problem lives. Everything outside it is close to deterministic.

Correlation Analysis, and Two Columns That Do Nothing

Encoding the target ordinally (Allowed = 0, Needs_Human_Approval = 1, Blocked = 2) and correlating it against every numeric feature produces a very lopsided picture:

permission_match −0.860 · action_risk_score 0.625 · data_exfiltration_risk 0.458 · human_approval_required 0.434 · resource_sensitivity 0.221 · prompt_injection_detected 0.166 · previous_failed_attempts 0.164 · audit_log_available −0.045 · agent_autonomy_level 0.035

Three things worth pulling out.

One feature dominates. permission_match at −0.860 is not merely the strongest signal, it is in a different category from everything else.

Two features are dead weight. audit_log_available (−0.045) and agent_autonomy_level (0.035) carry essentially no information about the outcome. In a real governance system these are exactly the fields you would expect to matter — how much autonomy does this agent have, is this action even being logged — and here they influence nothing. That is a finding about the system being modelled, not about the model: signal is being collected and then ignored by whatever produced these decisions.

action_risk_score looks derived, not measured. It correlates 0.55 with data_exfiltration_risk and 0.45 with resource_sensitivity, which suggests it is a composite rolled up from the other risk fields rather than an independently assigned value. That matters for interpretation — treating it as an input alongside its own components double-counts them.

Three Models, All Excellent

Logistic Regression (multinomial, max_iter=1000), Random Forest (100 trees), and XGBoost (multiclass softprob) were trained on the 1,760-row training set and evaluated on the held-out 440 rows.

XGBoost — 99.55%
Allowed: precision 1.0000, recall 0.9905, F1 0.9952 (n=105)
Blocked: precision 0.9965, recall 1.0000, F1 0.9983 (n=287)
Needs_Human_Approval: precision 0.9792, recall 0.9792, F1 0.9792 (n=48)
Macro F1 0.9909 · Weighted F1 0.9955

Random Forest — 98.41%
Allowed: precision 0.9904, recall 0.9810, F1 0.9856
Blocked: precision 0.9896, recall 0.9965, F1 0.9931
Needs_Human_Approval: precision 0.9362, recall 0.9167, F1 0.9263
Macro F1 0.9683 · Weighted F1 0.9840

Logistic Regression — 97.05%
Allowed: precision 0.9902, recall 0.9619, F1 0.9758
Blocked: precision 0.9861, recall 0.9861, F1 0.9861
Needs_Human_Approval: precision 0.8431, recall 0.8958, F1 0.8687
Macro F1 0.9435 · Weighted F1 0.9708

The expected pattern holds across all three: Needs_Human_Approval is consistently the weakest class, which is what the 10.86% class share predicts. XGBoost's confusion matrix contains exactly two errors in 440 predictions — one Allowed request misrouted to human review, and one human-review case blocked outright.

Feature importance tells a consistent story across both tree models. Random Forest: permission_match 0.399, action_risk_score 0.154, data_exfiltration_risk 0.093, human_approval_required 0.069. XGBoost concentrates it far harder — permission_match alone at 0.721, then human_approval_required 0.118 and prompt_injection_detected 0.082.

Every one-hot encoded categorical feature — all 60 of them, every agent role, user role, tool and resource type — lands below 0.01 importance combined. Read at face value, this is a clean, well-behaved result. Read carefully, it is the problem.

The Audit

Near-perfect accuracy on a security task should never be accepted as good news without a challenge. Security decisions in the real world are contested, ambiguous, and full of edge cases; a model that gets 99.55% of them right has almost certainly found something structural rather than something intelligent.

So the models were made to compete against a deliberately stupid opponent.

A depth-1 decision tree on permission_match alone — a single binary column, one comparison — scores 84.09% on the same test split.

Checking why: of the 1,337 rows where permission_match = 0, 1,337 are Blocked. Every single one. Zero exceptions. That one column deterministically resolves 61% of the entire dataset.

A depth-3 tree on three columns scores 95.00%. It uses permission_match, human_approval_required, and action_risk_score — no categorical features at all — and the rule it recovers is short enough to read aloud.

Against those baselines the results reframe completely. Logistic Regression's 97.05% is a two-point improvement over a three-line rule. XGBoost's 99.55%, the headline number, is 4.5 points. And the entire feature engineering effort — 60 one-hot columns encoding agent roles, user roles, tools, and resource types — contributes almost nothing, exactly as the importance chart already showed.

The label in this dataset is generated by a rule. The models are not learning to make security decisions; they are reverse-engineering a rubric that was applied to produce the labels in the first place. Once you know that, 99.55% stops being an achievement and becomes a measurement of how faithfully the rule was recovered.

This is not a criticism of the dataset — it is synthetic and a designed rubric is a perfectly reasonable way to build one. It is a criticism of reporting the accuracy without the baseline. A number is only meaningful next to the cheapest thing that beats it.

Problems We Hit, Including Our Own

A clean write-up hides the process. Here is what actually went wrong.

A hypothesis that turned out to be wrong. Going in, human_approval_required looked like obvious target leakage — a column that simply announces the Needs_Human_Approval label. Checking it directly killed that theory: 784 Blocked rows also carry the flag as 1, and only 141 of the 239 Needs_Human_Approval rows have it set. It is a legitimate request-time policy flag, not a post-decision echo. It does sit inside the generating rule, which is a subtler issue, but the leakage claim as originally stated was wrong. It is recorded here rather than quietly dropped, because the cost of a plausible-sounding hypothesis that nobody tests is exactly the failure this paper is about.

A convergence warning that was reported but not fixed. Logistic Regression emitted a convergence warning at max_iter=1000. It was correctly disclosed, and then nothing was done about it. The cause is straightforward — action_risk_score and data_exfiltration_risk run 0–100 while the one-hot columns are 0/1, and no scaling was applied before a multinomial solver. That untreated scale mismatch is a plausible part of why Logistic Regression trails on the minority class specifically (F1 0.8687 versus XGBoost's 0.9792). A StandardScaler in the pipeline is the fix, and it was not applied.

Charts could not be rendered inline. The assistant used for the analysis produces Plotly HTML and PNG artifacts that must be downloaded and opened separately. It cannot display a chart while explaining it, which makes iterative visual review — "that axis is unreadable, redo it" — considerably slower than it should be.

The one that matters: 99.55% went unchallenged. The analysis reported the headline accuracy, ranked the models, produced a confusion matrix, and concluded that the model had "learned a clear access-control decision hierarchy." Every individual number in that account is correct — all of them were independently verified against the raw CSV and all of them held. What was missing was the obvious next question: compared to what? No trivial baseline was proposed, and near-perfect accuracy on a security task was not treated as suspicious.

That is the pattern worth naming. The failure mode was not inaccuracy — nothing was fabricated, and every reported figure survived checking. The failure mode was incuriosity: producing a correct result and stopping, rather than treating a surprisingly good number as a thread to pull. The 84% one-column baseline took under a minute to run. Nobody ran it, because nothing in the output suggested it was needed.

Limitations

The dataset is synthetic and rule-generated. Nothing here transfers directly to production traffic. Real access logs contain contested decisions, human overrides, and policies that drift; this one contains a rubric applied consistently 2,200 times.

Accuracy is the wrong metric for this problem and was used anyway. With a 65/24/11 split, always predicting Blocked scores 65% while being useless. The per-class F1 scores carry the real information, and the class that matters operationally — Needs_Human_Approval, the human-in-the-loop route — is the weakest in all three models.

The costs are not symmetric and were never modelled. Wrongly blocking a legitimate request annoys someone. Wrongly allowing a malicious one is an incident. Every model here was trained as though those errors weigh the same. Class weighting or an explicit cost matrix is the minimum correction.

No calibration. For a system that routes borderline cases to humans, the probability estimate is more important than the argmax. None of the three models had their probabilities calibrated, so the confidence scores that would drive that routing are untrustworthy.

The genuinely hard subset was never isolated. The interesting problem is the 28–80 risk-score overlap band where the three classes actually collide. Everything outside it is resolved by the rule. Evaluating on the full test set lets the deterministic majority carry the score and hides performance on the cases that matter.

What Would Actually Improve This

Evaluate on the ambiguity band alone, and report per-class F1 rather than accuracy. Add class weights and a cost matrix reflecting the real asymmetry between a false allow and a false block. Calibrate the probabilities so human-review routing has a meaningful threshold behind it. Scale the numeric features before the linear model. And publish the trivial baseline next to every headline metric, permanently — if a one-column tree gets within 15 points of a gradient-boosted ensemble, the reader deserves to know that before they are told the ensemble is excellent.

Conclusion

The modelling exercise succeeded on its own terms. XGBoost reached 99.55% test accuracy with two errors across 440 predictions, the preprocessing was sound, stratification held to within 0.05 percentage points, and every reported figure survived independent verification against the raw data.

The audit is what made it worth publishing. A single binary column reaches 84%, three columns reach 95%, and the labels turn out to be generated by a rule that fits in five lines. The models were recovering a rubric, not learning security judgement — and none of that was visible from the accuracy scores, the classification reports, or the confusion matrix. It became visible the moment someone asked what the cheapest possible alternative would score.

That question generalises well beyond this dataset. EchoLeak passed its permission check. The GitHub MCP agent had valid access to both repositories. Neither system was broken in the way its own metrics could see. In agent security specifically, the dangerous failure is rarely the one that shows up as an error — it is the one that looks like a pass.

Two things to take away. First, on the systems: authorisation and trustworthiness are different questions, and checking only the first is how zero-click exfiltration gets through a correctly configured product. Second, on the analysis: a headline metric with no baseline beside it is not a result, it is a claim. The baseline here cost less than a minute to run and changed the entire conclusion.