Novelty Proposes: What Should an AI Pentester Try Next? 

By Nadav Barkai

An agentic pentester can generate plenty of plausible attack ideas while repeatedly covering the same ground. New payloads and rephrased hypotheses may still test the same underlying assumption. The activity continues, but the agent learns little more about the target.

At Alt, we believe the edge comes from understanding an application well enough to recognize combinations of weaknesses that individual checks may miss. We're exploring whether novelty can help direct that process by prioritizing hypotheses that test meaningfully different permission boundaries, workflow states, or combinations of weaknesses. Avoiding redundant attempts is the practical starting point. The broader question is whether this leads to more distinct, verified findings.

The finding was in the composition

The target was a ticket-management application with a role-scoped review workflow. Lower-privilege analysts opened cases and attached supporting documents. Higher-privilege reviewers read those documents and signed off. Upload and export features moved case material between them. Enumerating the application's upload and export routes reveals what functionality exists. Understanding how those routes interact requires something more: a model of the objects they operate on, who owns them, which roles can access them, and how content moves between users.

In this application, that distinction mattered. The vulnerability was not confined to a single endpoint. It emerged from how an unauthorized document write interacted with a legitimate review-and-export workflow.

The confirmed chain began at the upload endpoint. It authenticated the caller’s session but did not enforce object-level authorization on the target case. An analyst could therefore attach a document to a reviewer-owned case without the required authorization.

That document was a CSV containing a DDE (Dynamic Data Exchange)-capable formula. The application accepted the file, attributed it to the analyst, stored it, and later returned the same bytes through its case-export path as text/csv. When the reviewer opened the export, the spreadsheet interpreted the cell as a formula. Further client-side impact depended on the spreadsheet configuration.

Two familiar weakness classes were involved: missing authorization on the case write and CSV/formula injection. The finding lay in their composition. The export workflow preserved attacker-controlled content and carried it across the role boundary into a higher-trust context. The application did not execute the formula itself; the reviewer’s spreadsheet was the client-side sink.

Figure 1. An authorization gap lets a lower-privilege analyst place content in a reviewer-owned case. The export preserves the uploaded bytes and carries them to the reviewer’s spreadsheet, the client-side sink for formula injection. Further impact depends on spreadsheet settings

A rule-based test could identify either weakness independently. Understanding the attack path required connecting the unauthorized object write, the byte-preserving export, and the higher-privilege user who would open the resulting artifact.

The exploration archive already contained the obvious probes: formula-looking CSVs, direct object writes, and ownership checks on the upload route. The novelty was not a new primitive, but a previously unexplored connection between familiar ones in this target’s workflow.

A chain becomes testable when the target is modeled

To derive an exploit hypothesis like this, an agent needs a model of the application’s authorization and workflow semantics: what each route does, which roles can invoke it, which objects it reads or mutates, and how those objects change state.

That model can be represented as an evidence graph. Routes connect to actions; actions require roles and operate on objects; objects have owners and states. The graph also captures how content moves through the application, such as bytes supplied by one role passing through storage and export before reaching another role’s client-side sink.

From that model, the agent can derive candidate security invariants: rules that should hold across requests and state transitions. In the ticketing example, a document write should require authorization for the target case, and analyst-supplied content should not cross into a higher-trust workflow as if it were trusted application output.

An exploit hypothesis then proposes a sequence that could violate one of those invariants, with explicit preconditions, expected state transitions, and observable success criteria. Here, the hypothesis was straightforward:

• Preconditions: an authenticated analyst can reference a reviewer-owned case, and the reviewer can later open its export.
• Sequence: the analyst uploads the CSV, the application stores it, and the same bytes reach the reviewer through the export path.
• Success criteria: the unauthorized write succeeds, the content survives export unchanged, and the spreadsheet interprets the supplied cell as a formula.

Testing determines whether that path is actually reachable and whether the assumed boundary is truly violated.

The evidence graph gives the hypothesis a concrete basis: which relationships make the path plausible, which assumptions remain unverified, and what observations would confirm or refute it.

Why more payloads don’t mean more coverage

Some weaknesses expose recognizable patterns that scanners can test efficiently: reflected XSS, missing security headers, or injection reachable through standard payload mutation. But as those checks are exhausted, the harder findings become semantic rather than syntactic. They depend on how the target’s authorization, workflows, and state transitions actually work.

A request can be valid HTTP, carry an authenticated session, and use an expected parameter while still violating a security boundary. BOLA and IDOR depend on which objects a user is allowed to access or manipulate. Broken Function Level Authorization (BFLA) depends on which functions they are permitted to invoke. Business-logic flaws and multi-step chains may depend on ownership, tenant boundaries, request order, or object state across several requests.

More signatures and payload variants can improve coverage of recognizable weakness patterns. But they do not, on their own, reveal the application-specific conditions that make a chain possible. In the ticketing example, varying the formula payload would not show which case the analyst could write to or how the content eventually reached the reviewer.

Coverage still matters. But once the obvious checks are exhausted, the next useful hypothesis increasingly comes from understanding what the target’s roles, relationships, and workflows reveal that previous tests did not.

Novelty depends on what you compare against

Novelty search offers a framework for directing exploration. In its classical form, it favors candidates that differ from previously encountered behaviors, rather than those that simply make progress toward a predefined objective. This does not require a claim about reasoning beyond the model’s training data; novelty here is a property of the search process, relative to what has already been explored

Applied to pentesting, that raises a harder question: what counts as meaningfully different?

Different wording may describe the same test. But changing a role, object relationship, request order, or state transition can alter the preconditions enough to expose a boundary earlier tests never exercised.

Representation therefore matters. Semantic similarity is useful, but comparison also has to account for actions, authorization context, state transitions, and expected effects. Otherwise, a distinct test can look redundant simply because the descriptions resemble each other.

Figure 2. Similar hypotheses cluster around previously explored approaches. A distinct candidate proposes a different behavior, but still requires verification. Distances and contour bands are illustrative, not measured novelty scores or confidence levels.

Novelty is always relative to a reference set:

• Within a run: not a duplicate of something already attempted.
• Within a target: adds something beyond issues already known on that system.
• At the root-cause or chain level: introduces a distinct weakness mechanism, precondition set, or path to impact.
• Relative to public knowledge: goes beyond relevant disclosures, research, and known techniques.

For exploration, the first three matter most. They help determine whether another test adds meaningful coverage rather than more activity.

Novelty is also independent of severity. A finding can be novel and low-impact, or familiar and critical. Novelty tells us what is different. Verification and impact determine whether it matters.

From avoiding repetition to directing exploration

Duplicate suppression and novelty-guided exploration solve different problems. Duplicate suppression filters retries and hypotheses that add nothing beyond what is already known. Novelty-guided exploration would go further by influencing where the agent spends its next unit of testing budget.

The research direction is to compare candidates against what exploration has already covered: prior request sequences, attack-graph edges, validated root causes, and similar hypotheses. The goal is to estimate where another test is most likely to add meaningful coverage while preserving differences that matter to exploitation, such as authorization context, preconditions, and reachable state transitions.

The open question is whether that selection policy produces more distinct, verified findings within the same testing budget than alternative strategies. The ticketing case shows the value of target-specific composition. It does not, by itself, prove that novelty-guided selection caused the discovery.

Useful hypotheses can still come from recombining known weakness mechanisms with observed capabilities and inferred security invariants. A strong human pentester works similarly: build a model of roles, ownership, state transitions, and trust boundaries, then test where enforcement may fail. The engineering goal is to make that process inspectable: what evidence shaped the model, why a candidate was selected, and what execution established.

Novelty earns a test, not a finding

In the ticketing case, each step could be checked: unauthorized document placement, byte-preserving export, and formula interpretation at the client-side sink.

The same run also produced noise. Plausible hypotheses can depend on incorrect assumptions, unreachable states, or behavior that does not actually violate a security boundary. Verification has to reject those cases, reproduce successful paths, and determine whether the result is truly distinct.

At Alt, this is the loop we are pursuing: build an evidence-backed target model, derive exploit hypotheses, use exploration history to guide subsequent tests, and verify the resulting paths.

Novelty proposes. Verification disposes.

Part 2 will examine that second half: how candidate hypotheses become reproduced, distinct findings, and how the target model itself gets tested along the way.