# AI pentest vs manual pentest: what each one finds

Published: 2026-11-12 · By HackerGPT Team
Canonical: https://hackergpt.org/blog/ai-pentest-vs-manual-pentest/

> We sent 28 anonymous requests to one CyberGym target and then read the answers the slow way. What the breadth pass caught, what only context could decide, and what neither saw.

**TL;DR:** An AI pentest is fast, broad and repeatable: it requests every route it can find, reads every response and flags the pattern-shaped problems, such as exposed files, missing authentication, verbose errors and known versions. A manual pentest is slower and costs a tester's time, but it brings what the responses do not contain: what the business intended, which data matters, the agreed scope, and attack chains that run across a whole engagement. They find different things, so the useful question is which findings you need, not which one wins.

This is for learners who want to know where automation stops and judgement starts, and for site owners deciding what to run on their own app. We took one practice target, [CyberGym](https://cybergym.hackergpt.app), ran the breadth pass an AI agent runs, and then read the same 28 responses the way a tester would. You will see which findings each approach produced, and you can repeat both passes yourself. Test only targets you own or have written permission to test; CyberGym exists for exactly this.

## What is the difference between an AI pentest and a manual pentest?

An AI pentest is an agent that maps your app, sends requests, reads the answers and decides what to try next, in a loop, at machine speed. A manual pentest is a person doing the same work with tools, inside a scope agreed in writing, and signing their name under the conclusions. The loop itself, observe, decide, act, remember, is explained in [how agentic AI penetration testing works](https://hackergpt.org/blog/ai-driven-penetration-testing-the-shift-from-automation-to-agentic-workflows/).

The difference is not that one is smart and the other is not. It is where the information comes from. An agent knows what the app tells it: routes, responses, page text, error messages. A tester also knows what the owner told them: which data is confidential, which flows carry money, what is meant to be public, what is out of bounds. That second kind of knowledge is agreed before a test starts, which is why a [pentest scope](https://hackergpt.org/blog/pentest-scope-template/) is a document and not a URL.

Neither is a vulnerability scan in the narrow sense. A scanner runs a fixed list of checks; an agent chooses its next request from what it just saw. The [OWASP Web Security Testing Guide](https://wstg.owasp.org/v4.2/2-Introduction/) describes automated assessment tools as "useful as a first pass to find low-hanging fruit" but "ineffective at in-depth assessment". An agent goes further than the scanners that sentence was written about, because it reads what comes back. The part about depth still applies whenever the missing piece is the business's intent, not the app's behaviour.

## What does each one find?

Each approach is strongest at a different kind of finding. This is the pattern we see, scoped to web applications:

| Kind of finding | AI pentest | Manual pentest |
| --- | --- | --- |
| Exposed files, directory listings, debug and metrics endpoints | strong: it requests every path it can think of, every run | good, limited by the hours booked |
| Routes that answer without authentication | strong: it tests every route the same way | good, but repetitive work for a person |
| Version disclosure and known vulnerable components | strong: matches versions against advisories | good |
| Known injection classes (SQL, NoSQL, XSS) in mapped inputs | good: many inputs, many payload families | good on the inputs the tester picks |
| Whether a public response is *meant* to be public | partial: it can infer from names and page text | strong: the tester can ask the owner |
| Business-logic flaws: prices, quantities, refunds, approval steps | weak: it has to guess the rules | strong: the tester is told the rules |
| Chains across many steps and sessions | limited by what it keeps in memory | strong across the engagement |
| Severity for *this* business | a generic rating | a rating argued from the business's risk |

The table matches what [OWASP's business-logic guidance](https://wstg.owasp.org/v4.2/4-Web_Application_Security_Testing/10-Business_Logic_Testing/00-Introduction_to_Business_Logic/) says about the bottom half: these flaws "cannot be detected by a vulnerability scanner", and finding them relies on the tester's knowledge of "the complete business process and its rules". An agent is better at inferring rules than a scanner. It still infers them.

## What happened when we tested one CyberGym target both ways?

On CyberGym, the breadth pass took seconds and returned 28 answers, and the triage of 19 of them needed facts that were not in the responses. Here are both passes.

### Pass 1: the breadth pass

This is the mechanical half of an AI pentest: build a list of routes, request each one, record what comes back. An agent gets its route list by crawling the app and reading its JavaScript; on your own app, [AppMap](https://hackergpt.org/features/appmap/) does that step and records each endpoint with its auth requirement. We wrote the list by hand from CyberGym's public pages so you can repeat the run with nothing but `curl`:

```bash
TARGET=https://cybergym.hackergpt.app   # or your own site
while read -r path; do
  printf '%-40s %s\n' "$path" \
    "$(curl -s -o /dev/null -w '%{http_code} %{size_download}B' "$TARGET$path")"
done < paths.txt
```

`paths.txt` holds one path per line. Our run, without any login, on 12 November 2026:

```text
/robots.txt                              200 28B
/.well-known/security.txt                200 426B
/ftp                                     200 15751B
/api-docs                                301 158B
/metrics                                 200 24915B
/encryptionkeys                          200 7951B
/support/logs                            200 9149B
/api/Products                            200 12355B
/api/Feedbacks                           200 1734B
/api/Challenges                          200 57647B
/api/Users                               401 964B
/api/Quantitys                           200 6258B
/api/Deliverys                           200 266B
/api/SecurityQuestions                   200 1934B
/api/Complaints                          401 964B
/api/Cards                               401 31B
/api/Addresss                            401 31B
/api/BasketItems                         401 964B
/rest/memories                           200 6183B
/rest/admin/application-version          200 20B
/rest/admin/application-configuration    200 19784B
/rest/languages                          200 4858B
/rest/products/search?q=                 200 12787B
/rest/user/whoami                        200 11B
/rest/basket/1                           401 964B
/rest/order-history                      500 3326B
/rest/deluxe-membership                  400 72B
/snippets                                200 75294B
```

![Bar chart: of 28 unauthenticated requests to CyberGym, 19 returned 200, 6 returned 401, one each 301, 400 and 500](https://hackergpt.org/blog/ai-pentest-vs-manual-pentest/sweep-status.webp)

*The breadth pass in one picture: most anonymous requests got an answer, and every 200 is a question only context can close.*

Some of this is a finding on sight, and any automated pass worth running reports it. `/ftp`, `/encryptionkeys` and `/support/logs` return directory listings to anyone. `/rest/admin/application-version` returns `{"version":"19.1.1"}`, which tells an attacker which advisories to read. `/rest/order-history` answers an anonymous request with a 500 and an HTML error page that names the server's internal address (`::ffff:127.0.0.1`). The six 401s are the app refusing what it should refuse, and a good report records those too.

This is where the AI pentest earns its place. The sweep is boring, exact work that a person does slowly and an agent does the same way every time, on every deploy.

### Pass 2: reading the same answers with context

Nineteen routes answered 200 to a stranger, and the status code does not say which of them matter. We took four and asked the question a tester asks: *is this meant to be public, and what is in it?*

![Four CyberGym endpoints that all returned 200: what a pattern match says, what context adds, and the verdict](https://hackergpt.org/blog/ai-pentest-vs-manual-pentest/triage.webp)

*Four identical status codes, four different verdicts. Field names only; no values are shown.*

**`/rest/admin/application-configuration` looks serious and is not.** A pattern match sees `admin` in the path and no login. But the app's own frontend requests it on every page; CyberGym's `main.js` contains `host=this.hostServer+"/rest/admin"` and `this.http.get(this.host+"/application-configuration")`. Its top-level keys are UI and challenge settings:

```bash
curl -s https://cybergym.hackergpt.app/rest/admin/application-configuration | jq -c '.config | keys'
# ["application","challenges","ctf","hackingInstructor","memories","products","server"]
```

The one value that looks like a credential is a Google OAuth client ID, which OAuth treats as public: the browser has to send it. Verdict: low. Move it off the `admin` path and trim what the frontend does not use.

**`/rest/memories` looks harmless and is the worst of the four.** It is the shop's photo wall, public by design, and its status line is identical to `/api/Products`. The body is not:

```bash
curl -s https://cybergym.hackergpt.app/rest/memories | jq -c '.data[0].User | keys'
# ["createdAt","deletedAt","deluxeToken","email","id","isActive","lastLoginIp","password","profileImage","role","totpSecret","updatedAt","username"]
```

Every photo carries its poster's full user row. To be fair to the agent, this is a finding an AI pentest that reads bodies flags as readily as a person: a field named `password` in an anonymous response is a pattern. Both approaches agree here, and the full write-up of this finding is the worked example in [our pentest report example](https://hackergpt.org/blog/pentest-report-example/).

**`/metrics` is a question for the owner.** It is a Prometheus endpoint open to the internet. Most of it is runtime data, but some lines are business counters:

```bash
curl -s https://cybergym.hackergpt.app/metrics | grep -E '^cybergym_(users_registered_total|orders_placed_total)'
# cybergym_orders_placed_total{app="cybergym"} 3
# cybergym_users_registered_total{app="cybergym"} 25
```

On a practice target those numbers mean nothing. On a startup's shop, live user and order counts can be exactly what a competitor wants, or something the founder posts publicly every month. Nothing in the response says which. An agent can flag the endpoint; the severity is a business decision, and a manual tester gets it by asking.

**`/api/SecurityQuestions` looks suspicious and is fine.** Account-recovery data without a login sounds bad. The response is only the list of questions ("Your eldest siblings middle name?"), no answers, and the sign-up form must load it before anyone has an account; the frontend service that fetches it is in `main.js`. Not a finding on its own, though it is worth knowing that the app uses security questions at all.

### What neither pass saw

Our 28 anonymous GETs found nothing about business logic, because business logic lives behind a login and inside the rules of a purchase. On the same target, a basket accepted a negative quantity, a rule nobody wrote down, which we showed in [business logic vulnerabilities: examples from a web shop](https://hackergpt.org/blog/business-logic-vulnerabilities-examples/). An AI agent that logs in and explores the basket can find that too, if it guesses that quantities should be positive. A tester who was told "customers can only buy, never be paid" knows to try it. That gap is the honest difference between the two, and it does not close by sending more requests.

## When should I use an AI pentest, and when a manual one?

Use an AI pentest often, and a manual pentest when what you need to know depends on the business. In practice:

- **On every deploy, or at least every week:** the AI pass. New routes, forgotten files, a debug flag left on, a dependency with a new advisory. These change with the code, and a pass that is cheap to run catches them when they appear rather than at the next yearly test.
- **Before you launch payments, accounts or anything regulated:** a manual test, scoped by someone who can explain the business rules. That is where logic flaws cost money.
- **After a large change to how the app works**, such as a new role model, a new checkout or a new integration: a manual test of that change.
- **Hand the AI results to the tester.** The UK's National Cyber Security Centre suggests that a current vulnerability assessment "should be shared with the testers", and frames penetration testing as "a method for gaining assurance in your organisation's vulnerability assessment and management processes, not as a primary method for identifying vulnerabilities" ([NCSC, Penetration testing](https://www.ncsc.gov.uk/guidance/penetration-testing)). The breadth pass clears the floor so the tester's hours go to the parts only a person can do.

Neither approach leaves an app without flaws. Each gives you findings; the next step is to verify, fix and retest them.

## How do I check what an AI pentest reports?

Treat every AI finding as a lead until you have reproduced it yourself. Re-send the request, read the actual response, and ask the context question: is this meant to be public, and does the data matter to this business? Language models can report things that are not there, a risk covered in the [confident-liar section of our agentic pentesting article](https://hackergpt.org/blog/ai-driven-penetration-testing-the-shift-from-automation-to-agentic-workflows/). A finding with a request, a response and a verdict you would defend is ready for the report.

## Key takeaways

- Run the AI breadth pass on every change. It is good at exposed files, unauthenticated routes, versions and known vulnerability classes, and cheap enough to repeat.
- Read every 200 an anonymous request gets as a question, not an answer: is it meant to be public, and what is in the body?
- Book a manual test where the risk is in the rules: payments, roles, refunds, approvals. Tell the tester the rules.
- Share your automated results with the tester so their time goes to logic and chains, not to work an agent already did.
- Reproduce every finding before you act on it, whichever approach produced it.

## FAQ

### Is an AI pentest as good as a manual pentest?

They are not the same test, so "as good" depends on the finding. An AI pentest is broader and more repeatable for pattern-shaped problems such as exposed files and unauthenticated routes. A manual pentest is stronger wherever the answer depends on business intent, such as logic flaws, data sensitivity and severity for that business.

### Can an AI replace a penetration tester?

Not for the parts of a pentest that depend on information outside the app: the agreed scope, the business rules and the judgement about what a finding means to the owner. It can take over much of the repetitive mapping and checking, which leaves the tester more time for those parts. Many teams run both.

### What does a manual pentest find that an AI pentest misses?

Most often business-logic flaws, where the app behaves exactly as coded but the code breaks a rule nobody wrote down, like a negative quantity in a basket. Also chains of small issues across a long engagement, and severity judgements that need to know what the business considers confidential.

### Is an automated vulnerability scan the same as an AI pentest?

No. A scanner runs a fixed list of checks and reports matches. An AI pentest agent chooses its next request from what it just saw and reads the responses, so it can follow leads a fixed list would not. Both are automated, and neither replaces a person deciding what matters.

### Can I run an AI pentest on any website?

No. Test only sites you own or have written permission to test, and agree the scope first. For practice, use a deliberately vulnerable target such as CyberGym.

## Sources

- [OWASP Web Security Testing Guide v4.2: Introduction](https://wstg.owasp.org/v4.2/2-Introduction/)
- [OWASP Web Security Testing Guide v4.2: Introduction to Business Logic](https://wstg.owasp.org/v4.2/4-Web_Application_Security_Testing/10-Business_Logic_Testing/00-Introduction_to_Business_Logic/)
- [NCSC: Penetration testing (published 8 August 2017, reviewed 10 January 2022)](https://www.ncsc.gov.uk/guidance/penetration-testing)
- Our own requests to [CyberGym](https://cybergym.hackergpt.app) (Juice Shop 19.1.1), 12 November 2026, and its public `main.js`

Try the breadth pass on your own site with [HackerGPT](https://hackergpt.app), or practise both passes on [CyberGym](https://cybergym.hackergpt.app).