TL;DR: An AI pentest is fast, broad and repeatable: it requests every route it can find, reads every response and flags the pattern-shaped problems, such as exposed files, missing authentication, verbose errors and known versions. A manual pentest is slower and costs a tester's time, but it brings what the responses do not contain: what the business intended, which data matters, the agreed scope, and attack chains that run across a whole engagement. They find different things, so the useful question is which findings you need, not which one wins.
This is for learners who want to know where automation stops and judgement starts, and for site owners deciding what to run on their own app. We took one practice target, CyberGym, ran the breadth pass an AI agent runs, and then read the same 28 responses the way a tester would. You will see which findings each approach produced, and you can repeat both passes yourself. Test only targets you own or have written permission to test; CyberGym exists for exactly this.
What is the difference between an AI pentest and a manual pentest?
An AI pentest is an agent that maps your app, sends requests, reads the answers and decides what to try next, in a loop, at machine speed. A manual pentest is a person doing the same work with tools, inside a scope agreed in writing, and signing their name under the conclusions. The loop itself, observe, decide, act, remember, is explained in how agentic AI penetration testing works.
The difference is not that one is smart and the other is not. It is where the information comes from. An agent knows what the app tells it: routes, responses, page text, error messages. A tester also knows what the owner told them: which data is confidential, which flows carry money, what is meant to be public, what is out of bounds. That second kind of knowledge is agreed before a test starts, which is why a pentest scope is a document and not a URL.
Neither is a vulnerability scan in the narrow sense. A scanner runs a fixed list of checks; an agent chooses its next request from what it just saw. The OWASP Web Security Testing Guide describes automated assessment tools as "useful as a first pass to find low-hanging fruit" but "ineffective at in-depth assessment". An agent goes further than the scanners that sentence was written about, because it reads what comes back. The part about depth still applies whenever the missing piece is the business's intent, not the app's behaviour.
What does each one find?
Each approach is strongest at a different kind of finding. This is the pattern we see, scoped to web applications:
| Kind of finding | AI pentest | Manual pentest |
|---|---|---|
| Exposed files, directory listings, debug and metrics endpoints | strong: it requests every path it can think of, every run | good, limited by the hours booked |
| Routes that answer without authentication | strong: it tests every route the same way | good, but repetitive work for a person |
| Version disclosure and known vulnerable components | strong: matches versions against advisories | good |
| Known injection classes (SQL, NoSQL, XSS) in mapped inputs | good: many inputs, many payload families | good on the inputs the tester picks |
| Whether a public response is meant to be public | partial: it can infer from names and page text | strong: the tester can ask the owner |
| Business-logic flaws: prices, quantities, refunds, approval steps | weak: it has to guess the rules | strong: the tester is told the rules |
| Chains across many steps and sessions | limited by what it keeps in memory | strong across the engagement |
| Severity for this business | a generic rating | a rating argued from the business's risk |
The table matches what OWASP's business-logic guidance says about the bottom half: these flaws "cannot be detected by a vulnerability scanner", and finding them relies on the tester's knowledge of "the complete business process and its rules". An agent is better at inferring rules than a scanner. It still infers them.
What happened when we tested one CyberGym target both ways?
On CyberGym, the breadth pass took seconds and returned 28 answers, and the triage of 19 of them needed facts that were not in the responses. Here are both passes.
Pass 1: the breadth pass
This is the mechanical half of an AI pentest: build a list of routes, request each one, record what comes back. An agent gets its route list by crawling the app and reading its JavaScript; on your own app, AppMap does that step and records each endpoint with its auth requirement. We wrote the list by hand from CyberGym's public pages so you can repeat the run with nothing but curl:
TARGET=https://cybergym.hackergpt.app # or your own site
while read -r path; do
printf '%-40s %s\n' "$path" \
"$(curl -s -o /dev/null -w '%{http_code} %{size_download}B' "$TARGET$path")"
done < paths.txt
paths.txt holds one path per line. Our run, without any login, on 12 November 2026:
/robots.txt 200 28B
/.well-known/security.txt 200 426B
/ftp 200 15751B
/api-docs 301 158B
/metrics 200 24915B
/encryptionkeys 200 7951B
/support/logs 200 9149B
/api/Products 200 12355B
/api/Feedbacks 200 1734B
/api/Challenges 200 57647B
/api/Users 401 964B
/api/Quantitys 200 6258B
/api/Deliverys 200 266B
/api/SecurityQuestions 200 1934B
/api/Complaints 401 964B
/api/Cards 401 31B
/api/Addresss 401 31B
/api/BasketItems 401 964B
/rest/memories 200 6183B
/rest/admin/application-version 200 20B
/rest/admin/application-configuration 200 19784B
/rest/languages 200 4858B
/rest/products/search?q= 200 12787B
/rest/user/whoami 200 11B
/rest/basket/1 401 964B
/rest/order-history 500 3326B
/rest/deluxe-membership 400 72B
/snippets 200 75294B
The breadth pass in one picture: most anonymous requests got an answer, and every 200 is a question only context can close.
Some of this is a finding on sight, and any automated pass worth running reports it. /ftp, /encryptionkeys and /support/logs return directory listings to anyone. /rest/admin/application-version returns {"version":"19.1.1"}, which tells an attacker which advisories to read. /rest/order-history answers an anonymous request with a 500 and an HTML error page that names the server's internal address (::ffff:127.0.0.1). The six 401s are the app refusing what it should refuse, and a good report records those too.
This is where the AI pentest earns its place. The sweep is boring, exact work that a person does slowly and an agent does the same way every time, on every deploy.
Pass 2: reading the same answers with context
Nineteen routes answered 200 to a stranger, and the status code does not say which of them matter. We took four and asked the question a tester asks: is this meant to be public, and what is in it?
Four identical status codes, four different verdicts. Field names only; no values are shown.
/rest/admin/application-configuration looks serious and is not. A pattern match sees admin in the path and no login. But the app's own frontend requests it on every page; CyberGym's main.js contains host=this.hostServer+"/rest/admin" and this.http.get(this.host+"/application-configuration"). Its top-level keys are UI and challenge settings:
curl -s https://cybergym.hackergpt.app/rest/admin/application-configuration | jq -c '.config | keys'
# ["application","challenges","ctf","hackingInstructor","memories","products","server"]
The one value that looks like a credential is a Google OAuth client ID, which OAuth treats as public: the browser has to send it. Verdict: low. Move it off the admin path and trim what the frontend does not use.
/rest/memories looks harmless and is the worst of the four. It is the shop's photo wall, public by design, and its status line is identical to /api/Products. The body is not:
curl -s https://cybergym.hackergpt.app/rest/memories | jq -c '.data[0].User | keys'
# ["createdAt","deletedAt","deluxeToken","email","id","isActive","lastLoginIp","password","profileImage","role","totpSecret","updatedAt","username"]
Every photo carries its poster's full user row. To be fair to the agent, this is a finding an AI pentest that reads bodies flags as readily as a person: a field named password in an anonymous response is a pattern. Both approaches agree here, and the full write-up of this finding is the worked example in our pentest report example.
/metrics is a question for the owner. It is a Prometheus endpoint open to the internet. Most of it is runtime data, but some lines are business counters:
curl -s https://cybergym.hackergpt.app/metrics | grep -E '^cybergym_(users_registered_total|orders_placed_total)'
# cybergym_orders_placed_total{app="cybergym"} 3
# cybergym_users_registered_total{app="cybergym"} 25
On a practice target those numbers mean nothing. On a startup's shop, live user and order counts can be exactly what a competitor wants, or something the founder posts publicly every month. Nothing in the response says which. An agent can flag the endpoint; the severity is a business decision, and a manual tester gets it by asking.
/api/SecurityQuestions looks suspicious and is fine. Account-recovery data without a login sounds bad. The response is only the list of questions ("Your eldest siblings middle name?"), no answers, and the sign-up form must load it before anyone has an account; the frontend service that fetches it is in main.js. Not a finding on its own, though it is worth knowing that the app uses security questions at all.
What neither pass saw
Our 28 anonymous GETs found nothing about business logic, because business logic lives behind a login and inside the rules of a purchase. On the same target, a basket accepted a negative quantity, a rule nobody wrote down, which we showed in business logic vulnerabilities: examples from a web shop. An AI agent that logs in and explores the basket can find that too, if it guesses that quantities should be positive. A tester who was told "customers can only buy, never be paid" knows to try it. That gap is the honest difference between the two, and it does not close by sending more requests.
When should I use an AI pentest, and when a manual one?
Use an AI pentest often, and a manual pentest when what you need to know depends on the business. In practice:
- On every deploy, or at least every week: the AI pass. New routes, forgotten files, a debug flag left on, a dependency with a new advisory. These change with the code, and a pass that is cheap to run catches them when they appear rather than at the next yearly test.
- Before you launch payments, accounts or anything regulated: a manual test, scoped by someone who can explain the business rules. That is where logic flaws cost money.
- After a large change to how the app works, such as a new role model, a new checkout or a new integration: a manual test of that change.
- Hand the AI results to the tester. The UK's National Cyber Security Centre suggests that a current vulnerability assessment "should be shared with the testers", and frames penetration testing as "a method for gaining assurance in your organisation's vulnerability assessment and management processes, not as a primary method for identifying vulnerabilities" (NCSC, Penetration testing). The breadth pass clears the floor so the tester's hours go to the parts only a person can do.
Neither approach leaves an app without flaws. Each gives you findings; the next step is to verify, fix and retest them.
How do I check what an AI pentest reports?
Treat every AI finding as a lead until you have reproduced it yourself. Re-send the request, read the actual response, and ask the context question: is this meant to be public, and does the data matter to this business? Language models can report things that are not there, a risk covered in the confident-liar section of our agentic pentesting article. A finding with a request, a response and a verdict you would defend is ready for the report.
Key takeaways
- Run the AI breadth pass on every change. It is good at exposed files, unauthenticated routes, versions and known vulnerability classes, and cheap enough to repeat.
- Read every 200 an anonymous request gets as a question, not an answer: is it meant to be public, and what is in the body?
- Book a manual test where the risk is in the rules: payments, roles, refunds, approvals. Tell the tester the rules.
- Share your automated results with the tester so their time goes to logic and chains, not to work an agent already did.
- Reproduce every finding before you act on it, whichever approach produced it.
FAQ
Is an AI pentest as good as a manual pentest?
They are not the same test, so "as good" depends on the finding. An AI pentest is broader and more repeatable for pattern-shaped problems such as exposed files and unauthenticated routes. A manual pentest is stronger wherever the answer depends on business intent, such as logic flaws, data sensitivity and severity for that business.
Can an AI replace a penetration tester?
Not for the parts of a pentest that depend on information outside the app: the agreed scope, the business rules and the judgement about what a finding means to the owner. It can take over much of the repetitive mapping and checking, which leaves the tester more time for those parts. Many teams run both.
What does a manual pentest find that an AI pentest misses?
Most often business-logic flaws, where the app behaves exactly as coded but the code breaks a rule nobody wrote down, like a negative quantity in a basket. Also chains of small issues across a long engagement, and severity judgements that need to know what the business considers confidential.
Is an automated vulnerability scan the same as an AI pentest?
No. A scanner runs a fixed list of checks and reports matches. An AI pentest agent chooses its next request from what it just saw and reads the responses, so it can follow leads a fixed list would not. Both are automated, and neither replaces a person deciding what matters.
Can I run an AI pentest on any website?
No. Test only sites you own or have written permission to test, and agree the scope first. For practice, use a deliberately vulnerable target such as CyberGym.
Sources
- OWASP Web Security Testing Guide v4.2: Introduction
- OWASP Web Security Testing Guide v4.2: Introduction to Business Logic
- NCSC: Penetration testing (published 8 August 2017, reviewed 10 January 2022)
- Our own requests to CyberGym (Juice Shop 19.1.1), 12 November 2026, and its public
main.js
Try the breadth pass on your own site with HackerGPT, or practise both passes on CyberGym.