CyberLens

When Search Algorithms Go Looking for Vulnerabilities

Week 3 had no class meeting because of Labor Day, so the material was self-directed: Chapters 3 and 4 of AI Basics, plus an assignment and a lab on search and optimization algorithms. I worked through breadth-first search, depth-first search, A*, gradient descent, genetic algorithms, and Monte Carlo Tree Search. Nearly every textbook example involves mazes, maps, or board games. I wanted a source applying them to a security problem.

I selected “Reinforcement Learning for Automated Cybersecurity Penetration Testing,” a 2025 preprint by Daniel López-Montero and colleagues at GMV. The authors train a reinforcement learning agent to run penetration tests against web applications, deciding which tool to use next. The action space is large: 134 possible actions per URL, multiplied by every URL the agent discovers, spanning crawling, SQL injection, brute force, and cross-site scripting. The reward function grants positive reward for discovering vulnerabilities weighted by severity, and subtracts a cost for the time and computation each action consumes. The agent trains on simulated websites generated with a modified Barabási-Albert graph algorithm, then is tested against real vulnerable applications such as DVWA.

Reading this right after the algorithm assignment, the paper looked like a search problem wearing security clothing. The website topology is a graph, and the core difficulty is the one that makes uninformed search impractical at scale: the authors note that a site with 50 URLs produces an action space above 6,700 options. They handle it the way A* handles an unmanageable search tree, by adding structure that narrows what has to be explored. Here that structure comes from geometric deep learning, which builds priors that reduce the search space, leaving a model of only 69,304 parameters.

Two things stood out. First, the reward table is itself a security decision. The authors assign their highest values to stacked-query SQL injection and password brute force, so the agent’s priorities encode a human judgment about what matters. Change the table and you change what the tester looks for. Second, the authors state plainly that the system does not yet reach professional human-level performance and remains constrained by the actions available to it. That restraint is worth noticing in a field where tooling is often oversold.

The algorithms were the smaller lesson. The larger one came from the lab, where I asked two language models to explain the same six algorithms and compared them. One described AlphaGo as relying on a learned network alone to evaluate positions, when the 2016 system combined neural networks with Monte Carlo rollout simulations and AlphaGo Zero in 2017 removed them (Silver et al., 2016, 2017). I caught it while checking claims against the original Nature papers. The most detailed answer and the incorrect one came from the same place, with the same confidence.

Search and optimization stop feeling abstract once an agent uses them to enumerate injection payloads against a live target. What I want to carry forward is the verification habit. An explanation that reads well is not the same as one that is correct.

---

References

López-Montero, D., Morales-Martínez, A., Álvarez-Aldana, J. L., Gil-López, M., & Auñón-García, J. M. (2025). Reinforcement learning for automated cybersecurity penetration testing. arXiv. https://arxiv.org/abs/2507.02969

Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., van den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., Dieleman, S., Grewe, D., Nham, J., Kalchbrenner, N., Sutskever, I., Lillicrap, T., Leach, M., Kavukcuoglu, K., Graepel, T., & Hassabis, D. (2016). Mastering the game of Go with deep neural networks and tree search. Nature, 529(7587), 484-489. https://doi.org/10.1038/nature16961

Silver, D., Schrittwieser, J., Simonyan, K., Antonoglou, I., Huang, A., Guez, A., Hubert, T., Baker, L., Lai, M., Bolton, A., Chen, Y., Lillicrap, T., Hui, F., Sifre, L., van den Driessche, G., Graepel, T., & Hassabis, D. (2017). Mastering the game of Go without human knowledge. Nature, 550(7676), 354-359. https://doi.org/10.1038/nature24270