I just returned from L'Aquila, where I presented our paper The Ethics of Autonomous AI Agents for Offensive Security at FAIEMA 2026, written together with Jürgen Cito and Jasmin Wachter. The previous post ended with uncomfortable questions about the attacker-defender balance and the workforce. This one takes a step back and asks questions about the ethics of building offensive agents.

Does the Field Care?

Before writing about ethics, we checked whether the field talks about it. In a companion study we reviewed the ethics statements of 54 LLM-driven offensive-security prototypes. The good news: most papers acknowledge dual-use (that the same tool can be used by a benign user for increasing their security, as well as by a malicious user that uses the same tool to attack non-consenting third parties). Roughly half of the prototypes released their code, and the field is split on whether that is responsible or not. The most common argument for releasing prototypes is "we help defenders prepare", followed by "our prototype is not capable enough to cause real harm". The second argument has aged badly.

Three Things That Changed

Traditional tools like nmap or metasploit are passive instruments: a trained human picks the tool, and the tool does one predictable thing. Agents break this in three independent ways.

Their actions are indeterminate. You give the agent a goal and it decides how to get there. You pass the agency to the agent (d'oh) and the agent independently selects actions (attacks) to execute.

Their impact is indeterminate. Building an agent is closer to teaching methodology to students than to writing an exploit. With the latter, you write an exploit for a software, very often for a single specific version. With that, you can contact the author of the vulnerable software and warn them before releasing the vulnerability (responsible disclosure). You can't do this when you release an autonomous offensive security agent. Also related to unpredictable impact: Capability evaluations at both OpenAI and Anthropic have already produced real incidents where agents escaped their sandboxes and compromising unconsenting third-parties.

Their users are indeterminate. Usage-based billing plus a prompt now replaces years of training, including the ethical education and professional socialization that came with it. Onramp costs are lower when using LLMs than the, sometimes steep, license costs that we had before.

Who Is Responsible and Can Prevent Accidents and Abuse?

For users, the cleanest analogy is principal-agent liability: if you delegate authority to a system, foreseeable consequences stay with you. For tool-makers it is harder. Floridi's distributed moral responsibility reaches everyone causally involved, developers included. But consider the following scenario: you ship an agent with state-of-the-art safeguards, and a year later somebody swaps in a more capable model that bypasses them. Blaming you for the new agent backdates accountability to a moment when you could not have known about this. Responsibility diffuses. It does not disappear, but it becomes hard to pin down. In our companion study, we looked into how many papers were mentioning this future-looking scenario: only 4% of papers did. Our community is thus not aware of this problem (so it cannot be state-of-the-art or established-practise).

Tool-makers have two escape hatches. Withholding code buys control at the cost of reproducibility and defender access, and LLMs reverse-engineer binaries anyway. Delegating safety to the model provider works until it doesn't: after being breached by an agent during a safety evaluation, Hugging Face had to fall back to an open-weight model for forensics because their preferred frontier model refused the defensive work. And once open-weight models are capable enough, there is no gatekeeper left to delegate to.

Who Pays?

Mostly defenders and maintainers. cURL ended its monetary bug bounty after a flood of AI slop; now the reports are good, and the maintainers are drowning in those instead. Firefox fixed roughly seven times more security bugs per release in 2026 than in 2025. Consider that those maintainers are already overworked and often not-paid. This is not sustainable and a recipe for burnout. And, as discussed in the last post, automating the junior work erodes the pipeline that produces senior experts.

What We Recommend

We give longer recommendations within the paper, but want to include a short summary here: researchers should publish a threat model for their scaffold (what it refuses, what it cannot refuse, what a modification enables) and focus on providing good logs for analysis. Model-makers should default to open weights, but keep strongly offensive models behind structured access. Defenders should minimize attack surface: removed code cannot be exploited. Policy-makers should fund OSS maintainers, build substitute training pathways, and remember Wassenaar before regulating security research.

None of this resolves the open- versus closed-weight question; we don't think it has a clean answer. The short-term balance favours attackers. In the long run, software should become more secure.