Peer Review, and Attacks That Didn't Land

This week was prototyping peer review, which I found more interesting than I expected. Building a prototype is one thing, but having other people look at it and tell you what they see is a different kind of information. You know what you meant it to do, so you read it that way automatically. Someone seeing it for the first time does not have that, and what confuses them is the actual signal.

It follows on from Week 5 more than I realised at the time. The hard part of the interviews was listening without defending the idea, and peer review is the same skill in a different setting. The instinct is to explain why the thing they are confused about is actually fine. The useful move is to treat the confusion as the finding.

This was the week my research properly started. I am working on prompt injection and RL with Amulya Yadav, and on prompt injection with Saha. That is the direction I have been pointing at in these reflections all semester, so actually starting on it is a good feeling.

What I did not expect is how closely it lines up with this course. In Week 4 I built a custom GPT for Hydro Node and injected prompts into it to see whether it would hold to its instructions, and it did not. That was a side experiment on my own project. Now it is the thing I am researching. The journal has quietly tracked the whole path — Week 4 breaking my own assistant, Week 5 finding that Claude's personas handed back my own assumptions, Week 6 catching a sensor bug the code was never going to flag, and now this.

The common thread is that none of those failures announced themselves. The assistant still answered, the personas still sounded reasonable, the sensor still returned numbers. That is the part of agent reliability I keep running into, and it is why I wanted to work on it.

The results were not what I expected. I ran attacks against my agents through AgentDojo and several of them were flagged and failed rather than getting through. I had gone in assuming they would land, partly because of how easily my own GPT fell over in Week 4.

It is a useful kind of wrong to be. It means my mental model of how easy these attacks are was built on one informal experiment on a system I set up myself, not on anything measured. Running it properly against a benchmark is what showed me the difference, and now the question is why those attacks failed, which is a better question than the one I started with.

On top of that, starting research in the same stretch as coursework is a lot to hold at once. The research is where my attention wants to be right now, and I am still working out how to give the rest of it a fair share.