Also working on a product to build tasks from your own work for testing coding agents. Main thing I would offer is to look carefully at the agent trajectories - they love to figure out ways to cheat. Additionally, consider what "winning" means. If just using test pass rate, consider that tests might not encode what good means in your repo. I have been having success using "equivalence with merged PR" as judged by an LLM as a signal.
Yup, trying to be really strategic about testing. I didn't end up sticking with it, but I tried requiring test cases to cite a matching clause in the task assignment. But also: these tests are only indicative. Only a human can score a run.
To me, this comes down to verifiability. How do we measure the quality of what an agent is doing on our codebase rather than simply measuring task accomplishment?
> Verifying quality is orders of magnitude harder than "did the tests pass"
Agree that agentic grading is the future here. Cognition's Frontier Code is probably the best large public benchmark at this. You attribute agent quality issues to RLVR's binary pass/fail, however I wouldn't be surprised if labs are already supplementing that with rubrics as rewards to train more 'tasteful' models like Fable.
What can a practitioner do? I think there's promise in turning the optimization machine to the harness itself - building out a representative dataset of tasks on your repo, grading agent quality on them across various configurations, and optimizing [AGENTS.md / SKILLS.md / workflow / model / harness / tools] on that signal. High quality grading is still very hard, but it's more tractable at smaller, repo-level scale, and you can afford slower, more expensive verification for each task. You only need it to be right about your codebase's standards.
> In fact, it's not hard to imagine that if a model could reliably tell good code from bad, it might have written the good version to begin with
Pushing back slightly - detecting slop and discriminating quality is easier than generating it (why code review is so effective), and why grading is viable at repo eval scale even if it's much harder at RL scale.
Everyone is flying blind. For example, I am genuinely interested in trying HumanLayer, but would likely want some harder evidence (beyond anecdotes) that it's actually making my agents more effective before rolling out to an enterprise team.
I'm building this harness optimization loop @ https://stet.sh if curious
as a tip - models will always find a way to cheat, you will probably need to impose some restrictions on what they do / are able to access in the sandbox environment
It depends on what you're measuring. I agree that model resourcefulness is useful, but if you're trying to simulate real user sessions, then Claude looking at upstream Git and fetching the answer directly is somewhat worthless.
In my case, I'm trying to measure how coding agents perform under realistic scenarios when implementing tasks, as a proxy for how agents perform when used by actual users for those same tasks, so it's important to ensure the agents are behaving realistically instead of "cheating" and looking up answers.
Happy to share resources! I've been pretty deep in the space :)
- Mines tasks from your merged PRs/commits
- Replays them in Docker containers with different harness settings (change model / reasoning effort / AGENTS.md / etc)
- Grades the patches on various attributes (tests, equivalence with human patch, code quality)
The goal is to get a sense of how agents perform on your tasks, with your context, using the tools you do.
Thank you! Both of those papers are super new and super relevant.
The only big gap left is that they aren't using claude code/codex as harnesses. I'll try to reuse their constructed user sessions.
PS: Your work at stet is also interesting. That's definitely a problem right now that's hard to track. The only real solution is more robust CI/CD. I have since added harder validation like essentially running a full benchmark run on every prod push.
The most salient point here is the societal acceptance of consuming slop - somehow we've gotten to a point where the majority of people are ok with mediocre art. I feel that this is a trend that AI has only amplified. The commodification of attention has gradually led us to a point where we're optimizing for engagement instead of for intrinsic value of the content itself.
Personally, I will continue seeking out high-quality music/art/movies/books that speak to me, and most of my friends do the same. There will always be a demand for human-created art, regardless of any plagiarism or replication by labs.
Agree - all of this is based on vibes (I also use TDD based on vibes FWIW). The only way to settle "does TDD / caveman / [insert random skill here] help" is to replay real PRs from your repo and measure quality
> Seems like the progressive disclosure approach is the best for context efficiency; I wound up with a somewhat tight generic AGENTS.md, and the .cursor/rules individual files with glob matching for file names. Cursor honored those well.
This is also generally where I've landed - keep the AGENTS.md super light, and link out to docs as needed. Same idea with skills as well. Basically, preserve the context window at all costs.
The part I'm curious about is, when we're making the sorts of behavior changes you're describing on shared repos, how do we actually measure and quantify impact? It's one thing to tell the team that the agent should perform better, and it's another to say that you made the agent 5% better across a variety of tasks for every dev in the repo.
I didn’t have to share it or quantify it… so I didn’t care.
I just relied on different agents/models and kept asking a thorough prompt of “analyze the agents.md, cursorrules, etc and ensure its token efficient and enforces everything” (it was very specifically worded, I may have even asked an agent for how to ask agents for it) and just kept jumping from the 3 big models and medium and high thinking, each one kept finding little things and at one point moved entirely from one strategy to another, if I remember right.
Once I felt good enough I’ve been using it as my setup for my application and it’s been pretty good without any modifications or tweaks. Originally I decided to do this because I got tired realizing that it wasn’t honoring things I told it to. For example “restart the application after every modification to the server code” and it would “forget” to do that often… somehow now I’ve got it really well tuned for my particular codebase and approach to developing.
This has been bothering me for a while - the entire dev community is running on vibes when talking about AI. We're operating in an old paradigm, thinking that smart and logical additions to AGENTS.md result in good agent behavior, when in fact agents behavior is such a black box, that measurement is necessary.
> Even when all the rigging is controlled. (Which implies we need multiple experiments to compare against.)
Yes, agree that low n makes overclaiming a real risk with this sort of optimization loop. Low n results can be useful directionally but can't claim superiority without expanding the dataset. If I were running this for a shared repo with real consequences / value to improving AGENTS.md, instead of just as an experiment, I would expand n by a few factors for training / holdout, depending on expected variation on the tasks.
I'm also noticing similar patterns with needing to update AGENTS.md / skills per model release. E.g with Opus 4.6 -> 4.7, it became much more instruction adherent, so instructions written for the prior model generation might cause unexpected behavior in the new generation. I'm also convinced that an optimal AGENTS.md for Codex is not the same file as an optimized CLAUDE.md for Claude - the model personalities and behaviors are so different that we probably need to tune the instructions differently as well.
reply