Because a harness doesn't just "drive" the LLM. e.g., there's code in claude code that detects if the user's prompt shows they're angry, and they react to those prompts differently. (they use regex on "wtf", etc.!)
Thanks for the read. It seems to confirm that resource limits are an important factor for terminal benchmarks:
> The extra resources enable the agent to try approaches that only work with generous allocations, such as pulling in large dependencies, spawning expensive subprocesses, and running memory-intensive test suites.
> An agent that writes lean, efficient code very fast will do well under tight constraints. An agent that brute-forces solutions with heavyweight tools will do well under generous ones. Both are legitimate things to test, but collapsing them into a single score without specifying the resource configuration makes the differences—and real-world generalizability—hard to interpret.
So changing the resource limits changes the benchmark. Yet their score table claims their score to be for Terminal-Bench 2.1, not Terminal-Bench 2.1 with raised limits.
Probably because that extra 2% (or whatever figure) isn’t terrible valuable in the first place. Sometimes the best answer is “they’ll need to update their stuff”.
Perhaps? But I think this is more a case of just not seeking things out.
Music is as vibrant and diverse as ever, but not if you're only looking at the top charts run by the music industry.
Same deal with games, there's more experimentation and interesting concepts in gaming than ever before, but not from the AAA studios.
Now I can't speak for how you vacation, but I've had wonderfully different experiences between Hamburg, Berlin, Vienna, Prague, Rome, Paris, Montpellier, London, Amsterdam, Oslo, and Florence. I just don't go to the starbucks and instead wander around a bit, optionally picking from a few hit destinations if I feel like it. But also, it's not like this was created for nothing: https://www.itchyfeetcomic.com/2018/10/omnimappus-europeus.h...
On cities, I guess my point is that I didn't have to seek out new experiences. Simply going there guaranteed new experiences. I spent a lot of time in Vienna growing up and everything was different. I literally couldn't have a familiar experience even if I'd wanted to. In fact, I remember having to spend lots of money and travelling a long way to find a specialised shop to get a familiar food item (baked beans) to relieve the homesickness.
I don't think people under a certain age can really appreciate how different it was.
> On cities, I guess my point is that I didn't have to seek out new experiences.
Neither have I. Literally just wandering around. The one exception specifically was Prague's Old Town district having several of the same tourist trap shops. But all you have to do is walk around a minute outside of the most touristy area of Old Town and it's all gone. Not to mention all the awesome, different cafes and shops along other landmarks.
> Same deal with games, there's more experimentation and interesting concepts in gaming than ever before, but not from the AAA studios.
I just wanted to add on to this, I wouldn't really classify modern games as "lower quality" than those of the 80s. I'm really not a fan of AAA games, I think the last one I played was Elden Ring, but I would never suggest that they're actively low quality. Uncompelling? Absolutely. But I also have spent a lot of time playing games from the 80s. Silver and golden age CRPGs, random simulators, DOS games that catch my eye. "Quality" isn't the first thing that jumps to my mind. Often they're ugly, terribly balanced, buggy, rife with all sorts of issues in any category you can think of. Games have come a very, very long way. 2400 AD (1988) and Champions of Krynn (1990) are relatively speaking highly polished masterpieces. They're still kusoge, honestly. I have very little experience with the consoles of that era, because pretty much nothing I see even remotely catches my eye.
>Perhaps? But I think this is more a case of just not seeking things out
"If you went for deep dives and looked at fringe stuff there's still variety" is totally compatible with "the world is way more honogenized".
There have always been fringe unique voices, but that's beside the point. The world is not defined by the long tail, but by the top 20% most listened/visited areas/games played/movies watched/etc, that all now look the same.
>Music is as vibrant and diverse as ever, but not if you're only looking at the top charts run by the music industry"
So it's not "as vibrant and diverse as ever" then, since the same "top charts" were much more varied in the past, and they reflect what most people listen to, not just what some music nerds seek.
>Same deal with games, there's more experimentation and interesting concepts in gaming than ever before, but not from the AAA studios.
"There is more experimentation and interesting concepts in gaming than ever before" just not in the games most people play. Well, used to be ALSO in the games most people play.
>Now I can't speak for how you vacation, but I've had wonderfully different experiences between Hamburg, Berlin, Vienna, Prague, Rome, Paris, Montpellier, London, Amsterdam, Oslo, and Florence. I just don't go to the starbucks and instead wander around a bit, optionally picking from a few hit destinations if I feel like it.
Yes, if you go out of your way you can find some places in Hamburg, Berlin, Vienna, Prague, Rome, Paris, Montpellier, London, Amsterdam, Oslo, and Florence that are not all alike. Meanwhile, you could go anywhere in those cities 30 years ago and you'd find uniquely different stuff, whereas now all major streets and areas have the same globo cafe and globo shops.
Missing in here is the nuance that Meta was absolutely not too late to the first AI boom with Llama, and at one point were the clear leaders of the OSS alternative LLM space. But they blew it up themselves with the disastrous Llama 4 release which included, amongst immense self-imposed delays, lies on benchmarks. No wonder nobody wants to use their tech.
Hard agree with the post overall, especially on LinkedIn and X. On Bluesky at least there's not nearly as much of this crap, but it's a small community of practitioners and not flooded with sellers and resellers peddling some AI products.
For me, the big win is that it's very cheap to experiment with several approaches to something and pick what feels like a good winner. For UX work this is a boon because it shifts the bottleneck to evaluating designs, which is where the bottleneck should be. It has historically not been there.
It's a little more complicated than that, unfortunately.
If you use Claude via API in your own app, you're paying full price.
If you have an "API Plan" for Claude Code (i.e., free), you're paying full price.
If you have a Pro, Max, Max 5x, or Max 20x, your tokens are subsidized up until a rate limit. Then you pay full price for usage thereafter, until the end of the billing cycle.
The widespread belief in industry right now is that the per-seat pricing (which Copilot bailed from first) is going to go away in the near-term.
It's not more complicated? I referenced the subscription... I just added a small warning about it as some people may or may not be aware about the fact they're opening themselves up to serious consequences if they decide to use it on their employers code without explicit permission... Depending on their employers digression, eg largish entrenched employers which value their IP will be more willing to inflict damages on you. An upstart will likely not care unless the CEO sees an opportunity to profit personally.
Seems to be another great incremental update to the workhorse, nice!
I've been using Sonnet instead of Opus for almost all coding tasks for a while now. A little elbow grease to break down tasks and you can spend a lot less money for just about the same output quality.
Crazy. I just changed the default for our entire org to Opus because people were continually unimpressed with Sonnet's abilities. It's fascinating to think how varied people's experiences are when interacting with LLMs and how much the outcomes depend on how people approach interacting with the models.
Yeah I think people are sleeping on the smaller/faster models like Sonnet. As long as you have a detailed plan or small, well scoped individual tasks Sonnet can implement just fine. Opus will still do better at more open ended tasks or completely "vibe coding." Or spec/plan with Opus, and have Sonnet implement.
I would indeed be more inclined to use it if the tokens per second were better. Though I would be then using their more expensive Opus less though. Perhaps it is strategy.
I wasn't banned from the whole site but I definitely was perma-banned from the local subreddit. That's how reddit works. Moderators get to moderate as they see fit, and short of some egregious terms of service violations, reddit admins won't interfere.
I was also quickly banned from participating in my local sub. It is a shame that the local subs are the ones that are the most over moderated- they are effectively isolating those in the community that don't have an "acceptable" viewpoint on any given topic
> the local subs are the ones that are the most over moderated
The local subs are by nature ghost towns, and the first people to get them rolling will often establish some weird culture that will never be uprooted.
I disagree, though. The most over-moderated are any having to do with any product or service (especially particular websites or pieces of hardware or software), liberal-left subs, or subs related to any industries that are dominated by particular companies. This happens because companies make sure to control the modding of (what they consider) their own subs.
In the case of liberal-left subs, the Democratic Party has prioritized controlling all political discussion amongst the people whose votes it feels entitled to post-2016, when H. Clinton's campaign press released that it was assigning a budget of millions to "Correct the Record" anonymously online. In that year, the party apparatus was indistinguishable from Clinton's campaign organization, and those people continue to do the same thing except the budgets have gone way up. This was also when the Democrats and the CIA/FBI started to become mutual admiration societies (which I think has worn out to a certain extent after Gaza and ICE) but you can see how the personnel and tactics diffused over the past decade until their effects became overwhelming today.
Might be better to say that the local subs are the most organically over-moderated, but even then sometimes that first little clique that gets them rolling are actually connected in their local area, and engaging in the same kind of thoughtful, collaborative, profit-motivated manipulation as above.
It's also just that Reddit is the most centralized decentralized thing ever
Especially considering... look what Huffman did with Apollo and other third party extensions. The man doesn't give a fuck. How hard is it to brigade Reddit? Compared to if we had 20 different forums for (say) Austin Texas as opposed to the one with 6 mods that will ban whatever you post on ideological grounds?
Go take a gander at the sub for the most conservative leaning town you can think of. You will be surprised at how busy and “organically” left leaning it is.
Why in the world would you say this to a stranger about the account of a completely mundane event? I've seen people banned from subreddits for e.g. mentioning the wrong brand names, and even worse, been happy that they were banned because those brands were not the subject of the subreddit (and were a constant distraction because mentioning them was something provocative and easy to post if you didn't really know anything about the subject.)
And you think that "asian people like this brand of car" is something so obviously impossible to be banned for that you would denigrate a stranger with zero evidence, under your own name. I'm honestly shocked by people's bravery sometimes; if this is a professional account, people reading it who know you might think less of you (and will never mention this to you.)
It's also funny because the statement isn't even controversial and people still get insanely ass-mad about it.
Also consumer car buying by ethnicity is tracked data and billions of dollars in investment & marketing are allocated from this metric...It's an absolutely ridiculous thing for anyone to feel a certain way about being stated.
He said from a subreddit, not the whole site. There is a much lower and more variable threshold for banning from subreddits. Its very believable. People get banned from subreddits for being subscribed to different subreddits. Hence the “echo chamber.”
reply