Comment Re:Benchmarks (Score 1) 36
They are always gamed. The moment a "secret" test set is sent to a closed model, it's no longer secret. So every benchmark is compromised.
I suspect they even train adapters daily on github just for something hard to game like SWE-rebench. If I was launching a trillion dollar IPO I certainly would.