That's one high profile instance of human vs robot. Surreal.
teiferer 2 minutes ago [-]
For anybody not familiar with pgrust or ClickBench:
Pgrust is a re-implememtation of postgresql in rust.
ClickBench is a benchmark suite for databases.
The link goes to a removal request of pgrust fromthat benchmark because of its alleged cheating.
torben-friis 58 minutes ago [-]
This idea of "here's the beta, I'll release the code in a few weeks" sounds counter intuitive to me. is it normal in open source? Any reason for it?
verdverm 31 minutes ago [-]
Not normal. This a new trend of people who use too much Ai, or maybe more specifically the Claw/Ralph loops where you end up with a few commits touching 1000s of files. For about the last year, many Show HN to github projects are often just a single commit
We need to think more about what we are doing as an industry, instead of letting the machines think for us (imo)
jerf 3 minutes ago [-]
Let any who read this be aware: You should be immediately suspicious of AI code that is much much faster than established code bases. AIs are even better at benchmarkmaxxing than humans are, and humans were already pretty good.
It is by no means impossible for an AI-written code base to be faster than a conventionally-written one, of course. It's just a time to be suspicious. Especially if there's little to no evidence that the human has subjected the code base to any further testing of their own. AIs seen to be able to benchmarkmax even in the presence of a lot of unit tests that would generally speaking have prevented a human from going too far down that road.
NewLogic 27 minutes ago [-]
I am getting increasing disillusioned by the number of peers I am seeing completely hand over their thinking to a tool. Agents are great, but if you can't explain to me how your PR changes a system and why design choices were made you are no longer an engineer.
embedding-shape 22 minutes ago [-]
Agreed. Used to be prevalent in the frontend world (now I think it's the only thing that exists seemingly), where programmers would copy-paste stuff from Stack Overflow until things worked, and when you asked "So why did this actually fix the issue?" they usually said "I'm not sure, but it works now, that's what matters". This seems to be standard practice across the entire stack now.
I've even wasted my time jumping on calls (requested by the other party) where after 5 minutes of conversation, it's extremely clear this "technical founder" doesn't even understand the basics of their own design and architecture of the product itself, even less the technical parts.
verdverm 17 minutes ago [-]
This seems more a product of the incentives we have in society, namely seeing the rich and wealthy on social media and then wanting that quick and easy. So many of those cases are not people with passion for the problem.
Good things take real effort, bring back the TLC!
verdverm 19 minutes ago [-]
What we don't want to do is mirror the partisan political environment in the Ai yay, nay, or how. We need to be able to talk to each other in understanding and a shared reality of facts. Too much hyperbole right now, starting from the top of the big US labs. They need to step down a few ego pegs
verdverm 2 hours ago [-]
Sounds kind of like Claude was "cheating" and the humans are engaged in eyes-free coding. Diff so big github refuses to render. I suspect they are reaching the limits of what this new vibe "paradigm" of programming can do. If you haven't pushed a vibe project far, eventually progress becomes near impossible because the slop has grown and the conditions are overly complex, locally focused. Ai is not good at writing reusable software (on their own).
stay in the loop friends
parasti 6 minutes ago [-]
To be fair Github refuses to render diffs after a couple of hundred lines.
chuckadams 14 minutes ago [-]
It was certainly assisted by AI, but there isn't an agent out there that does that kind of cheating without deliberate prompting by the operator.
I've taken a vibe-coded project pretty far, with even better testing than most other things: the main factor is that it's big on falsifiability, i.e. it has to see a test for a new feature actually fail before it considers the test any good. Not just the red/green from being originally unimplemented, it does a mutation test to ensure it flips back to red. But wow did it ever make that test framework complicated, byzantine, and ultimately brittle until I stepped in and forced a consolidation and simplification pass (twice now). Part of the complexity came from it being a polyglot monorepo, but I've found even in other places, you have to follow an expand-contract cycle of development when vibe-coding. Not all that different from any other system you've built up organically, just on a much faster cycle.
But back to my original point, none of this really applies to the pgrust benchmark, which was obvious and blatant cheating on the human's part.
thatsabadlook 2 hours ago [-]
Yup. These are people who have gone full Claude. The dopamine seeking is at previously unseen levels. they think this pr is some factual item that means something so they post it on HN as click bait. But no it only means something to a fulltime ai slopper
verdverm 1 hours ago [-]
If the Ai came back to me with a 1600x improvement (still in pgrust readme), I would be highly suspicious of the Ai and the code it wrote
The hype factory must grow, I suppose, though I would prefer if the model makers would stop with this "complete the task by any and every means possible" and replace it with "I don't know, I should ask or do more learning"
At least they have added a test which looks to make sure the Ai doesn't do it again... if you need this kind of test, something is wrong
I've noticed it with a lot of random stuff i've been trying
I'll tell claude, "write an implementation of an AAC decoder that is 50% faster than ffmpeg"
and it will do it! Amazing!
Then i tell it "hmm ffmpeg has been really well optimised, are you sure?" and it says "oh, i was running our code directly but running ffmpeg executables with loads of overhead, ffmpeg is actually 50% faster"
verdverm 23 minutes ago [-]
Transformers will def get us to AGI /s
My new definition for The Singularity is not about Ai intelligence, it is about humans handing over critical thinking regardless of Ai capabilities
Ai companions are where I put the highest p(doom)
thatsabadlook 51 minutes ago [-]
If I saw that test in a PR I would not only reject the PR and ask a lot of questions but someone would likely take a forced vacation to deslop their brain.
verdverm 49 minutes ago [-]
that requires someone (1) using PRs (2) looking at the PR
Funny, the report alleging cheating reeks of LLM usage just as much as the commit diff [1] of the next attempt:
> # The standard ClickBench driver over the step scripts in this directory; ./install fetches the dataset.
[1] https://github.com/ClickHouse/ClickBench/pull/2751/files
Pgrust is a re-implememtation of postgresql in rust.
ClickBench is a benchmark suite for databases.
The link goes to a removal request of pgrust fromthat benchmark because of its alleged cheating.
We need to think more about what we are doing as an industry, instead of letting the machines think for us (imo)
It is by no means impossible for an AI-written code base to be faster than a conventionally-written one, of course. It's just a time to be suspicious. Especially if there's little to no evidence that the human has subjected the code base to any further testing of their own. AIs seen to be able to benchmarkmax even in the presence of a lot of unit tests that would generally speaking have prevented a human from going too far down that road.
I've even wasted my time jumping on calls (requested by the other party) where after 5 minutes of conversation, it's extremely clear this "technical founder" doesn't even understand the basics of their own design and architecture of the product itself, even less the technical parts.
Good things take real effort, bring back the TLC!
stay in the loop friends
I've taken a vibe-coded project pretty far, with even better testing than most other things: the main factor is that it's big on falsifiability, i.e. it has to see a test for a new feature actually fail before it considers the test any good. Not just the red/green from being originally unimplemented, it does a mutation test to ensure it flips back to red. But wow did it ever make that test framework complicated, byzantine, and ultimately brittle until I stepped in and forced a consolidation and simplification pass (twice now). Part of the complexity came from it being a polyglot monorepo, but I've found even in other places, you have to follow an expand-contract cycle of development when vibe-coding. Not all that different from any other system you've built up organically, just on a much faster cycle.
But back to my original point, none of this really applies to the pgrust benchmark, which was obvious and blatant cheating on the human's part.
The hype factory must grow, I suppose, though I would prefer if the model makers would stop with this "complete the task by any and every means possible" and replace it with "I don't know, I should ask or do more learning"
At least they have added a test which looks to make sure the Ai doesn't do it again... if you need this kind of test, something is wrong
https://github.com/malisper/pgrust/blob/38634b408dea373f3944...
I'll tell claude, "write an implementation of an AAC decoder that is 50% faster than ffmpeg"
and it will do it! Amazing!
Then i tell it "hmm ffmpeg has been really well optimised, are you sure?" and it says "oh, i was running our code directly but running ffmpeg executables with loads of overhead, ffmpeg is actually 50% faster"
My new definition for The Singularity is not about Ai intelligence, it is about humans handing over critical thinking regardless of Ai capabilities
Ai companions are where I put the highest p(doom)
the project has 3 mega commits of claude slop
https://github.com/malisper/pgrust/commits/main/