← Home

On Benchmarks

It's always sad to see the sorry state of the beautiful English language these days. Once upon a time, people used the right words, they knew "literally" didn't mean "figuratively", and that "they" isn't valid as a gender-neutral pronoun.

It's also particularly concerning that programmers, usually a smart bunch, are not immune to this. Just today, I heard someone saying I should "take a benchmark"(!) Their statement really set off the stickler in me. Sorry, but here's a rant.

The correct verb to use with "benchmark" isn't "take" or "make" or even "do"... and don't even think of using "benchmark" as a verb on its own. No, the correct form is "to fake a benchmark". If this is confusing, let me give you an example: (Any resemblance to real characters or conversations is purely coincidental)

Sara: The HHVM team loves faking benchmarks. For example, here's one that shows HHVM can be up to 10x as fast as php.net PHP!

Zeev: Today I faked a benchmark, it shows that actually, with phpng, php.net PHP can be almost as fast as HHVM!

Is this really so hard? It's literally the simplest thing to remember.

Jokes aside, seriously, never trust a benchmark. Though they're rarely outright lies, they are often deceptive, even if not intentionally. As Nikita Popov put it (though I'm certain he was hardly the first), "Never trust a benchmark you didn't fake yourself."

More importantly, unless you're Facebook (no, you're not Facebook), the latency of incrementing an integer variable in your PHP script is almost certainly not a problem. People worry too much about tiny things and neglect that they're spending 500ms doing a blocking read from a database or a file. Now, I'm not saying performance is unimportant - it very much is important. But as Knuth said, "97% of the time, premature optimization is the root of all evil".

P.S. While I was joking, it would be pretty funny if people started only referring to faking benchmarks, as if that's the proper way to say it.

Edit 2015-03-20: Just in case someone missed the sarcasm: No, I am not a prescriptivist, I very much like "they", and using "literally" is not wrong.