Comment by worldsayshi
10 hours ago
One: Can we test the tests using some somewhat formal specification of the why?
Two: my intuition says that exhaustively specifying the intended input output pairs would only hold marginal utility compared to testing a few well selected input output pairs. It's more like attaching the corners of a sheet to the wall than gluing the whole sheet to the wall. And glue is potentially harder to remove. The sheet is n-dimensional though.
I really don't understand the "exhaustive specification" thing. How else is software supposed to work but with exhaustive specification? Is the operator + not specified exhaustively? Does your intuition tell you it is enough to give some pairs of numbers and their sums, and no need for some words that explain + computes the algebraic sum of its operands? There are an infinite number of functions of two arguments that pass through a finite number of specified points. Without the words saying what + does, it could literally do anything outside the test cases.
Of course, for + it's relatively easy to intuit what it is supposed to mean. But if I develop a "joe's interpolation operator", you think you'll understand it well enough from 5-10 unit tests, and actually giving you the formula would add nothing? Again I find myself wondering if I'm missing some english knowledge...
Can you imagine understanding the Windows API from nothing but unit tests? I really cannot. No text to explain the concepts of process, memory protection, file system? There is absolutely no way I would get it.
The thing about Joe's interpolation operator is that Joe doesn't work here anymore but thousands of users are relying on his work and we need to change it such that as few of them scream as possible.
That's the natural habitat for code, not formally specified, but partially functioning in situ. Often the best you can do is contribute a few more test cases towards a decent spec for existing code because there just isn't time to re-architect the thing.
If you are working with code in an environment where spending time improving the specification can be made a prerequisite of whatever insane thing the stakeholders want today... Hang on to that job. For the rest of us, it's a game of which-hack-is-least-bad.
What's stopping someone from reading the code, studying it deeply, and then writing down what it does? That's what I do, but I see people struggle with it because they just want to get more tickets done.
2 replies →
I suspect we're thinking about quite different use cases for our testing code. If the input-output pairs are describing a highly technical relationship I would probably want a more rigorous testing procedure. Possibly proofs.
Most of the tests I write daily is about moving and transforming data in ways that are individually rather trivial, but when features pile up, keeping track of all requirements is hard, so you want regression tests. But you also don't want a bunch of regression tests that are hard to change when you change requirements, which will happen. So you want a decent amount of simple tests for individually simple requirements that make up a complex whole.