Comment by whoisthemachine
3 months ago
> Modelling the happy path is great for refactoring - even a necessity, but doesn’t help with finding bugs.
This is a common misconception (one that I also initially held). Unit tests aren't meant to find bugs, they're meant to protect against regressions, and in doing so, act as a documentation of how a component is supposed to behave in response to different input.
> Unit tests aren't meant to find bugs, they're meant to protect against regressions
That hasn't been the general consensus on unit tests for at least 30 years now. Regression tests are a small subset of tests, typically named for an ID in some bug tracker, and are about validating a fix. The majority of unit tests catch issues before a bug is even opened, and pretty much any random developer you talk to will consider that to be the point.
> Regression tests are a small subset of tests, typically named for an ID in some bug tracker, and are about validating a fix.
This is how I also tend to think of them, but it's not how the phrase is generally used. The general meaning of regression tests it to ensure known correct functionality doesn't break with a future change. There's no actual requirement it be tied to a known bug.
They do not "find" bugs in the way that exploratory testing or user operation might (or even in the way that broader integration tests might), that is they don't find bugs that are not in the known problem space. But they are very good at proving a method works correctly and covers the known execution permutations.
The majority of unit tests catch issues before a bug is even opened
The "issue" that is being caught is the bug the parent is talking about, not a "bug" in JIRA or something.
There's a few issues with this IMO:
1. Changes often require changing the functionality of a component, which means many of the current unit tests are bunk and need to be updated. Changes that are simply refactoring but should retain the same behavior, need to update/rewrite the tests, in which case again often requires significant refactoring of the existing tests.
2. Small isolated changes usually require testing everything which in a big org is very time consuming and slows down builds and deploys unnecessarily.
3. A lot of false confidence is instilled by passing unit tests. The tests passed, were good! Most of the production bugs I've seen are things you'd never catch in a unit test.
I really can't imagine a large refactor where we wouldn't end up rewriting all the tests. Integration tests are much better for that imo, "units" should be flexible.
Yes changing contracts implies updating tests. They should.
Refactoring under the same contract should not lead to refactoring of tests. Unless of course you introduce a new dependency you have to mock ? That's just one example.
If your code changes a lot it has nothing to do with tests being hard to change. It has to do with the code it tests changes too often. Poor contracts perhaps.
And just like the parent comment. Tests are not about finding or solving bugs, they are about regressions and making sure your contracts are correctly implemented.
If your refactoring includes changes to interfaces, different abstractions, logical changes, business logic, then most of your tests need to be effectively rewritten.
The only part where I see unit tests being useful for refactoring is making changes to the internals of a single unit. Its always been more trouble than its worth for me.
In some cases it makes sense, like testing small units that heavy in logic (function that calculates order prices for example, scientific computing, etc). But unit testing every single piece of code has always seemed dogmatic to me (unit tests are good engineering, write unit tests always everywhere). Everything has tradeoffs and as engineers I think our job is to understand the pros and cons and apply them effectively.
I think writing tests as a form of documentation is a waste of time. If I'm using a component I don't want to read unit tests to figure out what it should do.
Unit tests are most often used to cover a few more lines that need coverage. That's the value they provide.
A well designed API will generally allow users to understand usage without any additional documentation, sure. However, those who modify the API in the future will want to know every last detail that you knew when you were writing it originally. That must be documented to ensure that they don't get something wrong and break things – and for their general sanity. That is, unless you hate future developers for some reason.
You could do it in Word instead, I suppose, but if you write it in code then a computer can validate that the documentation you wrote is true. That brings tremendous value.
Undocumented and undefined are the same thing. If you are worried about changing undefined features then you must be in some sort of legacy hell.
3 replies →
Documentation for other developers of that code, not users of that code.
Also worth noting that "other developers" includes the original developer when they return to the code several months later.
Well tests are usually in English and easier to read then code IMH(uman)O
Regressions are bugs.
They can do both.
On that topic, static type checking can effectively be seen as a "unit test" that tests as well as documents the expected types for an interface.
No, static typing proves correctness (with respect to the types), which unit testing doesn’t do.
Unit testing proves correctness in regard to the test written (not necessarily the correctness of the application itself). They're similar in that they are both typically fast to run, and that they check an aspect of the program for correctness.
2 replies →