Comment by devit
16 days ago
Testing both your code and the message system is exactly what you want, since if the message system is broken in a way that upstream didn't catch, you want to learn about it during testing and not production, if possible.
I’m still mad about the time I was told to mock a payment gateway in tests even though they had a testing environment and then got a slew of bug reports from people whose company names had punctuation (and thus failed the name validation the payment gateway was secretly running).
You should be unit testing components that interact with the payment gateway. This could involve dozens of even hundreds of tests, where the gateway should be mocked. These tests should be fast and reliable. In addition, you should have a small suite of integration/E2E tests against a real test instance of the gateway. These tests may be slow and unreliable (because of the real network involved) but catch those hairy issues you failed to find in unit tests.
Also, when your integration suite (or customer reports) discovers that the payment gateway fails on punctuation, another unit test should be added with a mock that responds the same way, and an E2E test added with punctuation in the input data and a failure expectation.
What makes you so certain you would have included punctuation in the input data if the test had not mocked the gateway?
That reminds me that Stripe actually maintains (or used to) their own mock for their Ruby package. This puts the burden on maintaining the mock on the library owner, where it is more likely that they would implement the mock correctly, edge cases and all.
Could you just run some behaviour tests against the gateway?
You can get the best of both.
Keep in mind that there are different kinds of testing. What Beck called unit tests and integration tests.
Unit tests are really for purposes of documentation. They show future programmers the intent and usage of a function/interface so that others can figure out what you were trying to do. Mocking is fine here as future programmers are not looking to learn about the message system here. They will refer to the message system's own documentation when they need to know something about the message system.
Integration tests are for the more classical view on testing. Beck suggested this is done by another team using different tools (e.g. UI control software), but regardless of specifics it is done as a whole system. This is where you would look for such failure points.
Unit tests as a form of example code based documentation is where I could see unit tests complimenting documentation, yes.
However, depending on the industry, code coverage is a valuable tool to gauge the maturity of the software baseline and burning down software execution risk. One example of this is Airworthiness or Safety Critical Code.
Of course, there is no single code coverage metric. Code covered by unit tests does not count towards code covered by integration tests. They are completely separate systems. And, at least in Beck's opinion, should be carried out by completely different teams.
I would want to seperate those tests. You want to know what has failed. Also depends on how many tests you have.