← Back to context

Comment by stult

11 days ago

One of the nice things about the .NET ORM EntityFramework is that you can swap a mocked in-memory database for your prod DB with dependency injection, so without modifying your code at all and theoretically without affecting the behavior of the ORM. Which is to say, you're right, it's about using the right tools. Those tools of course vary by ecosystem and so in some cases mocking the database is in fact the correct decision.

8 comments

stult

Reply
bunderbunder  11 days ago

Probably the single most obnoxious production defect I ever found related to a database would never have made it into production if we had been using a real database instead of a test double. It happened because the test double failed to replicate a key detail in the database's transaction isolation rules.

After figuring it out, I swapped us over to running all the tests that hit the database against the real database, in a testcontainer, with a RAM disk for minimizing query latency. It was about a day's worth of work, and turned up a few other bugs that hadn't bit us in production yet, too. Also sailing past our test suite because the test double failed to accurately replicate the behavior in question.

Total time to run CI went up by about 10 seconds. (For local development you could chop that way down by not starting a fresh server instance for every test run.) Given how many person-hours we spent on diagnosing, resolving, and cleaning up after just that first defect, I estimated the nominally slower non-mocked tests are still a net time saver if amortized over anything less than about 50,000 CI runs, and even then we should probably only count the ones where an engineer is actually blocking on waiting for the tests to complete.

That said, there was a time when I thought test doubles for databases was the most practical option because testing against real databases while maintaining test isolation was an unholy PITA. But that time was 5 or 6 years ago, before I had really learned how to use Docker properly.

  • stult  11 days ago

    I simply don't think that I will ever be able to come up with anything even vaguely as comprehensive as the test coverage that Microsoft already has for ensuring their ORM behaves consistently across database providers. In my over 10 years of using EF, I have never once encountered a database bug like you describe. If I were to discover such a bug (which I'll admit does occasionally happen even though it hasn't happened to me), it would be easier and better by far to submit an issue to the EF team and let them figure out a fix (including the appropriate tests) than it would be to rework my own test infrastructure. I am not in the business of developing requirements or code for databases, and building an elaborate test model for what I consider the essential requirements for a database would be a distraction from developing code that is more valuable to the business.

    The same logic does not apply to all ORMs, of course, which do not all benefit from the same quality of professional dev support that EF receives from MS. But that's my main point from above: the correct design decision depends on the context. For services written in other languages with other ORMs or raw SQL, I absolutely will spin up a full Postgres test container because it is indeed trivial (have one running in the background on my laptop right now in fact). It just isn't necessary in the specific context of EntityFramework code.

    • smackeyacky  11 days ago

      I can think of a few things that will upset this particular apple cart, chief amongst them is the behaviour of different databases and sorting / collation which might not be generally categorised as the kind of bug a test suite will uncover, but certainly creates production bugs / issues.

      I love EntityFramework, it's easily the best ORM I have ever used but it has a few cross-platform footguns that require testing against the actual database service you're using.

      2 replies →

  • nickkell  10 days ago

    Using an in-memory database does not increase my confidence that in my software either. I also started using dockerised dependencies in tests a couple of years ago.

    Can you please explain what you did with a RAM disk to speed them up?