A Case for Mocking and Stubbing
So, when I hear people defend mocking and stubbing (I'll say m&s to save typing) it is for one of two reasons:
1. The controller tests should only be testing the controller. The model tests should only be testing the model. Your controller code should not touch the model so you must m&s out your models to keep things separate.
2. Hitting the database is slow. M&S speeds up the testing process allowing me to focus on writing Red/Green/Refactor rather than wait for the test suite to finish.
Let's look at the first reason...
This is how I develop.
1. I write user stories, write the steps watch them fail.
2. I write the functional tests, watch them fail
3. I write the unit tests, watch them fail
4. I write the models, unit tests pass
5. I write the controllers, functional tests pass
6. If I have done everything right then my stories should now pass
7. I can go back and refactor or continue by writing a new set of stories
So, if I am m&sing my way through my functional tests it defeats Red/Green/Refactor. My functional tests should rely upon the models being written properly. There are dependencies here that cannot be ignored or wished away for the sake of an idea testing environment. (separate but equal) What if I change something on my model down the line, my unit test will fail, I fix that. But my functional tests still pass because I m&sed that functionality. In the real world, the app will break.
As a developer I need my tests to fail if I change something. I need to be told exactly where the tests are failing so I can correct this. You do not get this when you m&s.
I'm sure many people will disagree and I am very interested in hearing the counter-argument or even an argument for why (if) I'm completely wrong.
So, for the second point... speed.
This is somewhat of a valid argument. Having fast tests is necessary for effective TDD. You need immediate feedback on what works, what doesn't. However, the price you pay for that speed is not worth it IMO. So, what would be the alternative?
What the TDD community needs is an in-memory ANSI compliant database that can be used solely in the testing environment. This way, nothing gets written/read to disk. The current practice of using MySQL or Postgres or SQLite3 for your testing environment is nuts. These databases are meant to handle millions of records, organize and serve them up. How many records within a given test are you using? For most of my tests I usually have no more than 3 or 4.
Another benefit is a much faster transactional rollback after every test. Just delete the memory, RAM is fast. Disk IO is not.
There is NullDB, but it is quite limited in that it cannot do ActiveRecord::Base.find. (I'm guessing this will cause a few problems for people)
SQLite3 can be an in-memory database. Problem solved, right? Not quite. SQLite3 is pretty limited. Most people are probably using MySQL and rely upon many of the SQL functions that are included.
What would be nice (and well beyond my ability) is to have a Gem that simulated the database you use, only it is in-memory. Optimized for small data sets. No need to go through a heavy hashing algorithm. Keep it light. Keep it fast.
8 comments:
Here is a project that is attempting just what you suggest using TreeTop as the SQL parser:
http://github.com/smtlaissezfaire/guillotine/
re: your argument
All points are valid but you seem to leave off the design benefit of mocking which is the primary reason to use them. Also, if you have stories (Cucumber features) driving your entire process like you say those will catch any breaks in between the layers. That said both approaches are valid with pros and cons to each.
I've used SQLite memory databases for many apps and the speed was on par with using MySQL through a native driver.
As the Merb guys say, "No code is faster than no code."
I started to leave a comment and found I had too much to say to respond properly here, so I posted my response on my blog. Feel free to comment here or there if you're interested in a dialog about this.
The short version is that it makes sense that mocking won't work for you because it is an isolation technique and your approach favors integration.
The long version is at http://blog.davidchelimsky.net/2008/12/11/a-case-against-a-case-against-mocking-and-stubbing.
Cheers,
David
Take off your rails colored glasses...
There are valid critiques of mocking, but you aren't really making those.
Your basic argument is that you can't test integration of components when you are just playing make believe.
I totally agree, but that doesn't mean mocking isn't a great tool to have in the proper context.
Functional tests are not one of those.
Thanks for all of the responses. This seems to have really gotten under the skin of some Rspec people. Not my intention but still interesting. I still maintain my opinion that within an integrated testing approach that mocking/stubbing is going to cause major issues. One valid reason to use mocking/stubbing in this context would be when accessing resources outside of your project. (Geocoded data for example)
I guess I should have made the database stuff a separate post because in all honesty I am more interested in that.
topfunky: thanks for the update on using SQLite for an in-memory database. True, no code is faster than no code. But, as I stated RAM is faster than disk IO. I wonder if MySQL could be adapted to write to memory instead of disk. SQLite is slow in general, but is a good place to start for an app considering how easy it is to setup.
"What if I change something on my model down the line, my unit test will fail, I fix that. But my functional tests still pass because I m&sed that functionality."
A technique that can help alleviate that problem is to use a stub/mock that will check the method you are stubbing/mocking actually exists with the correct arity on your model. Brian Takita's rr framework supports this technique by prepending your stub/mock with strong. This provides the benefit of not being concerned if the implementation of model#whatever is correct, while also providing you with errors if model#whatever is removed or it's signature changes.
I think the crux of your argument is the assertion that "functional tests should rely upon the models being written properly." I don't see the support for that assertion.
I also think that you're not accounting for integration tests, which will indeed fail if the model is not correct.
Put your test-DB on a ramdisk or on a SSD and you're halfway there.
Post a Comment