Property Based TDD at SPA 2013
Keith Braithwaite and I ran the Property-Based TDD workshop at the SPA 2013 conference. This time we ran two exercises at once. One half of the room wrote code to score a game of Tic-Tac-Toe, the other half wrote code to perform calculations with units of measure.
Languages used: Java, Python, Ruby, C#, C++ Scala and Haskell.
Observations, lessons learned and ideas came thick and fast at the end of the workshop. I wrote them down as fast as I could. Apologies if I missed anything. I've tried to group related observations.
Incremental Development:
- One property test drives out much more behaviour than a single example.
- Property tests did detect defects in the code under test that we didn't expect. For example, a property test failed with a divide-by-zero error for some inputs.
- We [programmers experienced in TDD] are familiar with the baby-steps to take when using example-based tests to test-drive a design. But we are not familiar with the baby-steps required when using property-based tests.
- When properties or generators duplicate the code under test, are we using the right tool for the job?
- Initially the property based tests duplicate the code under test. But when working incrementally, after a few tests the implementation of the code under test and the properties and generators start to diverge.
- Property tests would provide a clear description of behaviour that has a non-obvious implementation. E.g. if the implementation is highly optimised.
- Property tests can be useful for reverse engineering or reengineering legacy code.
- Could the properties and generators that emerge from incremental development with property-based tests be used as a real implementation? E.g. could you use property-based testing to characterise some existing legacy code and then throw the code away and just use the properties and generators?
Domain Modelling:
- General properties are quite easy to write. As code becomes more specific, properties become more complicated.
- It is much easier to apply property testing to units of measure than Tic-Tac-Toe. Units of measure are naturally algebraic. You can work out an algebra of Tic-Tac-Toe, but it takes significant effort.
- How applicable is property testing to the messy domains that we have to support in commercial software?
- You need to get the right representation of the problem early so that you can express properties.
- You can start with an empty class (in a dynamically typed language) and test-drive its API and implementation with properties.
- You need be careful about the choice of generators and understand what they represent in domain terms.
Generators:
- Filtering of acceptable input must be done in the generators. If done in the tests (e.g. with if statements or assume(...) calls in JUnit), there is a danger that the filter discards so many inputs that you end up with very poor coverage. The test will appear to succeed but actually hardly have tested anything at all.
- There is a danger that generators become very complex. You need discipline to split properties into different cases that each have simple generators. We were not used to that way of working.
- Complicated generators will themselves need tests.
Tooling:
- We did not trust what was being generated. We ended up using println statements to see what the test framework was doing. Maybe this is due to lack of familiarity with the tool (ScalaCheck) and limited documentation.
- The property testing tool is not very verbose about what was tested. E.g. how many tests were run. There's an opaqueness to many of the tools. Haskell QuickCheck reports more useful information when it runs than other tools.
- Some tools (Propr?) only shows the inputs that caused a test failure and not the assertion that failed. This makes it hard to understand why complicated property tests failed.
- Surprisingly, QuickCheck for C++ was simple to use.
Some people have put the code they wrote online:
- Units of Measure in Haskell by Jonathon Perret and Willem van den Ende.
- Units of Measure in Python by Andrew Seward.
- Scoring Tic-Tac-Toe in Haskell by James Byatt
Thanks to everyone who attended.