A few weeks ago, at StarEast, I was talking with Antony Marcano and Rob Sabourin about agile processes, and asked a question that has become a perrennial favorite; Can you be agile if you are not creating and executing loads of automated unit tests?
The reason this has become a favorite question of mine is that it is very germane to us at Sirius SQA. We try to do things in a very agile way (note that I always use the lowercase version of the word), and we have tried on 3 separate occasions to implement a strong automated unit testing program, and failed. While it ultimately does not matter…our processes are what they are, no matter what label you do or do not apply to them…it has long bothered me that we were missing such a large piece of what we aspired to.
Sadly, after a bit of discussion, Antony and Rob both agreed that without the unit tests, it really isn’t agile…whatever it may be. And so we began to talk about why it had failed for us in the past. To put it in a nutshell very tightly, we do a lot of non-OOP coding; we have many libraries that are purely libraries of functions and procedures. This is problematic because most of the prepackaged frameworks for unit testing are very strongly aligned with OOP methodologies, if not downright reliant on them. So as we tried to implement these frameworks, we were continually frustrated by an inability to get to the level of detail we wanted to get to, and produce the results we needed. That, at this point in the conversation, was my perception of why things had gone wrong. Rob Sabourin began to challenge a few of my beliefs, and simply pointed out that while many of the pre-packaged frameworks were heavily OOP aligned, unit testing was nothing new, and that it sounded like we just needed to grow our own framework. With this as a starting point, the conversation meandered through the evening over a few drinks, and in the morning I was once again committed to making this work. And so I began to pick through the still smoldering ashes of my efforts from less than 6 months ago, culling those parts that seemed valuable, and purging those that did not. By the end of the day, I had a reasonable framework and a small hand full of tests.
I showed what I had done to Antony, and he began to offer very specific advice on details of implementation, syntax, how test code should look relative to prod code, features, and, probably most important, the pain and common problems of retrofitting an agile unit test process into an existing code base that was not written to support it. It turns out that much of the pain we had experienced from our first efforts was not so much from a lack of OOP, as it was from a lack of designing and writing code explicitly to be tested by automated unit tests.
By the next day, I had greatly expanded and simplified the test code, and doubled the number of tests. Here is an example of the code for a simple test:
LogIteration(’1: Defect Report is Generated’);
MyReportForm := TfrmReport.Create(NIL);
IO_OpenTest(TestFileName);
MyReportForm.GenReport(Self);
ReportHasLines := (MyReportForm.rvReport.LineCount > 34);
CheckTrue(’A Report Has Content’, ReportHasLines);
FreeAndNil(MyReportForm);
DumpTempDir;
Note the first line; the output for the test is written to an editor for review, and the LogIteration marks the beginning of a different variation of a given test; within a test, there will normally be many of these variations, with different properties being manipulated to various valid and invalid values, to test error trapping and various states of valid execution. This particular test only checks that the report has more lines than the standard header; if it does, then a report with content was generated, which is all this is checking. In addition to CheckTrue, there is also CheckFalse, CheckString, and CheckInteger; Because of the highly visual nature of TestExplorer, we are contemplating a CheckBitmap as well.
At this point, I was fairly certain this could grow wings and fly. However, another challenge for us with unit testing is that through the years, at least 90% of the problems we have had in the real world have been with multithreading and performance constraints. Static unit testing would never find these issues…run one at a time, these routines all worked perfectly, which is why the problems got past us in the first place. So, for the first week after the conference, we tried to expand the idea to a multithreaded consideration, where we could run multiple instances of unit tests simultaneously, to force multi user load conditions. This ultimately turned out to be quite a challenge, since working with the code at the very low (individual function) level we wanted to test meant creating global state variables which were global to the entire test framework, and having multiple instances of the test manipulating those state variables led to chaos. This feature will require more architecture to be viable, and has been postponed for now.
But in truth, it may be that that was yet another false precept. The problems that were created by heavy load were problems; the fact that they only manifested themselves under extreme circumstances that were impossible to reproduce on demand and almost impossible to monitor does not change the basic reality of the fact. And with the framework that we now had in place, we found we could begin to emulate some of those situations, by forcing invalid state conditions and file conditions in process. In some instances, this required refactoring some of the code to be test - aware, and actually interact with the test framework when being tested; but this has become a very powerful tool in the last two weeks.
Our framework as it stands now is bare bones, but very functional, and returning tons of value for the investment so far. We have made adjustments on almost every function we have tested…many in ways so subtle and small that they will never be noticed by an end user at all, but we know that the code is getting cleaner and tighter. In others, we have made startling discoveries of minor logic flaws that have never showed up in production, but only by the grace of God.
As a tester turned developer, this has now become almost addictive. I won’t claim that we write the tests first yet, but it is an iterative concurrent process where we create the base code, then begin to test the outer edges, and refine from there. We are currently creating a new utility for V5, our Vista release, that will export stand alone videos from test sessions with an encapsulated player that will emulate the integrated video replay from TestExplorer; this will be fully unit tested as it is developed. As we refactor for Vista compatibility, we are striving to unit test everything we touch, at least to a happy path level. We hope to hit all public functions heavily, and dive into the lower level stuff as needed. It takes time, but it is a very satisfying experience to finally see our aspirations take flight in this regard.
So, other than bragging, what is the point here? Everyone knows unit testing is good, nothing new there, so what’s the big deal? Well, there is a philosophy that says there is nothing new under the sun, and I think sometimes in IT that is a good thing for us to remember. So even from the very title of this entry, I have not tried to present anything new, but rather have reached back and made allusions to something very old, ancient mythology. Mythology is about life teaching lessons to men, and I seem to be getting a fair amount of that this year…and the biggest lesson I keep getting is, don’t give up. Success can be found in the ashes of failure for those who don’t quit. This week my unit testing is finally working well…and that is good enough for me. I will take solace in that and face tomorrows battles tomorrow.
David
