Sirius Ruminations

As months go, this has been a tough one.

As weeks go, this has been a good one.

I have, for quite some time now, been involved with a very large client who is creating a massive SOA system.  Concurrently, the organization is moving from being very decentralized, to being much more centralized.  The two are not unrelated – this kind of project requires massive amounts of coordination that the organization previously did not support.  Growing pains abound.

In an odd quirk of luck, I am on the performance test team here, although it is safe to say I do a LOT of cross functional work, especially in the area of automation.  For months now, the test team has labored to test a particular application in this system.  While in some circumstances, this application may perform well, in the broader context of the entire SOA system it really has to work in, it is a hurtin puppy.  In almost 10 years of consulting, testing, and automating, this is the most challenging, frustrating, irritating circumstance I have ever found myself in.  At conferences, over dinner, on phone calls, and via Skype, I have spoken at length with colleagues about the problems, challenges, personal issues, and frustrations.  My thanx to all of them for helping me keep my head screwed on straight, and especially to Michael Bolton, who literally spent over 12 hours on Skype with me on one of my absolute worst days, convincing me not to throw in the towel and walk away.  Were it not for them, I may not be writing this…

Three days ago was a watershed day.  After weeks of chasing down a particular defect that the performance automation team considered a showstopper for further automation, repeating results, gathering data, refining our understanding, analyzing myriad charts and graphs to help the dev team understand root cause, the dev team finally pronounced that their app was fine, and our scripts were breaking it.  (Much of this blog will read like a classic tale of every testing parable we all know…except this really just happened to me)  So, stepping up to that challenge, three days ago we turned off all of our automation tools, took the entire test team for one day (all 10 of us) and began manually pushing the application into a corner.  With 10 people in a couple of hours, we logged hundreds of errors…not hundreds of defects, mind you, many of the errors were the same defect over and over, but that was exactly our point as performance testers…it keeps breaking the same way over and over under incredibly light load.  Complete vindication.  Absolutely every single bad behavior we had documented in our automated tests came to the fore under this manual assault.  And with that, we were suddenly on the map.  Now not only did our bosses boss know what we were finding, but his bosses bosses boss did as well.  We were suddenly center stage in the spotlight…in a metal chair…at a barren table…in a room with no windows…you get the picture.

The dev team, not to be taken that easily, immediately proclaimed it was our test environment that was to blame.  (Never mind that we had spent millions building a test environment that much more closely matched the target real world deployment than what they were developing in).  So two days ago, we switched off of the performance test environment onto the functional test environment, and did the same thing all over.  Same results.

The dev teams next response was that our testing methodology was flawed, and we were purposely setting the application up for failure.  They informed us that they were going to do their own version of our manual assault, in their environment, using their test methodology, to prove that we were purposely breaking their baby to make them look bad.  This announcement was immediately followed by a request from our bosses bosses bosses boss, politely asking if we would be so kind as to monitor the developers testing process.  Ouch.  So yesterday the dev team spent several hours proving that in any environment, no matter how delicately you touch it, it simply falls over and cries.

By now, you have probably inferred (correctly) that the relationship between dev and test is strained.  That is a whole different article, but yes, it is.  But even given that, this is a MASSIVE success for the testing team.  I am a firm believer that the role of the test team is to provide valuable information to management to allow them to make informed decisions about releasing software.  Until three days ago, I felt like this train was running to the station, no matter what we said.  Now, we have incredible visibility, management is listening, and I am fairly certain that this application will undergo extensive adjusting before it sees the light of day.  It is a brave new world.  Next week, the offsite contract developers have been invited in to help us “understand” this behavior that now cannot be ignored any longer.  It is not an invitation they can decline.  This test team has turned a juggernaut running at full throttle 90 degrees off course…an absolute miracle in my experience…and ultimately, a very satisfying experience.  This application ultimately will have a very direct impact on the quality of life of a large group of people, people I actually care about personally.  It really needs to be good.  And I finally feel like I may have made a difference.

In our industry, on the blogs, in the newsgroups, at the conferences, in conversations, we constantly decry how bad things are, how much we are ignored, how futile our efforts often seem.  I am very happy to present what I believe, at least for now, is a true success story.  I would like to see more of these.  I think it would help raise the spirits of some of us, and under the heading of “You catch more flies with honey” may actually help improve our professional lot in life.  So if you have your own success story, I invite you to post it here as a comment to this blog.  Large or small, anything that says “Hey, testing made a positive difference here”, bring it on.

And now, I am going to go bask in the newfound glory of my fame…because I know by Monday it will all be gone again.
Happy testing.

Last week, I went to the Better Software Conference in Vegas.  I hate Vegas;  but that’s not what this is about, the title notwithstanding.

As many of my posts keep saying, I love going to conferences.  Conferences are breeding grounds for new ideas and critical conversations with people who are not already infected with the context of your current situation.  (You ever try to talk with someone about how you’re doing things, when they have been right there with you doing it the same way for just as long?  That conversation gets stale soon…)  So it follows, one down side to conferences is that there are so many people to talk to, and so many ideas, that you run out of time before you run out of conversation.

This was the first time I had been to the Better Software Conference;  and it’s content is slightly different than my norm, and it is far from where I live.  Because of that, most of the folks I normally see at conferences would not be there.  But this meant that I would have more time to spend with those who were.  One of those was Michael Bolton.

Now if you don’t know Michael, he is a very interesting person;  smart, very well read, and a deep philosopher of testing.  His insights can be keen, appropriately sharp and sarcastic, and always thought provoking…all good things, IMHO.  

So I went to see Michaels presentation, “First to Market or First to Fail”.  It was really an amazing presentation, covering much more territory than it’s name implies…and it made some wonderfully descriptive corollaries between technology and biology, and why things catch on and thrive (like the flu) and why they eventually don’t anymore (like smallpox).

Near the end of the presentation, we did an exercise where we considered a new and emerging technology, and made lists of how we think it might affect and change the balance of the status quo in the future.  The technology we all, as a class, chose, was the iPhone.  

As we talked during the afternoon, and over dinner, I related some of my personal biases on this particular issue.  As a father of two young daughters, I have watched amazed as the cell phone has radically changed the entire social landscape of teens in our culture.  While describing this to Michael, I made the comment “It’s absolutely viral!”…which, of course, brought the context of the conversation right back to the comparisons he had made in his presentation.

So for me, for Sirius SQA and TestExplorer, the question becomes “How do we become absolutely viral?”  As bad as it sounds, it is the equivalent of unbridled commercial success.  How do you mutate and adapt until you have become something that no one can do without.  That is the challenge for us.

If you have looked at TestExplorer, if you are a customer, if you just glanced at the website…we would love to hear from you.  Right here…publicly, freely…What caused you to develop an immunity or a resistance to our message and our products or services?  What would entice you and compel you to look again?  What would make it so good that you absolutely could not live without it?  Evolve or die…an age old lesson, and we are feeling rather evolutionary at the moment.

Sadly, Michael left the conference early, so I will have to bug him over the next few months as I go over his material from the conference proceedings and come up with new questions.  But I did find other new ideas to fill my time…that will be another blog for another day, though.

Until then, get your shots updated, and be careful as you approach that next newfangled piece of whiz bang technology…it could be contagious!!

Sincerely,

David

Last week, I was driving around, and my 7 year old daughter was in the back seat with me.  We were chatting as we drove along (her mind tends to wander and bounce around) when she suddenly made some comment about something she had observed.  I was immediately struck with the depth of insight her comment carried with it, and thought to myself, “Oh, I must write a blog about that – that is so relevant to what we do!”

I can’t remember what she said.

So why am I writing the blog anyway?  Because there is still a lot of lessons to be learned here.

First, if you’ll just trust me for a moment in the fact that whatever it was she did say was in fact startlingly profound, is that fact that you never know when these things will happen…when an observation, out of context, will cause an earth moving jolt of realization about how the universe hooked something else together, and suddenly you have a seemingly unrelated insight that is huge.  So since you never know when these things are coming, you have to be on the lookout for them at all times…and that is no small task.

Second, and a corollary to the first, they will often happen at a most inopportune time.  You will think to yourself, “Oh, this is so profound, I will certainly remember it, so there is no reason to stop the very important thing I am doing right now and deal with it.”  I could have jotted down whatever it was my daughter had said…but somehow, continuing to drive the car through downtown traffic seemed more important at the moment.

Third, no matter how good you are, it always helps if someone has your back.  I consider myself a fairly observant guy…I am a software tester, trained to spot all those little idiosyncrasies that the developers miss.  I take pride in being good at what I do, and work diligently to train my mind to pay attention to little details, incongruities, broken assumptions and premises, and all manner of minutiae that drive my friends insane.  And yet, I missed whatever it was she saw.

Fourth, there is no time machine.  It really bothers me at a personal level that my little girl said something that I considered very intelligent and important, and now, I cannot remember what it was.  I would give almost anything to be able to dial the clock back, sit in the seat again, and try to pay enough attention this time to not forget, to somehow find a way to engrave it in my memory so strongly that it could not escape…but that simply will never happen.  It is gone, and gone forever.

Now I rarely use the blog to blatantly hawk our own products, but all of these reasons are exactly why TestExplorer was created in the first place.  Because as software testers, especially as manual, exploratory, sapient testers, we deal with these issues constantly.  In the middle of something that is crucial and has a deadline of last week we will suddenly see something unrelated and think we will get back to it later and when the dust finally settles all we remember is that there was something we wanted to get back to later.  TestExplorer logs these things for us, retains the context, lets us go back in time and review them when it is convenient for us, and with more than enough detail to thoroughly investigate them.  So if, as you read this, little voices in your head were yelling “Yes! Yes!  That is me exactly!”, then I would encourage you to take a look at TestExplorer, and see if you think it can help you in this regard.  And if it can’t, I would encourage you to tell me why, so I can change that.

Beyond that, on the heels of Fathers Day, I would encourage those parents out there to take heart…your not the only one who forgot one of those little gems that pop out every now and then.  Enjoy them in the moment is all I can say.

A few weeks ago, at StarEast, I was talking with Antony Marcano and Rob Sabourin about agile processes, and asked a question that has become a perrennial favorite; Can you be agile if you are not creating and executing loads of automated unit tests?

The reason this has become a favorite question of mine is that it is very germane to us at Sirius SQA. We try to do things in a very agile way (note that I always use the lowercase version of the word), and we have tried on 3 separate occasions to implement a strong automated unit testing program, and failed. While it ultimately does not matter…our processes are what they are, no matter what label you do or do not apply to them…it has long bothered me that we were missing such a large piece of what we aspired to.

Sadly, after a bit of discussion, Antony and Rob both agreed that without the unit tests, it really isn’t agile…whatever it may be. And so we began to talk about why it had failed for us in the past. To put it in a nutshell very tightly, we do a lot of non-OOP coding; we have many libraries that are purely libraries of functions and procedures. This is problematic because most of the prepackaged frameworks for unit testing are very strongly aligned with OOP methodologies, if not downright reliant on them. So as we tried to implement these frameworks, we were continually frustrated by an inability to get to the level of detail we wanted to get to, and produce the results we needed. That, at this point in the conversation, was my perception of why things had gone wrong. Rob Sabourin began to challenge a few of my beliefs, and simply pointed out that while many of the pre-packaged frameworks were heavily OOP aligned, unit testing was nothing new, and that it sounded like we just needed to grow our own framework. With this as a starting point, the conversation meandered through the evening over a few drinks, and in the morning I was once again committed to making this work. And so I began to pick through the still smoldering ashes of my efforts from less than 6 months ago, culling those parts that seemed valuable, and purging those that did not. By the end of the day, I had a reasonable framework and a small hand full of tests.

Few things in life are certain, and in software testing, even fewer.  Like our modern political system, the community of software testers is fractured and splintered.  And while we may draw bold lines around big camps such as Context Driven or Factory, Manual or Automated, Exploratory or Scripted, within any of those major camps there is fierce competition.  This can leave us poor individual testers wondering if there is in fact nothing solid for us to hold onto.  But there is… 

Now those who know me know that I am a strong proponent of conferences.  In the first place, they are usually a lot of fun.  But beyond that, they are a unique opportunity to get together with some fabulously interesting poeple and talk about all manner of things.  For this reason, for me, the best part of the conference is normally what happens down in the bar after the days official events are done.  But there is one conference where this is not true. 

 The Conference of the Association for Software Testing is a unique event among conferences.  At most conferences, the rules of etiquette of an elevator apply…you go in, pick your spot, be quiet for the duration of the journey, mutter something polite as you leave, and barely remember anything about the whole experience 5 minutes later.  So what makes CAST different?  In a word…Debate.  You see, before you’re even allowed to take the stage at CAST as a presenter, you have to agree to a few rules, specifically, that as soon as you begin speaking, you are fair game for anyone in the audience.  And the audience members, they are encouraged to challenge you and engage you in professional, respectful, but quite possibly heated and passionate debate.   

So why would any presenter want to subject himself to that? Well, for those of us who would, it is because having your ideas challenged is how you grow intellectually.  Being forced to defend your ideas, your beliefs and precepts, makes you confront them, examine them and quite possibly address their flaws and weaknesses.  This, folks, is what conferences are for…CONFERRING!  For my money, a day in this kind of emotionally charged, intellectually raw petri dish of ideas slugging it out in a Darwinian fight to the finish is far more valuable than a day in a staied, stoic tutorial.  This is where ideas are born, formed, forged and hardened.  There are some other unique things about CAST…it is young, and small, and not as vendor centric as many other conferences.  I also count these as pluses…in an environment like this, if you don’t make some new friends, and actually spend some time getting to know them, it is because you were trying not to. 

Being young and small and not so vendor centric has it’s drawbacks as well though…you’re not as well known, advertising is expensive, good speakers can be hard to entice…and so getting a good crowd to come to such a conference can be a challenge. 

I think Jon Bach, the Conference Chair for this year, has done an awesome job of overcoming those challenges.  In fact, I think if you look at the list of presenters from last years archives and this years schedule, you will be very impressed.  But I saw Jon last week at StarEast, and in one of our conversations I heard him mention that overall attendance was still not at the level he wants to see.  So I am trying to help get the word out.  Because, you see, in addition to whatever else I may be in a professional or business sense, at my heart and soul, I am a tester.  I want to understand things;  I want there to be structure and order;  and yet I have an almost perverse sense of wonder and awe at the way the universe routinely dashes those concepts against the cliffs and boulders of ambiguity and misunderstanding.  And the only reasonable thing I can do in the face of that is to try harder to understand more…to debate, to listen, to speak, to broaden my view of the universe, and hopefully maybe someone else’s as well.  To have…and to keep…an inquisitive mind. 

If this is how you think…if this is how you feel inside…well, then you are likely a good tester.  And that…THAT…is the solid thing for us to hold onto.  That above all else, software testing needs an inquisitive mind, and an inquisitive mind needs to be challenged and fed.  No matter what school you claim, no matter what method you use, no matter whose shirt you wear.  If you agree with this, then CAST has something for you.  And better yet, CAST offers something to you…a chance to be a part of something special at it’s inception;  to contribute in a unique way.  A chance to help it grow out of it’s infancy. If you have read this far, then you are at least cuirous about what kind of conference I could be so passionate about.  And so I would simply invite you to come and see for yourself.  Bring a friend.  Tell others on your team about it…heck, bring your whole team!!  Tell your online friends to drop by the site, blog about it, help get the word out.  It won’t always be this small;  it won’t always be this intimate;  it won’t always be this cheap.   

I was there last year, and I thought it was absolutely amazing.  In the middle of a presentation James Bach was giving on tester certification, someone with a different viewpoint rebutted what James was saying.  What followed was an intellectual exchange that puts any presidential debate you have ever seen to shame.  (By the way, each presentation has a facilitator to help manage the process)  As the ideas ebbed and flowed, it became apparent that this conversation was not going to conclude in its allotted time.  In accordance with their stated process, conference organizers quickly checked in with follow on speakers for that day, and schedules were adjusted and accommodations made so that the debate could continue on, and everyone with an opinion was given ample opportunity to express themselves.   

I will be there again this year, guaranteed. Consider it CAST in stone.

Star light, Star bright
First star I see tonight
I wish I may, I wish I might
Get some decent sleep tonight.

I just got back from STAR East late last night, and I am exhuasted yet energized, much like you might feel after a good hard workout. It was, as always, a wonderful time to reconnect with old friends, meet new friends, eat, drink and be merry.This year, I got to meet Antony Marcano, founder of Testing Reflections. He is an all around great guy, and I came away from our many conversations with a renewed dedication to active blogging again, and a fresh perspective on how my company can better deploy unit testing that has true value for us on our own applications, something that has long been a rough spot for me. I also got to meet Shrini Kulkarni, an active member of the Yahoo Software Testing Group. It was great to be able to put a face to a voice in the blogosphere, and in the few brief conversations we had I got the best new idea from the entire conference from Shrini…there will be more to come on this as I push from idea to implementation, but this is the kind of stuff that makes conferences great.

Good morning, class. Today, we are going to learn a new word – Heuristication. Heuristication is a state of sophistication in dealing with and applying Heuristics as an element in an overall test methodology.What does Heuristication look like? How can you identify it? Well, in order to answer those questions, first we have to understand what heuristics themselves are.

A heuristic is a fallible method for solving a problem. “Fallible?”, you may ask, “Why would I want to use something that’s fallible?” Well, lets look at a slightly different wording of that same idea…a heuristic is a general rule of thumb that works most of the time. Now we can begin to see the value. It is general, and works most of the time, so it can provide a solution quickly, without thinking very hard or very long about it.

Here is an example of a good heuristic…15% is a good tip for your waiter. Now, a waiter may immediately argue that 18% is more appropriate, but the point is to solve a problem quickly without having to think too hard, and 15% is much easier math. On the other hand, if you are wining and dining a major client in the fanciest restaurant in town, 15% may not be even close to appropriate. So in that case the rule fails. So it is a general rule that supplies a fast easy solution to the problem most of the time, but it may not provide the right solution all of the time.

So if that is a heuristic, then what is heuristication?

The first symptom of heuristication in a tester is the displayed ability to draw from a long list of heuristics they carry around in their head. Depending on the tester, they may display this very emphatically as they test, for example “I can’t reproduce that bug right now, but the Dead Bee heuristic tells me I can’t just ignore it until I have strong evidence that it has been repaired.” On the other hand, they may apply heuristics in a way that is not obvious to the observer, but when questioned about their tactics, can immediately justify them. For example, when given a product, with little experience with that product, and scant documentation, and being asked to ascertain it’s suitability for release, a tester may embark on a series of seemingly random activities; but when you ask him why he is doing what he is doing, he will reply “Well, I am applying a heuristic that says if I validate every functional claim made in the marketing material and on the web site, since I have no other better source of information to tell me what the application should do, then it is suitable for release to end users, who also will have no other more specific guidance.”

That is a decent level of displayed usage, but the term Heuristication also implies the ability to synthesize new heuristics out of your own experiences. A great example of this is the The Gilbertian Squirrel Dodge, a heuristic of mine that reminds me not to react out of panic. This heuristic was created when a squirrel ran right into my foot while trying to run away from me. This ability to create new heuristics is vital, because a heuristic needs to be a personal thing. It should not be just a checklist of things to do, but a collection of ideas, each of which embodies and evokes strong mental images and emotional and cognitive reactions. These additional elements help anchor the heuristic in our mind, and provide strong frames of references for it’s use, as well as cognitive mappings which should serve to bring the heuristic to our attention at the time when it’s use is appropriate. But they also help to transfer the knowledge to others. Just like stories have been used for centuries to transfer information, such as history and lineage, because they put that information in a frame of reference that has emotion and entertainment entertwined, so a good heuristic, when shared, will pass along a good idea, but it will also ensure the person hearing it remembers and understands by virtue of the way the idea is presented. So the ability to create strong heuristics is a requirement for Heuristication.

But we must not forget that a heuristic is a fallible method for solving a problem, and that possible fallibility must be evaluated before the heuristic is applied. Just as under tipping the waiter in the fancy restaurant could be disastrous for the business deal you are trying to win, applying any heuristic at the wrong time could create more problems than it solves. So another element of Heuristication is the ability to critically consider a heuristic in the context of it’s proposed usage, and determine whether or not it truly applies. This takes practice and experience…Heuristication cannot be purchased in a bottle. Like any skill, it is something that evolves from work. A heuristic encapsulates a problem, a frame of reference, and a plausible solution, all in one package. Before implementing the solution, it is imperative to insure that the problem and frame of reference are properly applicable. James Bach’s commented to me after reading the original post, “This is what I think elevates the phrase “heuristic reasoning” above the everyday idea of rules of thumb and guidelines.”

Finally, Heuristication requires a knowledge of other well founded heuristics, in addition to your own. Not only should you be able to create, and pass along, heuristics, but you should also be able to receive and assimilate them. Ultimately, groups of heuristicated testers will develop a common lexicon of heuristics that can be used in any testing environment, with any other sufficiently heuristicated testers, whether they have worked together in similar situations before or not. Much like the old joke where the office staff had been together so long that rather than telling entire jokes, they simply recited the punch line, and everyone laughes, a tester, when with a group of other heuristicated testers, pondering a test problem, should be able to simply say “Dead Bee”, or “ Remember the Squirrels”, and mental light bulbs should light up all over the room, because the heuristic carries such a strong mental model with it that everyone immediately sees and understands the relevance, the point being made, and the implications for what needs to happen next. There is very little debate…much of the debate has already occurred long ago, and the heuristic encapsulates that debate, and much of the proving out of the validity of the concept being put forth. By bringing us to a known starting point, at the very heart of the debate already, the heuristic becomes a powerful, time saving tool.

So when we Pirate the Cicada Killer Wasp
client that we didn’t see coming, and begin to Paint The Room, only to ultimately Remember The Squirrels, perhaps we will have better luck keeping in mind that FCC Cuts Vids, rather than Kid Tested, Mother Approved, SanFrancisco Depot, Krushpic Shtemple (A Romanian hockey all-star), and Feds Fescura! (A popular workers rights chant in ancient Rome).

Because the point, after all, is to quickly find a solution to the current problem.

Sincerely,
David

(Many thanks to James Bach and Mike Kelly for their comments and suggestions)

Another chapter in the continuing saga of my personal quest for good heuristics.

Earlier this summer, I had the honor of attending WHET, the Workshop on Heuristic and Exploratory Techniques. Leaving the workshop one afternoon, lost in deep thought over the day’s events, I was strolling down the sidewalk toward my car. In the grassy area off to my left, I was vaguely aware of some squirrels playing away. As I approached, they stood up and froze, suddenly leery of the giant monster who had just invaded their space. As I continued to walk along the sidewalk, at complete right angles to the squirrels, staring at the ground in front of me lost in thought, taking no notice of them whatsoever, one of them finally could not take the pressure any longer, and he bolted…unfortunately, he chose his escape route very poorly, and before I realized what he was doing, the poor little guy ran headfirst, full speed, directly into the side of my foot. Right into my foot!! I’m not kidding…he hit me so hard it hurt. He bounced off of my foot, rolled over, jumped up and picked a new direction to run. I was amazed.

And like the starburst that must have gone off in his poor little head, I suddenly saw clearly how this behavior gets played out all the time. And thus was borne “The Gilbertian Squirrel Dodge”, that frantic avoidance of danger not real, and the latest in my growing list of heuristics. (Side bar: At WHET, one of the amusing things going on during the week was a casual attempt to invent new words or phrases and attach your name to them, such as “The Bachtonian Transpection”, which I may use as a later topic if James does not beat me to it.) So what does “The Gilbertian Squirrel Dodge” help me to remember? That when things seem to be going very badly, and I am filled with a sense of panic and an overriding urge to do something, anything, take some action, what I should probably do…is nothing. Sit still. The danger…or at least the sense of impending doom…will pass, and I will be more able, in just a very little while, to rationally evaluate the situation and take proper action. That under those conditions, taking action may in fact cause harm…irreparable harm. While waiting may allow something bad to continue happening for a time, that is usually manageable and recoverable.

Okay, so that sounds all well and good, but how does it play in the real world? You’d be amazed. Since WHET, I have quietly been keeping track of how many times I see this principle violated, and the pain it causes. In our rush to get our weekly work done so we can get to the weekend, in our rush to make the project deadline, in our rush to be first to market…over and over, we become reactionary in really bad ways. I have lots of stories I could tell, but let me just give you a small example that has consumed the last two days of my life.

A few days ago, Erwin Trier, a friend and long time beta tester for our products, shot me an email telling me that our automatic update was not working for him. We checked, and of course he was right. The code for the update process had been stable, static, and functioning perfectly well over a year. But faced with the fact that this was a live defect already in production that potentially affected every single user of our product, and although did no harm, was incredibly embarrassing, we sprang to action. We quickly came to understand that the problem was related to the size of the download file for upgrades, which over time had grown. We spent two days refactoring code looking for the smoking gun of some hard coded size limitation that someone had thought would never get exceeded. We were also working with our server side webmasters and ISP team to see if the problem was on their side. This morning, I finally got an email…turns out there was a size cap for outbound anonymous FTP.

In retrospect, it is easy, but in the heat of battle, it is not. All we had to do was send our customers an email with a direct download link for the upgrade, and an apology for the inconvenience. Embarrassing, yes, but problem solved and all the time in the world to fix the issue. Instead, we did a very risky thing…we messed with code that we had proven out already. Because I forgot about the squirrels. And while in the end, the results are not devastating to us, we still have to mop up, figure out whether to throw all the refactoring away and stick with the production base, or finish what we’ve started, and then build in safeguards to prevent misconfigured FTP servers from biting us again. All told, this will cost me over a week of development time, and for a company like ours, that is real pain.

So the next time you’re sitting at your desk thinking “Crap, what the hell is going on here? I don’t get it. I’ve got to do something!!” – DON’T!!!

Remember the squirrels.

Sincerely,
David

Over the past few weeks, I have been working with James Bach on my BCRIT certification. Through that process, I have identified my use of heuristics as a good area for potential growth, and so I try to spend some time each day actively thinking about this. For those not familiar, a heuristic is a key word or phrase that encapsulates some larger idea, and triggers you to apply some strategy or behavior in your testing. You try to pick heuristics that will be easy for you to remember, something personal or interesting. Something amusing is always a good choice, James’ “Dead Bee” heuristic being a good example.So over the same time period, my daughter Sarah and I have been doing some reading and study of satire, and it’s use in comics and cartoons. We have looked at the work of Gary Larson (The Far Side) and Chas Addams (The Addams Family), and their use of animals and human caricatures to point out incongruent behavior in society. So the other day, Sarah brings the Sunday funnies to show me this cartoon. (The cartoon was Pearls Before Swine, I don’t know the authors name, and the panel ran several weeks ago. That is about the best I can do for giving proper credit) The cartoon depicted two kids discussing Ol’ Yeller, one who had seen it several times telling the other who had never heard of it what it was all about, and that it was showing at the local theatre. After hearing the story of how the young boy eventually has to put his own dog down, he says “Wow, I bet everyone in the theatre was bawlin about that”, to which his friend replies “Yeah, what other reaction could you possibly have?”

The last panel in the cartoon was the inside of the theatre, with everyone sitting around crying and sniffling…everyone, that is, except the two alligators in the middle of the theatre, who were jumping up and down cheering and yelling “Eat Da Dog!! Eat Da Dog!!” (We have two dogs, two cats, a snake, several fish, and a collection of horses we are part owner of on any given day, so please, no emails about what an animal hater I must be)

The alligators’ behavior is completely inappropriate…unless, of course, you’re an alligator. Because what is appropriate and inappropriate is very relevant to context. And sometimes, forcibly changing our context and perspective can help us discover things we did not see before. So the next time I am faced with a challenging testing problem, and I can’t see a solution, I am going to think to myself, “How can I eat dis dog?”

Now THAT’S an interesting heuristic!

Sincerely,
David

Recently, three unrelated events have given birth to a new idea in my head, and I wanted to share it with you. So, to properly set up the background, the three unrelated events:  At the recent CAST, James Bach presented a keynote, “Against Certification”. During that presentation, one of the things he did was to review some of the “Body of Knowledge” documentation upon which such certifications were based. As he reviewed this documentation, one of the things I was struck by was an underlying pattern of motive driving much of the documentation. That motive was the ability to predict, manage, and control the SDLC. Much of this “Body of Knowledge” for test certification was obviously written by managers, not testers.

In a recent blog, Michael Bolton talked about estimation and prediction for rapid testing efforts in projects. I thought it was a very good blog, and agreed with everything in it, but in the end it left me with a sense of a lack of resolution, and that bothers me.

I live in the Tampa Bay, Florida area. We recently had our first storm of the hurricane season. After the last few years, we who live here watch keenly the daily, and sometimes hourly, updates to the tracking models and projected paths of these storms.

So what do all of these events have in common? They all deal with trying to predict the progress and ultimate resolution of a process that is full of risk and ambiguity. But that’s not really the interesting part. What I find fascinating is that only one of them has a universally accepted solution…the hurricane trackers.

Now, when I say universally accepted, make no mistake…not everyone likes the model that is used. Many people will tell you they think weathermen may as well throw darts at the map as do what they do. What I mean is that it is used consistently across the industry, with very, Very, VERY little variation from one specific implementation to the other. It is very flawed, but it is good enough that any reasonable professional in that industry understands it is a tool, and when used properly, it is a tool that provides good value, and so they use it appropriately. That fascinates me…why can’t we reach that kind of consensus about estimating the development and testing process?

After thinking about this for awhile, I believe there are some concrete elements in the hurricane tracking models that are not present in any estimation model I have ever seen in software; and I believe it is the existence of these elements that make that model acceptable. Lets look at some of them.

First, hurricane tracking models always start from the current position of the storm. While they track the history of the storm for historical purposes, where the storm went yesterday has little to no bearing on where it will go later today, other than having put it into it’s current place. And more to the point, where we expected, or planned, for it to go yesterday, last week, last month, has absolutely NO bearing on where it is now, or where we expect, or plan, for it to go tomorrow. It does not even enter into the conversation. So the plan gets reset based on reality at a given time. No one holds onto the past.

From where it is today, the second element of the model is a path of where we expect it to go, and that path ultimately leads to an expected location and time for landfall, the ultimate result of tracking the storm. That path is based on many factors…how fast the storm is making progress, which direction it is going, where the prevailing winds are, what the temperature of the water is , how much time there is between now and landfall, etc. All of those factors are highly variable, and largely out of control of the hurricane trackers. This is the single element of the model that most resembles the current models we use for estimating development and testing efforts…a straight line, or at best a gantt chart, plotted against a calendar, with the end date carved in stone. Interestingly enough, this element of the model is the single element that most forecasters trust the least, and they all say so loudly.

Which brings us to the third element of the model, and I think this is the biggie. The Cone of Uncertainty. Starting at the point where the storm is now, following roughly along the predicted path of the storm, is an ever expanding cone. This cone represents where, given all the information currently available, the storm MAY go if something in the model changes. I have NEVER seen this in any model of software planning. Yet this is the part of the model that forecasters talk about the most, and encourage everyone to pay the most attention to. This is the part of that model that takes into account Risk and Ambiguity.

I think these elements are powerful in terms of creating a model that deals largely in truth, acknowledges uncertainty, visually depicts it’s status and communicates clearly, and updates rapidly and understandably. And I think it is a model we could emulate to predict, in a reasonable and acceptable manner, the software testing process, even when using rapid / exploratory testing.

It is the test teams job to provide information to management to better enable them to make decisions about releasing software. It is managements job to keep the software development and creation process on track, and release software by balancing a vast array of considerations and risks. While neither of us can know everything, or plan everything, in advance, the test team can also not simply throw up our hands and say “We cannot provide any answer to the questions your asking.” when the questions they are asking are perfectly valid. At least not if we want to be perceived as a team player adding value to the process. But a model such as this, that plots not only a course, but a reasonable expectation of possible variation around that course, and updates itself regularly, would let management see not only where the project is, and where it is headed, but it would also better help them understand the costs and risks with getting it back on track versus accepting the new point of impact. I believe that would be a far better answer than “We will test all we can until the day you shove it out the door.”

I am so intrigued by this, that I am going to begin working on such a model. I will be happy to publish both the ongoing work, and the final results, online. If you would like to collaborate on this with me, drop me a line at dgilbert@sirius-sqa.com.

In the meantime, batton down the hatches, and good luck weathering the storms of your own testing efforts.