We are quite familiar with the eternal debate and endless list of opinions about what constitutes quality.  We stated right up front, we were not trying to define quality for anyone but ourselves.  But, until further evolution, (which is quite possible) this is our definition.

We spent a fair amount of the morning discussing our own opinions and knowledge and scouring the internet.  Ultimately, we came away with 4 statements that we felt framed the starting basis for our discussion. They were –

1. Quality is value to some person.
2. Quality is a relationship between a user and the product.
3. Quality is fitness for use.
4. Quality is conformance to requirements.

From these four statements, we began to discuss the specific business goals of Raymond James, our corporate culture, our current processes, and the stated goals and vision of the business.  We talked about how we very specifically wanted any formal statement we made not to be just words, but to be an actionable statement that drove the behavior of our SQA organization.  We worked diligently to make that a reality.  Ultimately, this is what we came up with:

What is software quality?
Confidence that the product provides value to the business by meeting the needs of its users in a cost effective and efficient manner.

How is software quality achieved?
Confidence – Defined measures and metrics that are used to determine success of meeting the user needs

that the software solutions provides value to the business — Each business area must interpret and articulate the value that the product must/will provide based on the understood user needs

by meeting the needs of its users — The users’ needs must be defined in some way.  We do this by gathering requirements, doing customer surveys, interviewing, etc.

in a cost effective — Making the best choices based on the value to the business, working within the project management triangle ? time, scope, cost  (faster, better, cheaper)

and efficient manner — Using appropriate process to create products with value to the business and long term sustainability (maintainability, scalability, security)

I am very proud of what we have done.  I feel it lends real credence to the rest of what we claim we want to do.  It is my manager’s stated goal for Raymond James to be the premiere software testing organization in central Florida in the near future.  We want highly skilled testers to WANT to come and work with us.  And doing something this simple, yet this important, is a big step in the right direction, in my opinion.

I will try to get permission to post the entire vision statement in a follow up blog, when it is all finished.  In the meantime, I welcome any comments on our definition.

I would also like to hear anyone else’s experience in defining quality, and how that definition has stood the test of time.

David

become aware of something both amusing and enlightening.  In our parking lot, there are lights atop tall poles; nothing unusual about that.  But as the timing of my comings and goings varied around an

average, I noticed that the parking lot does not fill up from closest to the building outwards.  There are key positions that are left vacant, even when they are close to the building.  These spaces

are under, or very near, the light posts.  Now, if you live near a body of salt water, you know where this is going.  You see, salt water birds are, generally speaking, large…and messy…and uncouth…  so it

makes sense that the spot right under the lightpost would be NOT desired…unless for some reason you WANT your car covered in bird poop.

But as the days rolled on, and I continued to observe this, I realized something a bit surprising.  The spot UNDER the lightpost was not the worst.  The worst was the spot next to that…and always towards the direction of the bay.  Apparently, the birds go to the water to get food…then come sit on the lightpost to digest it…then, as they begin to fly away, some muscular reflex discharges

the excess.  But by then they are a few flaps into flight, headed back for more food, so more often than not, the target zone is the space adjacent to the spot under the lightpost…but not always.  So both

spots are to be avoided.

So the lessons I learn here are that problems don’t always show up near their source.  They also don’t always show up where we expect them.  And systems are way more interdependent than we normally believe.

I love learning lessons about software testing from nature;  in the first place, I love synchronicity in the universe, and I think we mimic nature more than we admit.  We create organic systems by design,

and even inject mistakes in them that have organic properties.  If you are a tester tasked with finding those mistakes, this is a valuable insight.  Binary thinking is NOT the best approach to software testing.

So observations from the world around me that help me understand how these complex systems we design may go awry are a favorite source of inspiration.

So this is all well and good…until the other day, when I walked out into the parking lot, and as I walked by a parking space, predictably vacant, something caught my eye.  It was a dead fish, lying in the

parking spot.  That was unexpected.  I immediately thought, “What the hell is that doing here?”  A mile from the water, yet there it is…a fish out of water.  Ok, so obviously, some bird dragged it here and

then dropped it.  But then what?  Flew a mile back to the bay to get another one, rather than just jumping off the lightpost and getting this one back?  Why?  This is the heart of exploratory testing…the

poop in the parking lot is reasonable and may have even been predictable…but this dead fish needs more data.  When the results of your last test drive the design of your next test, you are doing

exploratory testing.  And it is the most natural behavior on the face of the planet.

And while you may say, “Why is it so important?”, ask yourself this…would you rather come out and find a few splats of bird poo on you windshield…or a dead fish stuck in your windshield wiper?

See…now there’s a lesson about the value of risk based testing!

As you come and go, remember that software is designed by people for people, and while it is ultimately just lots of 1’s and 0’s chasing themselves around, the behavior it will display will at times be strikingly organic.  Learn lessons from the world around you when expectations get shaken up a bit;  question why, and try to learn what the insight says on a larger level, and how that may apply to these artificial universes we create.  You will be a better software tester for it.

And don’t park under the lightpost…

David

As this was going on, her team decided to use her in a testers role, and tasked her with creating and executing some test cases.

A few days later, some of us were having a working lunch, and Liz was commenting on the fact that she was executing some tests that she had created, and that it was kinda boring. Picking up on this, I began to go down a fairly standard path of context driven type questions. Why was she executing the tests? Did she understand that in the creation of the tests, the testing had already occurred? So what are you really testing?

“That’s a good question.” she responded. (Note to self: that phrase is a strong indicator that something great is about to happen) “I am really checking to see if I would write it up the same now as I did originally; so, I guess I am testing my ability to be a good tester.”

Wow! That is without question the best defense I have ever heard for doing a test you have already done, when you have no reasonable expectation of anything different happening. And it is the ONLY time in my career anyone answered a question about testing with an answer about thinking, specifically about their own ability to think critically about their own work.

This happened the week before I left to start my new position, so I don’t get to work formally with Liz now, and that saddens me. But I am not far, we all stay in touch, and in fact a bunch of us just had lunch together; and I hope Liz continues to grow and evolve as a tester, and I hope I get to occasionally help.

The software testing community needs more people like Liz…sharp, inquisitive, aggressive, critical but not mean spirited. I love finding people like that, and I love having the chance to work with them. And now, thanx to Liz, I have a new way of seeing value in something that I used to consider fairly wasteful.

David

Last week was mostly corporate orientation, welcome aboard, meet the team, etc.  As such, they have already proven themselves to be a great place to be; the corporate culture is warm and friendly, and no one is walking around the hallways shaking their head and grumbling to themselves.  But this is Monday, and I expect that this week, the gloves are off, I am an employee, and there is an expectation that I will be productive, and now.

Now for those who have never been an independent consultant, it may be difficult to understand, but for those who have, the question is, “Dude, WHY?!?!”  Having once walked away from the myth of corporate security, why on earth would you go back and be a slave to the man, having tasted the sweet freedom of independence?

To truly understand that, let me explain the process of how I came to be here.  It all began with an email from Michael Bolton;  seems a manager had seen he and James Bach at StarEast, and asked if he knew anyone who shared their viewpoints of software testing and may be interested in working for a company in the Tampa area.  As luck would have it, I live there, so he forwarded the email to me and suggested I call them.

Over the course of a few conversations with a few nice people there, it became apparent to all of us that I did not fit the cookie cutter mold for what they were looking for.  I don’t fit anyone’s cookie cutter mold…this is why I have not had a real job in over 12 years.  But here is where it gets interesting.  Rather than thanking me for my time and telling me I did not fit the mold…they went off and made a new mold, based specifically on our conversations.

Now, a company who will do that is serious about bucking the status quo to reach their goals.  A company who will do that is serious about getting what they want, not through the application of brute force, but by being smart, and thinking outside the box, and compromising when a really good solution is not necessarily what they thought it was gonna be.  THAT got my attention.

For six years I have worked at a large government contract, and while we have done some good work, the culture there is simply too large, too entrenched, too spread out and decentralized, and too political to make any large and meaningful changes.  On the other hand, this new position is ripe with opportunity to do really meaningful work, and I find myself at a level where I can be heard and effective, with the apparent backing of a group of folks we have already demonstrated a serious desire to achieve their goals above and beyond all other considerations.  I am a very, VERY goal driven person…so this resonates strongly with me.

So as I sit here looking out of my cube, I am incredibly excited at the possibilities that lie in front of me.  I am thankful for good friends who I have had the honor and opportunity to work with over the years who respect me as much as I respect them, and recommended me for this.  I am deeply honored that a lifetimes work, including a lot of swimming upstream, has finally gotten me to a place where that contrarian but tenaciously grounded viewpoint is valued and desired, and can be put to very good use.  And I am enthusiastic and ready to get started.

David

Anyway, here is my problem…every 3-5 days, performance on my machines gets really sluggish.  A quick check shows that about 90% of my 4 Gigs…yeah, FOUR GIGS…of memory are actively in use.  I then proceed to shut down every application running.  Nothing gets better.  Most times, by the time I notice the degraded performance, I cannot even get all the apps to shut down, and am forced to reboot…Hard.

Now I have researched this, and I understand the official position that Win7 actively seeks to not waste memory…if any memory is free, it will use it for cache or some such, always seeking to improve your performance.  Free memory is wasted memory.  And in theory, I agree with that…but when my computer consistently slows to a crawl every 3-5 days, then the theory and the reality are in conflict.

So it happened to me again today.  I did manage to get all the apps closed down this time, and got the resource manager open.  This is how things looked.

My Resource Manager with NO apps running

By the way, I shut down all the apps and then left the machine alone for over three hours to let it try and fix itself.  I find it really difficult to believe that with no apps running (and I know there are things running, but no ‘User Interface” apps, ok?) that Win7 needs an entire 3.5 gigs of memory to do something, anything, in the background.  And right after I saved this screenshot, everything locked up, and I had to reboot again.  When I tried to do a soft reboot, Windows refused…it said a background program would not shutdown.  The offending background program?  Program Manager.

Also, I have checked the top nine memory consumers in the above list…Notice, no matter how I sort them, Committed, Private, Working Set, or Shareable, the order may shift around, but these are always the top 9.  What do we have there?  5 SvcHost.exe, which translates into an application or group of applications that is explicitly running in such a way as to avoid being identified…a haven for a litany of crap windows keeps dumping on us as well as viruses, but because the mechanism is valid, it is nearly impossible for Joe Average to figure out and deal with what highly obfuscated stuff may be encapsulated therein.  Ekrn is my virus scanner, something I simply won’t be turning off, largely in part to the previous comment.  DWM, that part of windows that is responsible for all the visual magic such as Aero, task bar preview, and the Roll-O-Dex app switcher;  sadly, turning all that crap off will do next to nothing to decrease the actuall resources consumed by DWM.  SearchIndexer, which as much as I may not like it being there, and taking lots of resources, is the only practical solution I currently know to 1/2 TerraByte hard drives.  And finally PerfMon, which I had to run to get to Resource Manager to get this screenshot…Windows telling me how bad it is at telling me how bad it is.

So to summarize, that is Windows, Windows, Windows, Windows, My Virus Scanner, Windows, Windows, Windows and Windows.  Hmmm…

Okay, enough of that.  I don’t wanna complain…I wanna solution.  So, if you have any supporting stories that may shed light on my problem…if you have fixed such a problem on your own…or if you just wanna tell me I am full of crap…let me know what you think.  Thanx.

David

Now I have been tasked with creating the test plan…THE Test Plan…(Trumpets sound, the clouds part, a golden chalice descends from heaven containing a scroll upon which is written The Test Plan…you get the picture)… for a new upcoming project. And, since it is a large organization with lots of projects going on, and they like for their official documentation to have a consistent look and feel, there is a template for the test plan. Okay, no problem with that. So I go open the template to start crafting the test plan. The template…the empty structure, devoid of any meaningful input at all…is already 28 pages long. My response to this was not positive.

And to make matters worse, I really do not have much good information to put into the test plan, because, primarily, communication and collaboration among the teams is not strong right now. So I am frustrated that creating a test plan is deemed the highest priority at this point.

It is these kinds of situations that caused me, some time back, to come up with the theorem of the well formed test plan.

The amount of formatting and verbosity applied to a test plan is inversely proportional to the amount of actual good testing represented by such plan.

David

Last night, the Rebel Alliance gathered at StarEast. The original plan was to gather at Cafe TuTu Tango, a local watering hole that would have done the canteena scene from StarWars proud. But then, as happens in these epic stories, a stroke of unbelievable luck crossed paths with a stroke of genius. In a deus ex machina that even Shakespeare would be jealous of, one of the alliance members got comped into a massive suite; really, the official max occupancy for this room was 78. Why did this happen? I am not really sure…the story I heard was something along the lines of “The hotel just wanted to give him the room” But our plans instantly shifted, and we gathered in his suite, stocked the kitchen with beer and soda and snacks, and ordered Thai food for dinner.

After some general socializing, the lightning talks began. I must say, lightning talks are a marvelous icebreaker. The AA version of getting to know each other — “Hi, my name is David, and I am a software tester” — is feeble. But spend a couple hours listening to fast paced 5 minute talks about something someone is interested in, and then participate in the Q&A afterward, and in a couple hours, you really start to get to know people. I especially liked Lanette Creamer’s presentation, “Herding Cats”, where she used her experiences getting her collection of cats to live happily together as a metaphor for her professional passion, which is collaborative testing and development. Since my own home is somewhat like Noah’s ark, the metaphor resonated strongly with me, and I immediately felt like she and I could find so many things to talk about, rather than just how to write good tests.

She later proved she could not only talk the talk, but also walk the walk, as she paired up with Shmuel Gershon to work through The Dice Game, being proctored by Michael Bolton. Their ability to work very well together was noted as being exemplary of how such things should really be done.
Shmuel continued to work with Michael throughout the evening, sharpening his skills as Michael continued to challenge him more and more. His enthusiasm for learning, and meeting new people, and sucking every bit of goodness out of the evening that he could, was infectious. Kudos to Intel in Israel for believing in him, that he could come all the way from there to here, and actually find and bring back something of value, not only for himself personally, but for them as well. Trust me, he was successful…and he has also distinguished himself as a young tester of note.

It was these kinds of experiences that made the evening special. The room was just the right size, as was the crowd, thanx to Matt for being wise enough to cap attendance. Through the evening, people gathered in corners and at coffee tables, in small groups of a few to a dozen; they would talk for half an hour or so, then get a drink or snack, and find a new group. Through the evening, I was a part of probably a dozen or so different conversations. There was no pressure to watch the hall for someone else you might feel compelled to schmooze with, no other schedule to be mindful of, no other distraction; but there was plenty to keep your interest — no one was looking at the clock and wondering when it was okay to leave. It was a marvelous evening.
And lest you believe that I am just pimping my own opinionated view, I can tell you that based on the collective experiences, conversations, blogs, tweets and emails of the last 48 hours, as I write this I know of at least two other large national conventions which are suddenly putting together events modeled directly on what happened last night. I hope to be, and present, at one if not both of them. And if I am fortunate enough to make that, I truly look forward to gathering with another Rebel Alliance group.

If you are headed for a conference soon, jump on this bandwagon. Get in touch with someone from the original Alliance, and if the conference you are going to is not doing something like this, do your own…that is the Rebel part. Even if they are, still consider doing your own…again, be a Rebel. It will make a difference, trust me, I know. I Love conferences. I go to them as often as I can…but this…this was special.

Long live the Rebellion.

David (Ben Kenobi) Gilbert
Charter member, Master Brewer, Rebel Alliance

Ultimately, I have to justify the expenditure for going to a conference with some practical business benefit. And in this business climate, networking is always one possible benefit. But large conferences can be a double edged sword in this regard. While there are tons of folks there to meet, it can be difficult to meet them. And, there are tons of other folks there meeting them, so a week later, will they even remember you?
And if you have a network of colleagues you reconnect with at conferences, how do you manage to make new contacts as well?

Matt Heusser has come up with a wonderful idea to help resolve this…something he has referred to as a conference within a conference. You can listen to a podcast about it here. He has issued invitations to a collection of folks he knows, and they know, to create a quick sub group, which he has dubbed The Rebel Alliance. The whole point of this is that when the conference is over, in the evenings, for one or two evenings, we are going to get together and have a casual mini-conference, with our own lightning talks and possible presentations. We are all going to blog about it and tweet about it.
Participation is being purposefully limited, so that it does not just become another huge gathering (such as the conference sponsored socials), but rather, everyone will actually get to know each other, and learn a bit about each other, so that when we leave, we will have a collection of a dozen or so business cards that is unique from the 200 or so that we grabbed with a quick “Hi, do you have a card?” For that dozen or so, we will have dinner, lightning talks, conversation afterward, some are heading to Disney after the conference, all of us are already communicating online, and that hopefully will continue. Everyone knows someone in the group who knows someone…no one in the group will be a complete unknown, so introductions will flow naturally.

This is a first, and I am not sure how it will evolve moving forward; but I would imagine that in future conferences, the Rebel Alliance formal attendees will be different, since the point is to create new business relations, not simply create a new clique. So if you are looking to a conference in the future, keep your ears and eyes open for Rebels, and join up at the conference you attend; make some real new connections, and have some fun at the same time. Eschew the ice breakers and conference sponsored fun night out, and embrace something that can really provide some return on your investment. Be a Rebel!

David

When I first started in software QA, I was one of the “We are the gatekeepers of quality!” kinds of people, considering myself the last vanguard before unleashing unclean software upon an unwitting world.  I eventually gave up that belief, and came to a place where I viewed testing as an information service, providing timely and accurate information to management concerning the quality of a piece of software.  Whether or not that software got released was a business decision, and the current state of quality was only one of many considerations that went into that decision.  Then I became a member of the Context Driven school of testing.  And then I came to work here.

Now, in the time that I have worked here, I have seen a lot of good work done.  I have done a lot of good exploratory testing, I have even gotten to do exploratory load testing, and I have stopped a lot of bad software from landing in users hands, which ultimately is job satisfaction of a kind for us testers.  But I have also seen at least three different projects go incredibly bad, and when a government project of the size and scope we work on here goes bad, it goes really BAD.  Estimates of the cost in dollars are millions…not quite a bank bailout, but still pretty serious coin.

As I have looked around and tried to understand what causes these collossal failures, I have come to the conclusion that it is not just bad testing.  It is not just bad development.  It is not just bad program management.  I think it is a series of events that is almost a natural outcome of organizations that become this large.  Projects become interdependant;  communication through the cloud of beauracracy is unclear;  people come and go;  with almost inexhuastable resources, and very lax accountability, mediocrity sets in;  and one day, you wake up with two projects that seem to work fine independantly, but won’t talk to each other, yet are interdependant in the real world.  Welcome to the brave new frontier of Systems instead of Applications.  Soon after, the finger pointing begins, everyone circles the wagons, and both projects proceed to fall out of the sky and become grey smoking holes in the ground.  The casualty count is high.

So given that, how do I try to make things better?  This question has led me to a new field of study, and a new era in my continued professional development;  I, and my company, are now beginning to get involved in Scope Management.  Scope managers are not common, per se, in American organizations, although they have been gaining prominance for the last decade in other places such as Finland, The Netherlands, and Australia.  A scope manager is essentially a mediator between the various consumers and producers of IT products in an organization.  The skills involved come from diverse set of more commonly understood professions;  Development, Project Management, Business Analyst, Customer Advocate, and Quality Assurance Engineer.  Strong analytical skills as well as strong communication skills are essential.  If you have been an independent consultant for any amount of time and with some measure of success, you likely have all of these skills.

So why do I think a scope manager can make a difference, when a tester cannot?  Two reasons;  First, they answer a different set of questions, e.g. “How do we all, all stakeholders, define this project”  and “How do we monitor the evolution of this project and keep our discreet definitions in synch?”  Second, they function at a much higher level, where the input carries more weight, and is less likely to get lost in the local politics.  A scope manager never blames anyone for anything…he simply states truth and reality, and presents options;  management then makes choices.

Now, some will say, this is a very document heavy, process heavy thing; you have sold your soul and are no longer context driven;  to them I would say, you are partially right.  This can be, and likely will be, relatively process and document heavy.  But these last four years have taught me that that is simply the reality of the world, in some contexts…see, still context driven.  Context driven does not say document heavy and process heavy are always bad…it says there are no best practices, nor are there any worst practices…just practices that work better in some instances than others.  Think of it this way;  if you are out on the lake on your personal boat on bright summer day, and the boat starts leaking, process and documentation may not be important;  you have a few choices, including simply jumping over and swimming the 50 yards to shore.  But if you are on an oceanliner in the middle of winter miles from anything but more water and that ship starts leaking, wouldn’t you prefer that there was some documentation and process to keep you safe?  Size seems to shift the context that way…so I would not think a startup dot com would see value in scope management.  But huge organizations, like my current client, will.

Also, I am getting in on the ground floor more or less, and talking with some of the influential folks in the industry, and we have already begun to discuss how we may try to adjust the basic process into a variant designed with more agile organizations in mind.  And so I am excited about the opportunities ahead, and the chance to try and continue to make a positive difference.

If you are interested in Scope Management, and the training and services we will be providing, please visit our website for more information, and feel free to leave comments here about what you think of this new emerging concept.

David

I am writing this blog, because I have the time to do it.

I have the time to do it, because my test environment is down.

My test environment is down because someone is installing and configuring and troubleshooting a new testing tool.  They have been since around the middle of the day yesterday, and an email just came out reiterating how important it is that everyone stay the hell out of the system until further notice.

We love tools here.  We have gobs of them.  We have even documented, at least to the acceptable level of most of the testers, that we have so many tools they actually affect the accuracy of our test results at times.  The managers don’t necessarily agree with that.  But I think about the last thing we need is another tool.  And I say that as a tool maker and seller.  Yes, another tool is not always the right answer.

I know it is just my opinion (Okay, thanx to James and his introducing me to the concept of Epistemology, at this point I think everything I think I know is just my opinion…ouch, my head is gonna explode) but…

When you spend more time installing, configuring, and troubleshooting your tool than using your tool to learn about your application, you’re not testing anymore.

When the value of the 10% of information a given tool will give you is more important than the other 90% of the information you can get without it, I am not sure you’re testing anymore.

When being able to say “On line 10 of method Foo in class Bar you have a memory leak” is more important than being able to say “6 out of the 10 primary business processes I have tested failed to meet my expectations, with me being a reasonable proxy for an average user”, I am not sure you’re testing anymore.

To me, these activities smell a lot like analysis and development.  And more and more, I am beginning to believe that tool blindness is both a symptom and a cause of this kind of behavior. I love tools.  I make them, I sell them, and I use them.  But I am always cautious to keep in mind that the tool is only that, and that the key to testing is the active involvement of people with brains, and skill.  And when the behavior of myself or anyone else on my team begins to look like tool usage for the sake of tool usage, so that statements can be made such as “We are doing good testing because we are using Super Snooper” or “If we don’t use Super Snooper, then we can’t be doing good testing”, I think tool blindness is setting in.

Begin with the end – and I don’t think any management team every created a test team with the express purpose of having them tool up.  When you are holding up testing on a semi-permanent basis to play with your tool, I think you have lost sight of the end.

Play with that tool too much, and you may just go blind.