The Developers are delaying the release again...apparently!!

I’ll never forget the look on the face of one of the Directors of a former Employer. Picture the scenario – the project was way behind its promised delivery date, clients were threatening mutiny, pressure was high, requirements were being de-scoped, and quality was heading south as mistake after mistake was occurring. It was in this context when we were called into a team wide meeting with Management and the Directors to discuss the matter. Whilst this was done in an open and transparent format (which was how we always worked), the aim was to get assurances of a high quality release to specification and in timely fashion.

I patiently waited (like the typical, introverted, well behaved, passionate Tester that I am ;)) until it was my turn to speak. To everyone’s surprise I only had one comment: “We need to slow down Development”.

 

It was at this point I got the glare from hell from one of my Directors. You see, he didn’t quite understand how I was proposing that we work at an even slower rate. He probably couldn’t accept my disregard to the position of Management, and I suspect he also loathed the likelihood that I was putting thoughts into the minds of my team mate Developers to relax and slow down. He was spot on because this is exactly how I was feeling (more on that later).

 

Moments took years to pass when I eventually glanced around the room. I noticed my Development Manager was not frowning or shocked like everyone else. I sensed he understood my point. I gathered he had enough on his plate and so perhaps from his perspective my comment was a show of support and a reflection for how others were really feeling.

 

The scenario we had was: Partially delivered functionality which was not working; Defect fixes which were failing re-test at a rate of 2 in 3; Delivery of high level business requirements which were being analysed at the time of Development and Testing and which required a lot of rework; and Minimal validation on the impact of changes being integrated into the overall system (in other words Regression testing was nowhere near on track). To make matters more interesting, the same team was servicing multiple production support issues, working on other projects, managing their own Development and Testing platforms, dealing with company ownership changes, and dealing with other work matters in general.

 

If that wasn’t enough of a challenge, the project was conceived on the basis of unqualified, high level client needs. That is, with insufficient input to determine if it was possible to transform those needs into a real deliverable specification on time and to acceptable quality. It was the typical scenario whereby our Sales people were pushing boundaries and this meant doing whatever it took to secure contracts (and fair enough as this was fundamental in ensuring our future).

 

So why did I say slow down? I said this because I had seen the same level of errors and fail rates consistently in the past. I had seen the conflicting priorities, the unqualified solution, and the tight schedule. All of this was part of our day to day working environment and was systematic behaviour. It was therefore measurable, and this is exactly what I had done over the years! Hence my metrics were telling me that this was how we rolled. So I wasn’t literally saying to code less or to skip tests. What I was essentially saying was that we all needed to relax because we were on track.

 

I can see the problem with this approach today, although at the time I stood my ground. With more clarity I can see that we were on a different track to our peers and that this did not give us the grounds alone to continue at the same rate and quality. We didn’t have the right to undermine existing agreements on the basis of a sarcastic outburst which at the time had not been qualified.

 

The approach I should have taken was to present a high level overview of the results we were achieving. I think I did this after the meeting however I cannot recall this with certainty and given that the meeting experience is what remains clear in my mind to this day, perhaps my comment and not my follow-up was the stronger message conveyed.

 

There was also an element of frustration in my comment. In my estimation documentation was a clear demonstration of behaviour which had been previously conveyed to Management in the past. However reoccurrence of the same problems convinced me that nothing was being done, and so I stopped presenting a detailed case. I also started to believe that it was out of everyone’s hands as power shifted to higher levels of management beyond our team. I was also well aware that my ongoing constructive feedback about the level of Development quality was already known and so all I managed to communicate was negativity. I can now accept that my message of showing that we were dealing with a high level of complexity did not get through.

 

When I read through this story and ask myself whether I believe that Development was the reason why those projects were constantly delayed, my genuine belief remains as a “no”. The line of business we were in was tougher than any prior industry I had worked in, and it simply was not possible to speed up at the flick of a finger. We had no competitors to measure against, we were in a mission critical space 24x7, and our financial position meant that we could not afford the luxury of setting up long term team plans where skills transfer could be done efficiently, and where other continuous improvements could be pioneered. We were good at trying to do this but on a tight budget and with a strong market, holding down good performers to achieve these goals was made all the more difficult. All is good now with my former team I’m sure!

 

Are detailed steps in a Test Case the way to go?

To have steps or to not have steps. This is a topic which will always have divided views amongst Testers. Most of us will categorically state that a Test Case must always have steps. A minority of us will say that steps should never be written. A few more of us will say that it depends on the context. I am definitely in the third group.

I have worked in organisations where every test to be executed needs to have the detail defined. I struggle abiding by this rule for all circumstances for many reasons. 

The first concern I have with this approach is that the cost is high. It can take hours and days to design steps and then only seconds or minutes to execute it. Hence I see more value in other preparation tasks such as to conduct pre-release testing, play and explore with the in-development prototype, and get involved with test environment setup and configuration.

The second concern I have is that unless the system to be tested is mature and completely up to date with documentation, the steps are never accurate. This has downstream impacts during execution whereby Testers get lost in trying to interpret the Test Case detail and divert focus from the system to the test design. This also relates to my third reservation whereby precious time is spent opening a test, clicking run, and entering a result against every step. Some tools handle this better in my opinion (eg Seapine TestTrack) whilst others have big efficiency overheads (eg QC, MTM).

My fourth concern with this approach is the reuse value has not been considered. If the test case is not likely to be reused what is the point? Additionally, if retesting is likely but adequate time has not been reserved to improve the test case and to fix its errors, where is the value? Also on this point, even if there is time available, how disciplined is the Team / Tester with regards to performing this task? I've been in QA/Testing for years and I still struggle doing this! The same can be said for peer reviewing - ie how often do your fellow Testers review your detailed test case designs?

My fifth concern is the over-focus given to the non-execution consumers. I refer to Managers who need to report, Auditors who need to assess compliance, and other team members such as Business Analysts and Project Managers. Yes these are important tasks but should they outweigh the execution value?

My last and biggest bug bear is the lack of empowerment and onus given to the Tester. If the detail is put on a platter, the Tester will never learn. They will not develop fundamental qualities of a Tester to explore, experiment, play, inquire. They will be less likely to consult Developers to question technical aspects of the solution. They will also not gain independence from the designer. They will make errors, miss bugs, and log incorrect incidents.

Okay so I've had my whinge with the must design with detail always case. What about the case of never designing with detail? Well I'm not a fan of this approach either because in safety critical and other mission critical systems, detailed steps are mandatory. I also note that extensive scheduling and resourcing is placed on the review and maintenance processes with these systems to ensure 100% accuracy. I also believe that detail is useful for new starters, junior Testers, and other Testers who need some guidance and who need to be managed. There is also value for other consumers as mentioned above.

So in my ideal world how do we answer this question?

1) Determine reuse value

2) Estimate the effort required to design the initial versions

3) Estimate the effort required to improve initial designs to accurate, reusable work products

4) Assess cost of designing versus other preparation tasks

5) Assess Tester knowledge of the System and help / hindrance with having detailed test cases.

6) Assess overhead cost to schedule and execute a detailed test case in the test execution tool.

7) Determine minimum requirements for meeting standards, policies, auditing, reporting, etc. 

Hopefully this discussion provides a balanced opinion as to why context has to always be applied...ready to hear opposing views with this topic, I know you're out there :).