Thursday, July 3, 2014

Are detailed steps in a Test Case the way to go?

To have steps or to not have steps. This is a topic which will always have divided views amongst Testers. Most of us will categorically state that a Test Case must always have steps. A minority of us will say that steps should never be written. A few more of us will say that it depends on the context. I am definitely in the third group.

I have worked in organisations where every test to be executed needs to have the detail defined. I struggle abiding by this rule for all circumstances for many reasons. 

The first concern I have with this approach is that the cost is high. It can take hours and days to design steps and then only seconds or minutes to execute it. Hence I see more value in other preparation tasks such as to conduct pre-release testing, play and explore with the in-development prototype, and get involved with test environment setup and configuration.

The second concern I have is that unless the system to be tested is mature and completely up to date with documentation, the steps are never accurate. This has downstream impacts during execution whereby Testers get lost in trying to interpret the Test Case detail and divert focus from the system to the test design. This also relates to my third reservation whereby precious time is spent opening a test, clicking run, and entering a result against every step. Some tools handle this better in my opinion (eg Seapine TestTrack) whilst others have big efficiency overheads (eg QC, MTM).

My fourth concern with this approach is the reuse value has not been considered. If the test case is not likely to be reused what is the point? Additionally, if retesting is likely but adequate time has not been reserved to improve the test case and to fix its errors, where is the value? Also on this point, even if there is time available, how disciplined is the Team / Tester with regards to performing this task? I've been in QA/Testing for years and I still struggle doing this! The same can be said for peer reviewing - ie how often do your fellow Testers review your detailed test case designs?

My fifth concern is the over-focus given to the non-execution consumers. I refer to Managers who need to report, Auditors who need to assess compliance, and other team members such as Business Analysts and Project Managers. Yes these are important tasks but should they outweigh the execution value?

My last and biggest bug bear is the lack of empowerment and onus given to the Tester. If the detail is put on a platter, the Tester will never learn. They will not develop fundamental qualities of a Tester to explore, experiment, play, inquire. They will be less likely to consult Developers to question technical aspects of the solution. They will also not gain independence from the designer. They will make errors, miss bugs, and log incorrect incidents.

Okay so I've had my whinge with the must design with detail always case. What about the case of never designing with detail? Well I'm not a fan of this approach either because in safety critical and other mission critical systems, detailed steps are mandatory. I also note that extensive scheduling and resourcing is placed on the review and maintenance processes with these systems to ensure 100% accuracy. I also believe that detail is useful for new starters, junior Testers, and other Testers who need some guidance and who need to be managed. There is also value for other consumers as mentioned above.

So in my ideal world how do we answer this question?

1) Determine reuse value
2) Estimate the effort required to design the initial versions
3) Estimate the effort required to improve initial designs to accurate, reusable work products
4) Assess cost of designing versus other preparation tasks
5) Assess Tester knowledge of the System and help / hindrance with having detailed test cases.
6) Assess overhead cost to schedule and execute a detailed test case in the test execution tool.
7) Determine minimum requirements for meeting standards, policies, auditing, reporting, etc. 

Hopefully this discussion provides a balanced opinion as to why context has to always be applied...ready to hear opposing views with this topic, I know you're out there :).

0 Comments:

Post a Comment

Subscribe to Post Comments [Atom]

<< Home