Test Team Psychology

Test Teams often inherit negative reputations amongst greater IT departments. It's something I've grown deeply aware of over the years and use to my advantage when joining a new organisation and trying to build long-lasting relationships with the greater IT department.

I've often seen this topic discussed in the context of Test & Development Team relationships, and particularly from the perspective of a Developer. It's often stated that we slow down progress because of the level of support we require, because we raise lots of defects which cause noise rather than to improve quality, and because we lack the technical understanding needed to contextualise our approach with respect to the architectures, technologies, infrastructures, interfaces, and so on.Whilst I accept this and could present opposing arguments which fight for the challenge we face as Testers, this blog is not aimed at winning an argument. Instead, I am seeking to peal open the inner workings and mindsets of a Test Team to determine whether we are heavily contributing to our own negative perception. Or to put it in simplistic terms, do we only have ourselves to blame?

Yes.

This response will come as no surprise to some of you - I have been leaning towards this view for quite some time now. They say it takes about 10,000 hours of experience to become a Master at something so on that basis I think I have more than enough data in my head to talk about Test Team psychology and to come to this conclusion. 

I think we Testers are emotionally grouped into four main categories: 

Firstly there are the Nerds. These types are pure techos most of whom can backup their education with real work output. They can mostly cut code, do a job without fuss, self-manage, and hide away in the corner of the office behind their multitude of tools and servers. They do all of this with the perfect poker face so that no-one really knows what they think, even when they completely disagree with the approach. They neither smile nor frown, speak loudly, nor quietly, just assertively. Get them out for a beer and they'll slag every single person from the organisation, explain why the organisation is doomed for failure, and so on.

Secondly, there are the Analysts. These folk genuinely care about quality and just about everything else, so they're constantly letting their emotions get the better of them. They go to the nth degree reviewing documentation, escalating issues, designing approaches, mitigating problems, playing with prototypes, building relationships, and picking the brains of every single expert to get a full picture. They don't know how to play with a poker face so if they're angry or tense or happy, it will show.

Thirdly, and finally, there are the Cruisers. With exception to a hard-coded smile, these folk will never show emotion, will always appear relaxed (ie cruising), will always follow the instructions and guidelines (and never deviate), and will always present themselves to their peers as being perfectly under control. That means if they're supposed to be busy, that's the mode they'll present. If they're meant to be flat out, they can do that too because they have been around longer enough to know how to survive, thrive, and dominate. Some of them landed in testing because it is a nicely paid easy career to enter into. They have strong networks, are nice to be around, but they prefer to work and play only within their network as that's where their strength lays. Hence unlike Nerds and Analysts, giving feedback is risky hence what they provide is sometimes not of value. Many of them have their valid reasons for this be it from a previous experience, personal background, or other. Also to their credit, some of them have Nerd skills or were Analysts in previous lives but lost the inspiration. This can be reversed with effort.

Fourthly, and finally, there are the Combinations. These folk have combinations of the above three, so technically speaking there isn't really a fourth type of Tester other that one who roams around each emotion. Depending on the combination and depending on whether Management has been able to accurately categorise them, these people are either well valued assets, ticking time bombs (for implosion or explosion), or delicate creatures who need to remain under cotton wool.

So lets just say for example that we're a Test Team top heavy with Nerds and Analysts with the odd Combination and Cruiser. That would be a Team setting and achieving goals, measuring what real quality means, having genuine career paths which meet those quality needs, exploring new tools to be better, and expanding their capabilities for the future. They would probably need some self-tuning now and then to keep the Analysts happy and to make sure the Nerds are telling us what they really think. Other than that I'd say they're heading in the right direction and not copping negative sentiment.

At the other extreme lets say we're top heavy with Cruisers, have a bunch of Combinations, and a small number of Nerds and Analysts. It is this dynamic which creates that negative sentiment. Worst still is because all the stuff is hidden behind the cruisers smiling face, the world carries on unbeknownst that stuff is going to come out or something is going to give (ie critical defect, problem with Project team relations, lack of real Test Team culture and cohesion, no Collaboration, ineffective Process Improvements, etc).

Hence if I were looking into my crystal ball I would conclude by saying that Test Teams in their traditional sense and existing psychology is something which cannot be sustained as technology advances and needs evolve. It's a wake up call for us Testers to become more analytical and/or more technical for our own survival, it's a wakeup call for Management to start invoking change in anticipation of our changing environment, and its a call out for our future leaders to be one step ahead in recognising Tester and greater IT Psychology so that we can be suitably performance managed and rewarded (or reprimanded) appropriately.

Controversial? Offensive? Full of sh!+? Tell me what you think.

Why I love testing on projects derived from prototypes

Ok. Over years I've noted that Testers rarely blog about projects they really enjoyed and more importantly why that was so. Yet when I look back and analyse some of the common characteristics in my project history, it has come to my attention that those which applied prototype development are right up there as some of my best.

My definition of prototype development is probably not what you'll find in a textbook, and I'm sure my readers will differ in their opinions. However for me, it's a trial run (sometimes to production). It's having something which has look and feel, something which demonstrates concepts, something which illustrates functionality, and something which has already gauged an active community as to it's usefulness and effectiveness. As a Tester, I cannot think of a better basis for gathering knowledge, defining risks, gaining high level understanding of scenarios, and setting the foundations for genuinely understanding requirements as they are being specified.

In a previous blog I've mentioned how I would much prefer to commence early learning about a system in development rather than to await documentation. I'm like a kid in a candy shop when a Developer trusts me enough to do this, which is mostly the case other than when the working environment just doesnt get it and spoils the party (that's for another blog!)

For a switched on Tester, getting a free pass to a Developer's sandpit is an awakening that genuine engagement in QA has commenced. It's an early insight into the building process and a real-time specification of the roots of what will eventually be a final Production version. It's an empowering activity which allows all key stakeholders to partake in validating objects, fonts, colours, workflows, speed, integration, and so on. It's collaboration in its' most effective form, and one which draws on a Tester's creative side to explore boundaries, conduct ad-hoc experimentation, gather notes, and contribute to requirements elicitation. I'm a non believer that a detailed specification must always follow a business requirement (full stop). Instead, I'm a big believer that an in-progress specification (ie prototype) can follow a business need which then applies back to an improved need (ie to the ultimate requirement). In other words, we're talking about a flexible, continuous, improving process (some might call this Agile :) ).

In promoting the above methods, perhaps I have presented a somewhat negative view for waterfall styled, formal process which requires Testers to wait for official documentation. However that's not my aim. I'm simply highlighting that it was in these types of projects where my biggest attributes as a Tester emerged. It was where I gained expertise earlier and so I found problems earlier and built up my value to the team earlier. However these projects also resulted in delays from their original deadline, required multiple releases to production, and created headaches for management looking after budgets, schedules, and resources across multiple projects. Hence I must concede that I view my success in a project not by whether it goes in on time and on budget, but whether I was influential on the end user needs being met to the utmost quality. I also concede that in referring to the "User" need versus the "Business" need I'm neglecting the importance of those aspects which are crucial to management.

I truly wish I could break protocol and share with my readers details about which projects I succeeded in based on these principles, and which ones I didn't. At least this would let me settle the air with those who have managed and are managing me. It will give them another opening to this brain of mine! 

A lot of related thoughts have also come to mind on this topic which are probably blogs in their own rights. For example:

1) Prototyping can get really messy when Testers are not involved

2) Testers will fail if they rely solely on a prototype and not on the emerging product encompassing endorsed documentation.

3) Moving towards a more Agile SDLC is no coincidence - it's what works (when done properly) and it's what Development and Business prefer.

4) Prototyping when the team is not located in the same building or when there are third parties involved is dangerous.

5) Prototyping or making a project Agile so that documentation does not have to occur is also dangerous!

Well this was my 5th blog, I hope you enjoyed it. I'm getting a few hits out there now so don't be shy with your feedback. :)

The Developers are delaying the release again...apparently!!

I’ll never forget the look on the face of one of the Directors of a former Employer. Picture the scenario – the project was way behind its promised delivery date, clients were threatening mutiny, pressure was high, requirements were being de-scoped, and quality was heading south as mistake after mistake was occurring. It was in this context when we were called into a team wide meeting with Management and the Directors to discuss the matter. Whilst this was done in an open and transparent format (which was how we always worked), the aim was to get assurances of a high quality release to specification and in timely fashion.

I patiently waited (like the typical, introverted, well behaved, passionate Tester that I am ;)) until it was my turn to speak. To everyone’s surprise I only had one comment: “We need to slow down Development”.

 

It was at this point I got the glare from hell from one of my Directors. You see, he didn’t quite understand how I was proposing that we work at an even slower rate. He probably couldn’t accept my disregard to the position of Management, and I suspect he also loathed the likelihood that I was putting thoughts into the minds of my team mate Developers to relax and slow down. He was spot on because this is exactly how I was feeling (more on that later).

 

Moments took years to pass when I eventually glanced around the room. I noticed my Development Manager was not frowning or shocked like everyone else. I sensed he understood my point. I gathered he had enough on his plate and so perhaps from his perspective my comment was a show of support and a reflection for how others were really feeling.

 

The scenario we had was: Partially delivered functionality which was not working; Defect fixes which were failing re-test at a rate of 2 in 3; Delivery of high level business requirements which were being analysed at the time of Development and Testing and which required a lot of rework; and Minimal validation on the impact of changes being integrated into the overall system (in other words Regression testing was nowhere near on track). To make matters more interesting, the same team was servicing multiple production support issues, working on other projects, managing their own Development and Testing platforms, dealing with company ownership changes, and dealing with other work matters in general.

 

If that wasn’t enough of a challenge, the project was conceived on the basis of unqualified, high level client needs. That is, with insufficient input to determine if it was possible to transform those needs into a real deliverable specification on time and to acceptable quality. It was the typical scenario whereby our Sales people were pushing boundaries and this meant doing whatever it took to secure contracts (and fair enough as this was fundamental in ensuring our future).

 

So why did I say slow down? I said this because I had seen the same level of errors and fail rates consistently in the past. I had seen the conflicting priorities, the unqualified solution, and the tight schedule. All of this was part of our day to day working environment and was systematic behaviour. It was therefore measurable, and this is exactly what I had done over the years! Hence my metrics were telling me that this was how we rolled. So I wasn’t literally saying to code less or to skip tests. What I was essentially saying was that we all needed to relax because we were on track.

 

I can see the problem with this approach today, although at the time I stood my ground. With more clarity I can see that we were on a different track to our peers and that this did not give us the grounds alone to continue at the same rate and quality. We didn’t have the right to undermine existing agreements on the basis of a sarcastic outburst which at the time had not been qualified.

 

The approach I should have taken was to present a high level overview of the results we were achieving. I think I did this after the meeting however I cannot recall this with certainty and given that the meeting experience is what remains clear in my mind to this day, perhaps my comment and not my follow-up was the stronger message conveyed.

 

There was also an element of frustration in my comment. In my estimation documentation was a clear demonstration of behaviour which had been previously conveyed to Management in the past. However reoccurrence of the same problems convinced me that nothing was being done, and so I stopped presenting a detailed case. I also started to believe that it was out of everyone’s hands as power shifted to higher levels of management beyond our team. I was also well aware that my ongoing constructive feedback about the level of Development quality was already known and so all I managed to communicate was negativity. I can now accept that my message of showing that we were dealing with a high level of complexity did not get through.

 

When I read through this story and ask myself whether I believe that Development was the reason why those projects were constantly delayed, my genuine belief remains as a “no”. The line of business we were in was tougher than any prior industry I had worked in, and it simply was not possible to speed up at the flick of a finger. We had no competitors to measure against, we were in a mission critical space 24x7, and our financial position meant that we could not afford the luxury of setting up long term team plans where skills transfer could be done efficiently, and where other continuous improvements could be pioneered. We were good at trying to do this but on a tight budget and with a strong market, holding down good performers to achieve these goals was made all the more difficult. All is good now with my former team I’m sure!

 

Are detailed steps in a Test Case the way to go?

To have steps or to not have steps. This is a topic which will always have divided views amongst Testers. Most of us will categorically state that a Test Case must always have steps. A minority of us will say that steps should never be written. A few more of us will say that it depends on the context. I am definitely in the third group.

I have worked in organisations where every test to be executed needs to have the detail defined. I struggle abiding by this rule for all circumstances for many reasons. 

The first concern I have with this approach is that the cost is high. It can take hours and days to design steps and then only seconds or minutes to execute it. Hence I see more value in other preparation tasks such as to conduct pre-release testing, play and explore with the in-development prototype, and get involved with test environment setup and configuration.

The second concern I have is that unless the system to be tested is mature and completely up to date with documentation, the steps are never accurate. This has downstream impacts during execution whereby Testers get lost in trying to interpret the Test Case detail and divert focus from the system to the test design. This also relates to my third reservation whereby precious time is spent opening a test, clicking run, and entering a result against every step. Some tools handle this better in my opinion (eg Seapine TestTrack) whilst others have big efficiency overheads (eg QC, MTM).

My fourth concern with this approach is the reuse value has not been considered. If the test case is not likely to be reused what is the point? Additionally, if retesting is likely but adequate time has not been reserved to improve the test case and to fix its errors, where is the value? Also on this point, even if there is time available, how disciplined is the Team / Tester with regards to performing this task? I've been in QA/Testing for years and I still struggle doing this! The same can be said for peer reviewing - ie how often do your fellow Testers review your detailed test case designs?

My fifth concern is the over-focus given to the non-execution consumers. I refer to Managers who need to report, Auditors who need to assess compliance, and other team members such as Business Analysts and Project Managers. Yes these are important tasks but should they outweigh the execution value?

My last and biggest bug bear is the lack of empowerment and onus given to the Tester. If the detail is put on a platter, the Tester will never learn. They will not develop fundamental qualities of a Tester to explore, experiment, play, inquire. They will be less likely to consult Developers to question technical aspects of the solution. They will also not gain independence from the designer. They will make errors, miss bugs, and log incorrect incidents.

Okay so I've had my whinge with the must design with detail always case. What about the case of never designing with detail? Well I'm not a fan of this approach either because in safety critical and other mission critical systems, detailed steps are mandatory. I also note that extensive scheduling and resourcing is placed on the review and maintenance processes with these systems to ensure 100% accuracy. I also believe that detail is useful for new starters, junior Testers, and other Testers who need some guidance and who need to be managed. There is also value for other consumers as mentioned above.

So in my ideal world how do we answer this question?

1) Determine reuse value

2) Estimate the effort required to design the initial versions

3) Estimate the effort required to improve initial designs to accurate, reusable work products

4) Assess cost of designing versus other preparation tasks

5) Assess Tester knowledge of the System and help / hindrance with having detailed test cases.

6) Assess overhead cost to schedule and execute a detailed test case in the test execution tool.

7) Determine minimum requirements for meeting standards, policies, auditing, reporting, etc. 

Hopefully this discussion provides a balanced opinion as to why context has to always be applied...ready to hear opposing views with this topic, I know you're out there :).

I love metrics

I love metrics. I love crunching out numbers, making calculations, combining them to validate outcomes, and using them to make assumptions. I love doing this when the raw data is difficult to process and requires a lot of manual analysis. I even enjoy making experienced-based, best guesses and then reuse the method to improve it over time.

I think my love of metrics has grown over time. It probably commenced with my university education whereby I was taught principles, given tools, learnt methods, and applied logic to solve complex problems and to prove theories. However I didn't really know what it was all about back then, in that I didn't quite appreciate the need because I had no use for this stuff. 

This was the case until I found myself at my first employer - a small company which was struggling to survive and under-resourced to organise themselves in quality assurance, project management, and so on. I had a lot of fun learning new technologies, writing test procedures, and learning more about myself. I also enjoyed the security of working for a good Manager who provided lots of guidance. That was until he resigned and I found myself out of my comfort zone on many occasions.

Looking back I'm glad I had this experience because I quickly learnt the need to be self-organised, to know how to measure my own performance, to not assume that there are structures and systems in place to support staff, and to get involved in shaping how things are done. The need for metrics had arrived.

Throughout my career I have cycled from small companies to big ones. When I'm at a small company I get to use all of my skills, am at the forefront of the decision making process, and share in both the positives and the negatives of the trials and tribulations. It also means I can explore the boundaries of learning from our challenges to implement procedures which fix problems, apply context, and adhere to best practices.

Then when it all gets a bit too much (or shall we say when the love wears out), I find myself needing a big company fix. They have their policies, strategic objectives, standards, methodologies, procedures, templates, guidelines, and so on. Moving from a small company to a big company is easy in this regards because despite one arriving with real front-line experience, they no longer have to worry about how stuff is done. They just have to do it the way they're told. Hence provided they can handle this massive culture change, all is good....and if they can't, they'll seek other incentives such as training or promotion....or they'll probably leave and cycle back to a small company :)

From a metrics perspective this cycling between small and large companies makes sense. From my past experiences small companies enabled me to experiment with procedures, define metrics, work out how to measure them, evaluate what they mean, and then use them to benchmark future performance. At a large company I get to demonstrate this knowledge when appropriate, or when needing to try and influence change.

In a large company culture where you're meant to follow a procedure without question it isn't always easy to influence change. However I truly believe a reverse-engineering approach can work. This entails shadowing formal progress with your own metrics and using them to prove or dis-prove an outcome. If this is done with a positive frame of mind, with the best of intentions, and within a team where Management are open to new ideas, metrics based process improvements can occur. Surely Colonel Sanders measured and documented how long it took to fry same-sized, same-prepared chicken, whilst the local chicken shop which went out of business sometimes ran out of chicken, over cooked it, and served inconsistent quantities of fries!

I conclude where I began. For the number of QA/testers out there who loathe metrics (and other IT professionals), there are more of us who love metrics!

I’ve got certification. That makes me a Tester! :|

Since the mid 90’s I have managed to survive and thrive as a Software Tester despite having no piece of paper with the word “certification” written on it to give my prospective Employers comfort in the knowledge that I must in fact be a Tester. That was until the seventeenth of my eighteen years as a Tester when it suddenly dawned on me that most hiring agencies were expecting certification as a mandatory requirement.

My awakening to this fact now meant I could no longer take the moral high ground. In other words, I could no longer pick the jobs I wanted based solely on my own beliefs and principles, that I could no longer present myself as a suitable candidate on the basis of my work experience and skills alone, and that I could no longer stand out from the crowd. I needed to conform.

Sure, part of my refusal to gain certification was driven by the fact that my previous Employers had no money for training and no time to spare. However underlying this was my lack of engagement to pursue this path. In my view back then I couldn’t get past my self-centred opinion that I would not become a better Tester with the certification.

Some of this was certainly driven by fear – Fear that I might fail the exam, fear that I might not be as good as I think I am, fear that the profession has evolved in a direction that I was not aware of, and fear that I might actually learn something. Imagine that!

I don’t hold these views today and perhaps this is why I am ultimately writing this blog. I have realised that experience and certification do not need to be competing characteristics and that they actually complement each other in helping to validate the knowledge I have and continue to gain. It is also wrong of me to assume that I am no longer taking a moral high ground because my values in my profession have not changed. I still root for high quality, best practice, and process improvements. I still collaborate with my peers about how to best approach testing, how to resolve issues, how to manage risk, and how to meet deadlines to the satisfaction of my stakeholders. Perhaps the final nail in the coffin towards this train of thought was when I considered other professions. Would I for example hire a plumber to do some major repairs in my bathroom if he/she was not formally certified by the relevant industry authorities? Would I allow a stranger to teach my child piano lessons on a gut feel alone rather than to sight a Police Clearance? No I would not.

Despite the above there were however, other motivations which contributed to this view, many of which still exist. For example, I struggle having to accept wording interpretations like the difference between a Test Approach and a Test Strategy or between a Test Case and a Test Script. I struggle to accept that many folk in the industry really do believe that certification is all that is required for a Senior Manager to suddenly own the portfolio of QA/Test, or for a random job applicant to be suitably employed as a Tester despite no prior employment history. I dislike the fact that governing bodies of Testing Certification have little (or no) influence over IT, SDLC, Project Management, or the Business Stakeholders for whose requirements we are delivering. There is no better way of witnessing this than by attending a yearly Testing Index Seminar which demonstrates how the dataset for gathering QA/Testing metrics has very little input by other professions. In other words it is the very QA/Testing professionals attending these Seminars who are rating their own performance and taking an active interest in fostering better awareness for the challenges faced in QA/Test. In a perfect world this would not be the case. Rather, it would be those with whom we collaborate in delivering requirements who conduct the appraising and who provide feedback about how we are really doing. This also reflects in our resultant processes in that we in QA/Test ultimately determine how we should operate (for example in what format we accept requirements, the change control processes we adhere to for managing deviations to requirements documentation, management of defects, and so on). This is also why I believe conflict occurs during Project delivery because all we’ve done is missed the problem until a point in time when it becomes exposed (ie when doing the actual analysis, coding, testing, and QA).

Another example as to why I have been previously motivated against certification is due to the personalities I have come across over the years. Unlike most other disciplines in IT, Testers come from very diverse walks of lives. Some of us are degree qualified, scientifically driven Engineers. Some of us are technical coding Computer Systems Analysts. Some of us worked in Support or Help Desk for a year on a specific Product and jumped across when the offer was made to test an upgrade. Some of us come from the non-technical, Business side. Some of us went from school or some other educational institution straight to a Junior Tester. Some of us got certification before knowing what Testing was.

I can also recall how in the mid 90’s when I was employed as a Test Engineer in the telecommunications industry that it was considered very privileged to have Cisco Certification. Taking the path of becoming a certified expert in IP Routing and Data Multiplexing (amongst other functions) was very tempting to me, and it was a natural path to take given I was testing these very functions in complex embedded systems. I am relieved that I did not pursue this direction. Besides the fact that our forever changing technical world is high layer, software driven with less emphasis towards data and networking layers of the past, it is also quite evident that certification in Cisco is no longer held in the prestige that it once was. I realised this when at the turn of the century it was quite common for Tertiary Educators to offer certified Cisco and Networking certification to undergraduates and without the need for prior experience in the industry. This awareness certainly drove my opinions about Testing Certification.

In my blogs I am going to try and offer a balanced discussion and hopefully I have done this in presenting opinions both for and against certification. However the fact is, I really do sit on the fence with this topic and hence it is easy for me to debate with my own contradicting opinions. Sure, I have more relaxed views from my prior leftist attitude of “no certification at all cost”, but perhaps not considering credit for my eminent certification should be given to my current employer (and Manager) for their endorsement rather than for my enthusiasm or lack of. Does that make me a hypocrite or an opportunist taking whatever training I can get my hands on? Perhaps so ;) .....Twofishtoday