Testing without specs: Unblurring the fuzzy corners of Perception-based Design

Got spec?

Yea, me neither.

We all have our favorite flavors of "Agile" (and ice cream). Our fondness for whiteboards, sticky notes, wikis and one-pagers continues to increase as we immerse ourselves in this new paradigm. Over time we have become more willing to incorporate change in our design and code. One of the first things we do in the mornings is our 15 minute stand up meeting. We are getting more comfortable with not having a lot of upfront planning. Actually, we're almost scared of doing too much planning, thinking that we might lose our valuable time if we are required to adapt to something later on. This new approach has at many instances proven to be our vehicle to many fast-paced yet high-quality customer focused software projects.

Here's what my office monitor looked like a few months ago:

But there are challenges, especially for testers. One of the big ones being: testing without requirements. For waterfall testers or brand new ones, it can be a nightmarish task. But agilists aren't huge fans of comprehensive documentation, their argument against it is that it keeps the team from producing actual work. Where actual work equals working code. The argument is valid for the most part. Documentation does tend to erode over time and is costly to maintain. This is however, unfortunately, sometimes used as an excuse to completely get rid of documentation. Simply abandoning formal planning doesn't make a team Agile. When formal documentation is completely ignored, design and implementation is based more on individual perceptions. Implementation could then really be a manifestation of what the developer thinks the product should be. That is scary.

Any solutions? We could go back to doing minimal documentation to help testers do their job. Maybe a list of capabilities that the team decided to enable in the software and have the testers verify them. But that's not testing, that's checking. If you're in a true Agile team the developer has probably already done that. The real problem here isn't so much with what has been determined as a capability or feature but really what hasn't been determined, omitted or incorrectly assumed as true or false in an attempt to be terse and nimble. How does one uncover those fuzzy corners?

Well, let's start with getting our hands on some information that is relatively easier to get.

12-minute Stakeholder Interviews

Depending on whether a stakeholder is a chicken or a pig, they will either be able to make a claim or have an expectation. A developer or designer for example has the capability to make a statement of how the product will behave or look like because they are writing the code or designing the UI. Similarly a customer has the right to make a demand and have an expectation. The purpose of a Stakeholder interview is to discover what those claims or expectations are. Don't worry we're not going to stop there. There are a few tools that will help us clear the haze that surrounds these statements.

The idea is to keep these interviews short and specific. They can be verbal and pre-planned, but should be changed on the fly if need be. One thing to remember is: specific questions will yield specific answers. To get most value it's a good idea to include qualifiers like "most important", "worse possible in your opinion", "5 best", "top most" etc. They will guide the stakeholder in their thought process. We're assuming that we are not committing to too much work per iteration, hence a 12-minute interview will be sufficient. General test sources can be used as mental triggers to design Stakeholder interviews e.g. questions around input data, record handling, file processing, security, performance etc.

Here are a few sample questions:

1. Can you give me a three examples of input data and corresponding output or resulting behavior?

2. What is the most frequently used feature of this product going to be?

3. Does timing matter for capability X?

4. If error condition Y occurs what would happen?

5. What is the maximum time that Operation M can take?

6. Which feature are you most nervous about/is most stable?

The questions are designed to be somewhat tough and intend to make the stakeholder think and make claims (or state expectations). It is imperative, though, that our stakeholder remains comfortable. We are not trying to grill them, we are learning about their perceptions. It is possible that some questions themselves will make the stakeholder take a step back and rethink a claim they had made earlier.
To decide whether a question is a candidate for a 12-minute Stakeholder interview question ensure that:

1. the question is seeking to get a statement claiming something or expressing an expectation.

2. the question is seeking to get a measurable statistic about the product. Something you can "check" (see above).

Questions about environment setup or background and motivation for building a feature should not be included in Stakeholder interviews. Once we have interviewed a few stakeholders, we can move to the next step i.e. poking holes in their claims and expectations.

The Agile Philosophy: Individuals and interactions over processes and tools

Empowering individuals is a superb idea. Something to be careful about, however, is the fact that individuals are inconsistent and not perfect, not to say that processes and tools are. They are only as perfect as the ones who developed them. We often make arguments that are fallacious. I'm sure there are a few statements in this post that fall perfectly into a certain category of informal fallacies as well. Our goal however is not to prove that someone's claim or expectation is wrong or fallacious, moreover it is to point out that there is some missing information or test case that we need to figure out and validate.

We, the software testers, couldn't be luckier. We appear to be tied very tightly to software and technology yet we have the luxury of employing learnings from other fields of art and science to reach our goals. Actually you could argue we work as much or more with people as we do with software. Today our saviors are scholars of Logic. How? Well, they have given us frameworks to detect fallacies in claims, arguments and general statements. Sometimes without even understanding the context of an argument and based purely on their structure. And we are going to use them to illuminate the shady borders of software.

A lot of what I'll be talking about in the remainder of this post really comes from the study of Logic. For some motivation, here's an example of a fallacy:

If it's raining, then the streets are wet.
It isn't raining.
Therefore, the streets aren't wet.

(from http://fallacyfiles.org/examples.html)

This fallacy is called "Fallacy of Propositional Logic". In this particular example, the fallacy is rather obvious but here's a real-world (okay I had to remove some specifics) example of the same:

Excerpt from a One-Pager: If the Global Validation Mode is on, Local Validation Mechanisms are effectively ignored.

From the troubleshooting guide: Turn Global Validation Mode off to make sure Local Validation Mechanisms are invoked. (Wait it's already off!!!)

Incorrect Assumption: The only way Local Validation could be bypassed is by turning Global Validation Mode on. (Turns out it isn't so)

A few ways to attack ambiguity

A great resource on fallacies are the "Fallacy Files". If nothing they're just fun to read. But for today let's talk about how to apply what we know from Logic to the claims and statements that we collected about our product. Here are just a few heuristics derived from fallacies: (If you're looking for them, there's a good chance you'll find them)

1. The Dangling Else: This one is a classic. Ex: The system will do operation X when A happens. Sooo … what if A doesn't happen? Is A the only possible thing that could happen?

2. Ambiguity of Reference: Ex: If condition A is true, the default behavior will kick in. And what's the "default behavior" again? Is it well defined?

3. Scope of reference: Ex: For all other cases, the system will ignore the input. Is there a finite set of "other cases"? What makes us so confident about this?

4. Omissions: Important details left out.
   • Causes without effects
   • Effects without causes

5. Random Organization
   • Mixed causes and effects: Ex: If A and B happen, C and D will occur. So do A and B need to happen    together for C and D to happen? Is C a result of A, or B or both.
   • Random case sequence

 6. Ambiguous precedence relationships: what comes first? Is that always the case?

 7. Implicit cases: Ex: The system takes two inputs A and B. Are there any environmental inputs like a path variable or a windows registry entry?

 8. Temporal Ambiguity: Does the statement hold true at all times? If we don't know, we have a case of Temporal Ambiguity.

 9. Boundary Ambiguity: Where does the influence of one feature end and where does that of the other begin?

You get the idea. There are a few others that can be listed here but I'll leave it up to you guys to think of more. And if you can I'd love to hear about them.

Summary

Let's do a quick recap. Rapid software development necessitates the abandonment of a lot of formal documentation techniques like writing Requirements Specifications or creating UML artifacts that we may be used to. Sometimes documentation is just verbose and hence not very useful but when done right it could prove to be vital for recording claims, responsibilities, edge-cases etc. It could also act as a forcing function for teams to think about things they wouldn't have otherwise. But in any case, it takes time away from writing and testing code. So instead of fighting for more or better documentation, we talked about changing our approach a bit and trying to use our own way to do all the good that documentation can do without doing extensive documentation. We tried to understand stakeholder perceptions by interviewing them and collecting their claims and expectations in one place. Then we looked at a few heuristics that we borrowed from Logicians to poke holes into these statements and find out what's missing. By doing that we're not just verifying claims but actually unblurring the fuzzy corners that no one had seen or realized existed. A thing to note here is that these methods can also be used to improve existing documentation.

If you have encountered similar issues and solved them in your teams, I would love to know. If you can improve my approach I'd be obliged. Till next time dear readers, sayonara!