Over the past few years in teaching and consulting with testers and test managers worldwide, I have noticed something interesting.
On one hand, testers complain they hardly ever get user requirements adequate for testing. On the other hand, when discussing what to base tests upon, the main response is "user requirements."
When I suggest there are other ways to design and evaluate tests, the response is similar to the time I accidentally walked into the glass of the revolving door at the Portland airport- surprise, followed by intense laughter.
I'm serious, though. There really are other ways to design and evaluate tests besides requirements. First, let's look at why you may need to do this. By the way, let me be quick to say that I am a big believer in having solid, testable requirements for any project. Without them, you are at risk for a multitude of problems, including not knowing if an acceptable solution has been delivered.
However, I also know we must anticipate real-world situations in which we may not (or should I say "probably not") get testable requirements. Here are just a few of the situations that may require other techniques than requirements-based testing. "We don't do requirements well" - The title of this article is intentional.
The word "defined" explicitly indicates that requirements are in a documented form that can be discussed, reviewed, managed and all the other things you do with things that are written - including filing them away and forgetting them. Hopefully, we'll not do the filing away and forgetting part.
Failing to create and manage requirements well is a cultural and a process problem. No matter how important the testers see requirements as being, unless the project team and customers sees them as important, you're not going to get them or at least to the degree you need them. So, even though you may rant and rave, at the end of the day you still don't have testable requirements. Here are some example situations:
Commercial Off-the-Shelf Software
Let's take the example of buying a commercial software product. You should have defined requirements, but not for the purpose of building the software. Your concerns relate to the fitness of use. At the least, your requirements will be a list of things the software should do for you. Hopefully, the requirements will also define desired or needed attributes such as performance and usability.
However, these requirements will not describe the processing rules of the software or of your organization. Your task then becomes to design tests that validate the software or system will do what you need it to do, how you need to do it.
Stale, Outdated, Requirements
Perhaps at one time in the past you had a set of current and correct requirements. However, the passing of time was not kind to them. Business has changed, laws and technology have changed and now those requirements are not even close to how the system or your organization looks today.
Lack of User or Customer Input
The project team, including testers, may want testable requirements. However, the users and customers simply will not provide them. Perhaps they can't reach agreement on what is desired. Or, they will not give it their time - "We have a business to run here. You do your job and we'll do ours." Sure, they'll tell you when you've missed the mark - they just have a problem telling you where the mark is. That's very frustrating, indeed.
People Think They are Writing Good Requirements
To some people, requirements are sufficient even though they contain vague words and omit important details. These people seldom review these kind of requirements and in reality, little can be done with them. They are created as a step in a process to say they have defined requirements for the purpose of compliance. Incorrect Requirements - Remember the pie charts and research data that shows that most of the defects on a project originate in requirements? The problem is that even in well-defined requirements there are still errors.
Requirements Convey "What" not "How"
This is the correct nature of requirements. They should not convey how to implement a feature in the software. In addition, they should not convey how a user performs a task - that's what a use case does.
So, when you design requirement-based tests you are focusing on the processing rules, not on the sequence of performing those rules. This means that you will need more information than contained in requirements to design a complete test in any event.
Testing Using a Lifecycle Approach
In a lifecycle testing approach, items are reviewed and tested as they are created. In addition, the tests and their evaluation criteria are based on previously defined items such as concept of operations, requirements, use cases and design (both general and detailed).
For example, when testing code at a unit level, you can base some tests on requirements, but you will also need to use the detailed design for things such as interface testing, integration testing and detailed functional testing. You will also need to know the coding constructs for structural testing.
Other Methods of Test Design and Evaluation
Taking our eyes off requirements for a moment can reveal some interesting things. Here are some other places we can look for test sources.
User Scenarios
This is probably my favorite source of non-requirement based tests and is very well suited for user acceptance testing. You may be thinking, "Isn't this a good application for use cases?" You would be correct, except you may not have adequately defined use cases. User scenarios are based on business or operational processes and how they are performed. Of course, this requires that you or someone else in the organization understand those processes.
Even better, it would be great if those processes could be in a documented form someplace. It is amazing that when the entire functional process for a task is defined, the individual paths or scenarios can become very apparent. Each of these paths or scenarios can be the basis for a test script. If test scripts don't appeal to you (and that's okay), then you can think of a series of test cases or conditions that will accomplish the testing of the process based on user scenarios.
The big question is where to find the knowledge of the processes. There are several possible sources of process information in most organizations:
User knowledge
You will have to interview them and work with them to map the process. It helps to speak with multiple users to get the full picture.
Training materials
Sometimes you will find process flow charts, etc. in training documents. Business process re-engineering documentation - At some time in the past there may have been an effort to re-engineer processes. This is one deliverable from that type of effort.
System documentation
Yes, you may find documented processes there as well. Just make sure it is still current.
It takes a little practice to get good at mapping out the major process. I prefer to use flowcharting as the technique, since there are so many good flowcharting tools on the market. One of my favorite tools is SmartDraw (www.smartdraw.com). If you visit their web site, you will see some examples of defining processes with flowcharts. There are other great flowcharting tools available as open source, such as Yed.
One important thing to remember is to keep the level of detail general enough to manage the complexity. My guideline is to limit the major decisions shown in the chart to about five or six. Even this low number of decisions can yield twenty-five or more scenarios.
I do not advocate the need to test all scenarios unless the risk is high. You will need to prioritize and choose the scenarios that are the most likely and most critical.
Here is how a typical process flow might look.
Figure 1 - Applying a Late Payment Penalty
If we identify some of the scenarios paths, we can see: