Archive for December 2006

Is Counting a Bad Idea for Test Effort?

December 28, 2006

I was going through the post Why counting is a bad idea on Shrini’s Blog on Software Testing. Shirni has listed a classic example on why counting is a bad idea for Testing.

Even though i like this post, I won’t agree fully on the views expressed there. Probably we might need to improve our selves with the process of measuring.

Certain artifacts like test cases, requirements and bugs are not countable things and any attempt to count them can only lead to manipulations and ill-informed decisions.

Wait — Are there any things at all in testing that we can count without loss of effectiveness and usefulness of the information that a counted number might reveal?

Let me describe my stand clearly wrt to this Why counting is a bad idea.

I do agree that it’s not posssible to count all the activities that we do in testing. But then that doesn’t mean like all the Test related metrics are bad. May be, We can’t generalize them since it leads to lot of subjective debate.

When it comes push the usefulness and effectiveness of the information by the metric, this is more of a improvement process. All these metrics are subjective to the respective projects / environments / organizations. Any attempt in making this as a genaral metric will lead to manipulations and ill-informed decisions.

  1. Bad Idea for whom ?
    Is this for Customers / Management / Developers / Testers / Others
  2. If Counting is a bad Idea, how do we measure the effort spent for the Project ? Do we need to go by the gut feelings of te people ? Will there be any data to justify our decisions ?

Most of the metrics used for Software Development for that matter are subjective. It varies based on the application, technology, domain, global support etc. It’s tough to make them general like we do in other industries. So the same will be applicable for Testing too.

Some thing like below are very much subjective and won’t be accepted in genaral.

Productivity

  1. No of Test cases prepared Per Person Per hour = 5
  2. No of Test cases executed per person per hour = 15

Look at the subjectivity in the above scenario.

  1. What is the structure of the Test Case ?
  2. How many user actions will be there in genaral per TestCase ?
  3. What is the Volume of Test Data associated with each TestCase ?
  4. What information available for these Test Cases (SRS, Design, UseCases etc) ?
  5. What kind of test cases are they writing ? Are they writing for Unit, Integration, System, Accepatance, Performance etc..
  6. What kind of Test Design Tech are they following (BVA, EP, DT, OA, CE etc)

There will be so many queries that will be come up here and it’s tough to measure some one just based on the number in genaral. But then since it’s subjective to the project (team), the respective stake holder should be able to take some decision based on that number.

But then, we can’t make this number generic. Probably this might be useful to the similar projects to have some rough estimate.

At the end, i shouldbe able to give qualitaive (again this is subjective) to the stake holders. If some one expects 100% qualitaive info here, that’s an ideal world to which we can’t go as of now.

What does that mean ?

I should be able to give a number. My metrics should tell me the number of Test Cases, Issues etc wrt the requirement.

Otherwise we may not have clarity on what we are doing & what are we trying to deliver. Things will be going on the pure gutfeelings

Pradeep has come up with nice replies to my comments on the post.

@Pradeep,

Thanks for sharing your views.

Let me look at your requirement

” This field should take only alphabets and nothing else other than that “

In the first place, the requirement is not clear for me

  1. What kind of Application are we taking about ?
  2. What is the size of this field (Some thing like password. It should have a Min and Max length)
  3. When you say Alphabets, Do you mean to support all the alphabets in all the languages or just english or some thing else?
  4. ????

See Pradeep, my queries will go on like this and all these will help me in getting the clarity on the requirement. This is how i try to to understand the system.

This process helps me in knowing on what is the requirement, it’s advantages, it’s limitations and most possible scenarios where this can be used.

All the above will help me in capturing the required tests for the above requirement.

But this may not be generic on the whole but then it can be generic to the Organization or at least to the project people are working with.

See there should be some way to measure on what we are doing. This measure should help the stake holders to take a decision on the quality of the application under test.

I do agree on the fact that we need to improve ourselves in coming up with measures that have qualitative information for the stake holders.

@ Shrini, Thanks for coming up a post that has good debate.

That’s expected behavior and Not a Bug

December 26, 2006

I keep hering the following phrases from our Development Team here

  1. That’s an expected behavior and Not a Bug
  2. That’s by Design

Today, I have an interesting debate with our Product Manager on a Issue. He put is as “That’s an Expected Behavior and Not a Bug”

I have been looking at root causes of these problems & why do run into this kind of debates always. The following limitations came

We could have avoided the same incase if

  1. Clarity on the Requirements across all the teams 
  2. Limitations of the Technology
  3. Usability Principles (UI Guidelines)

The above two arguments can be avoided if both teams have clarity on the above three.

Testing World 2006

December 8, 2006

Hi All,

I have attened the Testing World 2006 conference today & the response was tremendus. The session on Testing Skills That No One Talk About by Michael Bolton was too good and put me in different thought process on the areas left out to explore.

He talks about

  1. General Systems thinking
  2. Critical Thinking
  3. Debating Skills

Testing World 2006 is the conference for building your expertise in testing projects execution & various products. It’s two power-packed days of practical information you can take back to your job and put to work immediately.

These kind of conferences helps testers to know what’s happening around and how can we benefit from the same.

Applying Static Analysis for Software Testing

December 7, 2006

In this post, I will be sharing my views on Static Analysis and how the same is useful for Software Testing. Using Static Analysis in Testing will also be called as Static Testing.

The generic definition for Static Testing is that

Testing of an Application at the specification or implementation level without execution of that Application.

Static Analysis helps a lot for better Code Quality and we can perform the following set activities against the source code.

  1. Capture Metrics
  2. Complexity of the System (eg. Cyclomatic Complexity)
  3. Compliance against Standards (eg. check for the Java Coding Guidelines)
  4. Exception Handling (Captures the code that contains Unhandled exceptions)
  5. Infinite Loops
  6. Copy Paste Dectors (Most of the developers used to reuse the code via a copy paste and introduce some issues in the process)
  7. Duplicate Code
  8. Dead Code

We should be able to restrict most of the above said issues at very early stage of SDLC by applying Static Testing techniques and make these part of the build process.


%d bloggers like this: