Archive for the ‘Test Documentation’ category

The Life Cycle of a Bug – Different Stages in it.

September 10, 2007

In this post, i will explore  different stages of the Bug from it’s inception to closer. The Bug has been found and logged into the Bug Tracking System. It’s my fourth post in the Bug Life Cycle series.

  1. The Bug has been found and logged into the Bug Tracking System. It will be treated as New Bug in the System.
  2. The Bug will be assigned to the concerned Developer for a Resolution.
  3. The developer looks in to the possibilities of the resoultion & takes a call on Resolution by fixing it or differing over the information provided.
  4. Tester validates the resolved issue in the build & checks for the regression scenarios over the fix.
  5. If the issue found fixed, then he choose to Close the issue else he / she will Re-open the same.
  6. The Cycle follows again for the re-opened issue till it get’s closed.

Bug Life Cycle

It worth doing the following activities

  1. Capturing the required and re-usable info to the Bug Report at it’s each stage.
  2. Check for all the closed bugs of Severity 1 & 2 against final build for the release.

In the next post, I will share my thoughts on the useful metrics over the Bug Tracking Repository.

Happy Testing..

Advertisements

Bug in the BSNL Portal

August 10, 2007

Today, i have come across an interesting issue with BSNL Portal. I have been using this site for online billing and payments for about an year.

Followed the usual steps and my objective is too see the bill and pay it online. To my surprise, the system says

This error message ” Invalid object name ‘RECEIPT_MASTER’ “ is strange and there is no information for me on how do i proceed further. The application might be unable to get the specified object from the Database.

In case if this happens on the last day of the payment, on line users are liable for late payment charges due to this issue.

Users are familiar with the messages like, Unable to process the information as of now & please check back after some time.

The error messages must be meaningful to the business users and we need to make sure that the technology related terms / phrases are not part of the same.

Happy Testing…

Powered by ScribeFire.

Context driven information for Bug Reports

July 26, 2007

Context driven information is the need of the hour and there is a huge value associated with the same. It’s good to capture the context driven information in the bug reports. My initial experiences with bug reports way back in early 2000 have taught many lessons to improve upon.

Bug reports used to capture what is the problem with the system familiar to the user (tester) who reported the same. People spend very less time to capture all the details required and there are many reasons for the same. I hope some of the upcoming testers will also be in the similar situations.

Some of the Reasons people quote here

  • We need to test more and less time to capture & write more information in the Bug Reports
  • It’s tough to capture all the information required
  • System is complex & It’s tough for the novice users to understand the bug reports
  • You know capturing all the info is process driven & it may not be worth of efforts
  • It some times boring activity to collate the info & push it
  • I can reproduce it on my machine if developer needs it.

This list can go on…

I hope you have come across this situation at least once in your career.

This is my third post in the Bug Life Cycle Series and it’s good to know the different users of your Bugs and their context with them. The mission of your bug report is to provide details and context of the problem and convey the importance of it with a user driven stories.

Your bug report must be the voice of customer and it need play the role of an advocate against the problem. Bug Advocacy from Cem Kaner is an excellent source to begin with. If the bug report unable to specify the need of the context, then it’s better not to write any report

It’s good to explore & capture some of the following problems

  • Productivity
  • Performance
  • Usability
  • Migration
  • Stability etc

Try to link your issues with most suited functions listed above. It may not be obvious to other users in the system to explore & analyze the issues in that fashion.

There is another context associated with Bug Reports. That’s with the stake holders of the project. The Bug Tracking system must give the right trends and identify the hot spots. Testers must capture the right kind of data to derive better valuable metrics over the bug repository.

Care must be taken to capture

  • Capture all the Test Environment details
  • Detailed classification on the feature. Classify to the maximum possible sub feature/component of the system
  • Clarity on Severity & Priority
  • Versions and Build Numbers (Affected & Fixed)
  • Bug Classification (Requirements / Design / Implementation etc)
  • Bug Types (Functional, Performance, Usability, Security etc)
  • It can go on…

The above info helps a lot to identify the trends in bugs and focus on the unstable components / environments.

Final Thoughts
Push the entire context driven information to the bug repository at least for a release cycle and observe the results. Check back with your repository to identify the trends and risk associated with the release and I am sure that it will be in the similar lines of end user feedback.

Happy Testing…

The Bug Life Cycle Series

May 11, 2007

I have been talking about Bugs in the Software Applications in my recent posts. They play an important role in Software Development & Testing. The biggest value addition is to uncover the critical issues of the system at the early stage.

I have already written some posts on the Bug Tracking front and find the list below. They talk about how to handle the Bugs, avoid conflicts & the lessons to be learned for the testers.

 

The above posts doesn’t cover all aspects of a Bug.  I feel that it’s better to write more on the Bug Life Cycle & explore the same.

The Bug Life Cycle series will have the following posts and i will be writing them in frequent intervals.

  1. The Role of Software Testing

  2. An insight into Bugs in the Software Applications

  3. Context driven information for Bug Reports – It’s the need of the hour

  4. Life Cycle of a Bug – Different Stages in it

  5. Communicate the Value of Bug Reports

  6. The End Game

Stay tuned & check out the blog for upcoming articles.

Open Source Test Management Tools

April 22, 2007

 

In this post, i will share my views & findings with Open Source Test Management Tools. Usage of tools in the Test Management is becoming the basic need & It will be tough to manage the activities associated there with out any support from the tools.

 

The commercial tools in this segment are costly and may not fit into upcoming organizations budget. So i have been looking at Open Source Test Management tools. In the last week, I have evaluated some open source test management tools.

 

The following are my requirements for Test Management Tool

  1. Capture the Requirements

  2. Design Test Cases

  3. Map Test Cases with Requirements

  4. Link Bug reports with Test Case ID after the test execution

  5. Test Execution Reports

  6. Version Management for the Test Cases

  7. Search Feature

The following tools are evaluated after the initial screening.

The above tools work with Apache, MySQL & PHP. Both the tools looks promising and the Test Link has some additional benefits in-terms of better reports and linking with popular issue trackers.

 

Taking on “Testing Triangles – A classic excercise”

March 24, 2007

Solving puzzles, sudokus & the analytical problems is one of my hobby. This set of puzzles always helps me with some brain food which mandatory to stay tuned in the current trends.

I was going through this Testing Triangles post from Elisabeth Hendrickson from Quality Tree Software. The triangle program is very much popular to the world of testers and many of the books and blogs have already covered the same.

Elisabeth posted this as a testing puzzle and asks the people to uncover the bugs in her program. The beauty of this Triangle program is that whole code is available for the tester since it’s written using JavaScript.

I couldn’t stop my self exploring the program and discovering some issues with Testing Triangles and logging them into comments sections while in hurry too.

I thought for a while on this program in the evening and felt that there are many similarities for this even though it’s very very small when compared to the real world applications we test today. So let me take a re-look at this program and apply my test approach for the problem.

Step1: What is the Problem ?

We need to test a program that takes the three sides of a triangle and tell the user about what kind of triangle is it. Here is that Triangle program which has been written in Java Script.

Find the description from the post here

This version of the program takes as input three numbers representing the size of the sides of a triangle. When the user clicks “Draw” the program draws a picture of the triangle with the size of the sides shown in proportion and also displays the type of triangle.

Step2: Question the problem & get the required information for the Test.

The Testers first need to understand the application and the scenarios in which the same might be used.

So with this perception, let’s re-look at the information available for the application. This is true that the above information provided for the application, acts as Requirements, User Manual and any other info we can call it as.

Like many applications that we test now, even this doesn’t have clarity on the information provided. This is also one of the reason for me to call it as a classic example for testing. If we dig more into it the clarity on the information is missing and there are lot of implied requirements (at least from the developer perspective) /assumptions made by the developer.

It’s not clear from the above description that

  • What is a Triangle and how to define the same ?
  • How many types of triangles are there and how to define each one of them ?
  • Are there any boundary limits for the size of the sides that makes triangle ?
  • Can any given three numbers make a triangle ?
  • etc..

Since the info is not available and the Triangles are very generic. Let’s try to Google around the same and get some info to explore the application.

So get the info here, here, here, and here

Step 3: Sanity / Smoke Test for Triangle Program

By now, I know what a triangle is to begin with and how the same can be drawn. Let me do some sanity checks here.

  • For the Test Data of 1, 1, 1 as the values for three edges of triangle the output of the application is “Triangle Type: Equilateral”
  • For the Test Data of 2, 2, 3 as the values for three edges of triangle the output of the application is “Triangle Type: Isosceles”
  • For the Test Data of 5, 6, 7 as the values for three edges of triangle the output of the application is “Triangle Type: Scalene”

 

I have started with some very basic positive tests to make sure that the application is inline as per requirements. It’s not good to test the application against negative tests to begin with.

 

Let’s explore some more tests.

  • Try with 0,1,2 – Invalid – wow // might have checked for 0
  • Try with -2,3,4 – Invalid – wow // might have checked for -ve values
  • Try with 1,2,3 – Invalid – // what ? why is this invalid

For the Test Data of 1, 2, 3 as the values for three edges of triangle the output of the application is “ Triangle Type: Invalid

The Application says it’s invalid. But as per the above spec it should be a Scalene Triangle since all the edges are different from each other.

The following queries came to my mind

  • Is this a bug with Application ?
  • Am I missing any more info that’s required for the test ?
  • Can any given three edges make a triangle ?
  • Is there any logic behind the same ?
  • But it could have been good,if the application says why the data is invalid. I have no clue on this and it’s a usability issue.

 

The general tendency that i see in many of the testers is to report it as a bug the moment they see some different behavior with the application and run into many discussion over the same. I have already blogged on it with a post as That’s Expected Behavior and not a Bug. A little more depth into information gathering will be more effective.

Let’s see why this test data is invalid for the application.

I got some more information again over the web on the edges of the triangle that will help me to understand the issue. And this is the logic behind the same. Get more info on this here.

In order to construct a triangle, the sum of any two edges must be greater than the third one.

 

So with the above data of 1,2,3 we can’t construct a triangle and the data is invalid. But a error message here on why this is invalid from the application could have saved us lot of time in exploring the same.

One more test with 3,4,5

For the Test Data of 1, 2, 3 as the values for three edges of triangle the output of the application is “ Triangle Type: Right”

 

Oh the behavior is different again. i am expecting of Scalene Triangle. Some more info via message could have been helpful for users on why the same has been labeled as that particular type. Might be the developer expecting all the users to have the domain knowledge.

 

Step4: Thumb Rules of the game

By now, we have capture a bit more info on the application & let’s list the same here

  • Rule 1 : Sum of all the angles inside a triangle must be 180 degrees.
  • Rule 2 : In order to construct a triangle, the sum of any two edges must be greater than the third one.
  • In case of sides, they are called as equilateral triangle, isosceles triangle, right angled and scalene triangle
  • In case of angles, they are called as right-angled triangle, obtuse triangle and cute triangle

 

It’s a good practice to update this section as and when we get the more info about the app behavior & fine tune the same.

 

Step5 : Black Box Testing on the Application

We have the following info as of now through above test observations.

  • Application doesn’t have any validations for the input value to make sure weather it’s a number or not .
  • The is at least one test case that has passed for all the types of triangles

Many testers with so much information available now for testing still trying to attack on the system with negative tests (people used to call it as negative testing) to uncover some issues on the first phase itself. Even though we uncover many issues there it won’t be worth trying out the same without exploring the basic functionality of the application.

Let’s explore the application with some more Tests

The side of the triangle must be a numeric value. So the application should allow only that kind of data. Application must draw the triangle with in the area given for the same.

  • Try with 6,8,11 – draws out side the box
  • Try with 3,4,5 – Right Angled
  • Try with 6,8,10 – Scalene- // Why so. why can’t it be right angled ?
  • Try with 3,5,10 – Scalene // what ? Basic violation of thumb rules. it must be invalid. more over it just draws a single line and not a triangle
  • Try with 6,7,15 – same as above

There are couple of critical bugs here. For an input data which supposed to be invalid, treated as a triangle and other one is that it draws the triangle out of the box.

So there must be some goof up happened while handling the conditions on input data to satisfy them as valid or invalid. This has been proved now with the way that the above tests revealed the application behavior.

Let’s do some code based testing now since it’s a java script and the whole code is available.

Step6: White Box testing on the Application

Let’s look at some important piece of the code.

if (this.s1 >= this.s2 + this.s3) {

 return "Invalid";

}if ((this.s3 == this.s2) && (this.s2 == this.s1)) {

 return "Equilateral";

}

if (Math.pow(this.s1, 2) == (Math.pow(this.s2, 2) + (Math.pow(this.s3, 2)))) {

 return "Right";

}

if ((this.s3 != this.s2) && (this.s2 != this.s1)) {

 return "Scalene";

} else {

 return "Isosceles";

}

The above code helps us to get the type of triangle based on the input data. Some observations from the above code

  • If there are no validations for the input values, then it’s going to be a major problem. Think of the scenarios like (0,0,0) or (-2,3,4)
  • For this (this.s1 >= this.s2 + this.s3) condition we need to assume that the input data must be sorted from max to min being the s1 as max value and s3 as min value. Else this is going to be big blowup

Look at this bunch:

Triangle.prototype.setSides = function(s1, s2, s3) {
	sides = [s1, s2, s3];

 sides = sides.sort();	this.s1 = Number(sides[2]);

 this.s2 = Number(sides[1]);

 this.s3 = Number(sides[0]);

this.scale = 310/this.s1;

 this.scaledSide1 = this.scale * this.s1;

 this.scaledSide2 = this.scale * this.s2;

 this.scaledSide3 = this.scale * this.s3;

};

It’s reading the input data, sort it and assign the same for the values for the edges. I am concerned about the sort() method here. It’s a block box for us and need to explore on what it does. Other piece of does simple operations.

Let’s ask the google again for some help here. Find some important reading here on the sort method.

Oh my god. This is the problem. It’s doing a generic sort and not tuned towards the numeric sort. So this is the root cause of all the issues. It’s such an issue that even a programmer might over look the same by it’s nature

Find some info on how JavaScript sort works below from the link

Example

In this example we will create an array containing numbers and sort it:

<script type="text/javascript">
var arr = new Array(6)arr[0] = "10"arr[1] = "5"

arr[2] = "40"

arr[3] = "25"

arr[4] = "1000"

arr[5] = "1"
document.write(arr + "<br />")document.write(arr.sort())
</script>

The output of the code above will be:

10,5,40,25,1000,11,10,1000,25,40,5

Note that the numbers above are NOT sorted correctly (by numeric value). To solve this problem, we must add a function that handles this problem:

<script type="text/javascript">
function sortNumber(a,b){return a - b

}
var arr = new Array(6)arr[0] = "10"arr[1] = "5"

arr[2] = "40"

arr[3] = "25"

arr[4] = "1000"

arr[5] = "1"
document.write(arr + "<br />")document.write(arr.sort(sortNumber))
</script>

The output of the code above will be:

10,5,40,25,1000,11,5,10,25,40,1000


Step7: Summary:

I have taken a this approach for this application since it’s simple to apply & explore in this context. It’s good to always start exploring the applications by gathering information from all walks.

Keep getting the info, fine tune your tests and observe the application behavior. Set the base ground right and dig more and keep digging. You are bound to uncover all the important issues with the application.


%d bloggers like this: