Archive for the ‘Code Based Testing’ category

Non linear paths from Application Code

October 4, 2007
The applications become complex as their code base increases. This has challenges for the testers to determine the nonlinear paths in the application.

Most of the Static Analysis tools over the application code helps us to  identify the  cyclomatic complexity (nonlinear paths) at a method level. These might be helpful to validate those methods and to achieve good code coverage over the same.

But the Code coverage at a Unit Level may not be a big help since most of the end user scenarios won’t run after unit level paths. These paths is an integration of the above unit level paths.

Since the Testers focus on simulating the end user scenarios, it will be good to identify all the possible nonlinear paths around the application code base and capture the code coverage based on these paths.

You might want to go through some discussion around this on Linkedin Answers

In case you have similar experiences over white box testing drop me a mail at

Happy Testing…

Powered by ScribeFire.

Whitebox Testing – Is it really white ?

September 7, 2007

The popular myths around Blackbox & Whitebox Testing are by it’s name. It’s black since we can’t see it (don’t have access to the code) & it’s white since you have access to all the code. But then, With in the code there are many black boxes inside and it’s tough to have access to that code base.

  • We don’t have access to code of a language API. Most of the applications have been built on top of a API & assume that the API works fine
  • Most of the application do integrate some third party tools over it’s API. We don’t have access to that code base.
  • We don’t have access to the code of Compiler
  • We don’t have access to code of rum time engine that executes our application code
  • We don’t have access to the code of Operating System Services on top of which the application runs

The list goes on and there are many black boxes in side our code too. We are just testing the code written for the application and it’s better to call it as Code Based Testing rather than Whitebox Testing

— Happy Testing..

Use Cyclomatic Complexity to determine the Risk and Test Scenarios

July 19, 2007

Cyclomatic Complexity (CC) is a software metric (mostly code based metric) used to the number of independent path executions in the application. Introduced by Thomas McCabe in 1976, it measures the number of linearly-independent paths through a program module.

It helps the developers to determine the independent path executions and base line unit tests that they need to validate. Using this, the developers can assure that all the paths have been tested atleast once. It’s a great comfort for the developer and their respective managers.

It’s better to write JUnit Tests for all these linearly-independent paths and integrate it with any code coverage tool. These reports help to focus more on the un covered paths and improve the code coverage.

It also helps to evaluate the risk associated with the application. The following are the results published by SEI and they are being followed widely to determine the health of the code base.

Cyclomatic Complexity      Risk Evaluation
1-10                                              A simple program, without much risk
11-20                                            More complex, moderate risk
21-50                                            Complex, high risk program
Greater than 50                          Un testable program (very high risk)

Explore more at Cyclomatic Complexity in Software Technology Roadmap from SEI.

Further Reading on the topic

Use metrics to evaluate the risk early in the cycle & improve your test coverage.

Happy Testting …

Hey Testers, Communicate the Value of Testing

June 15, 2007

It’s almost a month since my last post on this blog and busy with my upcoming release of QuickRules BRMS. I have been talking to the people around on the Software Testing and felt that it’s not communicated well. Though there is enough information on this subject, i would like to describe my own version of the same here.


Testing is about making things better by providing constructive criticism based on the context (we can also say qualitative information and not being nice) at the right time and in the right direction too.

I like the phrase, “Testers, you are the headlights of the project” from the book Lessons Learned in Software Testing.

I have been thinking about this concept helped for the individuals. This revealed lot of crucial information and i hope this helps my fellow testers to motivate them & their teams.

There are many real life testers (incase if we need to list all of them) who contributed & still contributing a lot for us in every phase of life to grow and improve upon (fix the imp bugs among ourselves).Let’s explore some of them below.

My Parents are the first testers in my life. They contributed invaluable information at each stage (Milestone release) of my life. Instead of saying “Sshhhhhh you can’t do that, they used to tell me further implications that might arise”. An insight into this tells us that it’s not an order, but there is context based information for informed decisions.This helped me to stop for a while, analyze the information and work on the required steps to improve upon the current state.

My teachers helped me a lot by providing the constant feedback (just not being nice) through assignments, tests and covey the areas which are good and bad for me in the respective subjects. They are the best testers because they are the ones who taught the concepts and observed my execution towards the same.

There is a tremendous scope for the improvement, incase if we have acted upon the feedback at the right times.

My Boss at work used to evaluate (Test the Tester) me & provide the feedback on the tasks performed by me. This information helps to analyze to identify the next set of steps to be taken for the improvement.

If we look back, there are many testers around us providing the qualitative information to make things better and improve upon.

The Value of this information is tremendous since it came from people who are more experienced and passed through the current stage where we stand. The value lies in the fact that most of the successful people around, learnt a lot from others (learn from others mistakes too rather than your own) and they have become experts in their own fields.

How does this helps Software Testing

Software testing too comes under the similar lines and its role is to provide context driven information for the stake holders to make informed decisions over the application under test (AUT).

So as being testers, we need to provide the constructive criticism at each stage of the Development. If we look back at the above scenarios, the value addition is more because the people involved there have better skills over the context.

That being said, the current industry lacks skilled testers. The true value addition in Software Testing will be more, if and only if the people involved there have better skills over the context they working with.

Do share your views here or send them to me at

Happy Testing…

Open Source Tools to test your Java Code

April 16, 2007

Open Source tools are being used aggressively and there are many tools that can help Java Applications to validate their code.

The following issues can be uncovered

  • Unhandled exceptions
  • Infinite Loops
  • Dead code
  • Validations against Java coding guidelines from Sun

The following Metrics can be captured

  • Source Code Metrics
  • Coverage Metrics
  • Dependencies with the Design

Couple of resources listed below helps in implementing the same.

Taking on “Testing Triangles – A classic excercise”

March 24, 2007

Solving puzzles, sudokus & the analytical problems is one of my hobby. This set of puzzles always helps me with some brain food which mandatory to stay tuned in the current trends.

I was going through this Testing Triangles post from Elisabeth Hendrickson from Quality Tree Software. The triangle program is very much popular to the world of testers and many of the books and blogs have already covered the same.

Elisabeth posted this as a testing puzzle and asks the people to uncover the bugs in her program. The beauty of this Triangle program is that whole code is available for the tester since it’s written using JavaScript.

I couldn’t stop my self exploring the program and discovering some issues with Testing Triangles and logging them into comments sections while in hurry too.

I thought for a while on this program in the evening and felt that there are many similarities for this even though it’s very very small when compared to the real world applications we test today. So let me take a re-look at this program and apply my test approach for the problem.

Step1: What is the Problem ?

We need to test a program that takes the three sides of a triangle and tell the user about what kind of triangle is it. Here is that Triangle program which has been written in Java Script.

Find the description from the post here

This version of the program takes as input three numbers representing the size of the sides of a triangle. When the user clicks “Draw” the program draws a picture of the triangle with the size of the sides shown in proportion and also displays the type of triangle.

Step2: Question the problem & get the required information for the Test.

The Testers first need to understand the application and the scenarios in which the same might be used.

So with this perception, let’s re-look at the information available for the application. This is true that the above information provided for the application, acts as Requirements, User Manual and any other info we can call it as.

Like many applications that we test now, even this doesn’t have clarity on the information provided. This is also one of the reason for me to call it as a classic example for testing. If we dig more into it the clarity on the information is missing and there are lot of implied requirements (at least from the developer perspective) /assumptions made by the developer.

It’s not clear from the above description that

  • What is a Triangle and how to define the same ?
  • How many types of triangles are there and how to define each one of them ?
  • Are there any boundary limits for the size of the sides that makes triangle ?
  • Can any given three numbers make a triangle ?
  • etc..

Since the info is not available and the Triangles are very generic. Let’s try to Google around the same and get some info to explore the application.

So get the info here, here, here, and here

Step 3: Sanity / Smoke Test for Triangle Program

By now, I know what a triangle is to begin with and how the same can be drawn. Let me do some sanity checks here.

  • For the Test Data of 1, 1, 1 as the values for three edges of triangle the output of the application is “Triangle Type: Equilateral”
  • For the Test Data of 2, 2, 3 as the values for three edges of triangle the output of the application is “Triangle Type: Isosceles”
  • For the Test Data of 5, 6, 7 as the values for three edges of triangle the output of the application is “Triangle Type: Scalene”


I have started with some very basic positive tests to make sure that the application is inline as per requirements. It’s not good to test the application against negative tests to begin with.


Let’s explore some more tests.

  • Try with 0,1,2 – Invalid – wow // might have checked for 0
  • Try with -2,3,4 – Invalid – wow // might have checked for -ve values
  • Try with 1,2,3 – Invalid – // what ? why is this invalid

For the Test Data of 1, 2, 3 as the values for three edges of triangle the output of the application is “ Triangle Type: Invalid

The Application says it’s invalid. But as per the above spec it should be a Scalene Triangle since all the edges are different from each other.

The following queries came to my mind

  • Is this a bug with Application ?
  • Am I missing any more info that’s required for the test ?
  • Can any given three edges make a triangle ?
  • Is there any logic behind the same ?
  • But it could have been good,if the application says why the data is invalid. I have no clue on this and it’s a usability issue.


The general tendency that i see in many of the testers is to report it as a bug the moment they see some different behavior with the application and run into many discussion over the same. I have already blogged on it with a post as That’s Expected Behavior and not a Bug. A little more depth into information gathering will be more effective.

Let’s see why this test data is invalid for the application.

I got some more information again over the web on the edges of the triangle that will help me to understand the issue. And this is the logic behind the same. Get more info on this here.

In order to construct a triangle, the sum of any two edges must be greater than the third one.


So with the above data of 1,2,3 we can’t construct a triangle and the data is invalid. But a error message here on why this is invalid from the application could have saved us lot of time in exploring the same.

One more test with 3,4,5

For the Test Data of 1, 2, 3 as the values for three edges of triangle the output of the application is “ Triangle Type: Right”


Oh the behavior is different again. i am expecting of Scalene Triangle. Some more info via message could have been helpful for users on why the same has been labeled as that particular type. Might be the developer expecting all the users to have the domain knowledge.


Step4: Thumb Rules of the game

By now, we have capture a bit more info on the application & let’s list the same here

  • Rule 1 : Sum of all the angles inside a triangle must be 180 degrees.
  • Rule 2 : In order to construct a triangle, the sum of any two edges must be greater than the third one.
  • In case of sides, they are called as equilateral triangle, isosceles triangle, right angled and scalene triangle
  • In case of angles, they are called as right-angled triangle, obtuse triangle and cute triangle


It’s a good practice to update this section as and when we get the more info about the app behavior & fine tune the same.


Step5 : Black Box Testing on the Application

We have the following info as of now through above test observations.

  • Application doesn’t have any validations for the input value to make sure weather it’s a number or not .
  • The is at least one test case that has passed for all the types of triangles

Many testers with so much information available now for testing still trying to attack on the system with negative tests (people used to call it as negative testing) to uncover some issues on the first phase itself. Even though we uncover many issues there it won’t be worth trying out the same without exploring the basic functionality of the application.

Let’s explore the application with some more Tests

The side of the triangle must be a numeric value. So the application should allow only that kind of data. Application must draw the triangle with in the area given for the same.

  • Try with 6,8,11 – draws out side the box
  • Try with 3,4,5 – Right Angled
  • Try with 6,8,10 – Scalene- // Why so. why can’t it be right angled ?
  • Try with 3,5,10 – Scalene // what ? Basic violation of thumb rules. it must be invalid. more over it just draws a single line and not a triangle
  • Try with 6,7,15 – same as above

There are couple of critical bugs here. For an input data which supposed to be invalid, treated as a triangle and other one is that it draws the triangle out of the box.

So there must be some goof up happened while handling the conditions on input data to satisfy them as valid or invalid. This has been proved now with the way that the above tests revealed the application behavior.

Let’s do some code based testing now since it’s a java script and the whole code is available.

Step6: White Box testing on the Application

Let’s look at some important piece of the code.

if (this.s1 >= this.s2 + this.s3) {

 return "Invalid";

}if ((this.s3 == this.s2) && (this.s2 == this.s1)) {

 return "Equilateral";


if (Math.pow(this.s1, 2) == (Math.pow(this.s2, 2) + (Math.pow(this.s3, 2)))) {

 return "Right";


if ((this.s3 != this.s2) && (this.s2 != this.s1)) {

 return "Scalene";

} else {

 return "Isosceles";


The above code helps us to get the type of triangle based on the input data. Some observations from the above code

  • If there are no validations for the input values, then it’s going to be a major problem. Think of the scenarios like (0,0,0) or (-2,3,4)
  • For this (this.s1 >= this.s2 + this.s3) condition we need to assume that the input data must be sorted from max to min being the s1 as max value and s3 as min value. Else this is going to be big blowup

Look at this bunch:

Triangle.prototype.setSides = function(s1, s2, s3) {
	sides = [s1, s2, s3];

 sides = sides.sort();	this.s1 = Number(sides[2]);

 this.s2 = Number(sides[1]);

 this.s3 = Number(sides[0]);

this.scale = 310/this.s1;

 this.scaledSide1 = this.scale * this.s1;

 this.scaledSide2 = this.scale * this.s2;

 this.scaledSide3 = this.scale * this.s3;


It’s reading the input data, sort it and assign the same for the values for the edges. I am concerned about the sort() method here. It’s a block box for us and need to explore on what it does. Other piece of does simple operations.

Let’s ask the google again for some help here. Find some important reading here on the sort method.

Oh my god. This is the problem. It’s doing a generic sort and not tuned towards the numeric sort. So this is the root cause of all the issues. It’s such an issue that even a programmer might over look the same by it’s nature

Find some info on how JavaScript sort works below from the link


In this example we will create an array containing numbers and sort it:

<script type="text/javascript">
var arr = new Array(6)arr[0] = "10"arr[1] = "5"

arr[2] = "40"

arr[3] = "25"

arr[4] = "1000"

arr[5] = "1"
document.write(arr + "<br />")document.write(arr.sort())

The output of the code above will be:


Note that the numbers above are NOT sorted correctly (by numeric value). To solve this problem, we must add a function that handles this problem:

<script type="text/javascript">
function sortNumber(a,b){return a - b

var arr = new Array(6)arr[0] = "10"arr[1] = "5"

arr[2] = "40"

arr[3] = "25"

arr[4] = "1000"

arr[5] = "1"
document.write(arr + "<br />")document.write(arr.sort(sortNumber))

The output of the code above will be:


Step7: Summary:

I have taken a this approach for this application since it’s simple to apply & explore in this context. It’s good to always start exploring the applications by gathering information from all walks.

Keep getting the info, fine tune your tests and observe the application behavior. Set the base ground right and dig more and keep digging. You are bound to uncover all the important issues with the application.

%d bloggers like this: