Archive for the ‘Code Metrics’ category

Non linear paths from Application Code

October 4, 2007
The applications become complex as their code base increases. This has challenges for the testers to determine the nonlinear paths in the application.

Most of the Static Analysis tools over the application code helps us to  identify the  cyclomatic complexity (nonlinear paths) at a method level. These might be helpful to validate those methods and to achieve good code coverage over the same.

But the Code coverage at a Unit Level may not be a big help since most of the end user scenarios won’t run after unit level paths. These paths is an integration of the above unit level paths.

Since the Testers focus on simulating the end user scenarios, it will be good to identify all the possible nonlinear paths around the application code base and capture the code coverage based on these paths.

You might want to go through some discussion around this on Linkedin Answers

In case you have similar experiences over white box testing drop me a mail at venkatreddyc@gmail.com

Happy Testing…

Powered by ScribeFire.

Advertisements

Ability to identify the hot spots of release from Bug Database

September 24, 2007

Bug Database for the products might have thousands of issues over a period of time against various builds and releases. Though these issues fixed over a period of time, it might be hard to derive meaningful metrics over the release.

We need to support these releases over the production systems & it might be helpful to capture the hotspots / risk elements with the release. Most of the issues here to deal with the respective features, compatibility with other features / technologies & performance related issues.

The usual metrics of number of issues against a module and their severity levels may not be of help always.

How easy is it to derive the following from the Bug Database for a given release

  1. Identify the issues have originated from Requirements, Design & Implementation
  2. Identify the issues over their category (Functional, Performance, Security, Compatibility, Usability etc)
  3. Identify the issues along with their origin & category over the features rather than the modules / components

Powered by ScribeFire.

Use Cyclomatic Complexity to determine the Risk and Test Scenarios

July 19, 2007

Cyclomatic Complexity (CC) is a software metric (mostly code based metric) used to the number of independent path executions in the application. Introduced by Thomas McCabe in 1976, it measures the number of linearly-independent paths through a program module.

It helps the developers to determine the independent path executions and base line unit tests that they need to validate. Using this, the developers can assure that all the paths have been tested atleast once. It’s a great comfort for the developer and their respective managers.

It’s better to write JUnit Tests for all these linearly-independent paths and integrate it with any code coverage tool. These reports help to focus more on the un covered paths and improve the code coverage.

It also helps to evaluate the risk associated with the application. The following are the results published by SEI and they are being followed widely to determine the health of the code base.

Cyclomatic Complexity      Risk Evaluation
1-10                                              A simple program, without much risk
11-20                                            More complex, moderate risk
21-50                                            Complex, high risk program
Greater than 50                          Un testable program (very high risk)

Explore more at Cyclomatic Complexity in Software Technology Roadmap from SEI.

Further Reading on the topic

Use metrics to evaluate the risk early in the cycle & improve your test coverage.

Happy Testting …

Is Counting a Bad Idea for Test Effort?

December 28, 2006

I was going through the post Why counting is a bad idea on Shrini’s Blog on Software Testing. Shirni has listed a classic example on why counting is a bad idea for Testing.

Even though i like this post, I won’t agree fully on the views expressed there. Probably we might need to improve our selves with the process of measuring.

Certain artifacts like test cases, requirements and bugs are not countable things and any attempt to count them can only lead to manipulations and ill-informed decisions.

Wait — Are there any things at all in testing that we can count without loss of effectiveness and usefulness of the information that a counted number might reveal?

Let me describe my stand clearly wrt to this Why counting is a bad idea.

I do agree that it’s not posssible to count all the activities that we do in testing. But then that doesn’t mean like all the Test related metrics are bad. May be, We can’t generalize them since it leads to lot of subjective debate.

When it comes push the usefulness and effectiveness of the information by the metric, this is more of a improvement process. All these metrics are subjective to the respective projects / environments / organizations. Any attempt in making this as a genaral metric will lead to manipulations and ill-informed decisions.

  1. Bad Idea for whom ?
    Is this for Customers / Management / Developers / Testers / Others
  2. If Counting is a bad Idea, how do we measure the effort spent for the Project ? Do we need to go by the gut feelings of te people ? Will there be any data to justify our decisions ?

Most of the metrics used for Software Development for that matter are subjective. It varies based on the application, technology, domain, global support etc. It’s tough to make them general like we do in other industries. So the same will be applicable for Testing too.

Some thing like below are very much subjective and won’t be accepted in genaral.

Productivity

  1. No of Test cases prepared Per Person Per hour = 5
  2. No of Test cases executed per person per hour = 15

Look at the subjectivity in the above scenario.

  1. What is the structure of the Test Case ?
  2. How many user actions will be there in genaral per TestCase ?
  3. What is the Volume of Test Data associated with each TestCase ?
  4. What information available for these Test Cases (SRS, Design, UseCases etc) ?
  5. What kind of test cases are they writing ? Are they writing for Unit, Integration, System, Accepatance, Performance etc..
  6. What kind of Test Design Tech are they following (BVA, EP, DT, OA, CE etc)

There will be so many queries that will be come up here and it’s tough to measure some one just based on the number in genaral. But then since it’s subjective to the project (team), the respective stake holder should be able to take some decision based on that number.

But then, we can’t make this number generic. Probably this might be useful to the similar projects to have some rough estimate.

At the end, i shouldbe able to give qualitaive (again this is subjective) to the stake holders. If some one expects 100% qualitaive info here, that’s an ideal world to which we can’t go as of now.

What does that mean ?

I should be able to give a number. My metrics should tell me the number of Test Cases, Issues etc wrt the requirement.

Otherwise we may not have clarity on what we are doing & what are we trying to deliver. Things will be going on the pure gutfeelings

Pradeep has come up with nice replies to my comments on the post.

@Pradeep,

Thanks for sharing your views.

Let me look at your requirement

” This field should take only alphabets and nothing else other than that “

In the first place, the requirement is not clear for me

  1. What kind of Application are we taking about ?
  2. What is the size of this field (Some thing like password. It should have a Min and Max length)
  3. When you say Alphabets, Do you mean to support all the alphabets in all the languages or just english or some thing else?
  4. ????

See Pradeep, my queries will go on like this and all these will help me in getting the clarity on the requirement. This is how i try to to understand the system.

This process helps me in knowing on what is the requirement, it’s advantages, it’s limitations and most possible scenarios where this can be used.

All the above will help me in capturing the required tests for the above requirement.

But this may not be generic on the whole but then it can be generic to the Organization or at least to the project people are working with.

See there should be some way to measure on what we are doing. This measure should help the stake holders to take a decision on the quality of the application under test.

I do agree on the fact that we need to improve ourselves in coming up with measures that have qualitative information for the stake holders.

@ Shrini, Thanks for coming up a post that has good debate.


%d bloggers like this: