Case I
In May 1995, 823 customers of a major bank in the US suddenly found $900 million credited into their accounts. They were not sure whom to thank till it came to light that a programming error was responsible for their swelling credit—perhaps the most costly error in banking history.

Case II
Nike’s shortfall of $100 million in the third quarter of 2001 was due to flaws in its supply chain management system. 45% of all system downtime in the US in 2000 was due to defective systems.

The situations described above are a nightmare for the software industry—for enterprises get paralyzed due to faulty software. Defective software results in huge losses. In the US alone it runs into billions of dollars and it is estimated that losses represent 1% of the country’s GDP.

Why do software programs or products fail in the first place? The answer probably lies in the multi-pronged issues that affect software quality itself. For instance, the software sold in the market comes out with disclaimers, end user license agreements (EULA) and limited liability clauses from the vendor. Software solutions also fail because the vendor and the user companies find great difficulty in measuring quality. While vendors base their products on common quality attributes like reliability, efficiency, and functionality, in reality there are many glitches that surface in user conditions. Oversight of these defects by the enterprises often leads to catastrophic consequences.

This view only gets reinforced, once we look at what Gary Chapman, director of the 20th century project conducted by the University of Texas noted, "Repeated experiences with software glitches narrows down the use of computers to familiar and routine tasks. Our studies have shown that most of the users rely on less than 10% of the features of common programs like word processors and browsers." Users remain unaware of the fact that they are using defective software.

As Sashi Reddi, CEO, National Software Testing laboratories (NSTL) India puts it, "Software has been historically plagued by defects. In other industries, where defects are considered as a serious flaw, a user of software is comfortable if the solution merely works. Some of this lax approach to software quality is changing as a result of growing recognition of the costs incurred due to defects in the software."

The evolution
Software testing for a long time was centered on fixing bugs in the program only when the user encountered it. This was more on a demand basis than pre-empting the problem at the project completion stage. As Watts Humphrey, the software quality guru claimed, "bugs are like landmines that can explode any moment." This observation sums up the consequences of defective software processes. According to KR Jayakumar, CEO, Amity Soft Technologies, "The concept has considerably evolved in the recent past. Testing which used to be an after thought has now evolved into a major activity. Moreover software tools have changed the landscape, and as a result testing has become a specialized discipline." "Independent third party testing is the peak of this maturity, it signals a new breed of software professionals and an industry by itself," adds Jayakumar. One of the biggest drivers that will infuse more quality into software will come when enterprises become more conscious of the deliverables promised by the vendor.

A recently published report of the National Institute of Standards and Technology (NIST) says, " Software testing is the process of applying metrics to determine the product quality. It is the dynamic execution of the software and the comparison of the results of that execution against a set of predetermined criteria." Analysts firmly believe that software testing is an infrastructure technology that provides three main quality attributes—conformance, performance, and interoperability—to a solution.

Comments A V Asvini Kumar, CEO, Thinksoft Global, "Testing is always a part of SLA’s and typically takes up 30% of all efforts spent on a software project. But in the last two decades it has assumed significance because, software now dominates our lives in myriad ways. Also software now has become mission critical, life critical, and we are dependent on complex systems for our survival and hence delivering quality systems has become a mandatory requirement."

The market scenario
Testing the solutions thoroughly before it reaches the end-user is a major mandate every software vendor faces today. The one way of measuring quality is through a wide range of testing tools available in the market. However, the market is still fragmented. Hence the testing procedures adopted by companies vary. This leads to issues of standardization and testing methodologies. Industry estimates put the market for software testing tools at $ 931 million and say that it will grow to $ 2.6 billion by 2004.

The market for testing tools can be divided into two broad streams—application and web testing tools. An application test tool evaluates the software’s source code, functionality etc. A web testing tool addresses requirements like HTML testing, load /traffic management among others. In the testing tool space, companies like Rational Software have carved a niche for themselves. Rational for instance has got a slew of testing solutions (Pure Coverage, Purify, Visual Test, Rational Robot etc) that improves the speed, quality, and the predictability of the software projects. Says Eric Shurr, chief marketing officer, Rational Software, "TestStudio and other Rational testing tools liberate development teams from the time consuming manual tasks by automating the testing processes" Other major players in the testing firmament include names like Segues’ SilkTest, WinRunner from Mercury and Compuware’s QA Run and AutoTester.

Closer home, the market for software testing is still nascent. The handfuls of companies doing this kind of work are off shore in nature. Comments Asvini Kumar, "In India we have not evolved to a stage where information is a valued as a commodity. Hence the low index of deployment, usage of IT applications puts quality issues on the back burner as resources, and investments become top priority. "But this situation is gradually changing and five years down the line we will see a lot of action in this space," adds Kumar.

Meanwhile Sashi Reddi of NSTL has a slightly different view. He believes that Indian enterprises are clearly aware of this concept and have already started demanding quality solutions. He goes on to say, "India has the largest number of CMM level 4 and 5 developer companies. These companies have clearly recognized quality benchmarks as a distinct competitive edge."

The economic impact
Let us look at couple of events that sums up the far-reaching consequences of defective software. In April 1999 a bug played truant with the $1.2-billion US defense satellite launch. This is one of the costliest unmanned accidents in recent times. Close on its heels, in October 1999, the $125 million NASA Mars Climate Orbiter was lost in space due to a small data conversion error. Investigations into the incident revealed that the Orbiter frayed out of the set path because the software used some data in English units that should have been in metric units. The third incident occurred in December 1999 when the Mars Polar Lander mission failed for unknown reasons. But critics dubbed that one also as due to defective software processes. The failed missions exposed the loopholes in the US space program, prompting the government to order a review of space launch programs including the software used and the testing processes. The result: the US government lost more than billion dollars due to software problems overlooked at the testing stage.

Analysts estimate that in the US alone inadequate software testing infrastructure leads to losses to the tune of $22 to $60 billion. Interestingly half of the costs are borne by the users who ultimately suffer due to errors in the software. Given the cost implications, both to the vendor as well as the user companies, they are now putting in place or seriously considering a software testing architecture as part of their MIS policy. Says Jayakumar, "Enterprises will no longer be able to absorb costs due to defective software due to the highly competitive and low margin economic situations. On the developer side we see a growing degree of realization in testing their solution before it reaches the end user. For instance we are undertaking three major testing projects this month alone, from major companies. This reflects how quality-conscious developers are becoming in recent times."

The Challenges
Despite growing awareness, software testing even today becomes an activity towards the end of the product development lifecycle. This leads to testing that is not comprehensive, and hence results in the release of sub-quality software. Reflecting on this Sashi Reddi says, " The biggest challenge is to convince developers and enterprises to have a QA department working with the IT teams right from the moment of conception to implementation. Another main challenge is to enforce a clear separation between the development team and the testing team that will truly foster independent testing."

Latha Ramesh, a testing specialist with Chennai-based HiQ Consulting says that the dearth of qualified testing professionals is a major challenge confronting the testing industry. She says that in India software testing is not looked upon as a career choice since many consider it as a mundane exercise. "This perception has to change, only then can we kickstart a quality revolution," she adds.

Other issues like lack of standardized testing procedures and competitive pressures in delivering a software solution in the shortest possible time are potential threats to software quality. As Asvini Kumar puts it, " The pressure to reduce both costs and the time to market, unwittingly results in shortsighted budgeting approaches and as a consequence, systems are delivered with invisible defects."

Also, software project managers and development teams are quite attached to the product and hence it takes a lot of courage to spell out the problems with the developers to correct them. "Test project management is the toughest job," says Jayakumar. In the end, what emerges is that there’re still a lot of inadequacies in the realm of software and testing. But across the world, vendors and users are becoming increasingly conscious about the solutions they are developing and implementing. The emergence of quality benchmarks like ISO 9001 and SEI-CMM defined levels have ushered in a quality culture. But still, invisible errors unknown to the developer create chaos after implementation.

Shrikanth G in Chennai