In the quest to create high-quality software, most software companies spend enormous amounts of money and resources on testing. Is this a good strategy?
Afew years back, I bought a costly smartphone, attracted by its ‘cool’ features. lnce I started using it, I realised that it was way too buggy. Let me give you two examples. The mobile came with supporting software on a Ca. After installing the software and starting it, it crashed, throwing a sisual C++ runtime error: “R6025: Pure virtual function call”! As a C++ expert, I could easily understand the bug: it is illegal to call a pure virtual function from a constructor, and when we make such a call, it will crash the application. But the question was: why hadn’t the testing team for that mobile phone software caught it before releasing it?
Another strange problem with my mobile is that it ‘freezes’ or ‘hangs’ if I talk for ‘too long’. After a few freezes, , fiJurHd RuW WhDW ‘WRR ORnJ’ PHDnV DpprRxLPDWHOy hDOI-Dnhour, which is not really that long! By freeze or hang I mean the screen would go blank, and it wouldn’t respond to any key presses. It meant I couldn’t restart the mobile (which required pressing keys). The only thing I could do was to rHPRYH WhH EDWWHry Dnd puW LW EDFk! /DWHr, , DOVR fiJurHd RuW D wRrk-DrRund; LI , pOuJJHd WhH PRELOH WR LWV FhDrJHr, LW wRuOd immediately spring back to life! This discovery showed me an important aspect of the problem—since it recovered when an event occurred (plugging in the charger, in this case), it is likely to be a software bug!
When I talked to my friends about this problem, most of them said they were not surprised—they said that bugs are very common in mobiles. Reading about the strategies of mobile phone companies and talking to my friends working in those companies was enlightening. Mobile phRnH FRPpDnLHV IDFH FuW-WhrRDW FRPpHWLWLRn Dnd RnOy WhRVH whR firVW dHOLYHr WhH ODWHVW Dnd FRROHVW IunFWLRnDOLWy to the market, survive! In this scenario, the main quality strategy to test the software is: if the tested features work, it goes to market! Maybe there is a bit of exaggeration in that statement, but it more or less captures the essence of how these companies focus on functionality, and how they use testing as the primary means to check quality.
The situation is not so different in the software industry, where testing is the main approach for improving software TuDOLWy. AFFRrdLnJ WR BRrLV BHLzHr (D wHOO-knRwn rHVHDrFhHr in testing), testing accounts for approximately half of the total software development costs! aepending on the size and type of the software company, the ratio of developers to testers ranges from 5:1 to 1:1! It is safe to say that software companies rely too much on testing.
Let us take a holistic view of testing, to understand why WhH IRFuV Rn VRIWwDrH WHVWLnJ LV nRW WhH wDy WR FrHDWH hLJhquality software.
( : DLMkVWrD, WhH wHOO-knRwn DuWFh FRPpuWHr VFLHnWLVW, once said, “Testing can show the presence of bugs, not their absence.” This statement appears to be a clever play of words—but, if you think about it, it is insightful. What he means is, with testing, you can only check if the software has bugs. If you encounter no bugs while testing, it just PHDnV WhDW WHVWLnJ dLd nRW unFRYHr Dny EuJV; hRwHYHr, yRu cannot say there are no bugs in the software. For this reason, by doing testing alone, you cannot say that software will wRrk finH EHFDuVH LW LV unIHDVLEOH Dnd LPpRVVLEOH WR WHVW DOO WhH pRVVLELOLWLHV. 7hLV PHDnLnJ LV rHflHFWHd Ln WhH 86 )RRd and arug Administration’s guidance statement on validation IRr PHdLFDO VRIWwDrH: “6RIWwDrH WHVWLnJ Ey LWVHOI LV nRW VuIfiFLHnW WR HVWDEOLVh FRnfidHnFH WhDW WhH VRIWwDrH LV fiW IRr its intended use.”
ves, there has been considerable progress in software testing research, and today there are sophisticated testing tools available, but the statement that aijkstra made, and the position of the FaA on testing, still holds. To get better clarity on the limitations of testing, let us discuss testing from two different perspectives.
5HDO-wRrOd VRIWwDrH LV FRPpOHx, Dnd LWV FRPpOHxLWy LV rising every year. For example, the size of Windows 3.1 was DpprRxLPDWHOy 4 PLOOLRn /2C Ln 1990; Ln 2002, LW wDV 40 PLOOLRn /2C IRr :LndRwV X3. , dRn’W knRw WhH FRdH-EDVH sizes of the latest releases of Linux or Windows, but you can make easy guesses. Many applications I know are more than a million LlC, and their size is increasing every year. 7hH huPRnJRuV VLzHV RI rHDO-wRrOd FRdH-EDVHV PDkH WHVWLnJ HxWrHPHOy dLIfiFuOW. /HW PH JLYH yRu D VLPpOH HxDPpOH.
A few years back, when I was writing code in bclipse 3.1, it crashed. I checked its stack trace, which revealed