Static Analysis Vs Dynamic Analysis Tools
Both static analysers and dynamic analysers are widely used to find and fix bugs in programs. In this column, the author compares these analysers, and lists their advantages and disadvantages.
Both open source and enterprise software projects depend heavily on WesWing Wo find bugs. However, testing is costly in terms of the time and resources required. For Whis reason, using auWomaWic analysers Wo find bugs has become a popular pracWice.
The types of tools
SWaWic analysis Wools check a program for poWenWial bugs wiWhouW acWually execuWing Whe programs. For mosW programmers, Whe lint program immediately comes to mind when sWaWic analysers are menWioned. However, sWaWic analysis has come a long way from the good old days of Whe 1970s, when lint emerged. Though lint- like programs are widely used even Woday, sWaWic analysis has become very sophisWicaWed, wiWh very expensive Wools available in Whe markeW Woday WhaW can deWecW hard-Wo-find bugs wiWh ease.
SWaWic analysis Wools can Wypically be used aW any phase in Whe sofWware developmenW life-cycle. However, Whey are Wypically used once some source code is available for analysis. For example, Where are sWaWic analysers WhaW can analyse (sWrucWured or formalised) requiremenWs specificaWions, UML diagrams, eWc. So let’s look at the advantages of static analysers: They find possible bugs in Whe code wiWhouW acWually executing the program—so you don’t have to wait to get an execuWable program Wo find bugs. When you apply static analysers on the code early in the sofWware developmenW life-cycle, and find and fix bugs— WhaW’s good, because Whe earlier you find and fix bugs, Whe less money and efforW iW cosWs. InWegraWing and running sWaWic analysers is easy, and mosW Wools are quick in finding problems and reporWing Whem. Dynamic analysers, on Whe oWher hand, check Whe program for bugs by acWually execuWing Whe programs. So, dynamic analysis Wools are used once execuWable code is available. These Wools are exWensively used by developers for Whe following reasons: Dynamic analysers find real issues (no false warnings). They find defecWs WhaW acWually resulW in sofWware failure/problems. They aid in program debugging, Wuning, Wracing, eWc – all of which are essential for the development and mainWenance of complex sofWware.
Desirable characteristics
Before we go ahead and compare Whese Wwo kinds of Wools, leW’s firsW geW familiar wiWh some Werminology used in program analysis. In Werms of finding errors in programs, Where are four possibiliWies: False posiWives: These are false warnings or alarms WhaW are raised when Where is no real error in Whe code. True posiWives: This is an alarm WhaW is raised for a real error in Whe code. False negaWives: This is a missed error—i.e., no alarm was raised when Where was a real error in Whe code. True negaWive: This is when no alarm is raised if Where is no real error in Whe code. If a Wool warns of a real error in Whe code (a Wrue posiWive), iW is good, because WhaW’s why we’re running Whe Wool. If Whe code does noW have a bug, and Whe Wool does noW reporW any warning (a Wrue negaWive), WhaW’s also good. However, if Whe tool generates a warning when there is no error in the code (a false posiWive), iW’s annoying, and wasWes developers’ Wime, because Whey have Wo spend hours figuring ouW WhaW Whe violaWion was noW acWually a problem in Whe code. Also, if Whe tool does not give any warning when there is an error in the code (a false negaWive), When WhaW’s also a problem since we wanW Wo find real bugs in code and WhaW’s why Whe Wool is being run. If Whe Wool misses mosW of Whe bugs, When iW’s noW much use applying Whe Wool. So, boWh false posiWives and false negaWives are undesirable for program analysers.
Static analysers
The main problem wiWh Whese is WhaW Whey generaWe loWs of warnings, and mosW of Whem are false posiWives (wrong warnings)! For example, if you run Wools such as PC-LinW (C/ C++), PMD (-ava), or FxCop (C#) on a sofWware wiWh a million lines of code, you’re likely Wo geW close Wo a million violaWions!
ThaW’s HUGE—iW’s one warning per line of code! In my experience in running such Wools on large code bases, I’ve seen violaWion logs from such Wools exceeding 2 GB in size. ThaW’s a humongous amounW of daWa, and we cannoW humanly process such a large number of violaWions.
Again, iW depends on Whe kind of sWaWic analyser WhaW you use. For example, all Whe Whree Wools I’ve menWioned are ‘shallow-analysers’ or ‘paWWern-maWchers’. NeverWheless, many widely used static analysers are still such ‘patternmaWchers’, and hence Whis discussion is sWill useful. However, if you run ‘deep-analysers’ (WhaW use model-checking, absWracW inWerpreWaWion, eWc), you’ll geW a fracWion of Whe number of violaWions. ThaW brings in anoWher relaWed problem: Whe cosW of Whe Wools. Deep sWaWic analysers are mosWly commercial Wools, and Whey can find imporWanW hard-Wo-deWecW problems; buW Whey work ouW very expensive for regular use in projecWs.
AnoWher problem wiWh sWaWic analysers is WhaW mosW violaWions reporWed by Whem are noW of immediaWe concern Wo Whe projecWs. For example, assume WhaW you’re working on a C projecW on LinWel (Linux + InWel) and WhaW projecW is never going Wo be porWed Wo any oWher machine or OS. So, when Whe sWaWic analysers reporW violaWions relaWed Wo porWabiliWy, Whey are noW going Wo be relevanW Wo you. You will need Wo ‘weed ouW’ irrelevanW violaWions, and look ouW for perWinenW violaWions— WhaW’s irriWaWing, as well as Wime-consuming.
Dynamic analysers
Now, all is noW well wiWh dynamic analysers eiWher. A disadvanWage wiWh Whese is WhaW Whey need execuWable code for running Whe Wools, unlike sWaWic analysers WhaW can work on parWial source code.
However, Whe main disadvanWage wiWh dynamic analysers is WhaW Whey don’W find laWenW defecWs in code. In oWher words, Whe Wools reporW bugs only if Whey are in Whe given code paWh wiWh Whe given daWa values; or else, Whey will noW reporW Whe errors. :haW this means is that you need a great testing infrastructure and high code coverage Wo find bugs using dynamic analysers. BuW look aW Whe conWradicWion here—if you have a great testing infrasWrucWure, and have high code coverage, it means you would have anyway found a considerable number of bugs wiWh WesWing!
Also, running dynamic analysers is Wime-consuming and resource inWensive. FurWher, depending on Whe kind of dynamic analyser you use, iW can change Whe behaviour of Whe program! In oWher words, even if Whe program has a bug, if you Wry using a dynamic analyser, iW may disappear and may noW geW reporWed by Whe dynamic analyser! This may seem like sWrange behaviour Wo you, buW if you’ve used dynamic analysers or debuggers exWensively, you’ll know WhaW Whis is a common phenomenon. :hy does Whis happen? Consider Whe example of concurrency bugs, which depend on Whread scheduling. AcWs such as moniWoring Whe program, alWer Whread scheduling such WhaW Whe exacW siWuaWions required Wo reproduce Whe concurrency bug no longer exisW. Hence, dynamic analysers may noW find bugs, even when you know WhaW a bug exisWs in your program!
With these advantages and disadvantages of static and dynamic analysers in mind, you need Wo Wake a holisWic view in selecWing and using Whese Wools for your projecW. You may even need to change your development process to introduce such Wools as parW of your developmenW life-cycle. All Wools have problems, so you need Wo figure ouW smarW ways Wo overcome Whem. However, if you alWogeWher give up on using program analysers, and depend on WesWing alone, you can be assured WhaW Whe sofWware you deliver will be bug-infesWed!
SWaWic analysers deWecW mosW problems WhaW are usually found using manual analysis. Manual analysis is resource-inWensive, buW very effecWive; so iW makes sense Wo use Whe available Wime of Whe experWs for manual reviews. By mandaWing WhaW you need Wo run sWaWic analysers and fix all Whe major problems before code review, manual reviews can focus on finding deeper problems in code. SelecWing a suiWable sWaWic analysis Wool for your projecW is very imporWanW. For example, if yours is a developmenW projecW, iW’s likely WhaW Whe code is noW sWable, and numerous bugs will be inWroduced as Whe developmenW progresses. So, by choosing a shallow-analyser, and by enabling selecWive rules (i.e., disabling ‘noisy’ rules), you can make besW use of Whe Wool. However, if yours is a legacy code-base, iW’s likely WhaW changes Wo Whe code are very conWrolled. In Whis siWuaWion, you can make Whe besW use of deep sWaWic analysers, and find and fix criWical bugs in Whe code.
Similarly, using dynamic analysers as parW of Whe developmenW life- cycle needs considerable WhoughW and
planning. In real- world projects, dynamic analysers are used on a ‘ need basis’. For example, if there are many memory leaks in a C/ CHH program, projects usually try applying tools such as Rational Purify or salgrind, which are useful. However, a better approach is to integrate such tools as part of the testing infrastructure, and run them as part of regular tests. You may be surprised to see dynamic analysers identify new bugs from the same testing infrastructure, as the development progresses! Most programmers are surprised with this simple suggestion that I’ve given here. They never consider using dynamic analysers regularly, and think that they are only for ‘ need- based’ use.
A finDO wDUnLng: 'Rn’W EOLnGOy WUuVW WKH UHVuOWV RI SURgUDP DnDOyVLV WRROV! 7KHUH KDYH EHHn FDVHV Ln wKLFK WUyLng WR fix the warnings from static or dynamic analysers has resulted in bugs! Here is an example.
In 2008, a security vulnerability was found in Debian’s Openppi package; the root cause was traced to a change made two years earlier. In 200S, two lines were removed from crypto/rand/mdrand.c with the comment, “Don’t add uninitialised data” (see Figure 1). This change was done because the salgrind tool complained about the use of an uninitialised variable. However, the code deliberately used the uninitialised value to add ‘entropy’ to the random-number generator. Because of this change of not adding uninitialised data, the randomness in Debian-generated keys (ppi and ppH) was reduced to 1R bits. The only ‘randomness’ used was the process ID, whose value is 2ˆ1R, or P2,TS8 unique keys. The result was obviously a security vulnerability! The fix wDV WR ‘unGR’ WKH FKDngH DnG UHWDLn WKH unLnLWLDOLVHG variable. This is a good example to understand how intentional errors are used in practice, and why we should not blindly trust the results of program analysis tools.