One of my favorite pastimes is lunchtime with software engineers where I like to pose the following question: on average, how many software defects exist per 1,000 lines of delivered code? I’ve gotten answers across the board, and research backs up similar results – answers vary widely. But no one says “none.” Results are subject to many factors, from programming language to developer skill, but some seem to suggest it is in the range of 10-20 per 1,000 lines.
For comparison, I just finished a small piece of C-code totaling 780 lines. Once all the obvious errors were dealt with and it compiled, the memory checker found four more serious issues. That’s roughly five in 1,000 lines – but those are just memory issues like uninitialized memory reads and array overruns by one, etc. It does not indicate completeness, correctness, or even testing for infrequent code paths not included in the run. 10-20 per 1,000 seems right on the money to me.
The problem is cryptography may be mathematical algorithms, but the math is implemented in code. Bugs in cryptographic code happen. Recall the Java 15+ certificate validation bug? The bug was failing to check that the integers used in the algorithm were of sufficient size. In fact, integers of 0 qualified just fine, and so a certificate of all zeroes would be accepted as valid for whatever identity you tried to assume. Big bug, big consequences.
In similar fashion, the Heartbleed bug resulted from the ability to read bytes out of the permitted range, bytes that were not set to zero, and therefore could leak memory which contained sensitive or private information. The particularly nasty use-case being public web servers then leaking their private key. A simple bug, potentially big consequences.
This begs the question: how bad is it?
The public cryptography library GCrypt has roughly 200,000 lines of code. Does that mean there’s 2,000 to 4,000 defects hiding? Probably not. In fact, when tracking public bug repositories many software defects are found and remediated. In some instances, code can go back a decade or longer. If the first time each bit of code was added to the repository, introduced with a defect rate of 10-20/1,000, it certainly did not stay that way over the years as bugs were found and remediated.
Since code is added all the time, we can realistically expect the older code to be significantly more defect-free than the newer contributions. This leaves different pieces of code in various stages of bugginess. A blanket estimate cannot be made, and certainly the totals are suppressed by this incremental bug fixing.
Another important factor is the fact that not every defect results in a security vulnerability such as the Java or Heartbleed examples. Most bugs simply lead to odd behavior like a crash, none of which would be exploitable. Similarly, in many cases the most common and frequently executed code paths are only a small fraction of the total program code, and defects out of the critical path may simply not be consequential or reachable at all. These are mitigating factors… but it only takes one.
Given the millions of lines of cryptographic code, combined with the mitigating factors discussed above, it is no surprise that very serious cryptographic bugs are few and far between. On the other hand, they happen, and we cannot predict when or where.
Another uncomfortable reality is that I have chosen well-known, frequently used, and even open-source libraries for my comparisons – those that are subject to constant scrutiny. Many cryptographic implementations exist that are private, in-house, closed-source, and used in small numbers. A lot of the mitigating factors for those libraries vanish and a much higher bug-rate can be expected.
We will not be able to quantify exactly which cryptographic implementation will fall on the sword next. There is no way of knowing where or when the next code defect will take down an entire crypto system and all the data and communications it protects. No matter how solid the mathematics, we must live with the realization that implementations will be imperfect. An uncomfortable reality for sure.
But there is hope. Software bugs will form singular failure points in cryptography, but only if we ignore the fact that software bugs are a fact of life. Defense-in-depth and redundancy – also cryptographic redundancy – can build a dependable system out of unreliable parts. This is a well understood engineering practice, and applies equally to software engineering, if we are willing to accept that defects will always be there in the code we write.
Read the complete Single Points of Failure blog series by Vince Berk, Chief Strategy Officer at Quantum Xchange, including the Human Factor, Weak Entropy, Asymmetric Encryption, Password Strength and Public Handshake & Key Derivation.
