(Suggested 📚) How to Measure Anything

Deets

• How to Measure Anything, 3rd Edition by Douglas W. Hubbard
• ISBN: 978-1118539279

Review

How to Measure Anything is a book on how to measure things that matter. Something matters if you either care about it or need to make a decision about it. How to improve “worker satisfaction”, how to choose between “clean water” or “clean air” improvements at a factory, how to know some current project is actually going to improve the cybersecurity posture of your product, these are all things that matter and where choices need to made to know what choices to make at the start and if those choices are successful.

You can’t measure a cover, but you can look at it.

The book opens with a solid piece of writing that resonated with me:

I wrote this book to correct a costly myth that permeates many organizations today: that certain things can’t be measured. This widely held belief is a significant drain on the economy, public welfare, the environment, and even national security. “Intangibles” such as the value of quality, employee morale, or even the economic impact of cleaner water are frequently part of some critical business or government policy decision. Often an important decision requires better knowledge of the alleged intangible, but when an executive believes something to be immeasurable, attempts to measure it will not even be considered.

Later on it had another great paragraph:

It’s not as if the proposed initiative was being rejected simply because the person proposing it hadn’t measured the benefit (which would be a valid objection to a proposal); rather, it was believed that the benefit couldn’t possibly be measured. Consequently, some of the most important strategic proposals were being overlooked in favor of minor cost-saving ideas simply because everyone knew how to measure some things and didn’t know how to measure others. In addition, many major investments were approved with no plans for measuring their effectiveness after they were implemented. There would be no way to know whether they ever worked at all.

Both of these resonate so strongly with me because they describe organizations that could be doing better. You can assume that everyone is acting in good faith, but people don’t always have a good way to reason through what matters, what does not, and how to make decisions about it. Oftentimes executives might impose personal opinions on top of development which leads down strange paths. A case in point is Tesla and their refusal to add LIDAR into their cars; as of 2025 Waymo has functional driverless cars and Tesla does not, with Waymo stating that LIDAR makes a massive difference. Why did Tesla not add LIDAR? Because Tesla’s chief executive doesn’t believe it will work,

Tying this back to day by day SWE experiences, how does one measure the “value” a particular project has? In general you need to try and ask what the project is intended to accomplish and if it is meeting that goal. If the project is a tool people should be using, what are the MAUs and DAUs? How does that compare with incumbent or alternative tools? For something like product cybersec, did implementing this new project reduce the occurrence of issues in a measurable way? Did less issues happen after, perhaps issues were caught and attributable to the project? For example, consider a CI test that catches an unsafe coding pattern. Is it possible to observe how often that test fails once introduced, and is then followed up by a change that causes the test to pass?

This philosophy of measurable improvements is illustrated by Hubbard’s comments on a MITRE program to establish a knowledge database called MII:

If quality and innovation really did get better, shouldn’t someone at least be able to tell that there is any difference? If the relevant judges (i.e., the customers) can’t tell, in a blind test, that post-MII research is “higher quality” or “more innovative” than pre-MII research, then MII shouldn’t have any bearing on customer satisfaction or, for that matter, revenue. If, however, they can tell the difference, then you can worry about the next question: whether the revenue improved enough to be worth the investment of over $7 million by 2000.

So now tying this to the prior points, MITRE wanted to make a new knowledge database that would be higher quality, better than existing alternatives, etc, etc. But there was a question that they had to answer about how to measure if that was really the case. The answer here wasn’t asking customers to fill out a survey on their opinion (subjective), nor looking at active user numbers (too easily gamed if this is mandated to be used), but by asking downstream customers if they are getting better results from teams using the MII database vs those not using it. Sadly this test was not actually run, likely because the CIO of MITRE had not yet read this book.

The final part of the quote “whether the revenue improved enough to be worth the investment of over $7 million by 2000.” moves onto the next core part of the book: you often need to try and make decisions in advance of a project starting as well. Most choices are made between multiple options where one is pursued. To make breakfast you need to decide on toast, cereal, coffee, energy drinks, cow milk, soy milk, and plenty more decisions. Then you commit to pursue a few of those and you buy the items from the store. The same holds true in Corporate America: You need to be able to make decisions among multiple choices. Where this is tricker is that you need to try and figure out what is going to be the most impactful choice, and you must operate with the detailed information you may have about your own life. With breakfast you may have a strong sense that you prefer cereal over toast, but this is not as clear for corporate projects. Even breakfast lacks the benefit of perfect information, and you may choose cereal, buy the cereal and milk, only to discover the milk spoils a couple days after opening it and doesn’t get you through the week. With the spoiled milk, you made the best decision you could, there was a risk of the milk being bad, and you chose that over the alternative of toast. A core part of How to Measure Anything is how to reduce your uncertainty before choosing which project to pursue, and how to do so in a quantifiable manner.

Quantification of how to measure is a key part as well. Hubbard talks several times about this, referring to a concept of a “calibrated expert” that can provide accurate estimates within a confidence interval, or range of possible values. Calibration is important for any sort of measurement, even things that appear completely measurable (ie even measuring how long a webpage takes to render on your computer would really involve several measurements to account for background noise), but for estimating the chances of a situation occurring, and the outcomes, this is especially important. I appreciated Hubbard’s calling out of the problems with subjective measurements, especially in cybersecurity. He had a particular example highlighting if the risk of an investment failing is “low”, “medium”, or “high”, but what the terms meant was poorly defined. To quote from his passage on this:

Is a 5% chance of losing more than $5 million a low, medium, or high risk? Nobody knows. Is a medium-risk investment with a 15% return on investment better or worse than a high-risk investment with a 50% return? Again, nobody knows because the statements themselves are ambiguous.

This example resonates with me well because it comes up fairly often in cybersecurity. You need to manage risk, which means scoring and sorting risks to handle, but how to properly score risks often feels vibes based. It is worth noting that Hubbard has a second book specifically focusing on cybersecurity risk that is on my reading list. It is also worth noting that becoming properly calibrated, as in being able to provide accurate measurements, is a hard process as well that the book goes into a great deal of detail on.

I’ll close with yet another quote from the end of the book, summarizing the contents:

1. It’s been measured before.
2. You have far more data than you think.
3. You need far less data than you think.
4. Useful, new observations are more accessible than you think.

(1) in particular is the strongest endorsement I can give for reading the book. What you need to know has likely been done before, studied to exhaustion, and you can probably find it in this book. Even if it is not perfect detailed information, well, to add onto point (3), you don’t even need perfect data, you just need to reduce your uncertainty.