Risk vs. Uncertainty

It's the planning season, "Risk" is one of the most discussed topics during this time. But when we say "Risk" we might mean different things. Therefore I took some time to reflect on the two most common confusions that we might have when talking about "Risk", that is Risk vs Uncertainty. I took a lot of inspiration from this note and I'm sharing some of my own thoughts and realization that might be useful.

What is risk?

Risks are bad things that can negatively impact the outcome, so we don't want them to happen.

• Risks are certain, therefore usually measurable / quantifiable.
• Given the nature of its certainty, we can usually mitigate them using tools like insurance, contingency and parallel paths etc.
• In the context of R&D, dealing with risks ultimately leads to solving a trade-off problem: you put in more effort, spend more money, extend the timeline or compromise the feature set, etc.

An intuitive example: You are on the roof of a 20 floor building and you want to get to the ground. If you jump off the building, the risk is that you just die from free fall, and the chance is almost 99.9%. To mitigate that, you can choose to walk down stairs instead (more time), you can carry a parachute (more effort), or even call in a helicopter to carry you down (more money). All the options have relatively certain cost and benefits.

What is uncertainty?

Uncertainty is as simple as "We don't know". Under uncertainty, we don't know if something is easy or hard, and we don't know whether something will positively or negatively impact the outcome.

• Uncertainty can not be measured or quantified.
• Uncertainty can not be mitigated by known tools, since by definition you can not find a solution to a problem that you don't know.
• In the context of R&D, dealing with uncertainty requires taking steps into unexplored territory to gather data, prove or disprove hypotheses, and eventually convert uncertainty into risks that can be mitigated.

Another intuitive example: You stand in front of a river, and you want to get to the other side. The water is really muddy so you can't see whether it's deep or shallow. To be even more precise, you don't know which segment of the river is deep and which segment is shallow. This is uncertainty: there is nothing you can really do to mitigate the uncertainty, and the only thing that you can do is to actually find methods to measure the depth of the river. With that information, uncertainty is converted to risk, then you can find mitigation plans in response to each segment of the river.

Treating Uncertainty as Risk

Given the definition above, the result of treating uncertainty as risk is that we add false certainty to our belief and wrong mitigation plans that don't reduce uncertainty. This will lead to unrealistic goals and incorrect investment which ultimately leads to the failure of a project.

Concrete example: as we are building a new AR device, there are lots of uncertainty around whether each machine perception component is actually buildable given the power requirement and the SoC that we have. The team needs to do deep investigation and small, rapid experimentation to gain the necessary information. And based on the new information, we will either gain enough confidence to convert uncertainty into risks, or we might end up pivoting from the current PRD.

However, if we are mistakenly treating these uncertainties as risks, we might want to keep the original PRD unchanged, and just add more capacity, more investment and more time hoping to power through aiming at building something that is not even possible.

Treating Risk as Uncertainty

The opposite is to treat risk as uncertainty, and the symptom is "analysis paralysis". This is when we already have sufficient data to commit to an action, we still feel that there is uncertainty that is preventing us from making the move. This will usually lead to inaction or unnecessary compromise of the goal.

Concrete example: when we are building a ML models for an object detection-based application, the performance was not good enough, and we were stuck in a state where we were trying to explore and evaluate different models and algorithms with very limited training and benchmarking data and we were not going anywhere, since all that we do leads to Shit-In-Shit-Out (SISO).

This is us treating a risk as uncertainty. The field of object detection is already well studied and various methodologies have been established. What was missing was the hard job of gathering data and annotating them, such that we can train models and measure in a meaningful way. That is a risk that can be well mitigated with more capacity, more time and more money.

Risk mitigation is a means to a goal, Uncertainty reduction can be a goal itself

This is a useful framing in the context of OKR planning, since I've seen both concepts being used in the "wrong way".

Risk mitigation on a specific component is only useful if the end goal is achieved. It means nothing to mitigate some of the risks in a project and ignore the others since ultimately it's the joint probability of all risks being mitigated that defines the chance of the overall project success. Within a total system / a project, risk mitigation should be a global optimization that considers the full picture. Treating individual risk mitigation as a stand-alone goal will very likely lead to suboptimal local optimization. People might celebrate their local success when the bigger project actually fails.

Another symptom that I see in planning is to define a research goal as "deliver" a result when actually the goal is to reduce uncertainty. When the goal is framed as "delivery", it gives the false impression to the team that not achieving a certain metric is a failure, while in reality, the most useful outcome for a certain research project is to unveil what works, what doesn't and why certain things work but others don't.