Velocity NY 2013: Richard Cook, "Resilience In Complex Adaptive Systems"

Richard Cook's writing and talks on systems have hugely shaped my thinking on systems. For those interested in more, he gave a highly entertaining talk on system resiliency here and introduces a helpful model. [VIDEO] I also wrote a post summarizing how I think about Rasmussen’s model in startups and web applications. [LINK]

> the staff had become “de-sensitized” to the risk of a serious accident. Yet again serious problems happen due to the Normalization Of Deviance. We really need to find a way to create working conditions that encourage correcting problematic situations immediately before the behavior becomes normalized. Regarding the management/other problems... I encourage everyone in ever industry that is ever involved with safety (i.e. most industries) to see Richard Cook's short talk about "Resilience in Complex Adaptive Systems"[1]. [1] [VIDEO]

I should have used more specific language; I'm not trying to argue that short selling is "bad". The benefits - such as dampening a crashing market - are large enough to accommodate some risk. I'm primarily addressing the claim that, "There's absolutely nothing wrong with short selling". Ignoring small, acceptable risks is normalizing deviance[1]. [1] [VIDEO]

> both individual and group ethics have to fail However, they don't have to fail at the same time or in a single step. This type of problem can creep in slowly over years as the "normalization of deviance"[1]. Richard Cook presented[2] a very useful model for how this type of problem creeps into complex system. The pressure from economic and workload concerns never goes away, so unless there is a proactive, explicit counter force, way to push back against that force, the system will inevitably be pushed toward failure. Therefor it's important to stop problems early when they are small. The magnitude of the counter force increases rapidly as behavior becomes increasingly deviant. [1] [LINK] [2] [VIDEO]

> it's a short and murky step ... to outright fraud. Staying vigilant against the normalization of deviance[1] is difficult even in ideal situations. Choosing to normalize deviance as a business strategy is practically guaranteed to rapidly accelerate deviations from acceptable behavior. In any system - including every business - resilience and safety means fixing small problems before they grow into bigger, more damaging problems[2]. > rationalizations for unethical behavior Often, it doesn't even need to be rationalized. When deviant behavior is slowly normalized in small steps, the human mind tends to only see the "small, insignificant problems", and often skips[3] over big changes that should be "obvious". [1] [LINK] [2] [VIDEO] [3] [LINK]

> evil Hollywood et al popularize the misconception that evil is fantastic and done with intent . Most of the time evil is banal [1]. The larger problems happen when the unremarkable, small deviances from acceptable behavior becomes normalized [2][3]. [1] [LINK] [2] [LINK] [3] [VIDEO]

Aka "Normalization of Deviance"[1]. Fix the minor problems now , or they become the new normal, eventually accumulating until a huge deviance from "normal" behavior isn't even noticed. The stories[2] about airline pilots completely ignoring checklists and unusual alarms while trying to take off with the gust lock still engaged are a shocking example of strong this effect can be. For a very good explanation of how this happens, see Richard Cook's short talk "Resilience in Complex Adaptive Systems"[3]. [1] [LINK] [2] [LINK] [3] [VIDEO]

Idealism isn't about living as a monk, it's about not compromising on the things that are often seen as "trivial" or "inconsequential". Most people don't jump from acting ethically to accepting work on that is obviously problematic. What actually happens is a long series of small, trivial compromises and other deviations that eventually become normalized [1]. Humans make mistakes; it isn't possible to live the "ideal monk" life. The real question is how small problems are handled? Compromising when the problem isn't a "big enough" ethical problem inevitably[2] pushes that boundary back in a cycle of normalized small, easily avoided deviances. Instead, if a hard line is taken against small problems when they are still small , a lot of larger problems can be avoided. [1] [LINK] [2] [VIDEO]

One of the most important talks I've ever seen is Richard Cook's "Resilience In Complex Adaptive Systems"[1], in which he explains the fundamental abstract mechanism that causes the normalization of deviance. [1] [VIDEO]

> But if you stop, they may continue increasing the dilution. There’s a normalization of deviance. For a very good explanation of the mechanism that causes this kind of normalization of deviance, I recommend Richard Cook's outstanding talk "Resilience In Complex Adaptive Systems"[1]. [1] [VIDEO]

> Indifference that does not cause a business problem remains indifference. More generally, indifference to small "trivial" problems eventually normalizes the deviant behavior[1]. Evil behavior doesn't happen in one step, it's end result of regularly being indifferent to "minor" problems[2]. For a very good explanation of why this happens, I recommend Richard Cook's excellent talk, "Resilience In Complex Adaptive Systems"[3]. [1] [LINK] [2] [LINK] [3] [VIDEO]

> I cannot believe that everyone is ethicality challenged The difficult ethical discussion probably never happens. The decisions being made in those meetings are usually seen as small/inconsequential. The problems caused by those "small" decisions are ignored. Eventually those problems become normalized allowing another "small" decision to be made. Humans seem to be very bad at recognizing how a set of "small" decisions eventually add up to major - sometimes shocking[1] - consequences that nobody would have approved if asked directly. Most of the time, nobody realizes just how deviant their situation had become. For a good explanation of the mechanism underlying the normalization of deviance (as an abstract model), I strongly recommend this[2] short talk by Richard Cook. [1] [LINK] [2] [VIDEO] ("Resilience In Complex Adaptive Systems")

The good news, yes. The bad news: it depends a lot of what you mean, it may not be the answer you search and i do not have a "perfect" reference, because it is considered mostly obvious in these fields... But before i get there, a few things. > (that we are building castles on quick sands with our leaky abstractions that try to fit into constantly changing reality) Nearly all the references i point to will tell you that this leakiness and this constant changing reality is exactly what makes them work in the real world. Now onto references. A lot of them can be found at [LINK] I particularly advise to start at [LINK] [VIDEO] For more in depth, i particularly like [LINK] And this is basically a whole meta study of all the research and ideas that came to this field since WW2. Beware it is long [LINK] For an example of how things go wrong when things are correct but interact well together, i advise these two papers on space systems. (I link to Adrian blog on them because they may be easier to read, but there is a link to the papers in there) [LINK] [LINK] If these things make you curious for more, there is a lot. Or if you want to talk about/have help navigating what is in the [LINK] link, feel free to reach to me on twitter or by email. I will be happy to talk more.

It's just a variation of the Normalization of Deviance. See this[1] short talk by Richard Cook for a very good explanation of the mechanism that causes the transition from "robust" to "superfluous". [1] [VIDEO]

Good comment about this very thing here: [LINK] > It's just a variation of the Normalization of Deviance. See this[1] short talk by Richard Cook for a very good explanation of the mechanism that causes the transition from "robust" to "superfluous". [1] [VIDEO]