The IT Skeptic - Comments for "IT operations must beware inappropriate automation"

great quotes on automation

skeptic — Tue, 14 Jun 2011 22:18:01 +0000

"If you automate the wrong process you get the wrong result faster" ITSMWP Episode 63

Variations:
“If you automate a mess, you get an automated mess.”
– Rod Michael

"Automation applied to an inefficient operation will magnify the inefficiency. "
- Bill Gates

coverage

skeptic — Tue, 07 Jun 2011 22:19:01 +0000

It's true major failures happen when the broken bits coincide - people and automation - but equally I agree that people and systems cover and correct each other too.

Oh yes they do

JamesFinister — Tue, 07 Jun 2011 17:09:18 +0000

Matt,

I think you'll find the combination of the human factor with the machine factor is more common than you might think, and the consequences possibly worse than putting trlience on one or the other. Think Homer in a powerstation.

James Finister
www.tcs.com
http://coreitsm.blogspot.com/

Isaac Asimov

Matt Beran — Tue, 07 Jun 2011 12:25:00 +0000

In "Prelude to Foundation" there are two bits of history that are recounted that speak to this:
1) When the humanoid robots become too powerful and we must destroy them all (more because it's funny)
2) When Hari visits the heatsinks (a thermal power source on another planet), they talk about how hot it is, and maybe it should just be automated.

The counterpoint by the manager of the heatsink is perfect for this. He says that automation will fail, probably because of some computer malfunction, or physical break, so we must have people here or it could take up to 15 minutes to get a person to the correct location. Then, he makes the part that applies: people fail, automation fails, but they rarely fail synchronously.

Provide a good balance (which I believe summarizes at least part of your article).

Regarding the topic of Automation

guerino1 — Mon, 06 Jun 2011 17:26:38 +0000

Hi Rob,

I think I agree with some of your statements but not all.

First, experience has shown that there are two consistent forms of Automation:

1) The automation of human labor
2) Automation to further improve existing automation (i.e. multiple automated processes can be re-engineered to be improved, even further)

(We can argue that there's a 3rd that is a combination of the two but, fundamentally, it's the above two.)

Both of these happen regularly and often, throughout enterprises in all industries, globally.

Given the above, and I believe to some of your points, there really are criteria for what should or shouldn't be automated, in a given enterprise. For example:

- Priority of Work and the Organization
- Repeatability
- Ease of Automation
- Supportability & Maintainability of Such Automation
- Internal Skill Sets
- Cost of Automation
- Value to the Enterprise (Not always a tangible and measurable financial return. Sometimes it's a "soft" return.)
- etc.

When it comes to automation, there are usually three types or patterns that occur:

A) Complex (requiring formal Product & Release Management disciplines to be applied to somewhat complex programming. FYI: Not ITIL Release Management but what developers call Release Management)
B) Moder via Workflow/BPM tools (a mix of some programming and configuration)
C) Simple (usually the type of automation that revolves around "scripting" instead of more complex programming)

I've also learned that "who" does the automation is very important. In most enterprises, the most technically capable resources that can be leveraged to perform automation are usually highly trained, highly experienced and very "current" Software Developers/Designers and Engineers. Such resources are also empowered to write code, regularly.

There are also risks associated with your enterprise's Outsourcing strategy. Software Code products, such as Programs and Scripts are the types of Assets that often require regular review and improvement. As a result, you need Skill Sets and Resources with such Skill Sets that are available to perform such functions. If your enterprise Outsources the resources and the skill sets required to create and manage such constructs, it will be far more expensive and complex, in the long run, to support and change such constructs, as needed, in the future. Most enterprises that are successful at automating usually invest in keeping such resources internal to their enterprise, so that they don' t have to retrain, from the ground up, every time someone has to go back in to review and/or modify existing automation solutions.

From my own experiences and regarding the strategy of "automating" (and this is not to say that there are others out there), the most successful companies I've seen perform automation usually had a general principle that was professed from the top down: "If you have the skills and the power to automate, using the tools and and technologies that have been sanctioned by the enterprise, and if automating specific tasks frees you and/or others up to perform other work, then automate, automate, automate! Don't wait for reviews. Don't wait for permission. Don't wait for funding. Just do it. The worst case is that what you automate can be undone and we can revert back to original state." Such enterprises have the attitude of "Automate it hard! Automate it fast! Automate it often!" These types of enterprises not only automate their Core Business Functions but they also automate their Core IT Functions, such as those associated with IT Deployment. They get that IT is way too expensive and work hard to drive IT costs down, through the automation of everyday IT activities.

I've also found that people who don't automate for a living, doing it day in and day out, every day, often lack the understanding of how to best automate, when to automate, why something should or shouldn't be automated, what to use to facilitate best automation, and where to best target automation. In my experience these are skills that most IT people do not have. And, the way many enterprises set up Dev & Engineering vs. Support, it's rare that IT Support / Ops teams are best equipped to do so (Again, this is noting that there are always exceptions).

It's my opinion that with the advent of Outsourcing, most enterprises that have tried to outsource their development and/or engineering functions are often outsourcing and eliminating their most innovative and most empowered IT resources... Those that get how to most effectively automate the right things, the right ways. Such enterprises usually realize this after it's too late.

It's also my opinion that automation is the foundation for IT innovation as well as for Business Productivity. If you're not automating, you're not innovating. And, in these modern times, if you're not innovating you're not competing. If you're not competing, you might as well give up because your competition definitely is... and they will eat you for lunch.

Anyhow, I hope this adds value.

My Best,

Frank

The International Foundation for Information Technology (IF4IT)
Open IT Best Practices

Overdesigning the design process

Charles T. Betz — Sat, 04 Jun 2011 19:35:35 +0000

Peter, with all due respect, there have been notable failures in the philosophy that "if we just have a better design process we'll get better quality systems." Essentially, that's the root of waterfall thinking. New processes, if they have any value, tend to present unexpected difficulties. You cannot have a master, meta-process that controls for that fact, a reality that goes back to Turing's proof of the Halting Problem.

We need a faster, more adaptable process that accepts the fact that product design is always risky, uncertain, and best handled iteratively. That's why I am an unfailing advocate of the DevOps movement.

Charles T. Betz
http://www.erp4it.com

Lack of rigor

Springs Marty — Fri, 03 Jun 2011 17:28:18 +0000

Automation consists of more than stringing together multiple logical units of work, often utilizing a dependency tree as in good old "make." Let's ignore the blossom of failure scenarios introduced by unattended automation where literally no one is paying attention.

IT is so proud of its ability to verify correct functioning of a code chunklet under ideal conditions. Unit testing is all the rage.

The next level is the ability of code to recognize that it's in trouble then raise an exception to the invoking system. Only high end code is fully validated for this functionality.

The next level is for the invoking system to handle all exceptions correctly, making the appropriate adjustments, backing out any partial processing, retrying the process or even raising an exception to the higher invoking system (often a human being). Precious few systems are validated to perform this function correctly under all conditions.

Instead, we find code chunklets which each work just fine "in the lab" and string them together with a script then declare that we have automated something.

It's no wonder that things fall apart.

Automation

Peter Brooks — Fri, 03 Jun 2011 15:26:04 +0000

You make some important points.

I'd say that the cost of testing is high, as with other costs being high, often because of poor design. Following the Service Design Package method allows for a proper holistic approach to be taken to the design of services. In the case of automation of, for example, standard changes, rather than design each automated change, design a process and tools to produce automated changes - design the design process. Get that right, and the actual production of new processes become more reliable, less costly and time consuming to test and has fewer defects in operation.

Because this is often not done, and, rather, things are automated piecemeal, you end up with incompatible tools as well - which adds to the cost of training and increases the risk of improper use.