The IT Skeptic - Comments for "Conducting a root cause investigation on a 3rd party "

Good enough for what

aroos — Sat, 18 Feb 2012 09:31:34 +0000

Rob,
please print that previous comment, frame and hang it on your wall. It is classic. Let me rephrase it in English:

Tracking faults and customer problems in a single process is a big issue for service desks. It confuses front office with back office, tries to do 2 processes at once off the one entity, and screws the stats. Yes, we agree completely.

You use a seriously flawed model so that you would not have to abandon holy ITIL? Your comment is a proof of why we need to unlearn ITIL. As I said, it is not easy. Remember, you don't have to abandon everything, just cut the bad parts away.

exact match

skeptic — Thu, 16 Feb 2012 18:25:00 +0000

Yes. The problem process exactly matches what u need to do to fix a fault.

Actually you know I'd prefer there were a Fault entity, and maybe a separate process. Unlike you, I dont see that as justification for abandoning ITIL. The Incident/Problem model is good enough.

Tracking problems as incidents is a big issue for service desks. It confuses front office with back office, tries to do 2 processes at once off the one entity, and screws the stats

What is the problem

aroos — Thu, 16 Feb 2012 16:45:07 +0000

Fixing faults is usually simple routine activity which must be done urgently. You think it makes sense to call it Problem management?

inExpert

skeptic — Thu, 16 Feb 2012 09:32:08 +0000

What the hell is (reactive) problem management if it isn't fixing faults. Perhaps the Finnish translation is different. Or maybe I'm an unITIL InExpert

sorry, no points

aroos — Thu, 16 Feb 2012 08:50:18 +0000

Rob,
There are three concepts to ITILs two. Customer problem and fault are different animals. In itil, they are just incidents. Fixing faults is not problem management.Try harder ;)

unlearning

skeptic — Thu, 16 Feb 2012 00:41:36 +0000

Aale, you are free to use a different terminology than ITIL if you like but I don't see that as "unlearning".

Sure ITIl is fuzzy about when to create a ~~problem~~fault ticket from an ~~incident~~problem, and sure it is weak in systematically describing risk management but it does recognise a risk register and the need to deal with risks, so I can't see anything you have said that is un-ITIL

in the real world

skeptic — Thu, 16 Feb 2012 00:37:10 +0000

James correctly says one of the problems with this blog is that it is hard to tell when i have my tongue in my cheek, which is why I use the ":)" symbol. I was being facetious about "a problem gone is a problem solved I say".

nevertheless I never met a service desk that wasn't understaffed except one miraculous one Chris and i visited. I'm well aware of and agree with all the theoretical considerations you cite. meanwhile in the real world there is only so much you can capture and deal with - that was my point.

That's a tricky edge case

Wraith — Wed, 15 Feb 2012 20:08:13 +0000

That's a tricky edge case which requires a bit of consideration.

In an ideal world, your service desk staff filter out issues which can be tracked to user misunderstanding and lodge them as "Queries" rather than incidents.

If the service desk reboots anything and solves the user's incident, that's fine - what you have there is a clear Known Error which has "reboot" as a workaround. You don't need to do anything with these except record them and tag them appropriately. If they hit critical mass, you investigate and perform root cause analysis. From my perspective "reboot" is usually an inadequate workaround because it occupies both service desk and user time. If I'm tracking these reboots, I can do something about them if they're proving particularly onerous.

Failing to capture these issues throws away data which - in hindsight - could prove to be useful or critical later on. Any root cause analysis is going to live or die on the quality of the data you feed it and knowing WHEN an issue first arises and which users first experienced it is damn useful. If the issue never recurs, then we simply don't care. The resulting Problem record can languish until the end of time without anybody giving a damn.

As long as you provide the Service Desk with tools capable of quickly matching user symptoms to Problem records, the impact on workflow is minimal and the Problem records themselves can provide the Service Desk with analytical steps, data capture requirements and solutions. Constructed correctly, this allows your KEDB to educate your Service Desk as part of their normal workflow.

> In the hurly burly of the usual understaffed service desk, a problem gone is a problem solved I say :)

I wish. Users only give the Service Desk a limited number of chances to solve their issue. If they can solve it themselves by rebooting, or if you fail to resolve their issue the first few times they call you, then they stop calling you and telling you there's a problem and start complaining to their peers and their management instead. Great for the Service Desk - terrible for the reputation of the IT organisation.

Unlearn ITIL

aroos — Wed, 15 Feb 2012 14:27:01 +0000

All this is crazy ITILspeak. I translate it to plain language.

1 A customer has a problem with service X. Customer service solves it by any means as soon as possible.
2 The customer problem may have revealed a fault. It must be fixed asap.
3 There may be a risk that the customer problem may reoccur. That risk needs to be managed.

Three tickets if necessary. Different processes, probably different teams. All stages may need PROBLEM SOLVING but nobody needs Problem Management.

If you got that, it was your first step in unlearning ITIL. Cheers, its worth -5 APMG points and all free. Should I create also a free certificate?

Aale

a problem gone is a problem solved

skeptic — Wed, 15 Feb 2012 05:33:13 +0000

I love your summation except that I never actually said "Every Incident record requires a corresponding Problem record." In theory i suppose yes. pragmatically no.

If the user is happy to close the incident and we have restored service with no idea what actually happened (something that happens many times a day) then the incident is closed, period. We may never know what happened.

if similar things happen often enough i trust the support staff to recognise a pattern and open a problem.

I think you are being too idealistic in setting hard and fast rules. just like a GP saying "take 2 aspirin and call me back in the morning", the service desk may reboot or restart something and the issue goes away. often the user has got themselves in a tangle, or there may not actually be anything wrong at all: the IT equivalent of hypochondria.

In the hurly burly of the usual understaffed service desk, a problem gone is a problem solved I say :)

Every Incident record has a Problem record

Wraith — Wed, 15 Feb 2012 03:03:55 +0000

It's possible to argue that incident and problem records are simple artefacts of the way in which any organisation has chosen to implement their incident and problem processes; nevertheless, there's no simple getting away from the reality that every (ITIL Definition) Incident has an underlying (ITIL Definition) Problem.

To clarify: I agree with the Skeptic. Every Incident record requires a corresponding Problem record.

Those who have difficulty determining what should be logged and when are confused and haven't separated the processes sufficiently to understand the points of interface between them. Incident records are managed by Incident management. Problem records are managed by Problem management.

Every time an incident is resolved successfully, a solution is applied. If that solution is a new one which the support staff have generated spontaneously, it requires analysis to determine if, for example, the solution is temporary (a workaround) or permanent. In other words, do we have a resolved problem or a known error? The solution may need to be examined for cost, practicality and potential unintended consequences.

The task of performing that analysis falls to problem management and the responsibility for maintaining that solution, determining the impact of the ongoing incidents which arise from this problem and justifying the resources necessary to investigate and implement a more permanent solution also rests with them.

Technical staff document their diagnostics and attempts to resolve the incident in the original record. The incident record is a record of the progression of the incident process. In other words, the management of the symptoms of the incident. The problem record is a record of the progression of the problem process. In other words the management of the underlying root cause of the incident.

Thus, Problem records are solely created, updated and maintained by the Problem Management team. That team is responsible for sifting through the original incident record to determine the salient facts which are fed into the analytical (Kepner-Tregoe etc..) stage. In extreme cases, the Problem management team may have to sift through dozens, hundreds or thousands of instances of these incidents to extract the data required.,

Now, while every incident requires a corresponding Problem record, it does not require a corresponding UNIQUE problem record. Incidents which are clear repetitions of an earlier incident merely need to be linked to the original Problem record.

Once you shift your thinking from wading through a morass of incidents to simply linking Incident and Problem records together, the reporting which becomes available allows you to clearly demonstrate which Problems are causing you the most pain.

A few final points:

* Incidents without a Problem record cannot be closed. This enforces the following-

* Resolved incidents which have no corresponding problem record should be routed to Problem Management who will create a Known Error.

* Unresolvable incidents become open Problem records. If an incident cannot be resolved at all, this information is relevant to the urgency of the Problem. A problem record linked to 1,000 unresolved incidents which continue to impact customers will gain rapid attention as the "minutes of impact" metric climbs.

You make sense. And I

Vespasian — Tue, 14 Feb 2012 14:48:34 +0000

You make sense. And I suppose ITIL want to keep things vague so they can clarify things later as a justification for a new version and make even more money from certs and exams eh?! hehe. We will use logic here on this one I think.

Thanks

a problem is a problem

skeptic — Mon, 13 Feb 2012 21:04:00 +0000

It already has a separate thread :) In fact several

PERSONALLY I think a problem is a problem. You should open a Problem ticket for every underlying cause.

Other folk think "you can do RCA and update the incident ticket with the progress and eventual outcome" and only sometimes - under ill defined circumstances - open a separate Problem ticket. and ITIL appears to back them up. I think it's rubbish. You might as well not have a separate Problem at all (yes Aale I hear you)

please explain

skeptic — Mon, 13 Feb 2012 20:58:56 +0000

It's a "please explain": what went wrong - including what went wrong with your procedures - and what you have done to ensure it won't happen again so we can have confidence to continue doing business with you.

With an important supplier I'd do it face to face - i have done so. They tried to blame a technical fault - IT people always do. Under interrogation we finally got them to the realisation that they had f***ed up for two reasons: their guy did lazy research and they didn't give our SAN upgrade the priority it deserved because they hadn't even considered the impact of it failing even though we went to great links to explain that. That is a very different explanation than "bad firmware from the manufacturer". it was a learning experience for our supplier whose service was improved as a result.

We told them it was "part of our Major Problem Review" but that was over. it was part of our Supplier Management.

report from 3rd party..

Vespasian — Mon, 13 Feb 2012 12:16:52 +0000

Hi Skeptic,

Just wondering what this report is that is requested by Supplier Management, is it like a Major Problem Report but just to do with the way the 3rd party handled the incident and problem? Does that report have a specific ITIL name?

Cheers

Root cause without a problem ticket

Vespasian — Mon, 13 Feb 2012 09:41:25 +0000

Thanks again for the response.

Regards to conducting root cause without a problem ticket; but if ITIL says "Problem = unknown root cause of one or more existing/potential incidents", I'm a bit confused as to how we are allowed to conduct root cause on an Incident ticket, I didn't think we could use incident tickets to track root cause (and hence problems).

Are you saying that for low impact incidents you can do RCA and update the incident ticket with the progress and eventual outcome? On our ticketing system, the incident ticket can be put into a 'resolved' state, and then it can go into an 'RCA' state, but there has been a whole debate around strictly keeping RCA with problem, and hence do RCA on a separate problem ticket.

This might need a separate thread?

Thank you

Vespasian — Wed, 08 Feb 2012 15:55:58 +0000

Thanks very much to the IT Wizard, the skep and everyones contribution.

I've got some questions in repsonse to the post but don't want to go off topic so I will ask them in other threads.

It sounds like I need to ruffle some feathers about the contractual obligations of 3rd parties that we have with management; I have but a little voice though.

Regards

Vespasian

nice in theory

Visitor — Mon, 06 Feb 2012 05:58:56 +0000

But I know of someone at Company A who buys $xM of IT service from Provider B. Provider B buys 10 times the $ amount of various (non-IT) services from Company A. It is not a reciprocal deal, but unwritten understanding is not to beat them up for crappy service quality, and accept a swiss cheese contract.

Taken out of this poor fellah's hands.

I might've seen it.

MichaelC — Wed, 01 Feb 2012 13:07:58 +0000

I might've seen it. Currently. Couldn't possibly confirm that though :D

I daredn't imagine the non-IT business units getting hold of Supplier Management!

As an aside, I've been involved in the IT world for 13 years (and thus share a decent level of skepticism) but am relatively new to ITIL/ITSM. Having recently found your blog I'm enjoying your writings, so may just hang around a bit :)

yes, imagine that ...

Visitor — Thu, 26 Jan 2012 09:28:27 +0000

Funny you should bring that up. Unfortunately, I don't have to imagine what it would be like; we actually see it quite a lot and it's causing a few headaches at the moment.