Troubleshooting Intermittent Problems

Intermittent problems are extremely difficult to troubleshoot. Any reproducible problem can be troubleshot, if for no other reason than that each individual component can be proven to not be the problem through experimentation. Problems that are not reproducible cannot be approached in the same way.

Problems present as intermittent for one of two reasons:

1. We have not identified the real cause of the problem.
2. The problem is being caused by failing or flaky hardware.

The first possibility should be addressed by going back to brainstorming hypotheses.

It may be helpful to bring a fresh perspective into the brainstorming session, either by bringing in different people, or by sleeping on the problem.

The second problem is tougher. There are hardware diagnostics tests that can be run to try to identify the failing piece of hardware.

The first thing to do is to perform general maintenance on the system. Re-seat memory chips, processors, expansion boards and hard drives.

Once general maintenance has been performed, test suites like SunVTS can perform stress-testing on a system to try to trigger the failure and identify the failing part.

It may be the case, however, that the costs associated with this level of troubleshooting are prohibitive. In this case, we may want to attempt to shotgun the problem.

Shotgunning is the practice of replacing potentially failing parts without having identified them as actually being flaky. In general, parts are replaced by price point, with the cheapest parts being replaced first.

Though we are likely to inadvertently replace working parts, the cost of the replacement may be cheaper than the costs of the alternatives (like the downtime cost associated with stress testing).

When parts are removed during shotgunning, it is important to discard them rather than keep them as spares. Any part you remove as part of a troubleshooting exercise is questionable. (After all, what if a power surge caused multiple parts to fail? Or what if there was a cascading failure?) It does not make sense to have questionable parts in inventory; such parts would be useless for troubleshooting, and putting questionable parts into service just generates additional downtime down the road.

This practice may violate your service contract if performed without the knowledge and consent of your service provider.

Regardless of the method used to deal with intermittent problems, it is essential to keep good records. Relationships between our problem and other events may only become clear when we look at patterns over time. We may only be confident that we have really resolved the problem if we can demonstrate that we've gone well beyond the usual re-occurrence frequency without the problem re-emerging.

Book Review: Time Management for System Administrators

Tom's book is a quick read, which is definitely what you want from a book about time management. He starts with hints about how to manage time in an operational setting. His suggestions are down to earth and eminently implementable.

His title states that the book is for SAs, but the tips can be helpful to anyone who works in a high-interruption, high-stress environment.

Resolving Conflicts

Resolving conflicts within a team is probably one of the least favorite tasks for any team manager. Some conflicts should be allowed to work themselves out, but other conflicts can consume the team if they are left unchecked.

Not all conflict is bad. Constructive conflicts can take place in an atmosphere of mutual respect, where the parties involved express opinions and work towards a resolution that is better than what any one of them would have proposed alone. Destructive conflicts, however, need to be watched and managed.

There are several different possible ways to approach a conflict. There are times when each of these is appropriate, but these are listed from most effective to least effective.

• Confrontation. The people involved directly confront the problem and try to work it through.
• Compromise. The parties use give-and-take to try to give everyone part of what they want.
• Smoothing. Areas of agreement are emphasized over areas of disagreement.
• Forcing. A “solution” is imposed from higher up the hierarchy. When this method is overused, a manager will probably be seen as autocratic.
• Withdrawal. Ignoring the problem is usually the least effective way to deal with a problem. Most problems get worse when ignored, not better.

There are times when each of these approaches is valid, but the strongest managers will try to directly confront problems when possible and reach compromise solutions when necessary. Forcing may be needed from time to time, but should be used sparingly.

Book Review: Agile Management

Medinilla has captured the essence of the new management paradigm. A lot of the old command-and-control sorts of tactics that we inherited from the industrial revolution simply do not work in the information age.

He does not get bogged down in the details fo different Agile methodologies, but espouses bringing the soul of the Agile Manifesto to the organization as a whole.

Small groups within companies have implemented Agile and benefited from it, but many Agile proponents are allergic to management. Medinilla's book contains good advice and information for organizations that are looking to take advantage of Agile on a larger scale.

Book Review: The Practice of System and Network Administration

System Administration as a Profession

"The Practice of System and Network Administration" is different from most of the other technical books on a professional SA's bookshelf. This book is about how to become a professional system administrator.

The profession is about more than knowning obscure options to different commands. To become a professional, a system administrator needs to change mindset from a straight-ahead techie to a member of the team who has specialized expertise.

System administration has not always been recognized as a profession. System administrators themselves are partly to blame for that. We have tended to focus strictly on technology and not on how to structure our work to benefit both ourselves and the organizations we work for. Limoncelli, Hogan and Chalup have put togeter a great standard reference for people who are ready to transition to being professional system administrators.