Effective System Monitoring

In order to maintain a reliable IT environment, every enterprise needs to set up an effective monitoring regime.

A common mistake by new monitoring administrators is to alert on everything. This is an ineffective strategy for several reasons. For starters, it may result in higher telecom charges for passing large numbers of alerts. Passing tons of irrelevant alerts will impact team morale. And, no matter how dedicated your team is, you are guaranteed to reach a state where alerts will start being ignored because "they're all garbage anyway."

For example, it is common for non-technical managers to want to send alerts to the systems team when system CPU hits 100%. But, from a technical perspective, this is absurd:

• You are paying for a certain system capacity. Some applications (especially ones with extensive calculations) will use the full capacity of the system. This is a GOOD thing, since it means the calculations will be done sooner.
• What is it you are asking the alert recipient to do? Re-start the system? Kill the processes that are keeping the system busy? If there is nothing for a the systems staff to do in the immediate term, it should be reported in a summary report, not alerted.
• If there is an indication (beyond a busy CPU) that there is a runaway process of some sort, the alert needs to go to the team that would make that determination and take necessary action.

In order to be effective, a monitoring strategy needs to be thought out. You may end up monitoring a lot of things just to establish baselines or to view growth over time. Some things you monitor will need to be checked out right away. It is important to know which is which.

Historical information should be logged and retained for examination on an as-needed basis. It is wise to set up automated regular reports (distributed via email or web) to keep an eye on historical system trends, but there is no reason to send alerts on this sort of information.

Availability information should be characterized and handled in an appropriate way, probably through a tiered system of notifications. Depending on the urgency, it may show up on a monitoring console, be rolled up in a daily summary report, or paged out to the on-call person. Some common types of information in this category include:

• "Unusual" log messages. Defining what is "unusual" usually takes some time to tune whatever reporting system is being used. Some common tools include logwatch, swatch, and logcheck. Even though it takes time, your team will need to customize this list on their own systems.
• Hardware faults. Depending on the hardware and software involved, the vendor will have provided monitoring hooks to allow you to identify when hardware is failing.
• Availability failures. This includes things like ping monitoring or other types of connection monitoring that give a warning when a needed resource is no longer available.
• Danger signs. Typically, this will include anything that your team has identified that indicates that the system is entering a danger zone. This may mean certain types of performance characteristics, or it may mean certain types of system behavior.

Alerting Strategy

Alerts can come in different shapes, depending on the requirements of the environment. It is very common for alerts to be configured to be sent to a paging queue, which may include escalations beyond a single on-call person.

(If possible, configure escalations into your alerting system, so that you are not dependent on a single person's cell phone for the availability of your entire enterprise. A typical escalation procedure would be for an unacknowledged alert to be sent up defined chain of escalation. For example, if the on-call person does not respond in 15 minutes, an alert may go to the entire group. If the alert is not acknowledged 15 minutes after that, the alert may go to the manager.)

In some environments, alerts are handled by a round-the-clock team that is sometimes called the Network Operations Center (NOC). The NOC will coordinate response to the issue, including an evaluation of the alert and any necessary escalations.

Before an alert is configured, the monitoring group should first make sure that the alert meets three important criteria. The alert should be:

1. Important. If the issue being reported does not have an immediate impact, it should be included in a summary report, not alerted. Prioritize monitoring, alerting, and response by the level of risk to the organization.
2. Urgent. If the issue does not need to have action taken right away, report it as part of a summary report.
3. Actionable. If no action can be taken by the person who receives the alert, it should have been defined to be sent to the right person. (Or perhaps the issue should be reported in a summary report rather than sent through the alerting system.)

Courage in a Corporate Setting

Sometimes leaders need to leave their comfortable "safe zones" in order to be effective. The reality is that the bulk of our jobs can be done by almost anyone. Most decisions we make are of the "no brainer" variety, especially as we become more experienced and comfortable in our role as leaders. But there are a few decisions that we need to make where we really earn the money and privileges that come with a management role.

When we voluntarily step outside of our comfort zone to do what we know to be right, we demonstrate the courage that distinguishes between someone who is a leader and someone who is merely a boss.

Battlefield analogies are very common when we speak about courage. An article by Peter Voyer in Ivey Business Journal suggests some important leadership traits that translate from the battlefield to a corporate setting:

• Don't ask subordinates to do something you would not do. Not only should you be willing to work alongside your team, you should be seen as someone who engages in the task at hand. (Of course, the way you engage the project will be somewhat different than the tasks you would assign a junior team member, but nobody on your team should feel like you are unwilling to dirty your hands to make the project succeed.)
• Demonstrate moral fiber. You can lose years of built-up moral capital in a split second with a morally dubious decision.
• React quickly, decisively, and fairly when presented with a moral question.
• Maintain dignity and respect within and between groups.

I recently saw an article about a manager who was allegedly fired because he stood up for an Indian employee's right to earn the same salary as American employees with a similar job. This is the sort of courage we need if we want to be leaders and not merely bosses. Who do you want to see when you look in the mirror in the morning?

Great leaders earn the loyalty of the people who work with them. They earn loyalty by demonstrating loyalty. This doesn't mean that you cover for one of your subordinates who does something wrong; it does not help someone's development to infantilize them. But make sure that the consequences are fair and are implemented with the long-term development of your employee in mind. This may mean that you stand up for someone who has made a mistake and demand fair treatment for that person. Yes, this is uncomfortable, but it is part of how you become the manager you want to be.

Make sure you stay informed of your team's progress towards goals, and work with them to overcome obstacles. This does not mean that you do your team's work for them; it means that you provide a sounding board. Sometimes a problem is escalated to you if it is something that requires a manager's approval or advice; make sure that you do what you need to do promptly, then return the task to its rightful owner.

Maintain your integrity. Make the best decisions you can, and abide by the results of those decisions. Don't pass the blame. Instead, identify how to fix the situation and propose solutions.

Demonstrate courage by making the right decisions, even when they are hard. Anybody can be a great boss when the going is easy. Being a great leader comes from doing the right things even when they are not easy.

The Promise and Peril of Self-Driving Cars

The research by Google and others into self-driving cars has been intriguing. The vast majority of traffic accidents are the fault of drivers, and being able to eliminate human error would be a huge win for traffic safety.

But if computers are driving cars, we have to take a serious look at information security in the context of a self-driving automobile. Unfortunately, most current automation does not have adequate safeguards to protect from malicious inputs.

In particular, components do not do checking or validation to make sure that commands are being issued from an appropriate source. Security researchers have demonstrated that they are able to issue commands to a Prius to control steering, braking, acceleration, and dashboard displays. They were also able to disable an Escape's brakes at slow speed.

Ford and Toyota both point out that the researchers were connecting directly to the car's CAN (Controller Area Network), which limits the impact of some of their demonstrations. But keep in mind that wireless controllers on on-board systems such as Bluetooth controllers on sound systems and telematics units on satellite roadside assistance services may provide an entry point into the automobile. Anywhere a wireless connection allows access to a component connected to a CAN is a possible entry point for malicious code.

The sorts of security measures we use for other network-connected items would still work inside a car. Provide air gaps between components that don't need to be connected. And provide for validation and authentication of commands from components that do need to be connected.

I remember discussions about PC security in the early days of the Internet, when most computer viruses were still spread by injudicious insertion of floppy disks. Way back when, we were told that PCs didn't need to have security programmed in from the ground up. I'm hoping we learn from the history of those poor decisions. A Blue Screen of Death is one thing, but a traffic fatality is another.

The Risk from Insiders

There has been a lot of coverage of the role of a system administrator in the recent release of information about the National Security Administration's intelligence gathering methods. Regardless what you think about the methods that were revealed, information security professionals need to take a hard look at the sorts of exposures that exist due to organizational insiders.

Snowden's position as a system administrator is just the most recent high-profile insider who betrayed his employer's trust. His removal of documents on a thumb drive was viewed as unsuspicious precisely because of his job function.

As long as we have an IT infrastructure, the people who manage it will be in a privileged position. IT professionals recognize the risk; four of five professionals in a recent survey list insiders as the greatest source of risk to the environment.

The same methods that are used elsewhere in the security landscape will help to control and mitigate the risk from insiders. At a high level, there are three steps that need to be taken:

1. Data Classification: Identify the types of data in your environment, and what the confidentiality, integrity and availability requirements are for each type of data. NIST 800-60 can provide some guidance here.
2. Establish Control Standards: For the different types of data, we need to describe the measures that are taken to protect the data.
3. Audit: The controls need to be evaluated for effectiveness, and the organization's compliance with the the controls must be verified on a regular basis.

Controls

There are several publicly available documents outlining control best practices and standards. Here are a few:

• NIST 800-53
• PCI Security Standards

Some common controls include:

• Physical access controls: Things like security guards, mantraps, proximity card systems, and combination locks on doors control physical access to sensitive areas and systems.
• Logical access controls: In general, people should only have the level of access required for their jobs. Access controls should be as granular as possible, and high-level access should require extra levels of approval and scrutiny. Two-factor authentication should be in place for access to sensitive facilities.
• Personnel management: Some common measures include criminal background checks, periodic security awareness training, contractual attestations, and organizational communications.
• Separation of duties: Where possible, access should be limited to particular functions, and functions should be defined to limit access to sensitive data. In general, developers should not have access to production, system administrators don't need database access, application administrators don't need system-level access, and only the people who manage the hardware and network need physical access to the systems.
• Network security: The network should be segmented appropriately, and firewall rules should be in place to restrict traffic between different security zones.
• Workstations and laptops: Hard drives should have robust encryption and strong password policies should be in place. The types of data that are permitted for local storage should be established and monitored. The ability of end users to install applications needs to be restricted. Patches, anti-virus updates, and security workarounds need to be applied regularly.
• Backups and continuity: Data needs to be protected by a combination of archival backups, long-distance replication, and local disk mirroring/RAID-ing.
• Logging and auditing: Logs need to be collected to measure the effectiveness of these controls, and the logs need to be reviewed on a regular basis.

Some controls should get particular attention as directly addressing the issue of insider-led breaches.

It is bad enough that the security training level of such government employees is not monitored by the DHS. (Most parts of the private sector also don't track administrator security training, for that matter). Beyond carelessness or incompetence, employers need to consider the direct risks posed by their most trusted employees.

Using Email Effectively

Email is mis-used and over-used in most organizations that I have worked in. Some organizations are going so far as to ban email in the workplace. That would be unwise; email is over-used precisely because it is such a useful tool that can be adapted to almost any purpose.

The trick is to use your email effectively and efficiently. Your team's effectiveness will increase if you can create a culture where email is used appropriately.

Email rules of thumb

• Any email you write should be concise and to the point. Don't use two sentences where one will do.
• Use proper grammar, capitalization, and spelling. Errors are distractions from the purpose of the email.
• Make sure that the email's subject line is brief, but relevant to the discussion.
• If you are requesting action (such as a reply), be specific about what you need, what form you need it in, and when you need it. Email is strongest when it can be used to dispose of an issue efficiently, in one exchange.
• Be courteous, even formal. Email and humor don't go together.
• Use the bcc field if the recipients should not necessarily have each others' email addresses.
• Use web links rather than large file attachments.

Email's strengths

Email has some strengths as a tool:

• It is asynchronous. You can write the email when you have time, and the recipient can read it as he or she has time.
• It is fast. Email is delivered almost instantaneously. (That is why it has almost completely replaced the old-fashioned snailmail letter.)
• It can be directed to a specific audience. The sender can define who will receive the email. (Keep in mind that it may be forwarded by those recipients!)
• It can be used to reference other information. The email itself may act as a summary or notification about other information that may be attached as a separate document, or that may be referenced as a web link in the body of the email.

Email's weaknesses

But email also has its weaknesses:

• It may not be read immediately. The recipient may not read or even see the email right away. Unlike a phone call or in-person communication, you may not know that the recipient has received the message right away.
• It cannot carry feeling. Even if you attach cute emoticons to your message, there is no good way to convey emotional context the way that body language or tone of voice is able to do.
• Communication can drag on. If the email goes through a couple of rounds of replies, one recipient or the other may not read or reply to the message right away, lengthening out the conversation.

When to use email

With these characteristics in mind, it is important to use email in a way that leverages its strengths and avoids its weaknesses. Email should only be a part of your communications plan.

Some topics are well-suited for email. These include:

• Pointers to documentation or resources. The written record of an email allows people to reference the resources later. This sort of communication is best used with links to the resources rather than attachments, since it is nearly impossible to keep everyone up to date with the latest version of a document.
• Discussions of strategy or architecture. The asynchronous nature of email may be an advantage, as each participant is able to think things through before responding. The written record of an email trail can also be useful to trace the evolution of an idea or reference concerns that were raised in the conversation.
• Updates about the organization. Email provides a way to communicate to multiple recipients at the same time, and allows people to digest changes before responding to them.
• News updates. These can be read at a time convenient for the recipient.

When not to use email

Conversely, some types of communications should not be handled by email:

• Discussions about expectations. Unless you are trying to create a written record as part of a disciplinary process, this will come across as cold. People want to be able to ask clarifying questions in the moment and get immediate feedback. Use a phone or an in-person conversation.
• Personal matters. These are best addressed in an environment where the emotional context of the conversation can be communicated clearly.
• Bad news. People perceive email delivery of bad news as being cowardly. Schedule a meeting or conference call, or set up an in-person conversation with the affected people.
• Instruction. If you are communicating something complex, like instructions for a complicated task, do so in a way that allows for immediate questions and feedback.