The Risk from Insiders

There has been a lot of coverage of the role of a system administrator in the recent release of information about the National Security Administration's intelligence gathering methods. Regardless what you think about the methods that were revealed, information security professionals need to take a hard look at the sorts of exposures that exist due to organizational insiders.

Snowden's position as a system administrator is just the most recent high-profile insider who betrayed his employer's trust. His removal of documents on a thumb drive was viewed as unsuspicious precisely because of his job function.

As long as we have an IT infrastructure, the people who manage it will be in a privileged position. IT professionals recognize the risk; four of five professionals in a recent survey list insiders as the greatest source of risk to the environment.

The same methods that are used elsewhere in the security landscape will help to control and mitigate the risk from insiders. At a high level, there are three steps that need to be taken:

1. Data Classification: Identify the types of data in your environment, and what the confidentiality, integrity and availability requirements are for each type of data. NIST 800-60 can provide some guidance here.
2. Establish Control Standards: For the different types of data, we need to describe the measures that are taken to protect the data.
3. Audit: The controls need to be evaluated for effectiveness, and the organization's compliance with the the controls must be verified on a regular basis.

Controls

There are several publicly available documents outlining control best practices and standards. Here are a few:

• NIST 800-53
• PCI Security Standards

Some common controls include:

• Physical access controls: Things like security guards, mantraps, proximity card systems, and combination locks on doors control physical access to sensitive areas and systems.
• Logical access controls: In general, people should only have the level of access required for their jobs. Access controls should be as granular as possible, and high-level access should require extra levels of approval and scrutiny. Two-factor authentication should be in place for access to sensitive facilities.
• Personnel management: Some common measures include criminal background checks, periodic security awareness training, contractual attestations, and organizational communications.
• Separation of duties: Where possible, access should be limited to particular functions, and functions should be defined to limit access to sensitive data. In general, developers should not have access to production, system administrators don't need database access, application administrators don't need system-level access, and only the people who manage the hardware and network need physical access to the systems.
• Network security: The network should be segmented appropriately, and firewall rules should be in place to restrict traffic between different security zones.
• Workstations and laptops: Hard drives should have robust encryption and strong password policies should be in place. The types of data that are permitted for local storage should be established and monitored. The ability of end users to install applications needs to be restricted. Patches, anti-virus updates, and security workarounds need to be applied regularly.
• Backups and continuity: Data needs to be protected by a combination of archival backups, long-distance replication, and local disk mirroring/RAID-ing.
• Logging and auditing: Logs need to be collected to measure the effectiveness of these controls, and the logs need to be reviewed on a regular basis.

Some controls should get particular attention as directly addressing the issue of insider-led breaches.

It is bad enough that the security training level of such government employees is not monitored by the DHS. (Most parts of the private sector also don't track administrator security training, for that matter). Beyond carelessness or incompetence, employers need to consider the direct risks posed by their most trusted employees.

Using Email Effectively

Email is mis-used and over-used in most organizations that I have worked in. Some organizations are going so far as to ban email in the workplace. That would be unwise; email is over-used precisely because it is such a useful tool that can be adapted to almost any purpose.

The trick is to use your email effectively and efficiently. Your team's effectiveness will increase if you can create a culture where email is used appropriately.

Email rules of thumb

• Any email you write should be concise and to the point. Don't use two sentences where one will do.
• Use proper grammar, capitalization, and spelling. Errors are distractions from the purpose of the email.
• Make sure that the email's subject line is brief, but relevant to the discussion.
• If you are requesting action (such as a reply), be specific about what you need, what form you need it in, and when you need it. Email is strongest when it can be used to dispose of an issue efficiently, in one exchange.
• Be courteous, even formal. Email and humor don't go together.
• Use the bcc field if the recipients should not necessarily have each others' email addresses.
• Use web links rather than large file attachments.

Email's strengths

Email has some strengths as a tool:

• It is asynchronous. You can write the email when you have time, and the recipient can read it as he or she has time.
• It is fast. Email is delivered almost instantaneously. (That is why it has almost completely replaced the old-fashioned snailmail letter.)
• It can be directed to a specific audience. The sender can define who will receive the email. (Keep in mind that it may be forwarded by those recipients!)
• It can be used to reference other information. The email itself may act as a summary or notification about other information that may be attached as a separate document, or that may be referenced as a web link in the body of the email.

Email's weaknesses

But email also has its weaknesses:

• It may not be read immediately. The recipient may not read or even see the email right away. Unlike a phone call or in-person communication, you may not know that the recipient has received the message right away.
• It cannot carry feeling. Even if you attach cute emoticons to your message, there is no good way to convey emotional context the way that body language or tone of voice is able to do.
• Communication can drag on. If the email goes through a couple of rounds of replies, one recipient or the other may not read or reply to the message right away, lengthening out the conversation.

When to use email

With these characteristics in mind, it is important to use email in a way that leverages its strengths and avoids its weaknesses. Email should only be a part of your communications plan.

Some topics are well-suited for email. These include:

• Pointers to documentation or resources. The written record of an email allows people to reference the resources later. This sort of communication is best used with links to the resources rather than attachments, since it is nearly impossible to keep everyone up to date with the latest version of a document.
• Discussions of strategy or architecture. The asynchronous nature of email may be an advantage, as each participant is able to think things through before responding. The written record of an email trail can also be useful to trace the evolution of an idea or reference concerns that were raised in the conversation.
• Updates about the organization. Email provides a way to communicate to multiple recipients at the same time, and allows people to digest changes before responding to them.
• News updates. These can be read at a time convenient for the recipient.

When not to use email

Conversely, some types of communications should not be handled by email:

• Discussions about expectations. Unless you are trying to create a written record as part of a disciplinary process, this will come across as cold. People want to be able to ask clarifying questions in the moment and get immediate feedback. Use a phone or an in-person conversation.
• Personal matters. These are best addressed in an environment where the emotional context of the conversation can be communicated clearly.
• Bad news. People perceive email delivery of bad news as being cowardly. Schedule a meeting or conference call, or set up an in-person conversation with the affected people.
• Instruction. If you are communicating something complex, like instructions for a complicated task, do so in a way that allows for immediate questions and feedback.

The Price of Insecurity

A recent article on the SANS web site investigated the costs associated with a security breach at Idaho State University.

John Pescatore reports that a breach at ISU's Pocatello Family Medicine Clinic is likely to cost the university $1 million over a 2-year period.

By comparison, implementing best practices in the infrastructure is likely to have defeated the attack, and would have cost around $75k. Even an aggressive security posture is estimated by Pescatore to have cost about $500k total.

Many organizations look at the costs of security breaches, but few consider the TCO related to avoiding a major breach.

Recovery Strategies

Besides cost, the key business continuity drivers for a recovery solution are the Recovery Point Objective and the Recovery Time Objective.

Recovery Point Objective

The Recovery Point Objective (RPO) refers to the recovery point in time. Another way to think of this is that the RPO specifies the maximum allowable time delay between a data commit on the production side and the replication of this data to the recovery site.

It is probably easiest to think of RPO in terms of the amount of allowable data loss. The RPO is frequently expressed in terms of its relation to the time at which replication stops, as in “less than 5 minutes of data loss.”

Recovery Time Objective

The second major business driver is the Recovery Time Objective (RTO). This is the amount of time it will take us to recover from a disaster. Depending on the context, this may refer only to the technical steps required to bring up services on the recovery system. Usually, however, it refers to the amount of time that the service will be unavailable, including time to discover that an outage has occurred, the time required to decide to fail over, the time to get staff in place to perform the recovery, and then the amount of time to bring up services at the recovery site.

The costs associated with different RPO and RTO values will be determined by the type of application and its business purpose. Some applications may be able to tolerate unplanned outages of up to days without incurring substantial costs. Other applications may cause significant business-side problems with even minor amounts of unscheduled downtime.

Different applications and environments have different tolerances for RPO and RTO. Some applications might be able to tolerate a potential data loss of days or even weeks; some may not be able to tolerate any data loss at all. Some applications can remain unavailable long enough for us to purchase a new system and restore from tape; some cannot.

Recovery Strategies

There are several different strategies for recovering an application. Choosing a strategy will almost always involve an investment in hardware, software, and implementation time. If a strategy is chosen that does not support the business RPO and RTO requirements, an expensive re-tooling may be necessary.

Many types of replication solutions can be implemented at a server, disk storage, or storage network level. Each has unique advantages and disadvantages. Server replication tends to be cheapest, but also involves using server cycles to manage the replication. Storage network replication is extremely flexible, but can be more difficult to configure. Disk storage replication tends to be rock solid, but is usually limited in terms of supported hardware for the replication target.

Regardless where we choose to implement our data replication solution, we will still face a lot of the same issues. One issue that needs to be addressed is re-silvering of a replication solution that has been partitioned for some amount of time. Ideally, only the changed sections of the disks will need to be re-replicated. Some less sophisticated solutions require a re-silvering of the entire storage area, which can take a long time and soak up a lot of bandwidth. Re-silvering is an issue that needs to be investigaged during the product evaluation.

Continuity Planning

Continuity planning should be done during the initial architecture and design phases for each service. If the service is not designed to accommodate a natural recovery, it will be expensive and difficult to retrofit a recovery mechanism.

The type of recovery that is appropriate for each service will depend on the importance of the service and what the tolerance for downtime is for that service.

There are five generally-recognized approaches to recovery architecture:

• Server Replacement: Some services are run on standard server images with very little local customization. Such servers may most easily be recovered by replacing them with standard hardware and standard server images.
• Backup and Restore: Where there is a fair amount of tolerance for downtime on a service, it may be acceptable to rely on hardware replacement combined with restores from backups.
• Shared Nothing Failover: Some services are largely data-independent and do not require frequent data replication. In such cases, it might make sense to have an appropriately configured replacement at a recovery site. (One example may be an application server that pulls its data from a database. Aside from copying configuration changes, replication of the main server may not be necessary.)
• Replication and Failover: Several different replication technologies exist, each with different strengths and weaknesses. Array-based, SAN-based, file system-based or file-based technologies allow replication of data on a targeted basis. Synchronous replication techniques prevent data loss at the cost of performance and geographic dispersion. Asynchronous replication techniques permit relatively small amounts of data loss in order to preserve performance or allow replication across large distances. Failover techniques range from nearly instantaneous automated solutions to administrator-invoked scripts to involved manual checklists.
• Live Active-Active Stretch Clusters: Some services can be provided by active servers in multiple locations, where failover happens by client configurations. Some examples include DNS services (failover by resolv.conf lists), SMTP gateway servers (failover by MX record), web servers (failover by DNS load balancing), and some market data services (failover by client configuration). Such services should almost never be down. (Stretch clusters are clusters where the members are located at geographically dispersed locations.)

Which of these recovery approaches is appropriate to a given situation will depend on the cost of downtime on the service, as well as the particular characteristics of the service's architecture.

Causes of Recovery Failure

Janco released a study outlining the most frequent causes of a recovery failure:

• Failure of the backup or replication solution. If the a copy of the data is not available, we will not be able to recover.
• Unidentified failure modes. The recovery plan does not cover a type of failure.
• Failure to train staff in recovery procedure. If people don't know how to carry out the plan, the work is wasted.
• Lack of a communication plan. How do you communicate when your usual infrastructure is not available?
• Insufficient backup power. Do you have enough capacity? How long will it run?
• Failure to prioritize. What needs to be restored first? If you don't lay that out in advance, you will waste valuable time on recovering less critical services.
• Unavailable disaster documentation. If your documentation is only available on the systems that have failed, you are stuck. Keep physical copies available in recovery locations.
• Inadequate testing. Tests reveal weaknesses in the plan and also train staff to deal with a recovery situation in a timely way.
• Unavailable passwords or access. If the recovery team does not have the permissions necessary to carry out the recovery, it will fail.
• Plan is out of date. If the plan is not updated to reflect changes in the environment, the recovery will not succeed.

Recovery Business Practices

Janco also suggested several key business practices to improve the likelihood that you will survive a recovery:

• Eliminate single points of failure.
• Regularly update staff contact information, including assigned responsibilities.
• Stay abreast of current events, such as weather and other emergency situations.
• Plan for the worst case.
• Document your plans and keep updated copies available in well-known, available locations.
• Script what you can, and test your scripts.
• Define priorities and thresholds.
• Perform regular tests and make sure you can meet your RTO and RPO requirements.

Insourcing Picks Up Steam

I recently read an interesting report on insourcing by Pace Harmon. We have previously discussed some of the elements that should go into a decision whether or not to outsource or to offshore. Some major companies such as GM are re-insourcing operations that had previously been outsourced offshore.

Outsourcing typically works best with commodity IT activities. If complex activities are outsourced over the long run, an organization runs the risk of losing the insight and expertise needed to leverage new opportunities as the technology landscape evolves.

Pace Harmon report on several facts that are driving the insourcing trend:

• Wage inflation in India and other prime offshoring locations have led to an erosion in the wage differential between onshore and offshore talent
• Management costs associated with maintaining an offshore or outsourced relationship. Tracking and resolving quality issues can be especially expensive.
• Lack of provider agility and flexibility. When you purchase an offering from another company, you are limited to either purchasing their standard offering or paying a premium for premium service.

Organizations that are considering insourcing need to keep several factors in mind:

• Make sure that you have accounted for all of the costs of the re-insourcing operation. This will include direct costs, such staffing costs and termination penalties, as well as indirect costs such as those associated with reduced stability during the migration
• Does your outsourcing contract specify that the vendor is required to provide you assistance with the insourcing, including training and process documentation? If not, find out what it will cost to get that assistance from your vendor.
• Is your organization up to handling the level of complexity that your environment demands?
• Will you be able to attract and retain the right staff? You may be able to re-badge some of your vendor's staff, but would you be able to retain them?
• Are your organization's processes mature enough to be able to manage your team's technical responsibilities properly? If your organization does not have the maturity to collect requirements and track progress properly, you may not be ready for this transition.

Book Signing

Thanks to everyone who came to the book signing last night. I look forward to doing more events going forward!

When Does Third Party Support Make Sense?

The sales pitch by third party support vendors is compelling. The up-front cost savings are enough to make anyone pay attention. But can they deliver?

In some ways, the answer has to be no. They do not have direct access to new updates or the expertise of the engineers that produce them. When you go with third-party support, make sure that you have a way to get needed security and functionality upgrades, and make sure that you have a path forward.

License compliance, and compliance with regulatory and contractual requirements are your responsibility, not the responsibility of the support provider. If you have not worked around those issues, you have more work to do before making the leap to third party support.

Some vendors have fee requirements to re-establish a relationship if you need to upgrade to a new version or pull in the primary vendor's expertise. Make sure you are looking at the total cost picture, not just the pretty up-front cost picture the sales rep is showing you.

Scott Rosenberg suggests some situations that may be ripe for third party support:

• You have a highly customized environment that is several updates behind and may never be able to be updated.
• You are pulling in extra expertise outside of the basic maintenance agreement, for tuning or design support.
• No updates or upgrades are expected to be needed.

If you do go with a third-party vendor, make sure to protect yourself with a well-written contract and guarantees. Understand what the vendor will do, and how they will handle issues that are beyond their expertise. What timelines, deadlines, and service levels are guaranteed by the vendor?

As with any other business decision, make sure you have the facts before you proceed.

Tips for a Successful Interview

I've posted a few pointers recently in answer to some questions I have fielded from people who are preparing for interviews. I thought it might be useful to have a quick list of a few ideas that I have found useful in interviews:

1. Be enthusiastic, positive, and pleasant. Employers are looking for someone who is going to bring positive energy into their workplace. This is the time to explain what you like about the new position, as well as to catalog the positive aspects of your current and former positions. Be likable and friendly throughout.
2. Bring solutions to the employer's problems. Do your research, ask questions, and listen to the answer. Apply your experience and expertise to the employer's problems, and engage in a dialog about how to solve them. Show your prospective employer what you bring to the table, and set a solid tone for your future relationship.
3. Bring your full attention to the interview. Whatever is happening at home or your current job, it needs to stay outside the room. Listen to what your prospective boss is saying--and listen to what is not being said. Your full attention needs to be here, now, in the moment.
4. Be strong. Don't allow yourself to get discouraged. Be relentlessly positive, even when the interview hits rough patches. Some interviewers deliberately inject difficult questions or sections into their interviews to see how you react to adversity.
5. Prepare good answers. Examine the job listing for clues about what the hiring manager is looking for. Research common interview questions as well as guessing what questions you will be asked, and prepare good answers. Practice them with a coach who can suggest how to improve your answers.
6. Prepare good questions. When the interviewer invites you to ask questions, know what you are going to ask, and ask it in a positive way. Your questions should be on point and professional.

Critical Thinking

One of the most important characteristics of a good IT professional is an ability to think critically.

Mature IT professionals will understand industry best practices, but will also understand why those practices are widely adopted. Professionals will discover the needs of their organization, and will analyze the available tools and practices to adapt them to the current situation.

Cookbooking an IT environment is easy. Analyzing the challenges in the environment and creatively applying solid structures and processes is the mark of a mature IT professional.

Is It a Quantum Computer?

I ran across an article on the new Dwave 2 "quantum" computer.

Some specialists are expressing skepticism about whether or not what is happening in the box is truly "quantum" in nature or not, but it does seem clear that something unusual is happening. The issue is that direct examination of a quantum system leads to a collapse of the wave function, at which point quantum effects are no longer observable.

As these machines are built and tested, it will be interesting to see what types of problems are solved more quickly with this device. The types of problems that are solved may help to indicate whether or not "entanglement" is being used in the way that the manufacturer claims.

Negotiating Service Level Agreements

Service Level Agreements (SLAs) are increasing in importance as enterprises rely on vendors for more and more services. Especially with the penetration of cloud computing in the enterprise, organizations need to protect their reputation by setting clear expectations with the vendors.

Vendors have to protect their own interests, so good vendors are not going to accept any agreement proposed by a single customer. Reaching a SLA will be a tough negotiating process, but it should be done as part of almost any contract negotiation.

There will be a lot of exceptions listed in any detailed SLA. This is the time to identify the exceptions that are of the most concern to your organization, and to demand mitigating actions to be taken by the vendor. (Keep in mind that the cost of these mitigations will be transferred to you in your bill from the vendor, but your organization's reputation needs to be protected against major risks.)

One of the attractive features of cloud computing in particular is that the costs can be spread out over time and across the vendor's customer base. To the extent that you request industry-standard protections, you may be able to spread the costs across your utilization rather than having to pay a lump sum or fixed fee.

A well-designed SLA will contain allowances for necessary maintenance activities and for force majeure. It is common for penalties to be granted as service credits, and it is also common for penalties to be capped at a fixed percentage of recurring costs.

Easing Pain from Office Work

Baseline recently published a slideshow based on an American Osteopathic Association study of pain among office workers. The leading causes of pain for office workers were:

• Sitting for extended periods: 64%
• Posture at the desk: 61%
• Uncomfortable seating: 58%
• Extended computer screen viewing: 46%
• Extended mouse usage: 38%

The study also suggested several tips to help office workers ease some of the causes of physical pain that come from office work:

• Stand up and move every 30 minutes or so. Set a calendar or cell phone reminder if you aren't remembering. When youneed to talk to people in the office, stand up and walk over.
• Excercise 30 or more minutes per day. Find an exercise routine that fits your schedule and life so that you will be able to maintain it over the long term. Even things like using the stairs instead of the elevator, parking a long way away, or taking a lunchtime stroll can make a big difference over the long term.
• Make sure you have a comfortable seat. Some companies will provide an improved seat with a doctors' note; others may allow you to bring in your own seat.
• Adjust your monitor so that the top of the monitor is at eye level when you are sitting up straight.
• Develop good posture. Sit up straight. If you keep your feet flat on the ground, the rest of your body will tend to follow.
• When you mouse, keep your elbows close to your body and try not to flex your wrist. You may need to adjust your seating area to make proper mouse usage easier.

Emphasizing the Value of Your Experience in a Job Interview

I recently sent some advice to someone who was trying to maximize the value of her experience in a job interview. This is the advice I gave her:

Apply your experience to specific problems faced at the company. If you can engage the hiring manager in a back-and-forth dialog about an intractable problem, then make workable suggestions based on your experience, you will set yourself apart from the competition. And you will be demonstrating the value of your experience in a way that is likely to stick in the hiring manager's mind.

The better the research you can do before talking to the hiring manager, the more likely you will be able to engage him or her in a dialog. Figure out what is happening in the group, think of a creative way to address it, and show the hiring manager why your experience is crucial for the team's success.

Data Protection Pointers

eWeek recently posted a slideshow describing several suggestions for planning a data protection policy in a Hadoop framework. The pointers apply to other data environments as well:

1. Account for data protection in the planning phase of the project. Take compliance and privacy concerns into account in the initial design and architecture.
2. Identify data elements that need special protection. What compliance or liability concerns apply to those elements?
3. Does the application need access to the complete, raw data set? Can the data be masked, obfuscated, or desensitized?
4. What encryption requirements do you have? What encryption solutions are available? Do they interoperate with the authentication methods in the environment?
5. Establish standards. It will be easier to keep all the data safe if it is kept in standard templates.

CIO Magazine estimates the value of a data breach at $184-$330 million. Given the cost to an organization's recommendation, there is a business imperative in placing adequate resources and thought behind protecting the data in our custody.

Are Your Budget Priorities Aligned with Business?

A recent article by Larry Bofante of the US Tennis Association raises some important points about making sure that your budget priorities are aligned with your organization:

• What is the return on investment for each item? Does that match with a priority of the organization's leadership? Does it provide a quantitative or qualitative benefit? If not, it needs closer examination.
• Lower your operational costs in order to be able to fund innovation and improvements rather than seeking additional funding.
• Does each project have a business sponsor? If it is not important enough to the organization to be able to recruit a sponsor, perhaps it is not important enough to be done.
• Can the project be done better or cheaper by an outside vendor? If so, make sure that the vendor will represent your organization right.

Are Your Employees Afraid of You?

Nobody likes to screw up. But your employees should not be afraid to bring problems to you. If they do not feel free to bring problems and discuss them, it will take longer to discover and fix problems, and the problems will fester in the meantime.

The key here is to look at how you react to problems. If your first impulse is to attack and blame, you really need to work on that. Make sure that your reaction to problems is positive and resolution-oriented.

Ask questions and make sure you get the information. Your first response to needs to be to contain the damage, and that is not going to happen if you reduce your employee to a quivering puddle of fear. Find out how bad it is, and what your employee thinks happened.

From there, communicate out to affected people. I usually ask someone who is on the team but who is not an expert in the technology in question to take the lead on communications and notifications. You should notify your own boss. That leaves the technical experts free to concentrate on identifying and troubleshooting the problem.

Set up a bridge line for communications. Don't let it be captured by people who are asking unanswerable questions. When that starts to happen, break in with a situation update and the current state of the resolution plan. (Sometimes, that may just be a list of the possible causes that you have ruled out.) Make sure that there is a sense of forward motion and progress. If people insist on asking unanswerable questions, tell them that there will be a postmortem to discuss the causes of the problem and the long-term resolution plan.

Part of a manager's job is to make sure to fly air cover for your technical staff when they are working a problem. They can't concentrate on a resolution if they are constantly being asked "are we there yet?" Ask them for regular updates, but all other communication and updates should flow through you. If necessary, set up a separate bridge line for technical experts and keep the non-technical people off of it.

After the immediate crisis is over, hold a postmortem session to dissect the problem, the team's reaction to it, and how to avoid the problem in the future.

Then, if it is necessary to reprimand your employee, do it humanely and professionally. Reprimands should be direct, clear, private, and over. Don't let the problem drag on, and don't let it fester.

Conducting Effective Reviews

CIO Insight recently had a slideshow article describing several suggestions for conducting performance reviews. Here are several of the key suggestions:

1. Collect information throughout the year. As one of your team members completes tasks, file information about those tasks in a folder or other location that will be easy to find at review time. That will allow you to avoid a recency bias problem when looking at a full 12 months of performance.
2. Communicate throughout the year. Make sure to provide feedback throughout the year. Your comments on the performance review should not come as a surprise to the employee; they should all be things that you have discussed previously.
3. Gather feedback from a broad range of project managers, customers and peers of the team member so that you are getting a balanced look at the strengths and weaknesses of a particular person.
4. Allow the employee to have enough time to fill out the self-assessment portion of the review. The employee should have at least a week to provide feedback.
5. Consider the environment for the review. If possible, hold an in-person review in a neutral location, away from email and phone interruptions.
6. Walk into the review ready to listen. Ask open-ended questions. Show a real interest in the sorts of progress that the team member wants to make.
7. Before the review, make it clear that this will not be a compensation review, just a performance review. Removing money from this discussion will allow a better discussion of employee growth and goals.

Happy Mother's Day!

Selling Yourself in an Interview

I recently gave some advice to an experienced professional who is wondering if her experience might be considered a liability. Age discrimination is a real problem, since there tends to be a bias towards hiring inexperienced people who can demand a lower salary.

This is the advice I gave her; I'd be interested in your feedback:

Explain to the interviewer what you are going to do to improve their organization. Convince them that you would be a bargain at your rate. Do your research ahead of time to find out what the boss and the organization need, and explain why your mix of experience is the best possible fit that they will ever find.

Experienced people can be far more productive than inexperienced people, but they need to demonstrate the attitude and drive to accomplish to their potential.

When Your Opinion is Ignored

I was recently asked how best to respond when you are asked for your opinion, then your opinion is ignored. The question was in the context of someone who had been asked to assist with some interviews for a position in another business unit, but then her fellow interviewers refused to take her input into account.

Here was my recommendation. I'd be interested in other peoples' comments.

It really stinks to be asked for your opinion and then have it ignored. I have sometimes seen that sort of dynamic set up when someone's manager tells them to ask for help, and they resent the order from their manager. (Frequently, they feel like it reflects on their own competence.) They can't disrespect their manager, so they disrespect the person whose advice they have been told to seek.

If you're stuck in that sort of dynamic, make your written recommendations, cc your boss and their boss so that they know that the recommendations have been made in a businesslike and thoughtful way. (You may need to let the report sit overnight and re-edit it in the morning to remove any lingering snarkiness left over from your own hurt feelings from being disrespected.)

Then move on. You have fulfilled your request in a businesslike way. What they do with their time and their responsibility is on them.

In the case where you are asked to interview a bunch of subjects, give a written evaluation of them (I usually use a letter grade like in school--A, B+, D-, etc as a way to summarize), and leave the decision to them.

(If you look at the NFL scouting reports on players from before the draft, you will see a really useful approach to looking at the qualifications of a candidate in a summary way. You probably won't be talking about how fast the candidate runs the 40-yard dash, so substitute in professional requirements instead.)

Your Relationship with Your Boss

Your relationship with your boss is your responsibility. No other relationship will be more important to determining how successful you are in your position. It is not your boss's responsibility to manage this relationship; it is yours.

There are several elements of a successful relationship with your boss:

• Keep your boss in the loop. If you don't tell your boss what you are doing, other people will. And you might not be happy with the quality of the information in the updates they provide.
• Alert your boss to problems, preferably before they blow up. It is better to communicate problems earlier than later. It is best if you can present your boss with your plan of action along with the alert.
• Don't expect your boss to adopt your working style. Everyone does things differently, and you will need to meet your boss's expectations, not the other way around.
• Be up front about what you can deliver and on what timeline. Negotiate expectations that you can meet.
• Consider your boss's priorities. Your job is to make your boss's job easier. This includes being aware of your boss's peer relationships and trying not to cause friction.

Cloud Computing for the Enterprise

Up to now, a lot of the focus around cloud computing has been on rapid deployment, cost, and flexibility. These are all important, but more is needed in an enterprise setting. Enterprise customers need the efficiencies gained from improved management and automation capabilities, as well as the flexibility to use converged services such as PaaS (Platform as a Service). A Peer1 Hosting white paper suggests some additional characteristics you should be looking at in an enterprise-class cloud provider.

1. What are the capabilities of the infrastructure underlying the vendor's offering? What redundancies and reliability measures are in place?
2. What level of support is available? Most enterprises need round-the-clock support by front-line experts who can quickly resolve performance or availability issues.
3. What facilities are available to ease deployment? How easy is it to set up best practices templates that can be leveraged for consistency across the enterprise?
4. How hard is it to manage the environment? In what ways does the vendor support methods to ease patch management, monitoring, or job scheduling across the environment?
5. What support is available for inventory management? VM sprawl is a real problem in an environment where the cost of setting up and abandoning systems is "cheap."
6. Is there support to allow different business units to manage aspects of their own environments separate from other departments or the overall enterprise?
7. Does the vendor offer additional layers of converged services, such as PaaS (Platform as a Service) or DaaS (Database as a Service)? If so, are the offerings aligned with the enterprise's overall technology direction?
8. To what extent has the vendor simplified the overall management process?

The High Cost of Bad Bosses

A recent workplace study by Evolv quantified the cost we pay as a result of incompetent managers.

The annual cost of bad management to the US economy is estimated as $360 billion. Three of four workers characterized dealing with their supervisor as the most stressful part of their day, and that employee/manager relationships were the best predictor of retention rates.

Supervisors who engage with their teams were found to have retention rates 5 to 6 times greater than those of "drill sergeant" supervisors.

Looking Around the Corner

Sometimes you can see the problem coming before it gets here. You've tried to warn people, you've raised flags, and the problem is coming. The people around you are in full ostrich mode, hoping that if they don't see the problem, it won't see them.

Or maybe you look out six months, and you see a trend that is going to need a different approach.

Do yourself and your employer a favor. Prepare.

When the problem is at the doorstep and people are looking for a solution, you will have a plan, and you will have laid the groundwork for its execution.

This doesn't mean that you sit passive-aggressively in your cube hoping that things will go south. Part of preparing is evangelizing your solution to other people. Don't harp on about it, but look for allies who also see the problem and want to think about possible solutions. When you take a solution with broad buy-in to your bosses, it is much more likely to be implemented successfully.

5 Soft Skills for Leadership

A recent article in CIO magazine outlined five soft skills that every leader should develop:

• Financial management
• Critical thinking and analytical skills
• Marketing and communication
• Innovation and collaboration
• Leadership

Financial Management

For most techies, there are very few things more boring than a financial statement. But money is power, and people who understand it are more effective.

Develop the skills needed to understand what your organization's financial goals are, and how priorities are selected to support those goals. Some of this will be from learning how to read and understand reports, but the most important piece is to take an interest in your organization's financial life.

Critical Thinking

Learn how to identify the quality of data and how to see whether the data support the conclusions that are being drawn. This means that you have to approach what you are told with a critical eye, and devote the energy to think about whether the information matches reality, and whether the conclusions are correct.

Why are things done the way they are? How could they be better? What information can you collect to support (or debunk) those conclusions? All of these are aspects of critical thinking that you need to practice.

Marketing

What techie doesn't make fun of marketers? Scott Adams' comics are replete with gags about marketers that have more than a little truth to them. But they have something to teach us.

Effective managers learn how to communicate and sell their ideas. If you can't bring other people along with you, your results will be limited, no matter how good your ideas are.

Innovation and Collaboration

We like technology. So why is it that IT professionals are so bad at applying it to our own organizations? Develop tools and practices that will promote collaboration and multiply the effects of everyone's efforts.

Leadership

Becoming the person that other people want to follow is the essence of leadership. Think about the people you want to follow, and identify the characteristics they have that appeal to you. Be fair to your employees and teammates, and be generous when there is credit to go around. When things go wrong, take the positive tack and look at what was done to fix it and what is being done to prevent the same problem from coming back.

Give Your Employees the Gift of Flexibility

As a manager, you have a tremendous amount of influence over the lives of your employees. On your own, you have the ability to make your team members feel like valued, respected professionals. Or you can make your employees' lives a living hell.

One of the best tools at your disposal is flexibility in setting peoples' work schedules. Let people come in at a slightly different time so they can drop their kid at school or avoid a particularly nasty recurring traffic snarl. A little human compassion can go a long way to making a team member's work experience much less stressful.

I'm not talking about coddling your team. Especially in IT, you will be asking your team members to work odd and inconvenient hours. During deployments or major troubleshooting exercises, you will be asking them to cancel personal plans and stay for long hours to get the work done. That is the nature of IT. People who don't like that need to be looking into a different career path.

What I'm talking about is treating people like respected professionals. Set clear expectations and hold your team members accountable. That is what it means to treat people like professionals. But treat them like respected professionals by giving them the flexibility to accomplish the organization's work in a time and manner that will work for everyone involved.

If you can do that, you will reduce their stress and increase their focus. The organization's work will be done more efficiently and reliably. And isn't that in everyone's best interest?

Using Contractors Intelligently

From executive management's point of view, contractors solve a number of problems:

• You can bring in expertise that your current staff doesn't have already.
• If the project is falling behind, you can bring in extra hands to try to speed up the work.
• You can cut them loose at the end of the project with no impact to your permanent staff's morale.
• You don't have to pay their benefits (not directly, anyway).

The world is not as simple as contracting vendors would have you believe.

• If your staff doesn't have the expertise to understand the implementation, how will you go about maintaining and upgrading it?
• Adding more hands to a project increases the complexity in communicating requirements and managing resources. The experienced (ie, most productive) members of the staff will need to spend time and energy bringing the temporary workers up to speed enough to be able to contribute. Frequently, the overhead of additional hands actually outweighs the benefit.
• If your staff is told they have to support something they don't understand, that is probably not so great for morale.
• Do you really pay your contractors less than the fully-loaded cost of any employee? How about if you include the ramp-up and ramp-down time costs of the contractors into your calculation?

The key here is to use your contractors effectively.

• Don't use them to be the "expert" on a technology. Use them to train your staff on the technology and to work alongside your staff in designing the solution. The role of an "expert" contractor should be as a teacher and a mentor, not as an implementor.
• For contractors who are an "extra set of hands," make sure to assign them the routine, well-understood work. The training time for these tasks will be lower, since they are probably well-documented. You will be able to have more junior team members instruct the contractors on their duties, leaving the more experienced staff to push the project forward during the training interval.

There is an appropriate use for temporary workers. Using them as long-term staff replacements is not a good solution, since it is almost certainly more expensive than using internal staff. Use temporary workers either as mentors or in appropriate lower-level roles. Treat them decently, let them become part of the team. But set clear expectations, and be open and honest about where the project begins and ends. And when the project is over, let the contractors go. If that leaves you short-handed, that means you need an additional long-term employee.

Tactical vs Strategic Spending

Alix Partners released a study discussing the importance of focusing on strategic vs tactical IT expenses.

When there are cost reduction pressures, it is sometimes easier to cut a strategic project than to go after expenses in operational, day-to-day IT. Unfortunately, that is exactly the wrong mind-set to take into budgeting.

Ongoing expenses need to be managed aggressively. Look for old assumptions and unused capacity. Clean up and reduce where possible. Negotiate tough deals with your suppliers. Look for projects that will reduce your ongoing operational expenditures.

For most of us, our employers view IT as an expense, not as an investment. Part of that is our fault. By focusing on eliminating strategic expenses rather than controlling operational expenses, we have limited our ability to make our organizations more competitive and efficient. As IT leaders, we have the power to turn things around by changing how we prioritize IT expenses.

Protect Your Employees' Time

As a leader, you are only as effective as your team is. Your team will only be as effective as they are allowed to be. One of the best things you can do for your team is to protect blocks of time when they can concentrate on non-trivial tasks.

Firefighting Duty

There are a lot of quick-hit tasks that come up in IT. Sometimes there are an overwhelming number of these short tasks, and sometimes it is easier for team members to hide behind the the sense of accomplishment and "scorekeeping" advantages of concentrating on these tasks. But if your team spends all its efforts on these firefighting tasks, your team is not effective. That means that you are not effective either.

One approach to dealing with this problem is to rotate firefighting duty among the team members. Each team member knows that someone else is watching the incoming tickets most of the time, making it easier to schedule blocks of time to be able to concentrate on non-trivial tasks.

Multi-tasking is a myth. The best you can actually pull off is rapid context-switching. But even that comes at a significant performance cost. The most efficient way to get through tasks is to work on them in chunks of time large enough to minimize context-switching's costs. By rotating firefighting duty, you give your team members permission to concentrate on bigger tasks.

Don't Micromanage

Robert L Sutton talks about the cost of well-meaning manager interference. There is a fine line to be walked between allowing your team member to feel unsupported and providing unneeded and disruptive advice.

You have to ask for status updates, but make sure you don't do that in the middle of a productive block of time. And when you get a status update, listen to what the team member says. Sometimes the team member needs a sounding board, sometimes he or she needs advice. The only way to discover which is to listen to what is being said.

The Human Shield

Every organization has its share of instability. Part of your job as a manager is to shield your team members from the chaos occurring in the organization. Set priorities that make sense within the overall direction of the organization, and make sure that the directions and priorities you set are left in place long enough for your team members to actually get something productive done. If you are constantlty re-directing your team based on the latest whim of someone in a staff meeting, your team will be stuck in interrupt mode, and it will never have time to carry a task to completion.

At the same time, you need to be responsive to the rest of the organization. If your priorities are not aligned with the overall direction, you are not contributing to the organization's momentum.

Where is the tipping point between consistency and stubbornness? The answer to that question comes from experience and self-examination. Sometimes you have to pull the plug on a project if you are seriously out of alignment with the organization's priorities. This should not be a common occurrence, or you have an even bigger problem. But you have to make sure that your team's energy is contributing to the organization's progress, and that can only be done by re-aligning from time to time.

Constructive Conflicts

Managers need to bring together team members in a way that good ideas have a chance to evolve and emerge. Frequently, this happens through constructive conflict. Good ideas can emerge when you have passionate, intelligent people disagreeing respectfully if definitely.

But conflicts can have a darker side. There are people who would rather play politics than address problems, and part of a manager's role is to protect your team members from destructive conflicts where possible. Backstabbing and finger-pointing should not be tolerated, and your team members should see you as someone who has their back. If they can concentrate on producing great ideas and solid results, your team will be working to its potential. That is good for your team, and that means it is good for you.

Enthusiasm and Job Interviews

I recently replied to a posting by a job seeker who was concerned that she was competing with a younger candidate for a position that might be viewed as a reduction in rank from her previous experience as a VP. She wanted to know how best to portray her experience without playing into what she felt the interviewer saw as her greatest drawback.

My feeling is that enthusiasm can cover a lot of initial prejudices. You need to excite the interviewer about the prospect of working in the same organization as you, and you need to display enthusiasm about the new position. Come into the interview brimming with ideas about how you can make the environment better, and show the interviewer how excited you are at the thought of carrying them out.

Some people may be put off by this sort of exuberance, but they would probably not be fun to work with anyway. You deserve better than them.

Here is what I replied:

I think the key here is that the interviewer wants to get a sense that the challenges of the new job represent a fresh and exciting perspective for you. Bring up your previous successes and describe ways that you have been able to apply your experiences to create something new.

Describe the parts of the new job that you find particularly intriguing and exciting. If you can communicate a feeling of enthusiasm, it will tend to overcome any concerns by the new employer that you are looking to use their position as a parking lot while you keep looking for a new VP gig.

Book Review: Difficult People at Work

Success in any business setting is going to involve dealing with difficult people. Bell and Smith have written an engaging field guide to several different types of difficult people and how to deal with them.

The first half of the book presents a modified Meyers-Briggs type of personality assessment. The labels chosen for the different axes are different from the standard, which I found to be confusing. (On the other hand, I have heard other people complain that the original labels are confusing. For those people, the labels chosen by Bell and Smith may be more accessible to them.)

I was a little disappointed that more was not done with tying the personality assessments to the types of problem employees presented in the second part of the book.

In the second section, we are presented with several identifiable types of problem people, a strategy for dealing with them, and a story about a real-world sighting of each type of problem person. The descriptions were recognizable, the strategies were reasonable, and the stories were interesting.

This is a quick and enjoyable read.

Sources of Power within an Organization

Success in any organization is going to be built on reciprocal relationships. Cooperation is needed in order for everyone to really succeed. When you need someone to cooperate with you, part of building a successful long-term relationship with them will involve sharing something of value with them.

Here are several examples of the sorts of things you can share, whether or not you directly manage the person in question:

• Inspiration. Explain why what you need is really the right thing to do. Most people want to do a good job and to contribute in a meaningful way.
• Resources. This may be money, or it may be access to equipment, expertise, or space that the other person would find useful.
• Learning opportunities. Technical people like to learn new skills.
• Faster response. Can you arrange for the something to be expedited for the other person?
• Information. Is there information the other person needs that you can share?
• Recognition. Thank you messages with the other person's boss cc'ed are free and very effective.
• Visibility. Is your task something that is being watched by higher-ups?
• Contacts. Can you introduce the other person to new, valuable contacts?
• Team membership. Everyone wants to be part of a successful team.
• Ownership. Can you provide an ownership opportunity to the other person?

Withholding any of the above is a negative currency you can use in negotiations. Try to avoid that. Positive interactions are more likely to get you the sort of cooperation you actually need.

The value of each of these items may be different from person to person. Try to select something that works for the person you are dealing with. Just because one particular currency is more valuable to you, that doesn’t mean that the other person feels the same way.

From Techie to Boss ebook Available

The final version of the From Techie to Boss ebook is available from the Apress web site:

From Techie to Boss Description

From Techie to Boss has been released!

Sometimes it feels like non-technical managers just don't understand what we do for a living. The good ones try really hard and stand up for their team, but they just don't feel it in their bones. If technology is not stamped into your DNA, you just don't get it.

So that means that only technical people should manage technical people, right?

Here's the problem: technical people frequently do not make good managers. It isn't that they aren't smart enough; usually the best technicians are the people who are asked to step into leadership roles. But the skills that make a good techie are not necessarily the skills that make a good leader.

When you become a leader, the focus shifts. It is no longer about what you can accomplish as an individual contributor. You will be judged by your team's accomplishments.

Good technical people have developed good study habits, a sense of responsibility, and a solid work ethic. All of these are important, and can translate into skills that will help you be a good leader. But you will only be an effective leader when you inspire your team members to reach their potential.

Moving into a leadership role can be a bumpy ride. But it can also be hugely rewarding. Make sure to approach it from the right frame of mind. It isn't about you anymore. It is about your team.

This book lays out some of the lessons I have learned during my own transition from a front-line techie to a manager. The technical community has always been all about sharing what we learn. I look forward to hearing you share your own stories and lessons from your own journey.

Book Review: Influence without Authority

Any project manager knows how hard it is to get things done when the people you need don't report directly to you. You're responsible for getting things done, but you haven't been given the organizational authority to carry it out directly.

Cohen and Bradford lay out strategies and techniques that you can use to make sure things get done. Different people require different techniques, and the authors explore different ways to work with different types of people.

The skills laid out in this book are something that every project manager needs.

PaaS Decisions

Platform as a Service (PaaS) is one of the types of cloud services that you can purchase.

Information Week produced an excellent buyers' guide evaluating different PaaS vendors and contrasting their offerings.

As contrasted with IaaS (Infrastructure as a Service), PaaS provides a in integrated programming platform. Usually this includes the underlying infrastructure layer, either directly from the PaaS vendor or from a partner.

The contrast with SaaS (Software as a Service) is that PaaS provides a level of transparency, configurability and programmability that is greater than SaaS provides. PaaS needs to provide a stable platform capable of running arbitrary customer code reliably.

As a practical matter, a PaaS vendor will need to offer only a limited number of Operating System/Database/Application Server/Web Server version combinations in their offering. This may result in increased vendor lock-in, since it may be difficult to match that stack even if the elements are industry standard (eg LAMP).

A differentiator between PaaS vendors who use the same stack may be an Integrated Development Environment (IDE). This may include things like a code versioning system, a test environment, libraries, or an online community.

Some PaaS vendors provide access to a proprietary platform. (One leading example is force.com.) These PaaS vendors are offering increased configuration capability over what would be offered from a straight-up SaaS vendor. Lock-in is a given for this type of vendor, since the customer's programming and configuration efforts will not be portable to another vendor.

As with any other sort of cloud vendor, make sure that security, auditing, and monitoring requirements will fit within your corporate standards. PaaS may allow you less flexibility for monitoring, especially performance monitoring, than IaaS will.

SLAs (Service Level Agreements) will almost certainly not provide penalties large enough to compensate for the real costs of any outage. Find out what the availability history is for the vendor candidates, and probe into the architecture underlying their offering to judge what the expected reliability will be. Then make sure that SLAs include penalties large enough that the vendor feels that they have skin the game when it comes to your uptime.

Turning Around a Team in Trouble

When you take over a team, it is likely to be one of the following situations:

• Start-up: You are building a new team, possibly from scratch.
• Turnaround: This team is in serious trouble.
• Realignment: The team has been successful in the past, but needs to change to sync up with the overall organization's direction.
• Sustaining success: The team is already successful and aligned with the overall direction.

Each of these scenarios has particular challenges and needs a somewhat different mindset.

In some ways, a turnaround is both the biggest challenge as well as the biggest opportunity. The challenges are apparent, but everyone recognizes that things are going to have to change.

Part of turning things around is to find early wins. What shape these take will depend on the exact situation, but the team needs to become accustomed to setting goals and accomplishing them. As the team succeeds and is recognized for success, a culture of accomplishment will start to replace the culture of failure.

Examine the causes for the teams failures. Be honest, but try to address the causes for failure in as non-judgmental a way as possible. A culture of failure is often associated with a culture of blame. Move away from the blame game and towards an honest evaluation of what events occurred when, and what the underlying causes were. Focus on the "what" and the "why" rather than the "who."

Create a culture of honesty, openness and respect. People want to be part of successful teams. Work with your team to define what success looks like, and how to get there. Spend time with your team members to find out what is getting in their way, and help them find ways to overcome it.

At the same time, seek to create a culture of accountability. Once someone accepts a task, he or she owns it. That means that person gets credit for the success, and the responsibility to make it succeed. Turnaround teams are too used to failing. Help the team member find a way to succeed. That doesn't mean that you take over the task; that means that you mentor the team member to identify ways to move forward and succeed.

Model the values you think your team should embody. Things like honesty, openness, continuous improvement, and respect should be evident in how you do your job. If you model good behavior, team members will gravitate towards the values you need the team to embody.

Book Review: Being Geek

The author of this book writes the popular "Rands in Repose" blog. IT people will recognize the situations Lopp describes, and may find some of his advice and insights to be useful.

The chapters are strongly influenced by posts on the blog. While the chapters are interesting of themselves, there is little coherence between chapters, and some of the chapter headings are not a good description of the contents.

Lopp's other book, "Managing Humans," may be a better choice for many readers. In that book, the chapters are more coherent, and the stories are more on-point.

This was an enjoyable, quick read, which is why I gave it 4 stars. I just think the other book is better, which is why I gave that one 5.

Overcoming Cultural Differences

Modern teams usually include people from a range of cultures, backgrounds, and nationalities. Frequently, team members are not even located in the same geographic region. All of these differences can impair a team's efficiency. Misunderstandings can easily become barriers.

Communication styles are different for people from different backgrounds. Here are a few frequently-noticed differences between Western and Indian team members:

• Indians engage in small talk less easily than their western counterparts. Some technical training courses actively discourage students from engaging in small talk in order to avoid embarrassing exchanges.
• Perception of time and schedules are different. When planning, Indian respondents typically do not include time to deal with any difficulties. Managers may need to probe to make sure contingencies have been forseen in the plan.
• People from the southern part of India use what is sometimes called the “Indian wiggle” in which the head is shaken side to side. This is frequently misinterpreted by westerners as indicating “no” (i.e., as in a western “head shake”), when it really indicates anything from “I agree” to “I hear you.”
• Indians frequently use a “wobbly yes” rather than saying “no.” Western managers need to watch for a less-than-firm “yes” and drill down if necessary.

These are all examples of differences in cultural norms. Unfortunately, it is all too easy to fall into the pattern of expecting that everyone else communicates the same way you do—after all, isn’t that what makes “sense?”

There are no rights and wrongs here, just differences. Managers need to learn to understand and value different team members from different cultures. And they need to set a tone within the group where people are not made to feel badly about who they are or where they live.

Folks are just folks. Technical people are, by and large, hard-working, dedicated people. It is not easy for anyone to achieve technical competence. Value your people, no matter what their background is. And make sure to promote a culture of respect and tolerance in your group.

Book signing in Jacksonville

I have a book signing scheduled in the Mandarin, FL Barnes & Noble on June 6 at 7pm.

11112 San Jose Boulevard Suite 8
Jacksonville, FL 32223

I'll be doing a brief presentation at 7pm, then signing and greeting after that. I'll post updates here as we get closer.

Setting Priorities

In an ideal world, every problem should be chased all the way to ground and completely resolved. But if you try to chase too many problems at once, you will be ineffective. It is only possible to really concentrate on one thing at a time; multitasking is a myth. The question is which problem is going to get focused attention first. In the real world, it is usually the case that we have to decide which problems are pursued first and most aggressively.

Two factors to consider are:

• The frequency with which events occur.
• The cost of each event

Counting the frequency is going to depend on having good incident reporting so that we can make valid comparisons between problems. Recurrences of the same problem may be described different ways, or even have different symptoms, so the analysis of incident reports needs to be sophisticated enough to group related incidents together.

The cost of each occurrence may be monetary, but it is most likely to be counted in terms of staff time to resolve and lost productivity while the symptoms are active.