Originally written for the StackPath blog.

Humans are a critical part of operating systems at scale, yet we rarely pay much attention to them. Most of the time, energy and investment goes into picking the right technologies, the right hardware, the right APIs. But what about the people actually building and scaling those systems?

In 2016, Server Density launched HumanOps. It started with an event in London to hear from some of the big names in tech about how they think about the teams running infrastructure.

How can you reach your high availability goals without a team that is able to build reliable systems, and respond when things go wrong? How does sleep and fatigue affect system uptime? System errors are tracked, but what about human error? Can it be measured, and mitigated?

With the acquisition of Server Density by StackPath, I am pleased that HumanOps now has a team dedicated to continuing to build the community. We’re open to anyone taking on responsibility for a local meetup but will also be running our own series of events in major cities around the world. The first of these kicked off this week in San Francisco.

 

What is HumanOps?

The problem

A superhero culture exists within technical systems operations.

Being woken up to fix problems, losing sleep to make an amazing fix live in production and then powering through a full day of work is considered to be heroic effort.

There is little consideration for the impact this approach has on health, family and long term well-being.

The aim

Running complex systems is difficult and there will sometimes be incidents that require heroic effort. But these should be rare, and there should be processes in place to minimise their occurrence, mitigating the effects when they do happen.

HumanOps events are about encouraging the discussion of ideas and best practices around how to look after the team who look after your systems.

It considers that the human aspects of designing high availability systems are just as important as the selection of technologies and architecture choices.

It’s about showing that mature businesses can’t afford to sacrifice their teams and how the best managed organisations achieve this.

If Etsy, Facebook, Spotify and the UK Government can do this. So can you.

How to implement HumanOps

The first step to implementing HumanOps is to understand and accept the key principles.

Key principles

1. Humans build & operate systems that have critical business impact.
2. Humans require downtime. They get tired, get stressed and need breaks.
3. As a result, human wellbeing directly impacts system operations.
4. As a result, human wellbeing has a direct impact on critical business systems.

HumanOps systems and processes follow from these principles.

HumanOps systems & processes

There are many areas of operations where HumanOps can be applied, but there are a few core areas which are worth starting with first. Each one of these could be a separate blog post so here are a series of questions to start thinking about your own process design.

• On call
  This is where the most impact occurs. Being woken up to deal with a critical incident has a high impact, so it is important to design the on-call processes properly. Some key questions to ask: how is the workload shared across team members? How often is someone on-call and how long do they get off-call? What are the response time expectations for people at different escalation levels (e.g. do you have to stay at home by your computer or can you go out but with a longer response time?). Do you get time off after responding to an incident overnight? If so, is there any pressure to forgo that e.g. it should be automatic rather than requiring an active request. Do managers follow the same rules and set an example? Do you expect engineers to support their own code? Do you consider additional compensation for each on-call incident or is it baked into their standard employment contract? Do you prioritise bugs that wake people up?
• Metrics
  You can’t improve something without measuring it. Critical out of hours incidents will happen, but they should be rare. Do you know your baseline alert level and whether that is improving? Do you have metrics about the number of alerts in general, number of alerts out of hours? Do you know if one person is dealing with a disproportionate number of alerts? Do you know which parts of the system are generating the most alerts? How long does it take for you to respond and then resolve incidents? How does this link to the business impact – revenue, user engagement, NPS? Are these metrics surfaced to the management team?
• Documentation
  Only the smallest systems can be understood by a single person. This means writing and keeping documentation up to date needs to be a standard part of the development process. Runbooks should be linked to alerts to provide guidance on what alerts mean and how to debug them. Checklists must form a part of all human performed tasks to mitigate the risk of human error. How do you know when documentation is out of date? Who takes responsibility for updating it? How often do you test?
• Alerts
  Most system operators know the pain of receiving too many alerts which are irrelevant and don’t contain enough information to resolve the problem. This is where linked documentation comes in but the goal should be that alerts don’t reach humans except as a last resort. Interrupting a human should only happen if only a human can resolve the problem. This means automating as much as possible and triggering alerts based on user-impacting system conditions, not just on component failures where the system can continue to operate. Are your alerts actionable? Do they contain enough information for the recipient to know what to do next? Are they specific enough to point to the failure without resulting in a flood if there is a major outage?
• Simulation
  A large part of the stress of incidents is the uncertainty of the situation coupled with the knowledge that it is business / revenue impacting. Truly novel outages do happen but much of the incident response process can be trained. Knowing what you and each of your team members need to do and when will streamline response processes. Emergency response teams do this regularly because they know that major incidents are complex and difficult to coordinate ad-hoc. Everyone needs to know their role and what to do in advance. War gaming scenarios to test all your systems, people and documentation helps to reveal weaknesses that can be solved when it doesn’t matter as much, and teach the team that they can apply haste without speed. How is the incident initially triaged? What are the escalation processes? How does stakeholder communication work? What happens if your tools are down too e.g. is your Slack war room hosted in the same AWS region as your core infrastructure?

The idea behind HumanOps principles is to provide a framework for focusing on the human side of infrastructure.

What’s the point of spending all that time and money on fancy equipment if the people who actually operate it aren’t being looked after? Human wellbeing is not just a fluffy buzzword – it makes business sense too.

The idea behind HumanOps events are to share what works and what doesn’t, and demonstrate that the best companies consider their human teams to be just as important as their high tech infrastructure.

Over the coming months I’ll be writing more about each of these topics and sharing the videos of other organisations explaining how they do it, too.

If you’re interested in attending, speaking or even running a HumanOps event near you, check out the website event listings and get in touch if there’s nothing nearby.

I recently attended a cyber security conference about the current preparedness and future of cyber crime and security in the UK.

One of the audience members made a comment about how seriously businesses take their own security. He thought that, as with annual financial returns, business should be required to certify their own security credentials on an annual basis.

Many incidents of fraud occur not through cards being physically stolen, but through breaches in security at the shops we buy products from. The 2013 breach at Target is an example, the result of which might be that we decide not to shop there again.

Where we can make these consumer choices, the market is operating as it should. But it’s more challenging if the problem exists further down the chain. Perhaps the vendor used by the store for credit checking is the one that suffers a breach, such as at Equifax in 2017. Or more recently, the Ticketmaster incident, which was blamed on a third party component in their customer support system. How can consumers check several orders down into the supply chain?

Of course this is the idea behind one of the GDPR requirements to provide a list of all the third parties that data is being transferred to. But with companies like PayPal sharing data with hundreds of organisations, is it reasonable to expect consumers to check them all? Or any of them? And what would they actually check?

The Government already runs a certification programme called Cyber Essentials. If you want to sell into certain areas of government then you have to have a Cyber Essentials certification. Requiring vendors to certify helps with the government’s supply chain assurance at the same time as encouraging adoption of a UK standard.

But only around 10,000 businesses have certified in the 4 years the scheme has been operating. Is it a lack of awareness about the scheme or do customers and suppliers outside of government just not care? Maybe a combination of both.

As a consumer, you can’t easily assess security from the outside. You can only go on whether there have ever been any historical incidents and even then, that doesn’t tell you much about the state of their security today. So perhaps that audience member was onto something with requiring annual reporting?

There is also a power dynamic at work. The UK Government can mandate all of its suppliers comply with a particular certification because they all want to sell to government. But what if it were the other way around? Or swap the Government for another big organisation. Good luck requiring your suppliers to implement something similar if you’re just a small business.

It is impossible to have 100% security and breaches are inevitable, but as a customer you want to know that companies are taking basic steps to protect you – things like using strong passwords and keeping their systems up to date. It sounds simple, but one of the more interesting statistics from the conference I attended was that 80-90% of instances of cyber crime could be prevented by people having strong passwords and by keeping their computers and devices up to date. Surely these are basic security precautions all businesses should be expected to take.

Companies are already required to submit financial reports and annual statements about company details to Companies House. Would adding a security questionnaire to that return make a difference?

Voluntary compliance is often the first step because the companies that don’t provide the information are liable to be asked: why not? But then Cyber Essentials is already voluntary and not many businesses have certified. Maybe more would participate if it was free (there’s currently a £300 fee) and it just asked you about the current status, rather than requiring active steps to achieve a certification. Perhaps a grading system could indicate what level of security a business has in place which could show on the Companies House search record.

How many people would actually check this? Financial information about companies is already available but how often are returns checked before signing a contract? Suppliers sometimes run credit checks before offering credit terms but then there are multiple outcomes, such as the length of credit. A security check could only really have two outcomes – to do business, or not.

Last year, I wrote about how the supply side of the market was broken in relation to the security of consumer devices. Consumers should be able to expect product security just like they expect product safety. The good news is that they can indeed now expect this. In March this year, a report was released by the Department for Digital, Culture, Media and Sport alongside a new code of practice. Device manufacturers now have an incentive to build their products with security by design. If they don’t, the next step is regulation.

This is good for assurance of the security of consumer internet of things devices, but at what point does not using a secure password and keeping your systems up to date become negligence? Is the next step extending secure by design from internet of things devices to day to day general company administration?

If you’ve ever applied for a job anywhere, you probably had a terrible experience.

Submitting an application into a black hole.

Waiting weeks without hearing anything. Maybe never hearing anything at all.

Vague instructions and trying to guess what the selection criteria are.

Delays getting an answer from early interviews.

Lack of any feedback if you get to later interviews.

More delays getting an offer…then, suddenly, time is of the essence and you must make a decision right now!

For most candidates at most companies, this is probably familiar. How does it make you feel about that company? They might be building awesome products, using the latest tech and working on a problem you really want to be part of. You start off with a great impression from their cool products, external marketing and great reputation, only to leave the process disappointed.

Recruiters are a waste of time – not only do they do a terrible job for their clients but they usually contribute to the reputation damage inflicted by badly run processes. But the companies themselves are just as bad. Once a recruiter hands the process over, then they could still run things properly.

Recruitment is odd in that it usually fails – the most common outcome is the failure of the candidate. That’s by design. Many more people interact with the company through the recruitment process than will ever be employed there.

So why not make them advocates? Or at least not detractors.

Even with the disappointment of not being selected for a job, the company can still leave the candidate with a positive impression.

A well run recruitment process should always send replies quickly and keep the candidate informed at all stages. The candidate should never have to chase for a response. It should be run quickly, with progression to the next stage happening over the course of days or within 1-2 weeks. Schedules sometimes don’t fit but with people being the most crucial aspect of the success of a business, making time for candidates should be a priority. And if a candidate dedicates time to the process, the least you can do is let them know why they weren’t successful in the end.

Every company uses a system to process applications. Communication should be built in, it can even be automated at the early stages. There is no excuse.

Why? Because the candidate might become a customer. They might tell their friends (who could be suitable candidates). They might apply for another position in the future.

Recruitment is another opportunity to build the company brand. To do some marketing. To enhance reputation and show off. It should be treated as such.

2009, when Server Density started, was very early in SaaS. Most software was still sold on-premise with licensing. Some well known products like Salesforce, Xero and GMail (G-Suite/Google Apps) were delivered SaaS-only but they were the minority.

This meant that the understanding of SaaS marketing was also very early. “Growth hacking” wasn’t a thing and a lot of marketing was still around AdWords and banner ads. Indeed, one of our more effective early campaigns was a banner ad on the newly launched Server Fault as part of the Stack Overflow community!

Content marketing was also new. I was able to build up a huge following over the years simply by writing good quality technical content that would appeal to my target audience. The Server Density blog was and remains the biggest source of traffic and leads to the product.

2018 is very different. We’ve reached saturation point for all of the above low-cost channels. You have to do them all but they are only a small part of the marketing mix.

The biggest component in SaaS marketing today is events and conferences. This has been growing over the last few years but attending, speaking at and sponsoring events is now a huge, if not the largest, aspect of SaaS marketing spend. You have to pay to play.

Regardless of who you’re targeting – from developers to small businesses and from startups to enterprise IT managers – being at conferences is a highly effective method of generating leads, and talking to your existing customers.

Potential customers use conferences to discover new vendors. It’s the new way to search for products to evaluate. This surprised me when I was manning our Server Density booth – the number of potential users who come up and ask about your product as part of an evaluation they’re starting. Or because they’re interested in what’s new. These are kind of people you’d expect to hate any commercialisation – that stereotype is outdated.

Existing customers are just as important. If you don’t have a stand, they’ll wonder why you’re not there. They want to see the vendor they picked with a huge presence and lots of marketing materials, and probably t-shirts and swag they can take home, too. It validates their past decision and is also another channel to market to them for cross selling new products or explaining new functionality. Conferences are a legitimate channel for customer success!

If you’re not at all the big industry events, you’re not being seen.

The challenge is that it is expensive.

The cost of sponsoring combined with travel, hotel and food for several team members in high, not to mention any marketing collateral, banners, swag and all the other booth materials. Just sponsoring for your logo to appear isn’t sufficient – you have to have the booth table, too. And you need a good location with plenty of traffic. If you don’t, your competitors will. That’s not cheap.

This is hard for startups. You need a team of people working the conferences and managing the logistics not just a few times a year but a few times per month. The spend quickly ramps up. But the reasons are obvious – it’s difficult to match the lead volume and quality, because you can qualify and demo on the spot. This is why all your competitors are doing it, and it’s why you need to be doing it too.

It’s also a big reason why you can’t do SaaS without significant funding. Without it, you simply can’t compete with the spending levels needed to get the conference machine going.

One of the lessons of the High Growth Handbook is that the most successful software companies start out with a single product, but soon shift to using their distribution advantage to offer a portfolio of products:

Startups tend to succeed by building a product that is so compelling and differentiated that it causes large number of customers to adopt it over an incumbent. This large customer base becomes a major asset for the company going forward. Products can be cross sold to these customers, and the company’s share of time or wallet can expand. Since focusing on product is what caused initial success, founders of breakout companies often think product development is their primary competency and asset. In reality, the distribution channel and customer base derived from their first product is now one of the biggest go-forward advantages and differentiators the company has.

This advantage is fairly clear when it comes to public cloud providers.

When AWS first launched, it began with basic infrastructure primitives: storage (S3) and compute (EC2). Over time, it has added a vast number of products into the ecosystem.

This is a classic enterprise model: if you buy one product in the suite, when you need something else you will look to the vendor you already have a contract with first. This is because it simplifies management interfaces, network configuration, security, support, billing and legal agreements.

AWS certainly has an advantage here – it has the biggest mindshare amongst developers. The ecosystem effects of people with the right technology experience are compelling. Google is competing hard, but AWS is ahead when it comes to the size of the portfolio.

Yet AWS has a weakness when it comes to the office productivity suite. This is already a massive lead generator for Microsoft and Azure, and it could become a big source of customers for Google too.

Microsoft has been leveraging its licensing advantage amongst the largest, enterprise customers who use their productivity products – Office, Exchange, Windows. For a long time, Azure was being pushed to be licensed as part of the deal. If you’re already using Microsoft products, it makes sense to consider Azure first.

Whilst Microsoft might have a good base within the enterprise, Google has a similar foothold within the technology community. Pretty much every startup uses G Suite for email, calendar, docs, etc. Most of these use AWS. But the improvements in Google Cloud Platform, and the security and identity products in particular, are making the G Suite to G Cloud cross-sell more compelling.

Hows does AWS compare? WorkMail and WorkDocs. Not particularly compelling products, and products which seem to have been neglected. I don’t know anyone using either of these. Why would you?

This is one major area that AWS is significantly behind.

The Microsoft / Azure demographic is quite different from those using AWS and Google, but as G Suite and GCP become more tightly integrated, it will become a big differentiator for them.

It might seem like Brexit the only thing the Government is doing right now but in the 2017-2019 Parliament so far, some 23 Bills have received Royal Assent with more than half of those in 2018.

Some of these bills have introduced big changes, such as the Data Protection Act or the Space Industry Act. The former implementing GDPR and the latter paving the way for the UK to enhance its position in the space industry through new launch capabilities.

However, Brexit is taking up a significant part of any policy discussions inside and out of government. Touching every possible area, it is the most important and challenging question of modern times, something which is unlikely to change any time soon. This presents an opportunity for new businesses.

I was recently at an investment forum where we saw 12 startups pitch for funding. The format was very similar to when I was pitching for an initial pre-seed investment into my own software as a service business in 2009: just a few minutes to explain the what, why and how of your idea. But what was different were the types of companies and their approaches to monetisation.

The old approach where the majority of companies focused only on user growth, dealing with revenue later, was gone. These were companies with real business models actually charging for the value their products deliver to the customer rather than relying on vague notions of maximising users and selling them to advertisers.

Everyone always looks at Google as the example of an amazing ad-driven business, and it is. But there are very few situations where you can mirror the user intent of actively searching for something right now. In that context, a relevant ad makes perfect sense. Or if you know so much about a user that you can predict what they might want whilst they browse a social network feed. But these opportunities are rare. Isn’t it actually easier (and better) to build something so useful your users want to give you money for it to continue to exist?

Not only that but most of the pitches were for businesses hoping to tackle what I like to call “real problems”: healthcare and mental health, cyber security, new takes on financial risk, insurance, and several others.

What stood out to me was how many of these startups were addressing challenges which actually attempt to solve some of the big problems in society today. Bringing the startup model of new, innovative thinking to areas which might typically have only been considered solvable by government or the charity sector.

With the public sector grappling with Brexit, it is encouraging to see the forces of competition, revenue and profit coming in to propose solutions to bigger issues than how many more clicks can we get on an ad.

Whilst Elon Musk is often held up as one of the few entrepreneurs tackling big challenges, if the small sample size of the investment forum I attended is anything to go by, there are actually many more. The tech industry shouldn’t just be associated with “eyeballs” or libertarian Silicon Valley culture – it should be about tackling the big problems. For me, this means cyber security, healthcare and space as the areas of biggest opportunity over the coming decade. All areas that were once exclusive to the public sector. What else might also benefit from this approach?

Everyone is asking whether there are any real opportunities in Brexit, for there are certainly obvious downsides. With the public sector busy dealing with the incredible difficulties of extracting ourselves from the EU, this is a unique time to be considering how startups can step up.

Unless you’re just starting a new business from scratch, it is difficult to force big security policy changes across everyone in the company.

There are lots of things you “should” be doing. Whether this is rolling out a new device management platform to ensure everyone has the latest software updates or moving everyone to use a single-sign-on platform for all company logins, if you don’t do it from day one then it simply takes time to change existing practices.

Various events might trigger a revamp of your approach to security. It might be a big customer asking for supply chain assurances, it might be trying to sell into a particular industry like finance of government, or it may even be a security incident.

Security is never “done”. Rolling out device management across all company computer equipment is a big, time consuming project. But there are small wins that employees can do that will set the organisation apart from most other businesses, because most companies are horribly insecure.

At Server Density, we used a simple checklist that everyone would verify every 6 months. Once the initial setup is done when an employee joins, it only takes a couple of minutes to verify. It addresses the basics of ensuring the doors are locked and doesn’t require any specialist knowledge for most steps. Here’s the checklist.

A basic startup employee security checklist

This is specific to the services we used at Server Density, so may need adjusting for your own environment.

1. Have you enabled 2 factor auth on key accounts?
   1. Braintree.
   2. Google.
   3. Github.
   4. [… All key company services listed here]
2. Do you have full disk encryption enabled?
3. Are you storing any sensitive or important files locally e.g. customer lists, strategy documents, private keys?
   1. If so, are they actually local or have they been placed into a cloud “dropbox” (e.g. Google Drive, Dropbox).
   2. If they are in a cloud dropbox, ensure they are either removed (and deleted from the cloud service) or encrypted (use PGP).
   3. If you subsequently encrypt a previously plain text file, be sure the cloud service has not just written a new version and you cannot restore the previous version!
4. Are you running the latest OS version?
5. Do you have a strong OS password?
6. Confirm the password activates on sleep / screensaver.
7. Are you running the latest browser version?
   1. Be sure to restart Chrome regularly so it can apply updates.
   2. Enable click-to-play to prevent browser plugin vulnerabilities.
8. Are you using a password manager e.g. 1Password?
   1. Do you have a strong master password?
   2. Is the master password different from your OS password?
   3. Are you using different passwords for every account?
9. Do you have a passcode on your mobile device?
10. Review your Google Account security
    1. If you set a backup email, make sure it also has multi factor authentication enabled.
    2. Install this Chrome Extension to protect against phishing on your Google account.

I just read a great post about what makes a good board member, from the perspective of the board members themselves. Last year I wrote about how to get feedback from your board, but that is just the last part of a good meeting.

Thinking back on the many board meetings I’ve attended as both a CEO and Non-Exec Director, there are a few characteristics which make for the best meetings.

The context is a technology startup (20 employees) with a board consisting of 4 members (x1 VC representative, x1 angel representative, x1 independent and x1 CEO). Things may be different at larger sizes, although the principles are likely similar.

The board are well informed

You can’t have a proper discussion if the participants do not have the same level of knowledge.

Part of being a good CEO is providing the board with the relevant briefing notes with sufficient time to read them, typically 48-72 hours before the meeting.

Part of being a good board member is ensuring you have read and digested all the briefing notes in advance. Any specific, factual questions should be asked in advance but comments and discussion of the items should be left for the meeting itself.

That is the minimum you have to do.

I also sent out monthly investor updates which went to the board as well. I often had discussions and chats about certain specific issues with individual board members. The balance is between having the board involved as operational executives (which is too much) and having them so high level they don’t have sufficient context to discuss and make key decisions. Only updating them in advance of the board meeting probably isn’t enough to ensure quality engagement.

The board debate a few key issues

The majority of the meeting should be taken up by detailed discussion of 2-3 major issues, briefed in advance.

Everyone needs time to consider the materials and apply them to the issues at hand. You want everyone to form an opinion and be able to discuss it in the meeting.

The best decisions happen when everyone if sufficiently informed to have a debate about the issue at hand and be able to effectively advocate for their point of view.

The worst decisions happen when the decision is made by default because some participants are defeated by someone who is simply better prepared.

There is a reason why we have an adversarial system for English jury trials and the UK Parliament – properly argued debate produces the best decisions. The same applies for company boards.

Good governance involves being a critical friend, and you can only be critical if you have the right information with which to criticise constructively.

The board meeting is structured

The agendas I provided were typically structured like this:

1. 5-10 minutes for questions on materials provided. This is specifically scoped to the numbers and written status updates.
2. 30-45 minutes for 2-3 key areas of focus. Briefing materials would have been provided in advance to provide additional context. I would try and provide a question that we were aiming to answer, so as to ensure we actually made a decision at the end rather than just had a nice chat with no resolution.
3. 10-15 minutes for CEO feedback.
4. 5-10 minutes for anything unscheduled.

The timings were important because it allowed me to provide guidance on how important an issue was and ensure that we kept on track. They were not 100% strict but did allow me to pull things together when we were drifting off track.

Note that there is no time allocated for status updates or reviewing materials – that is all provided in advance and time is only allowed for questions relating to them. It’s a complete waste of everyone’s time to be using the meeting to listen to status updates.

How far back does your email archive go? Years, decades? What kind of discussions, opinions and sensitive files would a search reveal?

What damage could be done if someone got into your email? Your thoughts on people your know? Maybe crucial negotiation documents? Things that probably shouldn’t be public?

Email is a terrible way to store information. It is not supposed to be a database – it’s a method of communication. It’s a single place to find out everything you have ever said to anyone.

Not only does email act like a repository of your own communications, documents and discussions, every single from, to, cc and bcc has a copy. You might delete something from your own inbox but it’s probably nicely replicated many times around the world.

At Server Density, we had a policy of automatically deleting all email after 1 year. We had documentation retention policies for types of files which needed to be kept and for how long e.g. financial records for 7 years. But they were all retained in systems designed for the purpose, not email.

Anything of any importance should be saved somewhere else. Dedicated cloud file storage allows you to control access, share links with expiry dates and manage versions. You can encrypt sensitive files and audit access logs.

Knowing your legal obligations to retain specific data types and deleting everything else is good practice. Combine this with sending expiring links to files in cloud storage rather than attachments and you mitigate the risk of other people’s poor security hygiene too.

Email is insecure. It’s not a good database, so shouldn’t be treated like one.

Apple are always used as the example of why design is a crucial differentiator. Having a well designed, easy to use product is regularly cited as why something is better than a competitor.

That was true in 2007 when comparing the iPhone to other phones.

It was true in 2010-2012 when comparing SaaS products to their on-prem, enterprise alternatives.

It’s not true today.

Having a well designed interface with consistent styling and a well thought through user experience is standard.

Consumer products are always a few years ahead of business products but even with SaaS products in 2018, poorly designed products only stand out because they’re now so rare.

It’s certainly not that creating good design is easy to do – it’s still just as hard as it always was. It’s just that customer expectations have changed in the same way that everyone expects “mobile” to include iOS and Android, websites have to work on all major browsers and public cloud infrastructure is the default.

If you find yourself focusing on your product being “beautiful”, “easy to use”, “design led” or leading with “look and feel”, you may need to rethink your competitive positioning.