Ryan Armstrong's technical blog

Dynamic stats in Go's expvar

Mon, 11 Jun 2018 00:00:00 +0000

Go’s expvar package enables a standardized way to publish runtime statistics for your application. Out of the box, by importing expvar, you will have a HTTP handler registered at “/debug/vars” that will print a whole mess of GC, memory and commandline stats as a JSON document.

Expvar includes many primitives for safely setting or incrementing various static metric types. This works great under normal circumstances, but there are occasions when you might need to dynamically compute the desired metric value on demand.

A key example is “uptime”. How long has my service been up? Using the traditional static approach, you would need to update this value periodically (wasteful) and you risk retrieving an outdated value on request.

The following code is the simplest way to export an “uptime” variable that is computed on demand, using expvar.Func and a closure function.

import (
	"expvar"
	"net/http"
	"time"
)

func init() {
	start := time.Now()
	expvar.Publish("uptime", expvar.Func(func() interface{} {
		return time.Since(start).Seconds()
	}))
}

func main() {
	http.ListenAndServe(":8080", http.DefaultServeMux)
}

If you had more complex needs - say, additional fields to consider - you could instead implement the expvar.Var interface

import (
	"expvar"
	"net/http"
	"strconv"
	"time"
)

type uptimeVar struct {
	start time.Time
	// other important fields...
}

func (v *uptimeVar) String() string {
	// other things to compute...
	d := time.Since(v.start)
	return strconv.FormatFloat(d, 'g', -1, 64)
}

func init() {
	v := &uptimeVar{
		start: time.Now()
	}
	expvar.Publish("uptime", v)
}

func main() {
	http.ListenAndServe(":8080", http.DefaultServeMux)
}

A possible tradeoff in using dynamic metrics is that you risk introducing latency to your monitoring tools or health checks if your dynamic metrics are too compute-heavy or if they depend on I/O.

Optimized abs() for int64 in Go

Sat, 13 Jan 2018 00:00:00 +0000

The Go programming language has no built in abs function for computing the absolute value of an integer. That is, the non-negative representation of a negative or positive number.

I recently needed an implementation of the abs function to solve the Day 20 challenge in Advent of Code 2017. If you want to learn something new and get a kick out of testing yourself, I strongly recommend you check it out!

Go does include an abs function in the math package: math.Abs. Unfortunately, this doesn’t fit my use case, as it only accepts float64 as input and output. I need int64. It is possible to use math.Abs via type conversion, but this introduces some overhead for the conversion to float64 and back to int64, as well as introducing truncation for larger numbers.

There is a great discussion on optimizing math.Abs for floating point integers, but unfortunately these optimizations don’t directly apply to integers, due to their underlying encoding.

There must be other options. So begins my adventure into optimizing a simple function in Go!

You can find the source code and tests for this article on GitHub.

Type conversion vs. branching

The most obvious solution to me for this problem was a very basic function which returns n if n is zero or greater, and returns negative n if n is less than zero. The double negative is, of course, always positive.

Since it relies on a branching control structure to calculate an absolute value, let’s call this function abs.WithBranch.

package abs

func WithBranch(n int64) int64 {
	if n < 0 {
		return -n
	}
	return n
}

It works! Curiously, this is currently (Go v1.9.x) how math.Abs is implemented for float64 numbers.

Great! Branching works. But, does it improve on a call to math.Abs using type conversions? Let’s implement that too.

package abs

func WithStdLib(n int64) int64 {
	return int64(math.Abs(float64(n)))
}

Here we are converting n to a float64, passing it to math.Abs and then converting the result from float64 to int64. Surely, this must introduce some performance overhead?

A couple of simple benchmark tests yield clear results:

$ go test -bench=.
goos: darwin
goarch: amd64
pkg: github.com/cavaliercoder/abs
BenchmarkWithBranch-8 2000000000 0.30 ns/op
BenchmarkWithStdLib-8 2000000000 0.79 ns/op
PASS
ok github.com/cavaliercoder/abs 2.320s

At 0.30 ns/op (nanoseconds per operation), WithBranch is more than twice as fast! But our benchmark doesn’t represent the real world…

Benchmarking

We know that branching code breaks the sequential flow of a program, meaning that pipelining processors must predict what will happen next to maintain similar performance to a sequential program.

In the above benchmark, we used a constant value as the input to each call to abs. Because of this, the processor is able to predict which branch to take every time, because it’s the same every time.

We can trip the processor’s ability to predict the flow of execution, and better represent real world use-cases by introducing a Pseudo Random-Number Generator. If the input is random, the output will be random and the processor’s predictions should be wrong more often.

Given there are only two possible futures, I assume a pipelining processor can compute both outcomes at once, without a performance penalty. Let’s test by experiment.

$ go test -bench=.
goos: darwin
goarch: amd64
pkg: github.com/cavaliercoder/go-abs
BenchmarkRand-8 1000000000 2.59 ns/op
BenchmarkWithBranch-8 200000000 6.36 ns/op
BenchmarkWithStdLib-8 500000000 3.13 ns/op
PASS
ok github.com/cavaliercoder/go-abs 6.692s

I’ve introduced a new benchmark, BenchmarkRand, to measure the overhead of the RNG. We can substract the resulting overhead from the other benchmarks for an approximation of their actual runtime.

It turns out, abs.WithBranch doesn’t perform very well with random input. In fact, abs.WithStdLib is nearly seven times faster.

Truncation

Before we throw out the branching approach, it should be noted that it has the advantage that it won’t truncate any large numbers by converting from a signed int64 to an IEEE-754 float64, losing bits to represent the decimal point.

By example, abs.WithBranch(-9223372036854775807) correctly returns 9223372036854775807, while abs.WithStdLib(-9223372036854775807) experiences an overflow during the type conversion and incorrectly returns -9223372036854775808. It even returns the same incorrect, negative result for a high positive input, abs.WithStdLib(9223372036854775807).

The type conversion approach is faster with random input, but the branching approach is more correct. Can we do better?

A non-branching solution

Chapter 2 of Hacker’s Delight by Henry S. Warren introduces us to a branch-free way of computing the absolute value of a signed integer using a little Two’s Complement arithmetic.

To compute the absolute value of x, first, we compute the value y which is equal to x ⟫ 63. That is, x right shifted by 63 bits on a 64 bit architecture. Finally, compute (x ⨁ y) - y. That is, x exclusive-or y, take y. This yields the absolute value of x.

Let’s implement this in assembly!

// abs.go
package abs

func WithASM(n int64) int64

// abs_amd64.s
TEXT ·WithASM(SB),$0
 MOVQ n+0(FP), AX // copy input to AX
 MOVQ AX, CX // y ← x
 SARQ $63, CX // y ← y ⟫ 63
 XORQ CX, AX // x ← x ⨁ y
 SUBQ CX, AX // x ← x - y
 MOVQ AX, ret+8(FP) // copy result to return value
 RET

We start above with a Go function declaration for WithASM that has no immediate implementation. In a separate ASM code file, we declare the body in Go’s Assembler language. Note that this implementation is only valid on amd64 architecture systems, so we advise the compiler by giving the file name the _amd64.s suffix.

And the benchmarks results:

$ go test -bench=.
goos: darwin
goarch: amd64
pkg: github.com/cavaliercoder/go-abs
BenchmarkRand-8 1000000000 2.52 ns/op
BenchmarkWithBranch-8 200000000 5.91 ns/op
BenchmarkWithStdLib-8 500000000 2.88 ns/op
BenchmarkWithASM-8 1000000000 2.78 ns/op
PASS
ok github.com/cavaliercoder/go-abs 9.454s

All right! We observe a minor improvement of 0.10 ns/op. That’s actually a ~38% speed up. Still, there could be more! It might help to understand how Go treats ASM code.

Compiler optimizations

We need visibility of the optimizations the compiler is applying to the Go functions. The compiler accepts the -m flag to “print optimization decisions”. This can also be enabled in go build or go test with the -gcflags=-m argument.

Here’s what we see:

$ go tool compile -m abs.go
# github.com/cavaliercoder/abs
./abs.go:11:6: can inline WithBranch
./abs.go:21:6: can inline WithStdLib
./abs.go:22:23: inlining call to math.Abs

For simple functions like ours, the Go compiler supports function inlining. Function inlining means that calls to our function are replaced inline with the actual body of our function.

By example,

package main

import (
 "fmt"
 "github.com/cavaliercoder/abs"
)

func main() {
 n := abs.WithBranch(-1)
 fmt.Println(n)
}

… might actually be compiled more similar to:

package main

import "fmt"

func main() {
 n := -1
 if n < 0 {
 n = -n
 }
 fmt.Println(n)
}

According to the compiler output above, WithBranch and WithStdLib are able to be inlined, but WithASM is not. Even the underlying call to math.Abs is inlined into WithStdLib.

Since our WithASM function cannot be inlined, each caller incurs the overhead of a function call. This means writing a stack frame, copying in arguments, branching the program pointer, etc.

What if we even the playing field and disable inlining on our other Go functions? We can do this with a simple pragma comment before the function declaration: //go:noinline.

For example:

package abs

//go:noinline
func WithBranch(n int64) int64 {
	if n < 0 {
		return -n
	}
	return n
}

Running the compiler again, we see many fewer optimizations:

$ go tool compile -m abs.go
abs.go:22:23: inlining call to math.Abs

And here are the benchmark results:

$go test -bench=.
goos: darwin
goarch: amd64
pkg: github.com/cavaliercoder/go-abs
BenchmarkRand-8 1000000000 2.40 ns/op
BenchmarkWithBranch-8 200000000 7.06 ns/op
BenchmarkWithStdLib-8 500000000 3.46 ns/op
BenchmarkWithASM-8 1000000000 2.86 ns/op
PASS
ok github.com/cavaliercoder/go-abs 10.032s

The performance of WithASM remains unchanged, while WithStdLib is three times slower thanks to the overhead of the non-inline function call.

The lesson learned for me here, is that the performance gained by implementing ASM by hand needs to outweigh the benefits of compiler type safety, garbage collection, function inlining, etc. In most cases, this won’t be the true, though there are exceptions; like taking advantage of SIMD instructions for cryptography or media encoding.

Running the benchmarks a few times it becomes clear that there is a reasonable performance gain to be had via ASM. But, can we somehow retain the benefits of inlining?

One inline function, please

The Go compiler cannot inline functions that are implemented in assembler, but implementing our Two’s Complement arithmetic in Go is easy:

package abs

func WithTwosComplement(n int64) int64 {
	y := n >> 63 // y ← x ⟫ 63
	return (n ^ y) - y // (x ⨁ y) - y
}

The compiler says our new function can be inlined:

$ go tool compile -m abs.go
...
abs.go:26:6: can inline WithTwosComplement

How does it perform? It turns out, when we re-enable inlining, it performs even better! A whole 0.18 ns better, or roughly 60%.

$ go test -bench=.
goos: darwin
goarch: amd64
pkg: github.com/cavaliercoder/go-abs
BenchmarkRand-8 1000000000 2.39 ns/op
BenchmarkWithBranch-8 300000000 5.71 ns/op
BenchmarkWithStdLib-8 1000000000 3.01 ns/op
BenchmarkWithASM-8 1000000000 2.88 ns/op
BenchmarkWithTwosComplement-8 1000000000 2.70 ns/op
PASS
ok github.com/cavaliercoder/go-abs 14.417s

Now that the overhead of a function call is gone, the Two’s Complement implementation in Go out performs the ASM implementation. Curiosity might lead us next to wonder how similar the compiler’s output is to our hand-crafted ASM?

There’s an app for that.

Passing -S to the Go compiler causes it to “print assembly listing”.

$ go tool compile -S abs.go
...
"".WithTwosComplement STEXT nosplit size=24 args=0x10 locals=0x0
 0x0000 00000 (abs.go:26) TEXT "".WithTwosComplement(SB), NOSPLIT, $0-16
 0x0000 00000 (abs.go:26) FUNCDATA $0, gclocals·f207267fbf96a0178e8758c6e3e0ce28(SB)
 0x0000 00000 (abs.go:26) FUNCDATA $1, gclocals·33cdeccccebe80329f1fdbee7f5874cb(SB)
 0x0000 00000 (abs.go:26) MOVQ "".n+8(SP), AX
 0x0005 00005 (abs.go:26) MOVQ AX, CX
 0x0008 00008 (abs.go:27) SARQ $63, AX
 0x000c 00012 (abs.go:28) XORQ AX, CX
 0x000f 00015 (abs.go:28) SUBQ AX, CX
 0x0012 00018 (abs.go:28) MOVQ CX, "".~r1+16(SP)
 0x0017 00023 (abs.go:28) RET
...

The compiler’s implementation is exactly the same. Only this time, it has the advantage of being correctly configured and being cross-platform portable! Running the compiler again with GOARCH=386 produces a more complicated program that handles the 64-bit arithmetic on 32-bit machines.

In this case I could be more trusting of the compiler.

One last note about memory allocations; all of our implementations exhibit the same ideal behavior. When I run go test -bench=. -benchmem, I observe that each of the functions result in zero allocation operations and zero bytes allocated within the function body.

Homework

A few exercises for the reader:

What if we implemented the assembly for abs as a compiler intrinsic, rather than a runtime function call?
What if we used the PABSQ instruction included in SSSE3? This is not currently supported in Go’s ASM.

Conclusion

The Two’s Complement implementation in Go offers portability, function inlining, predictable performance, zero allocations and no integer truncation due to type conversions. The benchmarks showed a significant speed advantage over the other approaches.

The end result:

func abs(n int64) int64 {
	y := n >> 63
	return (n ^ y) - y
}

Inline vs. discrete rules for AWS Security Groups in Terraform

Sat, 15 Jul 2017 00:00:00 +0000

There are two ways to configure AWS Security Groups in Terraform. You may define rules inline with a aws_security_group resource or you may define additional discrete aws_security_group_rule resources.

My first instinct was to define a “base” Security Group using inline rules and then extend on it using external rules. Bad idea. More on that later.

For the two valid options though, there are important implications and I found these were not clear at the time of writing (circa Terraform v0.9.11). After a little research and experimentation I have a much clearer understanding and hope to save you all the bother.

This article focuses on managing AWS Security Groups in Terraform but you will find that all of the principles explored here apply equally to Network ACLs and Route Tables - both of which allow inline or external rule management.

Two approaches

Here’s how an inline Security Group definition looks:

resource "aws_security_group" "allow_all" {
 name = "allow_all"
 description = "Allow all inbound traffic"

 ingress {
 from_port = 0
 to_port = 0
 protocol = "-1"
 cidr_blocks = ["0.0.0.0/0"]
 }

 egress {
 from_port = 0
 to_port = 0
 protocol = "-1"
 cidr_blocks = ["0.0.0.0/0"]
 }
}

Above there are two rules, an ingress and egress rule defined inside or inline with the aws_security_group resource block.

Here’s how the same idea can be expressed using external rules via the aws_security_group_rule resource:

resource "aws_security_group" "allow_all" {
 name = "allow_all"
 description = "Allow all inbound traffic"
}

resource "aws_security_group_rule" "ingress" {
 type = "ingress"
 from_port = 0
 to_port = 0
 protocol = -1
 cidr_blocks = ["0.0.0.0/0"]

 security_group_id = "${aws_security_group.allow_all.id}"
}

resource "aws_security_group_rule" "egress" {
 type = "egress"
 from_port = 0
 to_port = 0
 protocol = -1
 cidr_blocks = ["0.0.0.0/0"]

 security_group_id = "${aws_security_group.allow_all.id}"
}

The Security Group and each of its rules are defined as discrete resources, intimately linked together in loving union by the security_group_id attribute.

A reasonable person might posit that the outcome of both configurations would be the same, but they are different in subtle ways - ways that might hurt a bit if not clearly understood.

Option 1: External rules

I had hoped that external rules would function similar to Puppet’s concat module - gathering partial resources defined anywhere in the graph and then enforcing the sum state. The reality however, which does make sense, is that the desired state is managed non-destructively.

You can test this, by manually adding a rule to the Security Group created by the Terraform code above (the snippet with only external rules). If you run terraform apply, it will ignore this manually created rule.

What this means, is that you can add and enforce rules on a Security Group that was created elsewhere. Just be cautious of conflicts with existing rules, precedence and collisions in your rule numbers.

Pros

Rules can be added non-destructively to unmanaged Security Groups. This might be useful for Security Groups associated with shared services, such as a Bastion host. Terraform configurations from all over the kingdom can create rules on a common Security Group to enable the access they require.

Ideally, one would create and associate distinct, well-tagged Security Groups for each use-case but we are limited to only five Security Groups per network interface.
Security Groups can be “modularized” and extended upon. You might write a Terraform module that creates a Security Group with standardized naming, tagging, lifecycle and maybe even some base access rules. Additional rules could then be added to the Security Group returned by the module using aws_security_group_rule resources.

Cons

Security Group rules that are added by accident or by nefarious means will not be nuked by Terraform. It won’t even report that they exist.
Infrastructure code should succinctly describe and quasi-document your actual infrastructure. Defining rules for a single Security Group in disparate files in your code-base makes it difficult to see at a glance what the state of a Security group should be.
It’s difficult to know or manage for how externally created rules will effect the rules you are configuring in Terraform. This is particularly important for Network ACLs, where a rule defined elsewhere could have higher precedence than yours and might explicitly deny a port you intend to open.

Option 2: Inline rules

When rules are defined inline, a Security Group is managed destructively. That is, any rule not defined inline, including rules defined elsewhere in Terraform and rules added manually or via other tools, will be unapologetically destroyed whenever Terraform next runs.

Pros

Security Groups are for security. They need to be clear, understood and well configured. Using inline rules means your resource definition is complete, definitive and deterministically provisioned. You can have confidence that there are no rules being described elsewhere in the code-base or added outside of Terraform. If there are, they get destroyed.

Cons

There’s no way to share code between inline-style Security Groups. This might mean lots of repetition for Security Groups that are mostly the same; having a common set of rules with minor exceptions between them. Make use of the five available Security Group slots per network interface to allow re-use of shared base Security Groups - though this is difficult if you’re using least-privilege, point-to-point access rules.

You can’t use the count meta-parameter to described rules in a loop.

It works with external rules:

variable "http_ports" {
 default = ["80", "443", "8080", "8443"]
}

resource "aws_security_group_rule" "ingress_http" {
 count = "${length(var.http_ports)}"

 type = "ingress"
 protocol = "tcp"
 cidr_blocks = ["0.0.0.0/0"]
 from_port = "${element(var.http_ports, count.index)}"
 to_port = "${element(var.http_ports, count.index)}"

 security_group_id = "${aws_security_group.allow_all.id}"
}

… but not with inline rules:

variable "http_ports" {
 default = ["80", "443", "8080", "8443"]
}

resource "aws_security_group" "allow_all" {
 name = "allow_all"
 description = "Allow all inbound traffic"

 ingress {
 count = "${length(var.http_ports)}"

 protocol = "tcp"
 cidr_blocks = ["0.0.0.0/0"]
 from_port = "${element(var.http_ports, count.index)}"
 to_port = "${element(var.http_ports, count.index)}"
 }
}

# 1 error(s) occurred:
#
# * aws_security_group.allow_all: ingress.0: invalid or unknown key: count
#

You’re going to need to code up an inline rule for each port.

Why not both?

My naive first approach was to blend both approaches. I hoped to create a configurable aws_security_group “module” that contained some mandatory rules, like allowing ingress SSH, monitoring, etc. This module could then be extended with additional rules using aws_security_group_rules resources. A kinda pseudo-OOP-abstract-class approach.

Fortunately, the Terraform documentation contained a well-lit warning sign:

At this time you cannot use a Security Group with in-line rules in conjunction with any Security Group Rule resources. Doing so will cause a conflict of rule settings and will overwrite rules.

What happens when you combine both methods? Here’s a cool high-school science lab experiment for you!

The following Terraform code defines both inline rules, and an external ingress_http rule.

resource "aws_security_group" "allow_all" {
 name = "allow_all"
 description = "Allow all inbound traffic"

 ingress {
 from_port = 0
 to_port = 0
 protocol = "-1"
 cidr_blocks = ["0.0.0.0/0"]
 }

 egress {
 from_port = 0
 to_port = 0
 protocol = "-1"
 cidr_blocks = ["0.0.0.0/0"]
 }
}

resource "aws_security_group_rule" "ingress_http" {
 type = "ingress"
 from_port = 80
 to_port = 80
 protocol = 6
 cidr_blocks = ["0.0.0.0/0"]

 security_group_id = "${aws_security_group.allow_all.id}"
}

Instructions

Apply this code with terraform apply - it should create the ingress_http rule
Apply the same code again with terraform apply - it should remove the newly created ingress_http rule
Go to step 1

Terraform will create and then destroy the external rule on each alternating invocation - like that scene with Dormammu in Marvel’s Dr Strange.

Bug? No. It actually kinda makes sense. Jake Champlin from HashiCorp explains it on a related GitHub issue.

Summary

I’ve come to prefer using inline rules where possible. It means our Security Groups match the code and phantom ‘allow all’ rules can’t be introduced that would break our security model and the integrity of our tests.

It does mean we have to duplicate some rules in a few Security Groups, and keep these definitions in sync, but I share the opinion that a little copying is better than a little dependency.

For more information about AWS Security Groups in Terraform, please see:

Postmortem: a socket leak in the AWS SDK for Node.js

Tue, 27 Jun 2017 00:00:00 +0000

On May 18, 2017, thewest.com.au started serving 504 Gateway Timeout responses for images. The root cause was a socket leak in a months-old release of the AWS SDK for Node.js that had been pinned by accident. Connections to S3 hung after the response headers arrived, and once 50 of them had leaked the SDK’s connection pool was exhausted and every request for a static asset stalled. This is how we found it.

I led the Production Engineering team at Seven West Media in Western Australia. Our internal postmortem differs from this one only in formality, not in the facts.

While reports came in from colleagues and customers that images were failing to display, a team of developers and sysadmins invoked our incident response process. It borrows heavily from ITIL Incident Management and Google’s Site Reliability Engineering, and it calls for us to stay calm, objective and to seek out quality information before making decisions.

Network I/O deviations

Initially we discovered unusual network I/O patterns on our Content API servers.

The Content API is a Node.js/Express application fronted by a local instance of nginx. It is responsible for serving article content, listings, curations, etc. to both our internal web servers (for server-side rendering) and to our client-side Javascript which uses the Single-Page Application model (see: Isomorphic Javascript: The Future of Web Apps). It also serves static assets, such as images, which it pipes directly from Amazon S3.

The above figure from Zabbix shows network I/O on one of the Content API instances. Around 7 AM the bursty traffic flattened into a ceiling, which meant something was being serialised or capped.

We employed Brendan Gregg’s USE method to build a clearer picture of network performance. We observed:

throughput between each component was otherwise typical, with plenty of headroom
no dropped packets, interface errors or other known TCP/IP issues that would indicate saturation or congestion
no error messages in any of the counters or logs we knew to check
no network issues were being reported by AWS
Health checks on the Content API servers were passing

There was one metric we neglected to check for saturation. More on that below.

HTTP 499 errors

One fastidious engineer noticed something else amiss: a large number of 499 Client Closed Request responses being logged by nginx on the same Content API servers.

The above screen grab from Sumo Logic of the same instance shows the uptick in 499 status messages at the same time that the network I/O anomaly appeared. The uptick is obvious in this image, but it was visually masked on our dashboards by the thousands of successful requests occurring at the same time.

Reviewing the nginx logs on each server revealed an interesting pattern that helped us to understand the scope of the issue: all 499 responses were constrained to static assets only. Requests for dynamic content were unaffected.

This finding was easy to confirm locally on any of the production servers. A quick $ curl -iv http://localhost/... for any static asset would hang until the client cancelled the request. The same curl request for any other resource behaved as expected.

This test also validated that no error was logged via nginx or the Node.js application. The application logs did show an incoming request but no outbound response or error.

So here’s what we knew:

Either our CDN or Load Balancing layer was cancelling some requests and returning status 504 as no timely response was received from upstream
Nginx was logging a status 499 when the downstream components cancelled a request
Requests to the Content API for static assets were falling into a black hole which meant that image content on the site was failing to display

At this point we decided we had sufficient information about the scope of the issue to consider a mitigation plan.

Stem the bleeding

Our incident response process emphasizes small, non-destructive changes over wholesale repairs where possible. The mantra I repeat to my team:

Avoid corrective surgery - just stem the bleeding

Corrective surgery is best performed once the situation has stabilized, we have better information and everyone is cool, calm and collected.

We formed a hypothesis that freshly launched instances of the Content API would behave correctly. They did: the new servers produced no errors once marked in.

We did have the option to instead simply restart the unhealthy instances, but our process also encourages the preservation of evidence. Restarting the instances would have reset their state - and as we learned later - would have reset the key metric we neglected to observe. We might not have found the root cause until the issue surfaced again.

Besides that, servers should be treated like cattle, not pets. A freshly launched instance represents a known good state.

With the healthy instances in rotation, everything stabilized. The 499 log messages disappeared, our curl test started behaving correctly and users were confirming a positive outcome.

Root cause analysis

With clearer heads and happy customers, we had the opportunity to take a detailed look at the unhealthy servers and compare them to healthy servers.

A call to $ netstat -t4 shows all TCPv4 sockets on a Linux system. The following figure shows the results of this call on both an unhealthy (left) and healthy (right) Content API instance.

Notice that the unhealthy instance has a large number of sockets in a CLOSE_WAIT state and also that most sockets have a large quantity of packets in the Receive Queue (second column). All of these sockets list AWS S3 as the foreign end point.

By comparison the healthy server has only one CLOSE_WAIT socket and the Receive Queues are mostly empty, except for the lonely CLOSE_WAIT socket. Could this be the beginning of a very slow connection leak?

Remember that key metric that we neglected to observe during our network performance observations? We needed to measure the number of CLOSE_WAIT sockets on a server. Queue length and the other sockets states might also have been useful under differing circumstances.

According to Gordon McKinney’s TCP/IP State Transition Diagram, CLOSE_WAIT indicates that the socket is:

… waiting for a connection termination request from the local user

In our case the “local user” was the Node.js application. Adding -p to the netstat call printed the PID that owned the leaked sockets. The PID belonged to our application.

Somewhere in our application, sockets were being created for S3 requests, but never cleaned up. These sockets were exhausting some unknown upper limit, causing new requests to hang.

Now we know where to look and our picture of the control path has a little more detail. Here’s how it looked once the leaked sockets reached critical mass:

We set about tracing the code path for S3 requests in our codebase and found the following.

AWS SDK for Node.js

All of the connections to AWS S3 were being established via the AWS SDK for Node.js.

The SDK uses socket pooling for connections to S3. The default maximum size of the pool is 50. If sockets were leaking and there is an upper boundary to the number of active sockets, it stands to reason that new requests to S3 (in our case, for static assets) would fail in some way once 50 sockets had leaked. The number 50 anecdotally matched the number of CLOSE_WAIT sockets we had seen with netstat.

We also found that our version of the SDK had been inadvertently pinned in our codebase to v2.6.14, which was released shortly before we relaunched thewest.com.au.

Given the age of our SDK version, it was conceivable that the socket leak was now a known issue. We trawled through GitHub issues, PRs and the SDK Changelog and found a suspect note in the 2.50.0 release notes:

bugfix: Request: Updates node.js request handling to obey socket read timeouts after response headers have been received. Previously timeouts were being ignored once headers were received, sometimes causing connections to hang.

Inspecting the code also revealed a significant body of other work had also been released concerning sockets.

We formed a hypothesis that updating the AWS SDK would cause all sockets to be cleaned up correctly or emit errors that could be safely handled. There were strong coincidences that suggested this was our issue. Patching the SDK was low risk, easily tested and overdue. First, though, we needed to measure.

Measurements

It was important that we started measuring these socket metrics before implementing a fix. Before-and-after measurements are critical to validate any fix.

I wrote a Zabbix module named zabbix-module-sockets to capture the following metrics:

the number of sockets in each state
the sum of the receive queue of all sockets
the sum of the send queue of all sockets

With a little simulated load we were able to replicate the connection leak in pre-production environments and measure the results in Zabbix.

The fix

With a reasonable hypothesis, measurements in place and way to validate our plan in pre-production, we deployed the updated AWS SDK to our pre-production environments.

The fix held. The following figure from Zabbix shows the change in behavior around June 2nd, when the fix was subsequently deployed to production.

The gradual upward trend of the Receive Queue length (green) stops after the production deployment and the queue length stabilizes. The earlier sawtooth pattern you may observe is caused by application restarts that were triggered to remediate the socket pool being exhausted.

The following figure shows sockets states on the same server. The thin, red slice of CLOSE_WAIT sockets discontinues around the time of the fix deployment.

The max column for CLOSE_WAIT sockets reads 50: exactly the SDK’s default pool size.

Lessons learned

We trade time-to-market against technical debt like everyone else, and we make the best moves we can with the information and resources we have at the time. This incident showed where that trade had gone wrong, and a few places it could go wrong again.

Keep dependencies current

The SDK had been pinned by accident at v2.6.14 and nobody had looked at it since launch. The fix for our bug had been public for months. A dependency that nobody chose to pin is debt that nobody is tracking; versions need an owner and a review cadence.

Alert on 499 responses

For HTTP servers that support HTTP 499 like nginx, a small number of 499 Client Closed Request responses should be expected at the edge, where clients do have the prerogative to disconnect prematurely. But in higher frequency, or at other points in your signal chain, these responses might indicate that timeouts are incorrectly configured. So…

Get your timeouts right

Each component should be configured with a timeout that is longer than all its upstream dependencies. By example, if you have a 3 second timeout on database queries, then the application server should not cancel requests any sooner than this. Otherwise the source of a problem is obscured and all you are presented with is a timeout error from a component other that the one that misbehaved - as we experienced. In deadline propagation terms, a callee should give up before its caller does, so the error surfaces from the layer that actually misbehaved.

This solution is complicated by services that need multiple round-trips or have asynchronous behaviors, but…

Everything should be bounded

There is an expected operational behavior of every component in our system. Any operation that completes too quickly or too slowly should create an error that is meaningfully propagated, logged and alerted on.

In this case, our Content API servers should have been configured with a tightly constrained timeout on all requests to S3. Error handlers in our code would then have sent a meaningful 5xx error back downstream.

This principle should also apply to the incoming request to the API. We should cancel any request that runs too long, for any reason. This has operational advantages, but is also good security practice to mitigate denial of service.

Health checks must validate all dependencies

Everything fails, all the time.

Werner Vogels, CTO of Amazon

Each of our micro-services has a health check route that quickly validates each of its known dependencies (e.g. it can connect to the database, write to a log file, etc.). In this case we had neglected to include S3 in the dependency checks. As a result, each service was advertising itself as healthy.

If we had included S3 in the health checks, the checks would have failed, these instances would have been marked out of the load balancer, alerts would have escalated and new healthy instances would have been rolled in automatically.

The trade-off is that a deep health check ties every instance’s fate to a shared dependency. Had S3 itself failed, every instance would have failed its check at once and the load balancer would have had nothing left to route to. The check has to distinguish a sick instance from a sick dependency, or the balancer needs a floor below which it stops marking instances out.

Summary

Under the stress of trying to restore services, we never felt any hunch to check the battle-hardened AWS SDK for leaked sockets to the notoriously reliable AWS S3 service. But this was our root cause.

As unlikely as the issue was, having a response plan in place and sticking to it gave us the best possible outcomes:

We followed the facts and found the root cause
Services were restored quickly, without introducing unnecessary risk or destroying evidence
We uncovered other weaknesses in our systems that could be addressed before they led to disaster
We learned a whole lot about the software and systems we work with

Appendix: Leaks from other buckets

With our new socket monitoring in place in Zabbix, we noticed a much smaller number of sockets to S3 being leaked on almost all of our application servers.

We discovered that the AWS CodeDeploy Agent was the culprit. We use CodeDeploy to deploy our Node.js applications and S3 is used as the storage backend.

Fortunately, these sockets leak much slower (we’ve only seen up to 3 on any one instance). For now, routine restarts will suffice until AWS are able to address the issue.

Zabbix tools I've written

Sat, 17 Jun 2017 00:00:00 +0000

I’ve had the pleasure of working with Zabbix for many years now, (since 2.0). Over that time, I’ve built a few tools that have made life easier for myself, the teams I’ve worked and the Zabbix community.

A couple of these tools are well known, like the popular libzbxpgsql, g2z or mib2zabbix, but others are obscured in the depths of my GitHub repository list.

I hope by listing these tools here, that they are of some use to you, and you might even consider contributing more rad features to them.

Monitoring

PostgreSQL Module

A native Zabbix module for high performance monitoring of PostgreSQL servers.

Systemd Module

A native Zabbix module to monitor systemd units via D-Bus.

Sockets Module

A native Zabbix module for monitoring Linux sockets - including TCP, TCP6, UDP, UDP6 and UNIX.

Configuration

Zabbops

A Python package for automating Zabbix configuration using Amazon Web Services APIs and services.

PostgreSQL Partioning scripts

PostgreSQL scripts to create and manage partitions for massive Zabbix databases.

Deployment

Docker-Zabbix

Dockerfiles to build vanilla, all-in-one Zabbix containers of common versions. Mostly useful for testing integrations with vanilla Zabbix instances.

Puppet-Zabbix

A Puppet module to install, configure and maintain Zabbix on RHEL derivative operating systems.

Packer-Zabbix

A Packer script to build a Vagrant box running Zabbix.

Zabbix MSI WiX template

A WiX template to build a MSI package of the Zabbix agent and tools for installation on Windows systems.

Extending Zabbix

Go2Zabbix

A Go-lang library for creating loadable Zabbix modules, written in Go.

Embedded Python Module

A native Zabbix module that embeds a Python interpreter into the agent, to allow for high performance Python script execution. Also includes a Python library to simplify the creation of embedded Zabbix modules.

Testing

Zabbix Agent Bench

A simple binary for highly parallel stress testing of a Zabbix agent and its loaded modules or User Parameters. Useful for highlighting issues in code or for simple integration tests.

Templating

Zabbix Template Convertor

A Python script to convert Zabbix templates between versions. For example, you can convert a 3.2 template to be loadable in Zabbix 2.0.

MIB2Zabbix

A Perl script to automatically generate Zabbix templates from SNMP MIB files.

PS-ZabbixTemplates

A native PowerShell module with useful extensions to the Zabbix agent, and the ability to automatically generate templates from Windows PDH Performance Counters.

Zabbix Conference 2016

Sun, 18 Sep 2016 00:00:00 +0000

I’m finally recovering from jet lag after landing home from Zabbix Conference 2016 in Riga, Latvia. My good mate and colleague, James Cook and I had the pleasure of attending the conference this year to discover what’s new with Zabbix, how others are making use of it, and most importantly, to meet and greet with the superb team at Zabbix SIA HQ.

After my talk at Perth DevOps Meetup, Zabbix were kind enough to ask me to give a similar presentation at the conference. You can download a PDF copy here.

My talk covered a little about Kinetic IT and our client, the WA Department of Education. I then described some of the automation tools we have written to turn Zabbix into a fairly autonomous monitoring system. Finally I gave a demonstration of a selection of the automation tools that I have released or rewritten for open source.

Please also, check out the talks for all the other speakers at the Zabbix Conference 2016 homepage.

Downloading large files in Go

Tue, 19 Jan 2016 00:00:00 +0000

Note, 2026: this article describes grab v1. The package has since moved to github.com/cavaliergopher/grab and the v3 API differs from the code below — GetAsync is gone in favour of Client.Do and Client.DoChannel, and IsNoFilename is now the ErrNoFilename error value. The design discussion still holds; for working code, see the current documentation.

The net/http package bundled with Go provides a really solid HTTP implementation which excels particularly as a base for HTTP based API clients and servers.

If you’re writing software which needs to download large files from the Internet, such as ISO images or software packages, you may need to implement some client side logic to overcome some more use-case specific challenges such as:

multiple concurrent downloads
naming of downloaded files
UI feedback with progress indicators
clean cancellation of running downloads
resuming of interrupted downloads
validating downloaded files using checksums

This article will step you through using a Go package called grab which abstracts net/http to provide these features. We’ll build a simple ‘wget’-like binary to make use of all such awesomeness.

grab provides convenience methods grab.Get, grab.GetAsync and grab.GetBatch for simple operations. When you need more control over the HTTP session, you can use a grab.Client and configure it to your needs.

Examples for these functions and using a custom client are included below.

To get started, install the grab package with:

$ go get github.com/cavaliercoder/grab

Download a file

The simplest way to download a file is using grab.Get. It accepts two parameters; a destination file path and the source URL. grab.Get uses grab.DefaultClient as a HTTP client which has default settings. It will follow redirect responses from remote servers and use a corporate proxy if configured on the host system. Essentially, grab.Get is a wrapper for grab.DefaultClient.Do.

You may specify an existing or non-existing file path as the destination or you may specify an existing directory.

If a directory is given as the destination, grab will determine the filename using Content-Disposition headers if they are returned by the remote server or extract a filename from the source URL. If either of these features fails, an error is returned which can be identified using grab.IsNoFilename.

If the destination filename exists, grab assumes it is a complete or partially complete download and will resume downloading from the end of the file if supported by the remote server. Otherwise the file will be overwritten.

grab.Get and all other download functions return a grab.Response which includes context about the downloaded file; including the path where the file was saved.

grab.Get is a blocking, synchronous operation, which means that the function does not return a response until the download is complete or encounters an error. This is not terribly useful for lengthy downloads so I’ll solve this problem a little further down.

First, the following example will create a simple binary which will download a source file from a URL specified on the command line and save it to the current working directory (".").

package main

import (
	"fmt"
	"github.com/cavaliercoder/grab"
	"os"
)

func main() {
	// get URL to download from command args
	if len(os.Args) < 2 {
		fmt.Fprintf(os.Stderr, "usage: %s url\n", os.Args[0])
		os.Exit(1)
	}

	url := os.Args[1]

	// download file
	fmt.Printf("Downloading %s...\n", url)
	resp, err := grab.Get(".", url)
	if err != nil {
		fmt.Fprintf(os.Stderr, "Error downloading %s: %v\n", url, err)
		os.Exit(1)
	}

	fmt.Printf("Successfully downloaded to %s\n", resp.Filename)
}

Build it with:

$ go build -o grab-example

and run it with:

$ ./grab-example http://some-url

Add progress updates

The simple example above will download a file but it is not practical for lengthy downloads which should provide some feedback to the user with the progress of the download.

The following example uses grab.GetAsync which immediately returns a channel which will receive a *grab.Response and close as soon as the download has been negotiated with the remote server, before the file transfer has started.

grab.GetAsync is a wrapper for grab.DefaultClient.DoAsync.

Once the response is received, it can be polled periodically to monitor the progress of the file transfer until it is finished. This example simply prints a progress update every 200ms.

All of the grab.Response methods are thread-safe and atomic.

package main

import (
	"fmt"
	"github.com/cavaliercoder/grab"
	"os"
	"time"
)

func main() {
	// get URL to download from command args
	if len(os.Args) < 2 {
		fmt.Fprintf(os.Stderr, "usage: %s url\n", os.Args[0])
		os.Exit(1)
	}

	url := os.Args[1]

	// start file download
	fmt.Printf("Downloading %s...\n", url)
	respch, err := grab.GetAsync(".", url)
	if err != nil {
		fmt.Fprintf(os.Stderr, "Error downloading %s: %v\n", url, err)
		os.Exit(1)
	}

	// block until HTTP/1.1 GET response is received
	fmt.Printf("Initializing download...\n")
	resp := <-respch

	// print progress until transfer is complete
	for !resp.IsComplete() {
		fmt.Printf("\033[1AProgress %d / %d bytes (%d%%)\033[K\n", resp.BytesTransferred(), resp.Size, int(100*resp.Progress()))
		time.Sleep(200 * time.Millisecond)
	}

	// clear progress line
	fmt.Printf("\033[1A\033[K")

	// check for errors
	if resp.Error != nil {
		fmt.Fprintf(os.Stderr, "Error downloading %s: %v\n", url, resp.Error)
		os.Exit(1)
	}

	fmt.Printf("Successfully downloaded to ./%s\n", resp.Filename)
}

Batch downloads

The next example allows multiple URLs to be given on the command line and downloaded simultaneously using grab.GetBatch() which is a wrapper for grab.DefaultClient.DoBatch().

Files will be transferred three at a time, as 3 is given for the worker count parameter in the call to grab.Getbatch(). To download all files immediately (one worker per request), simply give 0 as the worker count parameter. Each download will be saved to the current working directory as "." is given as the destination parameter.

With a batch operation we don’t have immediate access to any grab.Response. These will be sent via the channel returned by grab.GetBatch() each time a worker starts a requested URL. We don’t know when all of these responses will arrive, but we want to monitor downloads which are already in process so we create a for loop and select between two channels; one to receive responses and the other (the ticker) to periodically print the status of the responses which have already been received.

package main

import (
	"fmt"
	"github.com/cavaliercoder/grab"
	"os"
	"time"
)

func main() {
	// get URL to download from command args
	if len(os.Args) < 2 {
		fmt.Fprintf(os.Stderr, "usage: %s url [url]...\n", os.Args[0])
		os.Exit(1)
	}

	urls := os.Args[1:]

	// start file downloads, 3 at a time
	fmt.Printf("Downloading %d files...\n", len(urls))
	respch, err := grab.GetBatch(3, ".", urls...)
	if err != nil {
		fmt.Fprintf(os.Stderr, "%v\n", err)
		os.Exit(1)
	}

	// start a ticker to update progress every 200ms
	t := time.NewTicker(200 * time.Millisecond)

	// monitor downloads
	completed := 0
	inProgress := 0
	responses := make([]*grab.Response, 0)
	for completed < len(urls) {
		select {
		case resp := <-respch:
			// a new response has been received and has started downloading
			// (nil is received once, when the channel is closed by grab)
			if resp != nil {
				responses = append(responses, resp)
			}

		case <-t.C:
			// clear lines
			if inProgress > 0 {
				fmt.Printf("\033[%dA\033[K", inProgress)
			}

			// update completed downloads
			for i, resp := range responses {
				if resp != nil && resp.IsComplete() {
					// print final result
					if resp.Error != nil {
						fmt.Fprintf(os.Stderr, "Error downloading %s: %v\n", resp.Request.URL(), resp.Error)
					} else {
						fmt.Printf("Finished %s %d / %d bytes (%d%%)\n", resp.Filename, resp.BytesTransferred(), resp.Size, int(100*resp.Progress()))
					}

					// mark completed
					responses[i] = nil
					completed++
				}
			}

			// update downloads in progress
			inProgress = 0
			for _, resp := range responses {
				if resp != nil {
					inProgress++
					fmt.Printf("Downloading %s %d / %d bytes (%d%%)\033[K\n", resp.Filename, resp.BytesTransferred(), resp.Size, int(100*resp.Progress()))
				}
			}
		}
	}

	t.Stop()

	fmt.Printf("%d files successfully downloaded.\n", len(urls))
}

Customizing requests

None of the previous convenience methods offer any control over the HTTP request or transport. Much like the net/http package, grab enable such controls as well as additional features such as checksum validation via grab.Client, grab.Request and grab.Response.

In the same way that the grab.Get* methods work behind the scenes, using a client requires that you define and configure a grab.Client, one or more grab.Requests and pass them to one of the grab.Client.Do* methods which then return a grab.Response for each request.

Let’s take a look at some of the configuration options available when creating a download request in the following example:

// create a download request
req, err := grab.NewRequest("http://some-url/my-file")
if err != nil {
	panic(err)
}

// set destination file path
req.Filename = "./my-file"

// set expected file size if known
req.Size = 1024

// set expected file checksum if known
b, _ := hex.DecodeString("b982505fc48ea2221d163730c1856770dc6579af9eb73c997541c4ac6ecf20a9")
req.SetChecksum("sha256", b)

// delete the downloaded file if it fails checksum validation
req.RemoveOnError = true

// request a notification when the download is completed (successfully or
// otherwise)
ch := make(chan *grab.Response)
req.NotifyOnClose = ch

You may also configure the HTTP request itself, including request headers, cookies, authentication, etc. using the http.Request nested in the grab.Request.HTTPRequest field, as demonstrated in the following code:

// set custom HTTP method
req.HTTPRequest.Method = "POST"

// set request headers
req.HTTPRequest.Header.Set("X-SOME-HEADER", "Custom value")

// set a cookie
req.HTTPRequest.AddCookie(http.Cookie{})

// set basic HTTP authentication headers
req.HTTPRequest.SetBasicAuth("username", "password")

Using a custom Client

grab provides a default client, grab.DefaultClient which is used by each of the grab.Get* methods. If you wish to customize HTTP transport rules such as connection timeouts, proxy configuration, redirect policies, etc. you may create a custom client with grab.NewClient.

The following code includes examples of customizing a client:

// create a custom client
client := grab.NewClient()

// set the user agent string for all HTTP requests
client.UserAgent = "MyApp"

// set a custom connection timeout
client.HTTPClient.Timeout = 3 * time.Second

// replace the HTTP client with one that bypasses any system proxy settings
client.HTTPClient = &http.Client{
	Transport: &http.Transport{
		Proxy: nil,
	},
}

Once you have configured a client and some requests, you pass the requests to whichever of the grab.Do* methods best match your use case. These methods are synonymous with the grab.Get* methods and behave as follows:

grab.Client.Do - blocks and returns a response once the download is completed or an error occurs
grab.Client.DoAsync - immediately returns a channel which will receive a single *grab.Response and close as soon as the download has been negotiated with the remote server; before the transfer has started
grab.Client.DoBatch - accepts multiple requests and executes them simultaneously. It accepts a workers parameter which determines how many downloads will be in process at any given time, while the remaining requests are queued until a worker is available. It returns a channel which will receive a grab.*Response for each request and close once they are all sent. The responses are sent through the channel as soon as the download has been negotiated with the remote server; before the transfer has started

The following and final example uses a custom client to download a batch of files with periodic progress updates:

package main

import (
	"fmt"
	"github.com/cavaliercoder/grab"
	"os"
	"time"
)

func main() {
	// get URL to download from command args
	if len(os.Args) < 2 {
		fmt.Fprintf(os.Stderr, "usage: %s url [url]...\n", os.Args[0])
		os.Exit(1)
	}

	// create a custom client
	client := grab.NewClient()
	client.UserAgent = "Grab example"

	// create requests from command arguments
	reqs := make([]*grab.Request, 0)
	for _, url := range os.Args[1:] {
		req, err := grab.NewRequest(url)
		if err != nil {
			fmt.Fprintf(os.Stderr, "%v\n", err)
			os.Exit(1)
		}

		reqs = append(reqs, req)
	}

	// start file downloads, 3 at a time
	fmt.Printf("Downloading %d files...\n", len(reqs))
	respch := client.DoBatch(3, reqs...)

	// start a ticker to update progress every 200ms
	t := time.NewTicker(200 * time.Millisecond)

	// monitor downloads
	completed := 0
	inProgress := 0
	responses := make([]*grab.Response, 0)
	for completed < len(reqs) {
		select {
		case resp := <-respch:
			// a new response has been received and has started downloading
			// (nil is received once, when the channel is closed by grab)
			if resp != nil {
				responses = append(responses, resp)
			}

		case <-t.C:
			// clear lines
			if inProgress > 0 {
				fmt.Printf("\033[%dA\033[K", inProgress)
			}

			// update completed downloads
			for i, resp := range responses {
				if resp != nil && resp.IsComplete() {
					// print final result
					if resp.Error != nil {
						fmt.Fprintf(os.Stderr, "Error downloading %s: %v\n", resp.Request.URL(), resp.Error)
					} else {
						fmt.Printf("Finished %s %d / %d bytes (%d%%)\n", resp.Filename, resp.BytesTransferred(), resp.Size, int(100*resp.Progress()))
					}

					// mark completed
					responses[i] = nil
					completed++
				}
			}

			// update downloads in progress
			inProgress = 0
			for _, resp := range responses {
				if resp != nil {
					inProgress++
					fmt.Printf("Downloading %s %d / %d bytes (%d%%)\033[K\n", resp.Filename, resp.BytesTransferred(), resp.Size, int(100*resp.Progress()))
				}
			}
		}
	}

	t.Stop()

	fmt.Printf("%d files successfully downloaded.\n", len(reqs))
}

Recommended books

Sun, 27 Dec 2015 00:00:00 +0000

For me, the best measure of a book is whether it changes the way I think. These are the ones that did, in no particular order. Most of them found me somewhere between running infrastructure teams and building storage systems, and they’ve shaped how I approach both.

Thinking in Systems

By Donella H. Meadows

Everything is connected, across time and space. Donella beautifully illustrates a number of different systems “Animals” - common patterns that appear in air-conditioner thermostats, the art of steering a car on the road and distributed computing systems. This book taught me to prefer small, calculated moves over wholesale systems changes and to better interpret and respond to measurements over time.

Lean Enterprise

By Jez Humble, Joanne Molesky and Barry O'Reilly

When an IT organization gets large enough, the need for predictability and stability starts to stifle innovation until fast moving, unencumbered startups disrupt them.

Lean Enterprise provides a prescriptive process and plenty of follow up reading to enable large organizations to overcome “inertia” while minimizing cost and risk. The business is divided into three horizons where first, exploration and experimentation take place. Experiments that prove to deliver value are transitioned into the ‘Exploitation’ horizon where an MVP is quickly transformed into a sustainable product. Finally, the ‘Stabilization’ horizon focuses on efficiency and stability; keeping the lights on so new experimentation can get underway.

The Open Organization

By Jim Whitehurst

It’s no secret I’m a big believer in the power of Free and Open Source Software and open source communities. This book offers an inside look at Red Hat’s success as the largest open source vendor in the world, from the perspective of their C.E.O., Jim Whitehurst.

Jim explores how Red Hat leverage the power of internal and external communities to build the world class software products that run the New York Stock Exchange and countless other high profile enterprises.

What I found most interesting was Red Hat’s adoption of a meritocracy; where community members and their contributions are evaluated on their merits, not their title or pay grade. This has changed the way I take on ideas from my colleagues and has also encouraged me to participate in discussions outside my remit.

Made to Stick

By Chip Heath and Dan Heath

Ever wondered why the latest company strategy is not getting any traction? Why your marketing pitch is falling on deaf ears? Why even clear, concise facts are not enough to persuade people to your way of thinking?

Made to Stick answers these questions in a common-sense manner that has profoundly changed the way I communicate. It offers a very simple checklist (the SUCCESs model) to apply to your own communications to ensure they will be understood, memorable and influence people’s thinking.

I’ve found this book particularly useful in presenting my ideas for new projects, and in writing for my blog.

The Five Dysfunctions of a Team

By Patrick Lencioni

From the teams on the service desk to the executives and the board, nothing is more critical to the success of a business than effective teamwork. In The Five Dysfunctions of a Team, Patrick Lencioni illustrates the reforming of an executive team using a simple, five tier model which highlights the most common obstacles to being effective as a team.

For me it highlighted some common sense cause and effect in interpersonal and inter-team politics in my own workplace. Where I might have previously strived to encourage people to focus on business outcomes and results, I learned we first need to create a culture of accountability, which first needs unanimous commitment towards common goals, which first needs constructive conflict to take place, which first requires people to trust one another enough to feel vulnerable.

The Phoenix Project

By Gene Kim, Kevin Behr and George Spafford

This book is the de facto entry point for anyone interested in understanding DevOps. What I love most, is that it is a management parable, not another inapplicable implementation of Continuous Delivery or Infrastructure As Code.

The Phoenix Project is a very deliberate homage to The Goal and is written in a very similar style, adapted for modern IT audiences. It incorporates elements of The Goal’s Theory of Constraints, Lean manufacturing, Toyota Kata, Agile and Continuous Delivery into a DevOps management recipe and way of thinking.

It focuses primarily on helping the business achieve its goals using IT as a core competency, rather than focusing on the Dev/Ops relationship and automation tools. It helped me to focus and realign my day-to-day activities to those that can be directly linked to successful outcomes for the business as a whole.

The Goal

By Eliyahu M. Goldratt

For a parable written in 1984 concerning manufacturing plants, the application of The Goal to modern IT development and operations is incredibly potent and is expanded on further in The Phoenix Project.

This book has really helped me to identify where to (and where not to) invest my time and energy, especially concerning IT automation and process improvement.

I’ve written a full review in The Goal: A challenge to IT Operations

Toyota Kata

By Mike Rother

Almost every “improvement” initiative I have ever participated in could be described as an “implementation” of a known methodology. We implement Scrum or Lean or war rooms or silo crushing meetings or Continuous Delivery, etc., etc., etc.

Sometimes the implementations are successful (i.e. a well run Scrum team) but don’t actually benefit the business in any tangible or profitable way and even moderately successful implementations tend to degrade very quickly.

Mike Rother takes a look inside Toyota’s successful Continuous Improvement culture to show why people who copy Toyota’s process implementations typically cannot emulate Toyota’s success.

Toyota’s Lean processes are actually an evolution, more explicitly, an adaptation to change, specific to their needs and environment. This book explores how Toyota approaches changes and develops improvements. Their way of thinking and “Improvement Kata” can be applied almost universally; including in IT.

This book has helped me to bake Continuous Improvement into every action I take, rather than approaching it as sporadic and typically unsuccessful projects.

Who Moved My Cheese?

By Spencer Johnson M.D.

It only took roughly an hour to read this profound little parable from cover to cover. Who Moved My Cheese follows four little creatures, Sniff, Scurry, Hem and Haw, who navigate their way through a maze searching for Cheese, an analogy for the things that make us happy. Most importantly, it explores how each character reacts to changes in the maze and the location of their Cheese.

Change is inevitable in our lives, and especially in IT. This book offers some simple ideas, expressed as Handwriting on the wall of the maze, for how to identify, cope with and adapt to change effectively.

I identify most with the character Sniff. I tend to sniff out change and typically get excited by it. This book has helped me to better understand that others approach change differently and that being a “forward thinker” can actually be perceived as threatening.

I’m making a point of reading this regularly to make sure I don’t get trapped in the arrogance of success or miss the writing on the wall when my comfort zones start to change.

The C Programming Language

By Brian W. Kernighan and Dennis M. Ritchie

Not a programmer? Doesn’t matter. The concepts in this book and the tutorials will be a real challenge for a first time programmer and you may even give up. In fact, I did give up. But I came back a few years later and conquered it cover to cover.

As a Sysadmin, it’s important to understand the difference between the heap and the stack, what a pointer is, what shared memory is for, how memory gets allocated and corrupted, how memory leaks happen, how code patterns affect performance and capacity, how shared libraries are really used, what stack traces mean, etc.

If you can grok this book, all of these concepts become much clearer and you’ll even be able to build small tools of your own, participate better in open source communities and communicate with developers in a common tongue.

Good to Great

By Jim Collins

Good to Great is probably the most thoroughly researched and well presented work I have ever encountered. Jim Collins dissects the differentiators between companies which outperform the market consistently over fifteen years to those that don’t. The findings are often counter-intuitive but are very well established in the presented facts.

Jim condenses everything down into a tangible framework called The Flywheel which offers stepping stones to replicating the success of the Great companies.

One stage of the flywheel, The Hedgehog Concept complements the idea presented in Made to Stick that succinct, unidirectional strategies are most effective at enabling a company to move forward in unison.

A later stage, First Who, Then What emphasises that “getting the right people on the bus” is the first and most critical part of building a successful company. This idea is echoed as the First Key in First, Break All the Rules.

I’d strongly recommend anyone read this book as it offers a great sense of context around organisational success. More importantly however, the principles of a Level Five Leader should be taken on board by team members at any level of experience or title.

First, Break All the Rules

By Marcus Buckingham and Curt Coffman

This book brings together the findings from interviews with more than 80,000 successful managers to find out what makes them so effective.

The following keys are found to be consistent across most of the managers who took part in the study:

They hire for talent and understand the recurring thoughts, feelings and behaviors that are needed on their team
They define clear and valuable outcomes and leave the implementation to their employees
They focus on developing strengths instead of trying to fix weaknesses
They work to fit people where they belong and can succeed

I’d recommend this book to anyone looking to move into a management or leadership role, anyone looking to expand their team, hoping to land on the right hire, or anyone hoping to understand why their current org-structure is letting them down.

Site Reliability Engineering

Edited by Betsy Beyer, Chris Jones, Jennifer Petoff and Niall Richard Murphy.

This book collects essays from key members of Google’s SRE teams that build, deploy, monitor and maintain Google’s large scale production systems. The essays cover the philosophies, processes, architectures, mantras and mistakes that enable Google to be so successful.

Many of the lessons are directly applicable to much smaller scale IT service providers, like their incident response procedures, on-call rostering and philosophies concerning risk management and eliminating low value “toil”.

Other lessons were less applicable, but equally interesting, such as the design challenges of distributed consensus systems, automated deployment and data processing pipelines.

I’d recommend this book to engineers and operations managers in any IT org. It’s a real privilege to be able to learn from the refined and battle hardened practices of some of the world’s most talented operators.

The Goal: A challenge to IT Operations

Sat, 12 Dec 2015 00:00:00 +0000

This week I took an adventure in 1980’s manufacturing plant optimization in Eliyahu M. Goldratt’s “The Goal: A process of ongoing improvement”. The book is featured in discussions around DevOps, particularly concerning its heavy reference in Kim, Behr and Stafford’s The Phoenix Project and all things Lean.

It was an unusual read for me but has certainly changed the way I approach problems in IT operations and even in getting my daughter to school on time.

In this post, I’ll aim to condense some of the principles taught in the book, their application in DevOps and finally offer a challenge to IT organizations where application deployment is not the primary line of business

Summary

The Goal is a fictional business parable that follows Alex Rogo, a manufacturing plant manager at UniCo manufacturing. His plant (and the business as a whole) are deep in the red, despite employing some of the best management wizardry of cost cutting exercises, order expedition, localized optimizations and even fancy new robots.

Alex has a chance meeting with a physics professor, Jonah, from his university days, who challenges some of Alex’s (and my own) most basic assumptions about productivity and the reason the business exists at all. Jonah leads Alex, via provocative Socratic inquiry, to approach his manufacturing and personal issues scientifically.

The result for Alex was saving the business, and a good deal more besides.

I was surprised how entertaining the book was. For me the subject matter of the plant was fairly foreign and uninteresting but the story telling is very human and relatable.

Manufacturing jargon is kept to a minimum, with only enough detail given to solidify some of the abstract ideas being taught.

The book didn’t feel dated at all. The only reminder that we are not in the current millennium is the mention of dot-matrix printers and smoking in the office.

Being a work of fiction, I did doubt whether the scale of some of the successes encountered in the book could be replicated in reality. While the turnaround of the fictional UniCo is extreme, the common-sense nature of the science described lends credibility. Fortunately for us, there is now also a long history of well published successes and failures with Eliyahu’s process and similar ideas from DevOps, the Toyota Production System or Lean manufacturing.

Socratic inquiry

All of the lessons taught in the book could be condensed into a two or three page cheat sheet; even with substantial context. Unfortunately, some of the lessons are so contrary to our modus operandi that in this form they would read as nonsense; much like the notion of the world being round once did.

For example, Jonah suggests that most efficiencies gained in local areas of production actually serve to cripple the entire system, contrary to my intuition.

The beauty of the relationship between Jonah and Alex, is that Jonah never provides answers to anything. He only provides questions and validation of Alex’s assertions. In doing so, much like the Greek philosopher Socrates, Jonah teaches Alex how to think. Essentially, Jonah could give Alex a fish and feed him for a day, but instead teaches Alex how to fish, so Alex can own his own destiny.

This form of dialog caused me to continually consider my own situation. Clearly the nuances of manufacturing do not directly apply to IT operations, so some of the answers Alex discovers have no relevance to me (I don’t use heat treat ovens in the datacenter). Instead the challenges Jonah offers to my assumptions and way of thinking lead me to answers of my own.

The goal

The first question Jonah poses to Alex is “What is the goal”? In my position in IT Operations, I might have answered “uptime” or “minimizing costs” or “information security” or “good customer service”. Jonah highlights that these sorts of answers are actually just a means to an end, not the end itself.

For UniCo manufacturing, the end, or the goal, is to make money. I work for an outsourcing company, who have assigned me to a government body concerned with education. For me the goal is a little unclear, so I’ll give it some more thought and perhaps conversation with the powers-at-be. Is the goal to make money for my employer? Or to improve education for my client? Or to make money for my employer by improving education for my client? Or is the goal to assist the current government in winning the next election (not educational outcomes) so we can keep the client and continue to make money?

The point is, without clarity on the goal, you might unwittingly work to undermine it. For example, UniCo purchased new welding robots that improved local efficiency, but actually impacted negatively on their goal; to make money.

Once the goal is clear, Alex devises a succinct list of critical measures of progress towards the goal which are used to classify everything in the plant:

Throughput: the rate at which the system generates money though completed sales
Inventory: the money currently invested in assets it intends to sell (unfinished products, work-in-process, etc.)
Operational expenses: money spent to turn inventory into throughput (wages, power, etc.)

These measures are specific to UniCo’s goal of making money through manufacturing but could be abstracted to other applications.

For example in education, throughput might be the number of student who graduate and proceed into successful work placements, inventory might be the number of students currently in the system and operational expenses remain; the cost of employing staff and keeping the lawns.

Dependent events, statistical fluctuations

Jonah highlights that a manufacturing plant represents a sequence (or many sequences) of dependent events. Meaning, some events cannot happen until preceding events have completed. E.g. assembly cannot begin until all required parts have arrived. Each step in the process is also subject to statistical fluctuations; where the duration of each execution of a discrete process will fluctuate. Welding or painting a part, will always vary in time.

This is fairly self-evident, and like me, Alex assumes that the average execution time of each production line station will determine the throughput of the system (i.e. time from customer order to dispatch and receipt). In practice however, the plant is failing to keep up and the issue is compounding over time.

While hiking a group of scouts in single file through a nature trail, Alex becomes frustrated that the group of boys cannot seem to stay together and are running behind time to complete the hike. Alex identifies that the hike is much like his plant:

there is a goal: everyone arriving on time, all together
one boy cannot move forward until the next boy has moved forward
each step of each boy fluctuates in distance and duration
the distance covered by the last hiker is the throughput of the system
the energy expended is the operational expense of the system
the distance between the first and last hiker, is the inventory or work-in-process.

Much like the factory, the work-in-process is in continual growth, with the distance between front and back of the line constantly expanding, increasing risk and the energy required to compensate for the distance.

Alex’s expectation was that the average pace of the boy scouts would determine the arrival time. Instead he discovers, that it’s actually the maximum accumulated, negative deviation from the average that determines the throughput (arrival time) and inventory (distance between front and back) of the group.

Each step that each boy takes will deviate slightly from the average. Every time a deviation is in the negative (slightly slower), that deviation accumulates as the boys behind cannot pass and slow down. Additionally the distance between the hiker in front increases. Compensating for this phenomena, requires significant effort and is often outside the capacity of the boys.

Alex further models the problem by passing matches along a sequence of bowls, based on the fluctuating rolls of a dice. The throughput of the last bowl is significantly lower than anticipated.

This misconception about the capacity of the hikers being based on average pace is shared at the UniCo plant and in my own estimation.

The illusion of local efficiency

What Alex learns (and Jonah later validates) is that the whole process is actually hinged on the bottleneck (or constraint), the slowest boy scout hiker. Alex forces all of the boys to maintain pace with the slowest boy, young Herbie, by placing Herbie at the front of the pack. As a result, the boys behind start operating below capacity which in turn leads to the entire pack walking closely together (a reduction in inventory or work-in-process). The boys are no longer required to jog to catch up the gaps between them and so operational expenditure is reduced.

Now that Herbie is setting the pace for the group, his capacity gets greater scrutiny and the group realize Herbie is carrying several kilos of obscure camping equipment in his ruck sack. The equipment (or load) is distributed amongst the boys with higher capacity and suddenly the entire group is moving quicker (throughput)!

Prior to understanding this metaphor, I might have hinged the success of the group on the fastest kids (highest capacity resources). Ultimately, any efficiency gained in the faster boys (strength, speed, etc.) is a complete and utter waste when considering the goal of arriving on time and together.

Likewise, the robots purchased by UniCo were a waste. Because of their impressive capacity, they only served to increase inventory (by piling up products in front of the next work station) and operational cost (staff working overtime to catch up). The robots did increase the local efficiency of their work centers, but undermined the goal of the company, to make money, by increasing inventory and operational expenditure without increasing throughput of the entire system.

The process of ongoing improvements

The author provides a process to solve these problems, which in the parable, is devised by Alex and his team of colleagues. This is the process Alex used to rectify the hiking boy scouts, and his manufacturing plant woes:

Identify the system’s primary bottleneck/s (Herbie, or the plant’s heat treat machine)
Decide how to exploit the bottleneck (move Herbie to the front or keep the heat treat machine running during lunch breaks)
Subordinate everything else to the above decision (all kids slow down to Herbie’s pace and all other workstations prioritize parts bound for the heat treat machine and remain idle if there are sufficient parts ready)
Elevate the system’s constraints (distribute the contents of Herbie’s rucksack or install additional heat treat machines)
If a bottleneck is broken (i.e. no longer the bottleneck) repeat from step 1, but don’t allow inertia to become a constraint (in this case, market demand became the new bottleneck for Alex’s plant, but inertia caused them to stockpile too many completed products)

Change

Once the manufacturing plant is once again profitable, Alex faces the challenge of responding to changes in the marketplace and planning ahead to prevent any further decline. With further prompting from his mentor Jonah, Alex determines there are three critical challenges for managers in applying his new knowledge:

What needs to change?
What does it need to change to?
How do you execute the change?

These steps are particularly critical when approaching changes to the most difficult of all business resources to manage: people. When trying to change the culture, values and thinking of people, the book highlights two important strategies:

Socratic inquiry: leading people to answers by simply invoking curiosity. Alex did this by sharing Jonah’s questions with his colleagues and working together to find the answers.
The scientific method: create a hypothesis, A/B test the hypothesis (with a control) and share the conclusion. Alex did this by comparing the results of his hypothesis to the results attained in other UniCo plants, or simply with “before and after” results.

Contribution to DevOps

Prior to reading this book and The Phoenix Project, I would have described DevOps as:

Enabling frequent production releases by making developers and sysadmins
play nice together and using some cool new automation tool chains.

I now have a completely different perspective. In fact, the cooperative culture and tools have taken a heavy dive in significance for me. They are simply a means to an end. They were a tailored response to a discrete problem. Truth is, in my own experience in adopting “DevOps”, the CI/CD (Continuous Integration/Continuous Delivery) tools simply became the overpriced, under performing, state-of- the-art, highly “efficient” robots in Alex’s manufacturing plant.

CI/CD tools and DevOps culture were born from a specific problem. That is, that successful implementations of Agile development methodologies have significantly increased the production capacity of development teams, typically without exceeding marketplace demand. Historically, operations teams held the title of “most efficient” as software releases were infrequent. Now the tables have turned and operations has become a bottleneck in businesses achieving their primary goal; make money first, fast and forever.

DevOps addresses this problem, using Goldratt’s principles in the following ways:

Production deployment is identified as the bottleneck in achieving the goal
The bottleneck is “exploited” in that more frequent releases keep operations functioning at full capacity. Operational activities that don’t directly progress the goal are deprioritized.
Work is buffered in front of the bottleneck using “feature toggles” that operations can enable via configuration when ready
Testing and quality assurance are moved before the bottleneck using Continuous Integration and automated testing tools. This elevates the bottleneck by preventing it from processing defective goods (i.e. buggy software)
Batch sizes are reduced, increasing the work of the testing, packaging and release work centers, but increasing flow through the bottleneck (smaller changes, lower risk, easier planning, etc.)
Automation is used to elevate the operations bottleneck. E.g. one touch deployments, infrastructure as code, containers, etc.
As bottlenecks are defeated (i.e. ops can keep up with dev), other bottlenecks (prioritized according to the goal) are identified and attacked such as operational outages, technical debt, infrastructure projects, compliance problems, etc.

Application in IT Operations

Deploying applications into production is actually a very small part of what my organization does. Our focus is more in service desk, process management and infrastructure services.

Essentially, CI/CD is of very little (if any) value to us in achieving our goals. For this reason, often my colleagues express that DevOps is also of very little use to us. Eliyahu’s parable illustrates that the Theory of Constraints, a core component of DevOps thinking, is actually more relevant than ever; it’s just that the application will look different to CI/CD.

Our goal, hypothetically, is to make money. We do this by increasing throughput (projects delivered, requests fulfilled, etc.) while minimizing operational expenses and inventory (incomplete works).

Our bottleneck is not the frequent deployment of new software features. What we need to determine, is where the primary bottleneck is in achieving our goal. Hypothetically, it could be:

Service outages
Change and release management
Skills shortages
Quality assurance
Inter-team politics
Project delivery
Etc.

Improving any one of these in isolation could actually serve to undermine our goal by increasing inventory or operational expenses, without increasing throughput.

The next step, the challenge to you and hopefully the subject of a future article, is the application of Goldratt’s process of ongoing improvement to more diverse IT Operations organizations and overcoming the challenges of organizational change.

dmidecode for Apple OS X

Tue, 20 Oct 2015 00:00:00 +0000

Dmidecode is a long standing, effective tool for reading manufacturer info from the SMBIOS tables present on most modern x86 based systems. It’s been available for many years on most Unix like operating systems and has also been ported to Windows. Until now, it had never made the leap to OS X.

From the dmidecode website:

Dmidecode reports information about your system’s hardware as described in your system BIOS according to the SMBIOS/DMI standard. This information typically includes system manufacturer, model name, serial number, BIOS version, asset tag as well as a lot of other details of varying level of interest and reliability depending on the manufacturer. This will often include usage status for the CPU sockets, expansion slots (e.g. AGP, PCI, ISA) and memory module slots, and the list of I/O ports (e.g. serial, parallel, USB).

Installation on OS X

The easiest way to install dmidecode is with the homebrew package manager:

$ brew install cavaliercoder/dmidecode/dmidecode

To build from source, sources are available from the project page on GitHub.

Once cloned or extracted from the release tarball, you may build and install with:

$ make && make install

The above steps install the dmidecode binary into /usr/local/sbin. If this is not in your $PATH, you’ll need to add it or call the full file path:

$ /usr/local/sbin/dmidecode

Getting help

For all issues related to dmidecode itself, please see the dmidecode website.

This port is thoroughly tested on x86_64 iMacs and MacBook Pros running 10.9+. All API calls have documented compatability since OS X 10.0 but without the opportunity to test on older machines, I’ll be relying on the community for feedback.

If you encounter any issues, please raise an issue on the GitHub page and include your Apple device details and OS X version. Please first check for existing issues which may match your problem.

The portability challenge

Dmidecode ports very nicely onto any Unix-like platform that exposes raw system memory (specifically the range where SMBIOS lives) as a file handle (such as /dev/mem). Unfortunately, a few years back, Apple disabled its memory file handle by default. Now the only way to access SMBIOS memory directly (on an out-of-box OS X machine) is to use Apple’s proprietary IOService API.

This port of dmidecode aims to minimize deviation from the upstream codebase to ensure that ongoing maintenance is as simple as possible. The challenge has been achieving this without butchering the code with a bunch of if apple... else... blocks but cleanly navigating around code that assumes SMBIOS data is accessible from a file path.

Sample output

Here’s some sample dmidecode output from my MacBook Pro 2010 running OS X 10.9.5; modified for privacy and brevity:

# dmidecode 3.0
Getting SMBIOS data from Apple SMBIOS service.
SMBIOS 2.4 present.
61 structures occupying 2888 bytes.

...

Handle 0x001C, DMI type 0, 24 bytes
BIOS Information
	Vendor: Apple Inc.
	Version: MBP61.88Z.0057.B0F.1112091028
	Release Date: 12/09/11
	ROM Size: 4096 kB
	Characteristics:
		PCI is supported
		BIOS is upgradeable
		BIOS shadowing is allowed
		Boot from CD is supported
		Selectable boot is supported
		ACPI is supported
		IEEE 1394 boot is supported
		Smart battery is supported
		Function key-initiated network boot is supported
	BIOS Revision: 0.1

Handle 0x001D, DMI type 1, 27 bytes
System Information
	Manufacturer: Apple Inc.
	Product Name: MacBookPro6,2
	Version: 1.0
	Serial Number: W80223VBDAW
	UUID: 677C7349-0762-4254-A9A5-6646B1BD4083
	Wake-up Type: Power Switch
	SKU Number: System SKU#
	Family: MacBook Pro

Handle 0x001E, DMI type 2, 16 bytes
Base Board Information
	Manufacturer: Apple Inc.
	Product Name: Mac-F22586C8
	Version: MacBookPro6,2
	Serial Number: W8022732FVUPA
	Asset Tag: Base Board Asset Tag#
	Features:
		Board is a hosting board
		Board is replaceable
	Location In Chassis: Part Component
	Chassis Handle: 0x001F
	Type: Motherboard
	Contained Object Handles: 0

...