Thorsten Ball

Home Posts Talks Projects Contact My Book

Unicorn Unix Magic Tricks

20 Nov 2014

This post is based on the talk of the same name I gave at the Arrrrcamp conference in Ghent, Belgium on October 2nd, 2014. You can find the slides here and the video recording here.

Unicorn is a webserver written in Ruby for Rails and Rack applications. When I first used it I was amazed. This is magic, I thought. It had to be. Why?

Well, first of all: the master-worker architecture. Unicorn uses one master process to manage a lot of worker processes. When you tell Unicorn to use 16 worker processes it does so, just like that. And now you’re looking at 17 processes when you run ps aux | grep unicorn — each with a different name, showing whether its the master process or one of the worker processes, which even have their own number in their process names.

$ pstree | grep unicorn
 \-+= 27185 mrnugget unicorn master -c simple_unicorn_config.rb -l0.0.0.0:8080
   |--- 27210 mrnugget unicorn worker[0] -c simple_unicorn_config.rb -l0.0.0.0:8080
   |--- 27211 mrnugget unicorn worker[1] -c simple_unicorn_config.rb -l0.0.0.0:8080
   |--- 27212 mrnugget unicorn worker[2] -c simple_unicorn_config.rb -l0.0.0.0:8080
   |--- 27213 mrnugget unicorn worker[3] -c simple_unicorn_config.rb -l0.0.0.0:8080
   |--- 27214 mrnugget unicorn worker[4] -c simple_unicorn_config.rb -l0.0.0.0:8080
   |--- 27215 mrnugget unicorn worker[5] -c simple_unicorn_config.rb -l0.0.0.0:8080
   |--- 27216 mrnugget unicorn worker[6] -c simple_unicorn_config.rb -l0.0.0.0:8080
   |--- 27217 mrnugget unicorn worker[7] -c simple_unicorn_config.rb -l0.0.0.0:8080
   |--- 27218 mrnugget unicorn worker[8] -c simple_unicorn_config.rb -l0.0.0.0:8080
   |--- 27219 mrnugget unicorn worker[9] -c simple_unicorn_config.rb -l0.0.0.0:8080
   |--- 27220 mrnugget unicorn worker[10] -c simple_unicorn_config.rb -l0.0.0.0:8080
   |--- 27221 mrnugget unicorn worker[11] -c simple_unicorn_config.rb -l0.0.0.0:8080
   |--- 27222 mrnugget unicorn worker[12] -c simple_unicorn_config.rb -l0.0.0.0:8080
   |--- 27223 mrnugget unicorn worker[13] -c simple_unicorn_config.rb -l0.0.0.0:8080
   |--- 27224 mrnugget unicorn worker[14] -c simple_unicorn_config.rb -l0.0.0.0:8080
   \--- 27225 mrnugget unicorn worker[15] -c simple_unicorn_config.rb -l0.0.0.0:8080

How would one build something like this? I had no idea.

And then there’s a feature called “hot reload”, which means that you can tell Unicorn, while it’s running, to spin up a new version of your application. As soon as you do, Unicorn starts a new master process, which is going to serve the new version of your application. All the while the old master process is still running, responding to requests with your old application. Of course, the old master now has “old” in its name. Now, as soon as the new master process is fully booted up, you can send a QUIT signal to the old master process, which will in turn shut down and let the new one take over. And just like that you’ve switched to a new version of your application — without any downtime at all.

Oh, and Unicorn uses a lot more than the QUIT signal! There are tons of signals you can send to it: TTIN to increase the number of workers, TTOU to decrease it, USR1 to rotate the log files, USR2 to perform hot reloading, HUP to re-evaluate the configuration file. I didn’t know half of these signal names and there were even more in Unicorn’s own SIGNALS file.

And then there’s “preloading”: a feature of Unicorn that allows you to spin up new worker processes in less than a second, a fraction of the time it takes to boot up my Rails application. Somehow Unicorn is able to preload my application in memory and make use of that when creating new worker processes. And I had no idea how that works! Not a clue! And as if that wasn’t enough I discovered that Unicorn even has a file called PHILOSOPHY in its repository. Who else has that?! I was sure that there was some black magic going on. Because: how could Unicorn work like it does without magic?

Unix

After my first encounter with Unicorn I learned quite a bit about Unix systems and after a while I came back to Unicorn — still in amazement. But this time I read through the source code and it turns out, that, well, the secret ingredient to Unicorn is not magic but plain, old Unix.

Now, most people know Unix from a “user’s perspective”: the command line, shells, pipes, redirection, the kill command, scripting, text files and so on. But there’s this whole other side of Unix, too, which we could call the “developer’s perspective” now. From this side of Unix you can see signal handling, inter-process communication, usage of pipes without the |-character, system calls and whole lot more.

In what follows we’re going to have a look at Unicorn. We’ll take it apart and see that it’s just using some basic Unix tricks, the ones you can use as a developer, to do its work. The way we’re going to do that is by going through some of these Unix tricks, basic building blocks of every Unix system, and see how they work and how Unicorn uses them.

At the end we’ll go back to the “magic” of the beginning: hot reload, preloading, master-worker architecture. And we will see how these features work and how they are just Unix and not magic.

So let’s get started.

fork(2)

fork is how processes are created. Every process after the first one (with PID 1) was created with fork. So what is it, what is fork?

fork is a system call. Most of the time we can recognize system calls by the 2 behind their name (e.g. fork(2)) which means that we can find documentation about them in section 2 of the Unix manual, nowadays known as “man pages”. So in order to see the documentation for fork(2) you can run man 2 fork on your command line.

But what’s a system call? A way to communicate with the kernel of our operating system. System calls are the API of the kernel, if you will. We tell the kernel to do something for our us with system calls: reading, writing, allocating memory, networking, device management.

And fork is the system call that tells the kernel to create a new process. When one process asks the kernel for a new process with fork(2) the kernel splits the process making the call into two. That’s probably where the name comes from: calling fork(2) is a “fork in the road” in the lifetime of a process. As soon as the kernel returns control to the process after handling the system call there now is a parent process and a child process. A parent can have a lot of child processes, but a child process only one parent process.

And both processes, parent and child, are pretty much the same, right after the creation of the child. That’s because child processes in a Unix system inherit a lot of stuff from their parent processes: the data (the code it’s executing), the stack, the heap, the user id, the working directory, open file descriptors, the connected terminal and a lot more. This can be a burden (which is why copy-on-write is a thing) but also has some neat advantages — as we’ll see later.

So how do we use fork? Since (deep down) making a system call involves putting parameters and the unique identifier of the call in CPU registers (which ones may change depending on the architecture we’re working with) and firing a software interrupt, most programming languages provide wrappers that do all the work and allow us to not worry about which system call is identified by which number.

Ruby is no exception here and allows us to use fork(2) with a method called, well, fork:

# fork.rb

child_pid = fork do
  puts "[child] child_pid: #{child_pid}"
  puts "[child] Process ID: #{Process.pid}"
  puts "[child] Parent Process ID: #{Process.ppid}"
end

Process.wait(child_pid)

puts "[parent] child_pid: #{child_pid}"
puts "[parent] Process ID: #{Process.pid}"

What we’re doing here is calling fork in Ruby and pass it a block. This will create a new process, a child process, and run everything inside the block in the new process and then exit. In the parent process we call Process.wait and pass it the return value of fork, which is the ID of the child process. We also need to wait for child processes to exit because otherwise they’d turn into zombie processes. Yep, that’s a valid Unix rule right there: parent processes need to wait for their children to die so they don’t turn into zombies.

When we run this we’ll get this:

$ ruby fork.rb
[child] child_pid:
[child] Process ID: 29715
[child] Parent Process ID: 29695
[parent] child_pid: 29715
[parent] Process ID: 29695

As we can see, the child process has a new process ID and its parent process ID matches the process ID printed in the parent process. And most interestingly child_pid is nil inside the child process but contains a value in the parent process. This is how we can check whether we are in the parent process or the child process. Since the child inherits the data from the parent process, both processes are running the same code right after fork and we can decide which process does what depending on the return value of fork.

If we put a sleep somewhere inside the block, run it again and use a tool like ps or pstree we’d see something like this:

$ pstree | grep fork
 |   \-+= 29695 mrnugget ruby fork.rb
 |     \--- 29715 mrnugget ruby fork.rb

Two processes, one parent and one child, with different process IDs. Just by calling fork. That’s not too hard right? And it’s certainly not magic. So how does Unicorn use fork?

Unicorn and fork(2)

When Unicorn boots up it calls the spawn_missing_workers method, which contains this piece of code:

worker_nr = -1
until (worker_nr += 1) == @worker_processes
  WORKERS.value?(worker_nr) and next
  worker = Worker.new(worker_nr)
  before_fork.call(self, worker)
  if pid = fork
    WORKERS[pid] = worker
    worker.atfork_parent
  else
    after_fork_internal
    worker_loop(worker)
    exit
  end
end

So, what happens here? Unicorn calls this method with @worker_processes set to the number of workers we told it to boot up. It then goes into a loop and calls fork that many times. But instead of passing a block to fork, Unicorn instead checks the return value of fork so see if its now executing in the parent and in the child process. Remember: a forked process inherits the data of the parent process! A child process executes the same code as the parent, and we have to check for that in order to have the child do something else.

Passing a block to fork does the same thing under the hood, but explicitly checking the return-value of fork is quite a common idiom in many Unix programs, since the C API doesn’t allow passing blocks around.

If fork returned in the parent process, Unicorn saves the newly created worker object with PID of the newly created child process in the WORKERS hash constant, calls a callback and starts the loop again.

In the child process another callback is called and then the child goes into its main loop, the worker_loop. If the worker loop should somehow return the child process exits and is done.

And boom! We’ve now got 16 worker processes humming along, waiting for work in their worker_loop, just by going into a loop, doing some cleanup and calling fork 16 times.

That’s not too hard, is it? So let’s go from fork to another basic Unix feature…

Pipes!

My guess is that most people even vaguely familiar with Unix systems know about pipes and have probably done something like this at one point or another in their lives:

$ grep ‘wat’ journal.txt | wc -l
84

Pipes are amazing. Pipes are a really simple abstraction that allows us to take the output of one program and pass it as input to another program. Everybody loves pipes and I personally think the pipe character is one of the most best features Unix shells have to offer.

But did you know that you can use pipes outside of the shell?

pipe(2)

pipe(2) is a system call with which we can ask the kernel to create a pipe for us. This is exactly what shells are using. And we can use it too, without a shell!

Remember the saying that under Unix “everything is a file”? Well, pipes are files too. One pipe is nothing more than two file descriptors. A file descriptor is a number that points to an entry in the file table maintained by the kernel for each running process. In the case of pipes the two file table entries do not point to files on a disk, but rather to a memory buffer to which you can write and from which you can read with both ends of the pipe.

One of the file descriptors returned by pipe(2) is the read-end and the other one is the write-end. That’s because pipes are half duplex – the data only flows in one direction.

Outside of the shell pipes are heavily used for inter-process communication. One process writes to one end, and another process reads from the other end. How? Remember that a child process inherits a lot of stuff from its parent process? That includes file descriptors! And since pipes are just file descriptors, child processes inherit them. If we open a pipe with pipe(2) in a parent process and then call fork(2), both the parent and the child process have access to the same file descriptors of the pipe.

# pipe.rb

read_end, write_end = IO.pipe

fork do
  read_end.close

  write_end.write('Hello from your child!')
  write_end.close
end

write_end.close

Process.wait

message = read_end.read
read_end.close

puts "Received from child: '#{message}'"

In Ruby we can use IO.pipe, which is a wrapper around the pipe(2) system call, just like fork is a wrapper around fork(2), to create a pipe.

And in this example we create a pipe with IO.pipe and then create the child process with fork. Since just after the call to fork both processes have both pipe file descriptors we need to close the end of the pipe we’re not going to need. In the child process that’s the read-end and in the parent it’s the write-end.

We then write something to the pipe in the child, close the write-end and exit. The parent closes the write-end, waits for the child to exit and then reads the message the child wrote to the pipe. To clean up it closes the read-end. If we run this we get exactly what we expected:

$ ruby pipe.rb
Received from child: 'Hello from your child!'

That’s pretty amazing, isn’t it? Just a few lines of code and we created two processes that talk to each other! By the way, this is the exact same concept a shell uses to make the pipe-character work. It creates a pipe, it forks (once for each process on one side of the pipe) then uses another system call (dup2) to turn the write-end of the pipe into STDOUT and the read-end into STDIN respectively and then executes different programs which are now connected through a pipe.

So how does Unicorn make use of pipes?

Unicorn and pipe(2)

Unicorn uses pipes a lot.

First of all, there is a pipe between each worker process and the master process, with which they communicate. The master process writes command to the pipe (something like QUIT) and the child process then reads the commands and acts upon them. Communication between the master and its worker processes through pipes.

Then there’s another pipe the master process only uses internally and not for IPC, but for signal handling. It’s called the “self-pipe” and we’ll have a closer look at that one later.

And then there’s the ready_pipe Unicorn uses, which is actually quite an amazing trick. See, if you want to daemonize a process under Unix, you need to call fork(2) two times (and do some other things) so the process is completely detached from the controlling terminal and the shell thinks is the process is done and gives you a new prompt.

What Unicorn does when you tell it to run as a daemon is to create a pipe, called the ready_pipe. It then calls fork(2) two times, creating a grand child process. The grand child process inherited the pipe, of course, and as soon as its fully booted up and everything looks good, it writes to this pipe that it’s okay for the grand parent to quit. The grand parent, which waited for a message from the grand-child, reads this and then exits.

This allows Unicorn to wait for the grand child to boot up while still having a controlling terminal to which it can write error messages should something go wrong between the first call to fork(2) and booting up the HTTP server in the grand child. Only if the everything worked the grand child turns into a real daemon process. Process synchronization through pipes.

That does come pretty close to being magic, yep, but this is just a really clever use of fork(2) and pipe(2).

sockets & select(2)

At the heart of everything that has to do with networking under Unix are sockets. You want to read a website? You need to open a socket first. Send something to the logserver? Open a socket. Wait for incoming connections? Open a socket. Sockets are, simply put, endpoints between computers (or processes!) talking to each other.

There are a ton of different sockets: TCP sockets, UDP sockets, SCTP sockets, Unix domain sockets, raw sockets, datagram sockets, and so on. But there is one thing they all have in common: they are files. Yes, “everything is file” and that includes sockets. Just like a pipe, a socket is a file descriptor, from which you can read and write to just like with a file. The sockets API for reading and writing is deep down the same as the file API.

So, let’s say we are writing a server. How do we use sockets for that? The basic lifecycle of a server socket looks like this:

First we ask the kernel for a socket with the socket(2) system call. We specify the family of the socket (IPv4, IPv6, local), the type (stream, datagram) and the protocol (TCP, UDP, …). The kernel then returns a file descriptor, a number, which represents our socket.

Then we need to call bind(2), to bind our socket a network address and a port. After that we need to tell the kernel that our socket is a server socket, that will accept new connections, by calling listen(2). So now the kernel forwards incoming connections to us. (This is the main difference between the lifecycles of a server and a client socket).

Now that our socket is a real server socket and waiting for new incoming connections we can call accept(2), which accepts connections and returns a new socket. This new socket represents the connection. We can read from it and write to it.

But here’s the thing: accept(2) is a blocking call. It only returns if the kernel has a new connection for us. A server that doesn’t have too many incoming connections will be blocking for a long time on accept(2). This makes it really difficult to work with multiple sockets. How are you going to accept a connection on one socket if you’re still blocking on another socket that nobody wants to connect to?

This is where select(2) comes into play.

select(2) is a pretty old and famous (maybe infamous) Unix system call for working with file descriptors. It allows us to do multiplexing: we can monitor several file descriptors with select(2) and let the kernel notify us as soon as one of them has changed its state. And since sockets are file descriptors too, we can use select(2) to work with multiple sockets. Like this:

sock1 = Socket.new(:INET, :STREAM)
addr1 = Socket.pack_sockaddr_in(8888, '0.0.0.0')
sock1.bind(addr1)
sock1.listen(10)

sock2 = Socket.new(:INET, :STREAM)
addr2 = Socket.pack_sockaddr_in(9999, '0.0.0.0')
sock2.bind(addr2)
sock2.listen(10)

5.times do
  fork do
    loop do
      readable, _, _ = IO.select([sock1, sock2])

      connection, _ = readable.first.accept
      puts "[#{Process.pid}] #{connection.read}"
      connection.close
    end
  end
end

Process.wait

That’s a 23-line TCP server, listening on two ports, with 5 worker processes accepting connections. Besides missing some minor things like HTTP request parsing, HTTP response writing and error handling it’s pretty much ready to ship.

No, but seriously, this actually does a lot of stuff in just a few lines with the help of system calls.

We create two sockets with Socket.new, which somewhere deep down in Ruby calls socket(2). Then we bind the sockets to two different ports, 8888 and 9999 respectively, on the local interface. Afterwards we call listen(2) (hidden by the #listen method) and tell the kernel to queue up 10 connections at maximum for us to handle.

With our sockets ready to go we call fork 5 times, which in turn creates 5 child processes that all run the code in the block. So every child calls IO.select (which is the wrapper around select(2)) with the two sockets as argument. IO.select is going to block and only return if one of the two sockets is readable (on a listening socket that means that there are new connections). And this is exactly why we use select(2) here: with accept(2) we would block on one socket and miss out if the other socket had a new connection.

IO.select returns the readable sockets in an array. We take the first one and call accept(2) on it, which is now going to return immediately. Then we just read from the connection, close the connection socket and start our worker loop again.

If we run this and send some messages to our server with netcat like this:

$ echo 'foobar1' | nc localhost 9999
$ echo 'foobar2' | nc localhost 9999
$ echo 'foobar3' | nc localhost 8888
$ echo 'foobar4' | nc localhost 8888
$ echo 'foobar5' | nc localhost 9999

Then we can see our server accepting the connections and reading from them:

$ ruby tcp_sockets_example.rb
[31605] foobar1
[31607] foobar2
[31605] foobar3
[31607] foobar4
[31609] foobar5

Each connection handled by a different child process. Load balancing done by the kernel for us, thanks to select(2).

Unicorn, sockets and select

Before master process calls fork to create the worker processes, it calls socket, bind and listen to create one or more listening sockets (yes, you can configure Unicorn to listen on multiple ports!). It also creates the pipes that will be used to communicate with the worker processes.

After forking, the workers, of course, have inherited both the pipe and the listening sockets. Because, after all, sockets and pipes are file descriptors.

The workers then call select(2) as part of their worker_loop with both the pipe and the sockets as arguments. Now, whenever a connection comes in, one of the workers’ call to select(2) returns and this worker handles the connection by reading the request and passing it to the Rack/Rails application.

And here’s the thing: since the workers call select(2) not only with the sockets, but also with the master-to-worker pipe, they’ll never miss a message from the master while waiting for a new connection. And if there is a new connection, they handle it, close it and then read the message from the master process.

That’s a really neat way to do load balancing through the kernel and to guarantee that messages to workers are not lost or delayed too long while the worker process is doing its work.

Signals

Let’s talk about signals. Signals are another way to do IPC under Unix. We can send signals to processes and we can receive them.

$ kill -9 8433

This sends the signal 9, which is the KILL signal, to process 8433. That’s pretty well-known and a lot of people have used this before (probably with sweat running down their face). But did you know that pressing Ctrl-C and Ctrl-Z in your shell sends signals too?

So what are signals? Most often they are described as software interrupts. If we send a signal to the process, the kernel delivers it for us and makes the process jump to the code that deals with receiving this signal, effectively interrupting the current code flow of the process. Signals are asynchronous — we don’t have to block somewhere to send or receive a signal. And there are a lot of them: the current Linux kernel for example supports around 30 different signals.

Sending signals is pretty good, and I’d bet we’ve all done it a bunch of times, but what’s really cool is this: we can tell the kernel how we want our process to react to certain signals. That’s called “signal handling”.

We have a few options when it comes to signal handling. We can ignore signals: we can tell the kernel we don’t care about a signal and when the kernel delivers an ignored signal to our process it doesn’t jump to any specific code, but instead does nothing. Ignoring signals has one limitation though: we can’t ignore SIGKILL and SIGSTOP, since there has to be a way for an administrator to kill and stop a process, no matter what the developer of that process wants it to do.

The second option is to catch a signal, effectively defining a signal handler. If ignoring a signals means “Nope, kernel, don’t care about QUIT.” then defining a signal action is telling the kernel “Hey, if I receive this signal, please execute this piece of code here”. For example: a lot of Unix programs do some clean-up work (remove temp files, write to a log, kill child processes) when receiving SIGQUIT. That’s done by catching the signal and defining an appropriate signal handler, that does the clean-up work. Catching signals has the limitations that ignoring signals has: we can’t catch SIGKILL and SIGSTOP.

We can also let the defaults apply. Each signal has a default action associated with it. E.g. the default action for SIGQUIT is to terminate the process and make a core dump. We can let that one leave it as it is, or redefine the signal action by catching it. See man 3 signal on OS X or man 7 signal on Linux for a list of the default actions associated with each signal.

So, how do we catch a signal? In Ruby it’s pretty simple:

# signals.rb

trap(:SIGUSR1) do
  puts "SIGUSR1 received"
end

trap(:SIGQUIT) do
  puts "SIGQUIT received"
end

trap(:SIGKILL) do
  puts "You won't see this"
end

puts "My PID is #{Process.pid}. Send me some signals!"

sleep 100

We use trap to catch a signal and pass it a block to define a signal action that will be executed as soon as our process receives the signal. In this example, we try to redefine the signal handler for SIGUSR1, SIGQUIT and SIGKILL. The sleep statement gives us time to send the signals to our process.

If we run this and then send signals to our process with the kill command like this:

$ kill -USR1 31950
$ kill -QUIT 31950
$ kill -KILL 31950

Then our process will output the following:

$ ruby signals.rb
My PID is 31950. Send me some signals!
SIGUSR1 received
SIGQUIT received
zsh: killed     ruby signals.rb

As we can see, the kernel delivered all of the signals to our process. On receiving SIGUSR1 and SIGQUIT it executed the signal handlers, but, as I said before, catching SIGKILL proved useless and the kernel killed the process.

You can probably imagine what we can do with signal handlers. One of the most common things to do with custom signal handlers, for example, is to catch SIGQUIT to do some clean-up work before exiting. But there are a lot more signals and defining appropriate signal handlers can distinguish well-behaving processes from rude ones. Example: if a child process dies the kernel notifies the parent process by sending a SIGCHLD. The default action is to ignore the signal and do nothing, but a well-behaving application would probably wait for the child, clean up after him and write something to a log file.

Unicorn and signals

Unicorn sets up a lot of different signal handlers in the master process, before it calls fork and spawns the worker processes. These signal handlers do a lot of things. Here are a few examples:

These signal handlers are like a separate API through which you tell the master and worker processes what to do. And it’s pretty reliable too, considering the fact that signals are essentially asynchronous events and can be sent multiple times. This just screams for race-conditions and locks. So how does Unicorn do it?

Unicorn uses a self-pipe to manage its signal actions. The pipe the master process sets up is this self-pipe, which it will only use internally and not to talk to other processes. It also sets up a queue data structure. After that come the signal handlers. Unicorn catches a lot of signals, as we saw, but each signal handler doesn’t do much. It only pushes the signal’s name into the queue and sends one byte through the self-pipe.

After setting up the signal handlers, spawning worker processes, and so on, the master process goes into its main loop, in which it checks upon the workers regularly and sleeps in between. But it doesn’t just sleep, no, the master process actually goes to sleep by calling select(2) on the self-pipe, with a timeout as argument. This way it can go to sleep but will be woken up as soon as a signal arrived, since the signal handler just send a byte through the pipe, turns it into a readable pipe (from the master’s perspective) and select(2) now returns. After waking up, the master just has to pop off a signal from the queue it set up in the beginning and handle the signals one after another. This is of tremendous value if you consider again that signals are asynchronous and you’ll never know what you’re currently executing when a signal arrives, and that they can be sent multiple times — even if you’re currently executing your signal handler code. Using a queue and a self-pipe in this combination makes handling signals a lot saner and easier.

Worker processes, on the other hand, inherit the master’s signal handlers – again: child processes inherit a lot from their parents. But instead of leaving them as they are, the workers redefine (most of) the signal handlers to be no-ops. They get their signals through the pipe which connects them to the master process. If the master process, for example, receives SIGQUIT it writes the name of the signal to each pipe connected to a worker process to gracefully shut them down. The worker processes call select(2) on this master-worker pipe and the listening sockets, which means that as soon as they finish their work (or don’t have anything to do) they will read the signal name from the pipe and act upon it. This “signal delivery from master to worker via pipe”-mechanism avoids the many problems that can occur if a worker process should receive a signal while currently working of a request.

Magic?

By now we have looked at fork(2) and how easy it is to spawn a new process. We saw that we can use pipes pretty easily outside a shell and without any use of the pipe character by calling pipe(2) and just working with the two file descriptors as if they were files. We also created sockets, worked with select(2), looked at a pre-forking TCP server in 23 lines of Ruby and had the kernel of our operating system do our load balancing for us. Then we saw that Unicorn has its own API composed of signals and that it’s not that hard to work with signals.

These were just some basic Unix concepts. Trivial on their own, powerful when combined.

So, let’s have a closer look at these features of Unicorn that amazed me so much, that I was sure were created by some wizards with long robes and tall hats, in a basement far, far away, on old rusty PDP-11s.

Let’s see how this “magic” is just Unix.

Preloading

If we put preload = true in the configuration file, Unicorn will “preload” our Rack/Rails application in the master process to spare the worker process from doing it themselves. As soon as the application is preloaded, spawning off a new worker process is really, really fast, since the workers don’t have to load it anymore.

The question is: how does this work exactly? Let me explain.

Right after Unicorn has evaluated command line options, it builds a lambda called app. This lambda contains the instructions needed to load our Rack/Rails application into memory. It loads the config.ru file (or uses default settings) and then creates a Rack application with Rack::Builder, on which it calls #to_app.

So what should come out of the lambda is a Rack application in which we just need to call #call to pass it a request and get a response. But since lambdas are evaluated only as soon as they are called, this doesn’t happen when the lambda is defined.

Unicorn passes this app lambda on to the Unicorn::HttpServer, which eventually calls fork(2) to spawn the worker processes. But before it creates a new process, the HttpServer checks if we told Unicorn to use preloading. If we did, only then it calls the lambda. If we didn’t, the workers would each call the lambda after the call to fork(2).

Calling the lambda, which hasn’t been called before, now loads our application into memory. Files are being read, objects are created, connections established – everything is somehow getting stored in memory.

And here comes the real trick: since the master loaded the application into memory, which can take some time if we’re working with a large Rails application, the worker processes inherit it. Yep, the worker processes inherit our application. How neat is that? Since workers are created with fork(2) they already have the whole application in memory as soon as they are created. Preloading is just deciding if the Unicorn calls a lambda before or after the call to fork(2). And if Unicorn called it before, creating new worker processes is really fast, since they are basically ready to go right after creation, except for some callbacks and setup work.

With copy-on-write, which works in the Ruby VM since 2.x, this is even faster. The reason is that “inheriting” involves copying from the parent’s to the child’s memory address space. It’s probably not as slow as you imagine, but with copy-on-write only the memory regions which the child process wants to modify are copied.

And the best part of it is this: the kernel is doing all the work for us. The kernel answers the call to fork(2) and the kernel copies the memory. We just need to decide when to create our objects: before or after the call to fork(2).

This comes in really handy when we now look at another great feature of Unicorn.

Scaling workers with signals

Unicorn allows us to increase and decrease the number of its worker processes by sending two signals to the master process:

$ kill -TTIN 93821
$ kill -TTOU 93821

These two lines add and then remove a new worker process. The signals used, SIGTTIN and SIGTTOU, are normally sent by our terminal driver to notify a process running in the background when it’s trying to read from (SIGTTIN) or write to (SIGTTOU) the controlling terminal. Since Unicorn doesn’t allow not using a logfile when running as a daemon, this shouldn’t be an issue, which means that Unicorn is free to redefine the signal actions (the default for both signals is to stop the process).

It does so by defining signal handlers for SIGTTIN and SIGTTOU that, as we saw, only add the name of the signal to the signal queue and write a byte to the self-pipe to wake up the master process.

The master process, as soon as it wakes up from its main-loop sleep, sees the signals and increases or decreases the internal variable worker_processes, which is just an integer. And right before it goes back to sleep, it calls #maintain_worker_count, which either spawns a new worker or writes SIGQUIT to the pipe connected to the now superfluous worker process to gracefully shut it down.

So let’s say we send SIGTTIN to Unicorn to increase the number of workers. What will happen is that the master wakes up (triggered by the write to the self-pipe), increases worker_processes and calls #maintain_worker_count, which in turn will call another method called #spawn_missing_workers. Yes, that’s right. We looked at this method before, its the same one that’s used to spawn the worker processes when booting up. In its entirety it looks like this:

def spawn_missing_workers
  worker_nr = -1
  until (worker_nr += 1) == @worker_processes
    WORKERS.value?(worker_nr) and next
    worker = Worker.new(worker_nr)
    before_fork.call(self, worker)
    if pid = fork
      WORKERS[pid] = worker
      worker.atfork_parent
    else
      after_fork_internal
      worker_loop(worker)
      exit
    end
  end
  rescue => e
    @logger.error(e) rescue nil
    exit!
end

Again, this is just a loop that calls fork(2) N times. Now that N is increased by one, a new worker process will be created. The other calls to fork are skipped by checking whether WORKERS already contains an instance of Worker with the same worker_nr.

Take note of worker_nr here, it is important. All worker processes have a worker_nr by which they are easily identified in the row of spawned processes.

If we now send SIGTTOU to the master process, the following is going to happen. First of all, the master is woken up by a fresh byte on the self-pipe. Instead of increasing worker_processes now, it decreases it. And again, it calls #maintain_worker_count, which doesn’t jump straight to #spawn_missing_workers. Since no worker process is missing, #maintain_worker_count now takes care of reducing the number of workers:

def maintain_worker_count
  (off = WORKERS.size - worker_processes) == 0 and return
  off < 0 and return spawn_missing_workers
  WORKERS.each_value { |w| w.nr >= worker_processes and w.soft_kill(:QUIT) }
end

It may not be idiomatic Ruby, but these 3 lines are still fairly easy to understand. The first line generates the difference between the number of currently running worker processes and returns if it’s zero. If the difference is negative, a new worker will be spawned (which is where the path of SIGTTIN ends in this method). But since the difference is positive after decreasing worker_processes, the master process now takes the workers with a worker_nr that’s too high and calls soft_kill(:QUIT) on the worker instance.

This in turn sends the signal name through the pipe to the corresponding worker process, which will catch that signal through select(2) and gracefully shut down.

After this, the master process calls Process.waitpid (which in turn calls waitpid(2)), which returns the PID of dead children (and doesn’t leave them hanging as zombies). The worker process with this PID now just needs to be removed from the WORKERS hash and Unicorn is ready to go again.

All of this is pretty simple: fork(2) in a loop, pipes, signal handlers and keeping track of numbers. Again: it’s the combination of that makes these Unix idioms so powerful.

The same can be said for my favorite Unicorn feature.

Hot Reload

This fantastic feature has many names: hot reload, zero downtime deployment, hot swapping and hot deployment. It allows us to deploy a new version of our application, while the old one is still running.

With Unicorn “hot reload” means, that we can spin up a new master process, with new worker processes serving a new version of our application, while the old master process is still running and still handling requests with the old version.

It’s all triggered by sending a simple SIGUSR2 to the master process. But how?

Let’s take a step back and say that our Unicorn master and worker processes are just humming along. The master process is sleeping, waking up, checking up on the workers and going back to sleep. The worker processes are handling requests without a care in the world. Suddenly a SIGUSR2 is sent to the master process.

Again, the signal handler catches the signal, pushes the signal onto the signal queue, writes a byte to the self-pipe and returns. The master wakes up from its main-loop-slumber and sees that it received SIGUSR2. Straight away it calls the #reexec method. It’s a fairly long method and you don’t have to read through it now. But most of “hot reload” is contained in it, so let’s walk through it.

The first thing the method does it to check if the master process is already reexecuting (reexecuting means that a new master process is started by an old one). If it is, it returns and its job is done. But if not, it writes the current PID to /path/to/pidfile.pid.oldbin. .oldbin stands for “old binary”. With the PID saved to a file, the master process now calls fork(2), saves the returned PID of the newly created child process (to later check if it’s already reexecuting…) and returns. The old master process adds “(old)” to its process name (by changing $0 in Ruby) and is now done with #reexec. But since a process created with fork(2) is executing exactly the same code, the new child process goes ahead with #reexec.

Right after the call to fork(2) the child writes the numbers of the sockets it’s listening on (remember: sockets are files, files are represented as file descriptors, which are just numbers) to an environment variable called UNICORN_FD as one string, in which the numbers are separated by commas. (Yes, it keeps track of listening sockets by writing to an environment variable. Take a deep breath. It’ll make sense in a second.)

Afterwards it modifies the listening sockets so they stay open by setting the FD_CLOEXEC flag on them to false.

It then closes all the other file descriptors it doesn’t need (e.g.: sockets and files opened by the Rack/Rails application).

With all preparations and cleaning done, the child process now calls execve(2).

The execve(2) system call turns the calling process into a completely different program. Which program it’s turned into is determined by the arguments passed to execve(2): the path of the program, the arguments and environment variables. This is not a new process we’re talking about: the new program has the same process ID, but its complete heap, stack, text and data segments are replaced by the kernel.

This is how we can spawn new programs on a Unix system and what every Unix shell does when we try to launch Vim: it calls fork(2) to create a child process and then it calls execve(2) with the path to the Vim executable. Without the call to execve(2) we’d end up with a lot of copies of the original shell process when trying to start programs.

That’s also why Unicorn needs to set the FD_CLOEXEC flag to false on the sockets before it calls execve(2). Otherwise the sockets would get closed, when the of the process is being replaced.

Unicorn calls execve(2) with the original command line arguments it was started with (it keeps track of them), in effect spawning a fresh Unicorn master process that’s going to serve a new version of our application. Except that it’s not completely fresh: the environment variables the old master process set (UNICORN_FD) are still accessible by the new master process.

So the new master process boots up and loads the new application code into memory (preloading!). But before it creates worker processes with fork(2), it checks the UNICORN_FD environment variable. And it finds the numbers of our listening sockets! And since file descriptors are just numbers, it can work with them. It turns them into Ruby IO objects by calling IO.new with each number as an argument and has thereby recovered its listening sockets.

And now it calls fork(2) and creates worker processes which inherit these listening sockets again and can start their select(2) and accept(2) dance again, now handling requests with the new version of our application.

There is no “address already in use” error bubbling up. The new master process inherited these sockets, they are already bound to an address and transformed into listening sockets by the old master process. The new master process and its workers can work with them in the same way the worker processes of the old master process do.

Now there are two sets of master and worker processes running. Both are handling incoming connections on the same sockets.

We can now send SIGQUIT to the old master process to shut it down and as soon as it exits the new master process takes over and only our new application version is being served. And all of this happened without the old worker processes stopping their work once.

It’s just Unix

All of this is just Unix. The master-worker architecture, the signal handling, the communication through pipes, the preloading, the scaling of workers with signals and the hot reloading of Unicorn. There is no magic involved.

I think that’s the most amazing part about all of this. The combination of concepts like fork, pipe and signals, that are easy to understand on their own, and leveraging the operating system is where the perceived magic and ultimately the power of great Unix software like Unicorn comes from.

Why?

You might be thinking: “Why? Why should I care about this low-level stuff? I build web applications, why should I care about fork and select?

I think there are some really compelling reasons.

The first one is debugging. Have you ever wondered why you shouldn’t open a database connection (a socket!) before Unicorn calls fork(2)? Or why you get a “too many open files” error when you try to make a HTTP request (sockets!)? Now you know why.

Knowing how your system works on each layer of the stack is immensely helpful when trying to find and eliminate bugs.

The next reason I call the design and architecture reason and boils down to having answers to questions like these: should we use threads or processes? How could these processes talk to each other? What are the limitations? What are the benefits? Will this perform? What’s the alternative?

With some understanding of your operating system and the APIs it offers, it’s far easier to make architectural decisions and design choices when building a system or single components of it.

One more level of abstraction. Someone somewhere at some time said that “it’s always good to know one more level of abstraction beneath the one you’re currently working on” and I totally agree.

I like to think, that learning C made me a better Ruby programmer. I suddenly knew what was happening behind the curtains of the Ruby VM. And if I didn’t know, I could make a good guess.

And I think that knowing deeply about the system to which I deploy my (web) application makes me a better developer, for the same reasons.

But the most important reason for me, which is a personal one, is the realization that everything Unicorn does is not magic! No, it’s just Unix and there is no secret ingredient. Which, in turn, means that I could write software like this. I could write a webserver like this! Realizing this is worth a lot.

Follow me on twitter: @thorstenball. Or send me an email to me@thorstenball.com. Or check out my book at interpreterbook.com.

I also maintain a mailing list, on which I sent out occasional updates about my book or this blog. I won't spam you and you can unsubscribe at any time.

Comments

comments powered by Disqus