Updated: June 12, 2015
Several days ago, I published my long, thorough guide on Docker, an operating system level virtualization technology based on LXC, which offers a fast, lightweight and secure method of provisioning containerized applications. Lovely.
Now, one of the problems we faced when testing our first services, SSH and Apache, was the control of these services. We did not have init scripts or systemd available inside the containers, and frankly, we might not want them. But we do want some kind of mechanism to start, stop and whatnot our services. Introducing supervisord, hence this tutorial. Follow me please.
Supervisord in a nutshell
Supervisord is a process control system, designed to monitor and control processes. It does not aim to replace init, instead it encapsulates processes inside its own framework, and can start them at boot time, just like we want. There is no reason to go into any more depth about the software at this point.
Supervisord setup
Basically, supervisord is a python module. It can be installed using easy_install, which is a part of setuptools, which itself is an extension of the Python distutils package. Yes, it does get complicated quite at this point. Luckily, most Linux distributions ship with easy_install, including CentOS, which is our test platform for the day.
To get underway, we will need to install supervisord inside a container. Then, we will commit the image and use it as a baseline for our builds, which will include services like the aforementioned SSH and Apache. If you try to install supervisord on your host, you should succeed without any problems:
easy_install supervisor
Searching for supervisor
Reading https://pypi.python.org/simple/supervisor/
Best match: supervisor 3.1.3
Downloading https://pypi.python.org/packages/source/s/supervisor/
supervisor-3.1.3.tar.gz#md5=aad263c4fbc070de63dd354864d5e552
Processing supervisor-3.1.3.tar.gz
Writing /tmp/easy_install-vbOcMG/supervisor-3.1.3/setup.cfg
Running supervisor-3.1.3/setup.py -q bdist_egg --dist-dir
/tmp/easy_install-vbOcMG/supervisor-3.1.3/egg-dist-tmp-i96mIs
warning: no previously-included files matching '*' found under directory 'docs/.build'
Adding supervisor 3.1.3 to easy-install.pth file
Installing echo_supervisord_conf script to /usr/bin
Installing pidproxy script to /usr/bin
Installing supervisorctl script to /usr/bin
Installing supervisord script to /usr/bin
Installed
/usr/lib/python2.7/site-packages/supervisor-3.1.3-py2.7.egg
Processing dependencies for supervisor
Searching for meld3>=0.6.5
Reading https://pypi.python.org/simple/meld3/
Best match: meld3 1.0.2
Downloading https://pypi.python.org/packages/source/m/meld3/
meld3-1.0.2.tar.gz#md5=3ccc78cd79cffd63a751ad7684c02c91
Processing meld3-1.0.2.tar.gz
Writing /tmp/easy_install-wnhLVS/meld3-1.0.2/setup.cfg
Running meld3-1.0.2/setup.py -q bdist_egg --dist-dir
/tmp/easy_install-wnhLVS/meld3-1.0.2/egg-dist-tmp-Lp88cX
zip_safe flag not set; analyzing archive contents...
Adding meld3 1.0.2 to easy-install.pth file
Installed /usr/lib/python2.7/site-packages/meld3-1.0.2-py2.7.egg
Finished processing dependencies for supervisor
Inside a container, you will get an error, because the Python framework added into the container is not complete, and some of the modules are missing.
# easy_install supervisor
Traceback (most recent call last):
File "/usr/bin/easy_install", line 5, in <module>
from pkg_resources import load_entry_point
ImportError: No module named pkg_resources
This means we will need to manually setup easy_install:
wget https://bitbucket.org/pypa/setuptools/raw/ ->
-> bootstrap/ez_setup.py -O - | python
Supervisord configuration
The next step is to create a configuration for the container. We can create the file on the host, and then we will copy it into our image during the build process by using the COPY instruction in our Dockerfile.
COPY ./supervisord.conf /etc/supervisord.conf
The configuration itself will be something like:
[supervisord]
nodaemon=true
[program:sshd]
command=/usr/sbin/sshd -D
[program:httpd]
command=/bin/bash -c "exec /usr/sbin/httpd -DFOREGROUND"
What do we have here? Each square bracket pair defines a section. For supervisord itself, we define that it should start in the foreground rather than daemonize itself, which would mean becoming a background service.
For the program named sshd, we execute the relevant command, essentially running SSHD in the background. For the program named httpd, we start the server in the foreground, in a separate shell. There are many other options available, but at the moment, this is the bare minimum we need to get underway with supervisord.
Running a container & troubleshooting
After we've built our image and run it, supervisord should kick in and start our processes. But first, let's tackle some common errors. One that you might see is:
docker run -ti -p 22 -p 80 image-3:latest
/usr/lib/python2.7/site-packages/supervisor-3.1.3-py2.7.egg/
supervisor/options.py:296: UserWarning: Supervisord is running as root and it is searching for its
configuration file in default locations (including its current working directory); you probably want to specify
a "-c" argument specifying an absolute path to a configuration file for improved security.
'Supervisord is running as root and it is searching '
Error: No config file found at default paths (/usr/etc/supervisord.conf, /usr/supervisord.conf,
supervisord.conf, etc/supervisord.conf, /etc/supervisord.conf); use the -c option to specify a config file at a
different path
For help, use /usr/bin/supervisord -h
If you have specified a wrong path for the supervisord.conf file, or used a wrong name, the service will not be able to run. You will have to edit the COPY instruction, rebuild your image and start over. Then, the next error you may encounter will be:
# docker run -ti -p 22 -p 80 image-3:latest
/usr/lib/python2.7/site-packages/supervisor-3.1.3-py2.7.egg/
supervisor/options.py:296: UserWarning: Supervisord is running as root and it is searching for its
configuration file in default locations (including its current working directory); you probably want to specify
a "-c" argument specifying an absolute path to a configuration file for improved security.
'Supervisord is running as root and it is searching '
372 CRIT Supervisor running as root (no user in config file)
375 INFO supervisord started with pid 1
378 INFO spawned: 'httpd' with pid 9
381 INFO spawned: 'sshd' with pid 10
602 INFO exited: httpd (exit status 1; not expected)
606 INFO spawned: 'httpd' with pid 11
606 INFO success: sshd entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
671 INFO exited: httpd (exit status 1; not expected)
676 INFO spawned: 'httpd' with pid 12
742 INFO exited: httpd (exit status 1; not expected)
749 INFO spawned: 'httpd' with pid 13
825 INFO exited: httpd (exit status 1; not expected)
826 INFO gave up: httpd entered FATAL state, too many start retries too quickly
gave up: httpd entered FATAL state, too many start retries too quickly
What we have here is Apache threads exiting, forcing supervisord to retry starting them again, and eventually giving up, which leads us to a fatal state. Our container is pretty much useless at this point.
Here, the resolution is to edit the supervisord.conf file and introduce additional directives to the httpd program section, which will handle the process spawning in a more graceful manner.
[program:httpd]
startsecs = 0
autorestart = false
command=/bin/bash -c "exec /usr/sbin/httpd -DFOREGROUND"
We added the startsecs = 0 and autorestart = false. The first directive tells us:
The total number of seconds which the program needs to stay running after a startup to consider the start successful. If the program does not stay up for this many seconds after it has started, even if it exits with an expected exit code (see exitcodes), the startup will be considered a failure.
The second one means that supervisord won't be handling Apache thread restarts, and these will be handled by the Web server itself, as it handles incoming HTTP requests.
May be one of false, unexpected, or true. If false, the process will never be autorestarted. If unexpected, the process will be restart when the program exits with an exit code that is not one of the exit codes associated with this process configuration (see exitcodes). If true, the process will be unconditionally restarted when it exits, without regard to its exit code.
Now, we can try running the container again. Remember the /run/httpd issue that we encountered in the original guide? Without the /run directory in place, you will see the following messages:
docker run -ti -p 22 -p 80 image-3:latest
/usr/lib/python2.7/site-packages/supervisor-3.1.3-py2.7.egg/
supervisor/options.py:296: UserWarning: Supervisord is running as root and it is searching for its
configuration file in default locations (including its current working directory); you probably want to specify
a "-c" argument specifying an absolute path to a configuration file for improved security.
'Supervisord is running as root and it is searching '
913 CRIT Supervisor running as root (no user in config file)
916 INFO supervisord started with pid 1
919 INFO spawned: 'httpd' with pid 8
921 INFO spawned: 'sshd' with pid 9
079 INFO success: httpd entered RUNNING state, process has stayed up for > than 0 seconds (startsecs)
105 INFO exited: httpd (exit status 0; expected)
120 INFO reaped unknown pid 10
122 INFO success: sshd entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
At this point, you must connect (attach or exec a BASH shell) to the container, check the logs, and try to understand why the service is not running as expected.
Successful test
And then, once we finally sort everything out:
Conclusion
This is a fairly nerdy tutorial. But it does introduce another useful tool into our arsenal, which we can now use to control services inside containers in a robust and elegant manner, without having to install complex, expensive frameworks like systemd. It also teaches some more about Docker, how to work and troubleshoot problems and such.
Supervisord is not strictly necessary, because we managed to run our sshd and httpd processes without it, but some people might prefer this method, especially if they have to start and restart their services quite often. Either way, being familiar with the utility helps build understanding and confidence into the Docker mechanism. I hope you find this guide useful, and please do send your requests as to what we should explore next.
Cheers.