Dec 05 2013

Docker container network portability

Tag: devops,Docker,open source,virtualisationSven Dowideit @ 4:18 pm

Rather than hardcoding network links between a service consumer and provider, Docker encourages service portability.

eg, instead of 2 containers talking directly to each other:

(consumer) --> (redis)

requiring you to restart the consumer to attach it to a different redis service, you can add ambassador containers:

(consumer) --> (redis-ambassador) --> (redis)


(consumer) --> (redis-ambassador) ---network---> (redis-ambassador) --> (redis)

When you need to rewire your consumer to talk to a different resdis server, you can just restart the redis-ambassador container that the consumer is connected to.

This pattern also allows you to transparently move the redis server to a different docker host from the consumer.

Using the svendowideit/ambassador container, the link wiring is controlled entirely from the dockerrun parameters.

Two host Example

Start actual redis server on one Docker host

big-server $ docker run -d -name redis crosbymichael/redis

Then add an ambassador linked to the redis server, mapping a port to the outside world

big-server $ docker run -d -link redis:redis -name redis_ambassador -p 6379:6379 svendowideit/ambassador

On the other host, you can set up another ambassador setting environment variables for each remote port we want to proxy to the big-server

client-server $ docker run -d -name redis_ambassador -expose 6379 -e REDIS_PORT_6379_TCP=tcp:// svendowideit/ambassador

Then on the client-server host, you can use a redis client container to talk to the remote redis server, just by linking to the local redis ambassador.

client-server $ docker run -i -t -rm -link redis_ambassador:redis relateiq/redis-cli
redis> ping

How it works

The following example shows what the svendowideit/ambassador container does automatically (with a tiny amount of sed)

On the docker host ( that redis will run on:

# start actual redis server
$ docker run -d -name redis crosbymichael/redis

# get a redis-cli container for connection testing
$ docker pull relateiq/redis-cli

# test the redis server by talking to it directly
$ docker run -t -i -rm -link redis:redis relateiq/redis-cli
redis> ping

# add redis ambassador
$ docker run -t -i -link redis:redis -name redis_ambassador -p 6379:6379 busybox sh

in the redis_ambassador container, you can see the linked redis containers’s env

$ env

This environment is used by the ambassador socat script to expose redis to the world (via the -p 6379:6379 port mapping)

$ docker rm redis_ambassador
$ sudo ./contrib/
$ docker run -t -i -link redis:redis -name redis_ambassador -p 6379:6379 docker-ut sh

$ socat TCP4-LISTEN:6379,fork,reuseaddr TCP4:

then ping the redis server via the ambassador

Now goto a different server

$ sudo ./contrib/
$ docker run -t -i  -expose 6379 -name redis_ambassador docker-ut sh

$ socat TCP4-LISTEN:6379,fork,reuseaddr TCP4:

and get the redis-cli image so we can talk over the ambassador bridge

$ docker pull relateiq/redis-cli
$ docker run -i -t -rm -link redis_ambassador:redis relateiq/redis-cli
redis> ping

The svendowideit/ambassador Dockerfile

The svendowideit/ambassador image is a small busybox image with socat built in. When you start the container, it uses a small sed script to parse out the (possibly multiple) link environment variables to set up the port forwarding. On the remote host, you need to set the variable using the -e command line option.

-expose 1234 -e REDIS_PORT_1234_TCP=tcp:// will forward the local 1234port to the remote IP and port – in this case

# first you need to build the docker-ut image using ./contrib/
# then
#   docker build -t SvenDowideit/ambassador .
#   docker tag SvenDowideit/ambassador ambassador
# then to run it (on the host that has the real backend on it)
#   docker run -t -i -link redis:redis -name redis_ambassador -p 6379:6379 ambassador
# on the remote host, you can set up another ambassador
#    docker run -t -i -name redis_ambassador -expose 6379 sh

FROM    docker-ut

CMD     env | grep _TCP= | sed 's/.*_PORT_\([0-9]*\)_TCP=tcp:\/\/\(.*\):\(.*\)/socat TCP4-LISTEN:\1,fork,reuseaddr TCP4:\2:\3 \&/'  | sh && top

(this is pull request so will eventually find its way into the Docker documentation)
[Slashdot] [Digg] [Reddit] [] [Facebook] [Technorati] [Google] [StumbleUpon]

Nov 27 2013

Docker 0.7 is here – welcome RPM distros (and anyone else that lacks AUFS)

The Docker project has continued its mostly-monthly releases with the long anticipated 0.7 release, this time making the storage backend pluggable, so fedora/redhat based users can use it without building a custom kernel.

I’m curious to see the performance differences between the 3 storage backends we have now – but I need to assimilate the wonders of Linking containers for adhoc scaling first.

Try it out – I’m even more convinced that Docker containers have an interesting future :)

[Slashdot] [Digg] [Reddit] [] [Facebook] [Technorati] [Google] [StumbleUpon]

Oct 15 2013

easy install of on Debian

Tag: debian,devops,Docker,new,virtualisationSven Dowideit @ 9:05 pm

UPDATE: for Docker 0.6.5, the ubuntu debian package also installs on Debian. You still need to enable IPv4 forwarding as below – then re-start the docker daemon

I’ve been doing some work on Docker – learning golang, Docker internals, and just some of the command line options that I didn’t know I needed to know about.

Because I was in a hurry, I threw an old unused disk into one of my old laptops and installed ubuntu. That was enough for me to learn that I wanted to know alot more about Docker.

So, I’m back to using the loaner T530 with my 128GB SSD in it – its been running Debian since the day I got the SSD, over 2 years ago.

it turns out that on Debian testing (with the 3.10-3-amd64 kernel), its incredibly easy to run docker:

sudo apt-get install lxc wget bsdtar curl golang git aufs-tools mercurial iptables
wget --output-document=docker
chmod +x docker
sudo su -
#enable IPv4 forwarding
echo "net.ipv4.ip_forward = 1" >> /etc/sysctl.conf
sysctl -p /etc/sysctl.conf
# set up and mount the cgroup mountpoint
echo 'none /sys/fs/cgroup cgroup defaults 0 0' | sudo tee -a /etc/fstab
mount /sys/fs/cgroup
#OK, you might need to reboot if it fails to mount?
./docker -d &

from there, you can run the docker cli like normal (except that its not in your path yet).

I’m going to pull over the apt pinning installation documentation I wrote for publican the other week and re-write it (and test) for installing docker on Debian Stable, and we’ll all be much happier.

[Slashdot] [Digg] [Reddit] [] [Facebook] [Technorati] [Google] [StumbleUpon]

Aug 21 2013

I wonder if Docker can replace Puppet.

I’ve finally spent a little time playing with Docker, and to be honest, the really simple

here’s a list of commands that get run to set up the image

feels awesome.

to test it out, I wrote the simplest steps I could think of to create a working foswiki installation into a Dockerfile:

FROM ubuntu
MAINTAINER    Sven Dowideit <>

RUN echo deb stable main contrib > /etc/apt/sources.
RUN echo deb precise main restricted universe multive
rse >> /etc/apt/sources.list
RUN gpg –keyserver –recv-keys 379393E0AAEE96F6
RUN apt-key add //.gnupg/pubring.gpg
RUN apt-get update
RUN apt-get install -y foswiki

#create the tmp dir
RUN mkdir /var/lib/foswiki/working/tmp
RUN chmod 777 /var/lib/foswiki/working/tmp
#TODO: randomise the admin pwd..
RUN htpasswd -cb /var/lib/foswiki/data/.htpasswd admin admin
RUN mv /etc/foswiki/LocalSite.cfg /etc/foswiki/LocalSite.cfg.orig
RUN grep –invert-match {Password} /etc/foswiki/LocalSite.cfg.orig > /etc/foswiki/Loca
RUN chown www-data:www-data /etc/foswiki/LocalSite.cfg

RUN bash -c ‘echo “/usr/sbin/apachectl start” >> /.bashrc’
RUN bash -c ‘echo “echo foswiki configure admin user password is ‘admin'” >> /.bashrc’


and then I can create the image with a simple:

docker build -t svendowideit/ubuntu-foswiki .

and run that image by calling:

docker run -t -i -p 8888:80 svendowideit/ubuntu-foswiki /bin/bash

Which (assuming that port 8888 is unused on my host computer) means I can do some testing by pointing my web client to http://localhost:8888/foswiki

When I exit the bash shell, which allows me to debug what is happening, everything is shutdown, and all changes are lost. If I make changes, I can commit them, but at this point, I prefer to make a new Dockerfile.

The interesting thing is that Docker seems to create an image tag for every command, so if I make add some RUN lines, or make changes, it doesn’t need to re-do steps that it has done before…. which sounds to me just like Rex, Puppet, Ansible etc – but more re-useable.

And so, I’m curious to see how hard it would be to push out Docker versioned configuration changesets over ssh to ‘anywhere’, with some kind of impotency via system ‘tags’.


PS, the docker image is available from , and uses my debian packages, so you should install new extensions using apt-get install

[Slashdot] [Digg] [Reddit] [] [Facebook] [Technorati] [Google] [StumbleUpon]

Jun 05 2013

inkChilli wiki to Sharepoint migration tool

Tag: enterprise,foswiki,sharepoint,twiki,wiki,windowsSven Dowideit @ 11:17 am

2 months ago I was asked for help migrating a company’s TWiki site to Sharepoint, and then again a few days later from someone who’s company had declared that they needed to shift their foswiki information to Sharepoint too.

In response, I spent the 2 months researching through much of Microsoft’s undocumented API’s, and implementing a tool to do the job.

If you need to convert from TWiki or Foswiki to Sharepoint (2007, 2010, 2012 and Office 365), please head over to, or email me



[Slashdot] [Digg] [Reddit] [] [Facebook] [Technorati] [Google] [StumbleUpon]

Mar 01 2013

HTML5 & RDFa-lite semantic templating engines

While working on round-trip html5 <-> Rdfa-lite query and editing, I’ve been struck by the primitive feeling of html template languages. {{mustache}} ,  <%escapes%> <–%ssi escapes %–> all work the same way as C preprocessor macros – string concatenation style – which ignores the underlying structure that we’re working with.

Thankfully, HTML5 is changing everything – it has / will have a template tag

Along the way to finding that, I came across Weld.js (dead?), Transparency or Plates and Pure and not forgetting Knockout.js.

None of them quite goes where I want – as they either use class/id, or their own data- attributes.

What I’m after, is leveraging the already existing semantic annotations of existing (or template tag) elements to add new ones.

For example, I might have the following Link menu, and then want to dynamically add others from a remote query

     <ul id="page-list">
         <template id="page-item-template" style="display:none;">
            <li  typeof="WebPage" resource="/">
                <a property="url" href="/" tabindex="-1" id="index">
                    <span property="name">Home</span></a>
             <li typeof="WebPage" resource="/">
                  <a property="url" href="/" tabindex="-1" id="index">
                    <span property="name">Home</span></a>
             <li  typeof="WebPage" resource="/SvenDowideit.html">
                  <a property="url" href="/SvenDowideit.html" tabindex="-1" id="SvenDowideit">
                    <span property="name">Sven Dowideit</span></a>

The following works for browsers that don’t support the new HTML5 template tag:

var node = document.querySelector('#page-list [typeof=WebPage]').cloneNode(true);
node.querySelector('[property=name]').textContent = 'TODO';
node.querySelector('[property=url]').href = '/TODO.html';

A nice start, but to me, there’s still something not right with the addressing scheme –
which is where some of the above template engines use the @ symbol to denote that the value should be set on the named attribute.

something like this might work

var node = document.querySelector('#page-list [typeof=WebPage]').cloneNode(true);
node.render( {
    '[property=name]@textContent':  'TODO',
    '[property=url]@href': '/TODO.html'
but setting up the relationships beforehand, and making the clone implicit
var template = getTemplate('#page-list [typeof=WebPage]', {
   name: '[property=name]@textContent',
   url:  '[property=url]@href'
 {name: 'Sven Dowideit', url: '/SvenDowideit.html'},
 {name: 'TODO', url: '/TODO.html'}

Which looks an awful lot like Weld.js’ (and the inverse of Pure?) API. And, should be trivial to implement using Transparancy.

in the process of thinking it through, I wrote a simplistic version that deserves replacing when I’m thinking about something else:

        getTemplate: function(templateSelector, map) {
            if (this.Template === undefined) {
                var template = document.querySelector(templateSelector);
                var template_map = {};

                for (var key in map) {
                    var address = map[key].match(/^([^@]*)@?(.*)?$/);
                    template_map[key] = {
                        node: address[1] || '*',
                        attr: address[2] || 'textContent'
                this.Template = function(values) {
                    var new_elements = [];
                    for (var idx in values) {
                        var elem = template.cloneNode(true)                
                        for (var key in values[idx]) {
                            if (template_map[key] !== undefined) {
                                var node =  elem.querySelector(template_map[key].node);
                                //TODO: detect if key is a method, and call it?
                                if (template_map[key].attr === 'textContent') {
                                    node.textContent = values[idx][key];
                                } else {
                                    node.setAttribute(template_map[key].attr, values[idx][key]);
                        //TODO: if it where a real template tag, apparently there would be an elem.content
                    return new_elements;
            return this.Template;
[Slashdot] [Digg] [Reddit] [] [Facebook] [Technorati] [Google] [StumbleUpon]

« Previous PageNext Page »

Positions by Seo-Watcher
Statistical data collected by Statpress SEOlution (blogcraft).