You’ve probably heard of Docker, as since its initial release in 2013, it has gained a lot of traction in the tech community. Not familiar with this yet? Well, that’s okay, we’ll be explaining some of its primary functionality and applications in this blog.
To begin with, Docker allows applications to run in isolation inside light-weight containers, separate from the host operating system. This is achieved through configuring an ‘image’ with the files and settings required to run your application. Once an image is configured, it can be pushed to a Docker image registry allowing it to be pulled down to any machine. Images are the environment run within a container. Therefore once an image exists in a remote repository, a container can be launched anywhere, on any machine with Docker installed, and your application will run in the same way. This completely removes situations where your app runs on your machine - but not in deployment.
Sound good? Keep reading.
Docker Stack sits at a higher level than Docker containers and helps to manage the orchestration of multiple containers across several machines. Docker Stack is run across a Docker Swarm, which is essentially a group of machines running the Docker daemon, which are grouped together, essentially pooling resources.
Stacks allow for multiple services, which are containers distributed across a swarm, to be deployed and grouped logically. The services run in a Stack can be configured to run several replicas, which are clones of the underlying container. This number will be maintained as long as the service is running. Adding or removing replicas of a service is simple. This makes Docker Services ideal for running stateless processing applications.
Docker Services are easily scalable and ideal for stateless applications. We have implemented them for event streaming and message-based process intensive applications, handling millions of messages daily. And since Docker Services can communicate easily and live on the same ‘network’, they lend themselves particularly well to a micro-service architecture.
The deployment of many inter-communicating microservices is where Docker Stack comes in. The stack is configured using a docker-compose file. This is a YAML file written in a domain-specific language to specify Docker services, containers and networks. Given a compose file, it is as simple as one command to deploy the stack across an entire swarm of Docker nodes.
Our case for using Docker stack is the processing of millions of user page visit events containing information on the page a user has visited. This information is processed by a series of microservices, each feeding data to the next to attempt to find a good categorisation within our taxonomy for the page visit. Eventually, a categorisation within our taxonomy is decided upon and this is written to the profile of the user who landed on the page. From then on whenever this user is seen within our network we can be sure that this person has an interest in the taxonomy node assigned by this process.
As I said earlier to get a stack up and running just requires a docker-compose file. The following is run in a docker-quickstart-terminal, an easy way to get started with docker on windows. Docker toolbox for windows can be downloaded here.
docker swarm init
version?: ?'3'
services?:
machine1?:
image?: ?busybox
environment?:
- MACHINE=1
entrypoint?: ?sleep 5d
machine2?:
image?: ?busybox
environment?:
- MACHINE=2
entrypoint?: ?sleep 5d
machine3?:
image?: ?busybox
environment?:
- MACHINE=2
entrypoint?: ?sleep 5d
This specifies 3 services, each running a busybox image, which will be pulled from docker hub, and each will just sleep for 5 days on startup.
docker stack deploy --compose-file {path-to-compose-file} my-new-stack
docker service ls
You should see something like this:
We now have 3 services running on a docker swarm.
When our services are running on a swarm it is trivial to scale the number of instances.
docker service scale my-stack_machine1=5
docker service ls
Should show:
And the underlying containers can be seen by running:
docker ps
Showing these containers:
my-stack_machine1.5.r4vi6qme6d0qyo5lt4nfcnvyb
my-stack_machine1.4.7ts1e06bio6ktj1kcus6p3n17
my-stack_machine1.3.c0vr05p6ki3clhalgbnabd8wn
my-stack_machine1.2.xbilowtxjh30bk2q2c2s6zo1k
my-stack_machine2.1.riq2krbk93z0if4flwh5re5u1
my-stack_machine1.1.j3ut8y99v0colo6yk3tbwpwdb
my-stack_machine3.1.nafwoz48jyvlfsnxs45c5p48v
We now have 5 instances of the ‘my-stack_machine1’ service and one of machine 2 and 3, but 2 and 3 can be scaled in the same way.
If you have docker-machine installed it is easy to demonstrate how we add workers to a swarm. A worker is another machine which can host containers, or service replicas. This section will show you how to add more machines to your swarm allowing their resources to be used to run your containers.
docker-machine create worker-node
docker swarm join-token worker
“join-token” ?should output a command to run on a worker node like the following:
docker swarm join --token {TOKEN} {MASTER-IP}:2377
docker-machine ssh worker-node
This node joined a swarm as a worker.
docker service scale my-stack_machine2=5
docker ps
You should not be able to see all 5 replicas of machine 2 on your master node
?docker-machine ssh worker-node
docker ps
You should be able to see the rest of your instances on this worker machine
You now have multiple services capable of using the pooled resources of a swarm. It is common practice for swarms to consist of multiple cloud computing machines and the services will be distributed between all of them.
To summarize, in the words of Docker themselves “Docker is an open platform for developers and sysadmins to build, ship, and run distributed applications”, and Docker Stack sits a level above this to orchestrate multiple associated applications. In the above tutorial we created multiple Docker services and distributed them across multiple machines, showing how easy it is to work with a Docker Stack.
If you wish to find out more about Docker there are plenty of resources on their website https://www.docker.com. There are also plenty of useful tutorials ranging in complexity at https://training.play-with-docker.com/alacart/.
We use docker to containerise our core pipeline. We have worker containers that process incoming events and capture the data to storage - these handle 10s of millions of events/day. We then push this data through to our categorisation flow which in a similar way uses docker and an orchestration service to manage a pool of workers that analyse and categorise events into our taxonomy.
The benefits of orchestration platforms such as Docker Swarm and Kubernetes are that they can be used to run different containers across different machines, scale up/down and add/remove new containers as required.
If you’d like to have an informal chat about a potential role at Clicksco, book in a call with our team or call 0191 300 6501 to talk through the opportunities available at the moment.
We are a leading niche digital & tech recruitment specialist for the North East of England. We Specialise in the acquisition of high-performing technology talent across a variety of IT sectors including Digital & Technology Software Development.
Our ultimate goal is to make a positive impact on every client and candidate we serve - from the initial call and introduction, right up to the final delivery, we want our clients and candidates to feel they have had a beneficial and productive experience.
If you’re looking to start your journey in sourcing talent or find your dream job, you’ll need a passionate, motivated team of experts to guide you. Check out our Jobs page for open vacancies. If interested, contact us or call 0191 620 0123 for a quick chat with our team.
Follow us on our blog, Facebook, LinkedIn, Twitter or Instagram to follow industry news, events, success stories and new blogs releases.
Back to Blog