Tags

Other

Digital Recruitment - Ronald James Group


Docker Stack

You’ve probably heard of Docker, as since its initial release in 2013, it has gained a lot of traction in the tech community. Not familiar with this yet? Well, that’s okay, we’ll be explaining some of its primary functionality and applications in this blog.

To begin with, Docker allows applications to run in isolation inside light-weight containers, separate from the host operating system. This is achieved through configuring an ‘image’ with the files and settings required to run your application. Once an image is configured, it can be pushed to a Docker image registry allowing it to be pulled down to any machine. Images are the environment run within a container. Therefore once an image exists in a remote repository, a container can be launched anywhere, on any machine with Docker installed, and your application will run in the same way. This completely removes situations where your app runs on your machine - but not in deployment.

Sound good? Keep reading.

How does Docker Stack help?

Docker Stack sits at a higher level than Docker containers and helps to manage the orchestration of multiple containers across several machines. Docker Stack is run across a Docker Swarm, which is essentially a group of machines running the Docker daemon, which are grouped together, essentially pooling resources.

Stacks allow for multiple services, which are containers distributed across a swarm, to be deployed and grouped logically. The services run in a Stack can be configured to run several replicas, which are clones of the underlying container. This number will be maintained as long as the service is running. Adding or removing replicas of a service is simple. This makes Docker Services ideal for running stateless processing applications.

Docker Stack in production

Docker Services are easily scalable and ideal for stateless applications. We have implemented them for event streaming and message-based process intensive applications, handling millions of messages daily. And since Docker Services can communicate easily and live on the same ‘network’, they lend themselves particularly well to a micro-service architecture.

The deployment of many inter-communicating microservices is where Docker Stack comes in. The stack is configured using a docker-compose file. This is a YAML file written in a domain-specific language to specify Docker services, containers and networks. Given a compose file, it is as simple as one command to deploy the stack across an entire swarm of Docker nodes.

Our case for using Docker stack is the processing of millions of user page visit events containing information on the page a user has visited. This information is processed by a series of microservices, each feeding data to the next to attempt to find a good categorisation within our taxonomy for the page visit. Eventually, a categorisation within our taxonomy is decided upon and this is written to the profile of the user who landed on the page. From then on whenever this user is seen within our network we can be sure that this person has an interest in the taxonomy node assigned by this process.

Setting Up Your Stack

As I said earlier to get a stack up and running just requires a docker-compose file. The following is run in a docker-quickstart-terminal, an easy way to get started with docker on windows. Docker toolbox for windows can be downloaded here.

  • To deploy a stack you need to be running on a swarm master node, so initialize a swarm:
docker swarm init
  • Create a file named “docker-compose.yaml” containing the following:
version?: ?'3' 
services?:
   machine1?:
        image?: ?busybox 
        environment?:
        - MACHINE=1 
        entrypoint?: ?sleep 5d
machine2?:
        image?: ?busybox 
        environment?:
        - MACHINE=2 
        entrypoint?: ?sleep 5d
machine3?: 
        image?: ?busybox 
        environment?:
        - MACHINE=2 
        entrypoint?: ?sleep 5d

This specifies 3 services, each running a busybox image, which will be pulled from docker hub, and each will just sleep for 5 days on startup.

  • Use the compose file to deploy your stack of 3 services
docker stack deploy --compose-file {path-to-compose-file} my-new-stack
  • Check that your services are deployed
docker service ls

You should see something like this:

We now have 3 services running on a docker swarm.

Scaling your stack

When our services are running on a swarm it is trivial to scale the number of instances.

  • Now we have our services running we can scale them:
docker service scale my-stack_machine1=5
  • Let's check our replica counts now:
docker service ls

Should show:

And the underlying containers can be seen by running:

docker ps

Showing these containers:

my-stack_machine1.5.r4vi6qme6d0qyo5lt4nfcnvyb 
my-stack_machine1.4.7ts1e06bio6ktj1kcus6p3n17 
my-stack_machine1.3.c0vr05p6ki3clhalgbnabd8wn 
my-stack_machine1.2.xbilowtxjh30bk2q2c2s6zo1k 
my-stack_machine2.1.riq2krbk93z0if4flwh5re5u1 
my-stack_machine1.1.j3ut8y99v0colo6yk3tbwpwdb 
my-stack_machine3.1.nafwoz48jyvlfsnxs45c5p48v

We now have 5 instances of the ‘my-stack_machine1’ service and one of machine 2 and 3, but 2 and 3 can be scaled in the same way.

Adding swarm workers

If you have docker-machine installed it is easy to demonstrate how we add workers to a swarm. A worker is another machine which can host containers, or service replicas. This section will show you how to add more machines to your swarm allowing their resources to be used to run your containers.

  • Create a new docker-machine:
docker-machine create worker-node
  • Get a worker join token from your master swarm node:
docker swarm join-token worker
“join-token” ?should output a command to run on a worker node like the following: 
docker swarm join --token {TOKEN} {MASTER-IP}:2377
  • Connect to your new machine:
docker-machine ssh worker-node
  • Paste the join command you received earlier
    If everything worked well you should see the following:
This node joined a swarm as a worker.
  • Now if you scale a service on your master node it should be distributed across the two nodes Leave your worker docker-machine and on your master run:
docker service scale my-stack_machine2=5
  • Check the containers running on your master:
docker ps

You should not be able to see all 5 replicas of machine 2 on your master node

  • Connect to your worker machine again
?docker-machine ssh worker-node
  • Check on your containers here
docker ps

You should be able to see the rest of your instances on this worker machine

You now have multiple services capable of using the pooled resources of a swarm. It is common practice for swarms to consist of multiple cloud computing machines and the services will be distributed between all of them.

Conclusions

To summarize, in the words of Docker themselves “Docker is an open platform for developers and sysadmins to build, ship, and run distributed applications”, and Docker Stack sits a level above this to orchestrate multiple associated applications. In the above tutorial we created multiple Docker services and distributed them across multiple machines, showing how easy it is to work with a Docker Stack.

If you wish to find out more about Docker there are plenty of resources on their website https://www.docker.com. There are also plenty of useful tutorials ranging in complexity at https://training.play-with-docker.com/alacart/.

How does Clicksco use Docker?

We use docker to containerise our core pipeline. We have worker containers that process incoming events and capture the data to storage - these handle 10s of millions of events/day. We then push this data through to our categorisation flow which in a similar way uses docker and an orchestration service to manage a pool of workers that analyse and categorise events into our taxonomy.

The benefits of orchestration platforms such as Docker Swarm and Kubernetes are that they can be used to run different containers across different machines, scale up/down and add/remove new containers as required.

Interested in a career at Clicksco?

If you’d like to have an informal chat about a potential role at Clicksco, book in a call with our team or call 0191 300 6501 to talk through the opportunities available at the moment.

Who Are Ronald James?

We are a leading niche digital & tech recruitment specialist for the North East of England. We Specialise in the acquisition of high-performing technology talent across a variety of IT sectors including Digital & Technology Software Development.

Our ultimate goal is to make a positive impact on every client and candidate we serve - from the initial call and introduction, right up to the final delivery, we want our clients and candidates to feel they have had a beneficial and productive experience.

Contact our Team

If you’re looking to start your journey in sourcing talent or find your dream job, you’ll need a passionate, motivated team of experts to guide you. Check out our Jobs page for open vacancies. If interested, contact us or call 0191 620 0123 for a quick chat with our team.

Let's be Friends!

Follow us on our blog, FacebookLinkedInTwitter or Instagram to follow industry news, events, success stories and new blogs releases.

 

Back to Blog

</Follow Us>