How DevOps maintains a steady infrastructure at Clay

How DevOps maintains a steady infrastructure at Clay

At Clay for over six years each and every day, we are determined to keep delivering a secure and scalable product. Our aim is to serve a million end customers with the help of a limited amount of Bricks. This means every single one of them constantly needs to deliver in high quality.

One of the main reasons Clay has a rock-solid foundation is because of it’s DevOps.

In this blog post, we want to elaborate on our DevOps quest empowering the developers to deliver at their best capacity while maintaining a steady infrastructure.

DevOps, the cornerstone of Clay Clay created an advanced access control solution, which allows users from all over the world to unlock their doors whenever and wherever they are. The development team plays a key role in grooming features and determining which technologies we’re using.

Clay develops all components in-house with its multidisciplinary team. DevOps helps Clay improving on a regular basis by providing the development team with state of the art tools like Jenkins, Docker, Terraform and Ansible. These tools are used to get releases to production in an agile and efficient way.

DevOps is focused on empowering the Clay developers and delivering a secure and scalable product.

DevOps vision

Transparency
At Clay we have a clear strategy: Building a solid and reliable product with a corresponding infrastructure. This is the reason we never proceed without questioning all possible outcomes of a decision: We keep our eyes on our main goal at all times.

The Clay team strives to be focused and is always aiming for higher quality. We make sure the team knows exactly where we stand, what direction we want to go and what our vision entails. Therefore transparency in the development team is one of our main priorities. We feel we’ve grown into a thriving corporation partly by providing this transparency between every (non) development team member.

New Bricks automatically get the full scope of what the product entails when they get on board. Transparency as a company is the foundation for a tight team. This way there’s no room for any type of ambiguity of the company and what it stands for. Fresh Bricks can kick-start their career at Clay with a clear view of the product and the team they (hopefully) become proud to be a part of.

By leading the product and keeping in mind to never let the product lead us, we avoid producing software where for example the design is state-of-the-art but the back-end isn’t sustainable. This is a painful example of what could happen when the product is leading a team instead of the other way around. 

This will eventually result in unsatisfied customers, employees and a lot of sleepless nights. With this statement as a mantra, we are convinced we tackle a lot of potential issues before they actually happen.

In the beginning there was infrastructure
Starting a company is equally accelerating and terrifying. There are a lot of challenges to conquer. These first years our focus was mainly on getting new features to production. Redundancy was something we had to think about in a less hectic time, infrastructure being one of the bigger dragons to slay.

Little did we know Murphy’s Law was just around the corner to bite us in the ass…

At that time the Clay platform was relying on a single private fiber connection to keep our IoT devices (IQ’s) online. So, you can imagine our state of mind when maintenance workers dug up that single fiber connection. We had to keep our customers up-to-date with phone calls and instantly decided we needed to scale ahead on demand and create a redundant infrastructure for our core components.

Today’s infrastructure
We’ve come a long way since those first sleepless nights. Today we’ve reached the stage our servers are setup redundantly. Local access management ensures functionality even without the M2M router connectivity. Every (core) component is now redundant, and at this stage there’s no single point of failure.

Our cloud infrastructure ensures we can keep up with our growth and we’re ready for anything that might get thrown at us. Traffic spikes for example won’t take down our platform. With our autoscale setup, we are able to react before anything can happen.

Infrastructure towards 2019 
We have an ambitious roadmap planned for next year with automation as one of the main priorities, which we are already working on today.

We would like to share some of our primary goals with you:

Become fully containerized:
We are looking forward to be fully containerized by the end of 2019 by using .NET core Docker containers: a joined effort between DevOps and the development team. We’ll manage these containers with Kubernetes, an open source platform for automating deployment.

Kubernetes also provides self-healing capabilities. It restarts containers that fail, replaces and reschedules them when nodes die and kill containers that don’t respond to their health check. Combining these capabilities with a microservices, provides us a resilient architecture ready for the future.

Kubernetes among other things provides self-healing. It restarts containers that fail, replaces and reschedules them when nodes die and eliminate containers that don’t respond to the health check. Combining these capabilities with microservices, providing us a resilient architecture ready for the future.

This also enables efficient application lifecycle management between development, QA, and production environments. Allowing for confident production deployment. Our goal is to deploy to production multiple times a day.

Automated testing:
We want testing to be automated. Automated testing should be integrated in the whole CI/CO pipeline so we eliminate manually tracking and testing. The same goes for security.

Security:
It goes without saying security is one of the most important aspects of Clay. When having security in mind from the start you avoid building something and realizing it might not be as secure as you had hoped when it’s too late. DevSecOps is based on the mindset everybody is responsible for security and helps us check if everything we do is safe automatically. We aim to have this integrated in 2019.

Serverless architecture:
We want to build a server less architecture so everything that is required to run and scale our application with will be taken care of for us. Provisioning, scaling and managing servers are therefore eliminated. One of the main goals we have for the development team is to become platform independent and reach the highest level of security

IaC (Infrastructure as Code):
By the end of 2019, we want to have the whole infrastructure in code, which means managing and provisioning our cloud infrastructure will be done through machine-readable definition files instead of physical hardware configuration or interactive configuration tools, in addition to automated processes. This will make deploying to production more efficient since we will be able to duplicate the process of constructing the environment by running the existing code.

Microservices:
One of the next steps is to phase out our current communication tool we use to communicate with the IQ’s from a specific part of the back-end. Our long-term plan is to split the application into microservices: A development technique that structures an application as a collection of loosely coupled services.

Dev manifesto

At Clay we now have the luxury to have two DevOps working hard everyday to keep our infrastructure save and secure, supporting the development teams goal to phase out our current communication tool.

Read more about their take on working as a DevOps at Clay:

Clay perceived by our fulltime DevOps
Since 2015 Niels is DevOps at Clay. Initially, he didn’t specifically choose to become a DevOps. He was a developer and had the ambition to broaden his horizon. At Clay Niels got the opportunity to combine development and DevOps, but quickly his job transformed into a fulltime DevOps adventure.

First and foremost, it is Niels’ responsibility to guarantee 100% uptime of our infrastructure. He safeguards the security of the hosting environment. He makes sure the infrastructure can scale when needed. 

Apart from maintaining the infrastructure, he also manages the CI/CD pipeline and ensures developers can deploy fast and confident to production.

Clay applauds using the latest technologies and developing new things, this empowers Niels to keep motivated at his job. Niels likes to alternate between several of his favourite DevOps tools: Jenkins, Ansible, Terraform, Kubernetes and HashiCorp Vault. It inspires him to work in an innovative environment like Clay’s.

Clay perceived by our freelance DevOps
Stephan Groot is our external DevOps helping us out for the last three months. Stephan worked at Liberty Global and Accenture. We asked him what his perception is of Clay’s infrastructure:

My experience with large corporations (like Liberty Global and Accenture) is that everything works really slowly. Internal processes, infrastructure, even though something small, it’s all really cumbersome. To get shit done you basically need to follow a certain flow/process, this immediately kills the enthusiasm/ambition of the project. Decisions can only be made if certain groups of people are aligned and all in favour of a decision. This ensures decision-making often is a very difficult and above all long process. Usually, it takes weeks before a real decision can be made.

Fortunately, this works very different at Clay! If you need something from someone you are probably sitting not more then a few feet away. Because just a few people need to make decisions within Clay the actual decision-making within Clay is very easy and agile. Considering Clay is a relatively small company for the product they’re building your input is besides very welcome also weighs heavier than in a large corporation. This means you have a lot of responsibility, but you get to actually make a difference: You are one of the reasons the product is the way it is. You have a big say over what you work on and how you are getting there.

There is no real hierarchy at Clay. The atmosphere is very open and relaxed which works really well for me! What is nice to see is that everyone at Clay is willing to lift each other to a higher level. ‘The Bricks’ Clay likes to call its employees are all very motivated to build the best product out there, and they are happy to do it together. If you run into an issue and you need to discuss this with someone, there’s always someone that is eager to help you out. This way you learn from each other and the company and it’s employees benefit from this way of working tremendously.

The ambition at Clay is to manage as many possible locks with a relatively small team. To nourish this ambition the DevOps teams challenge is to make processes smart, efficient and reliable. This means when you can prove the benefits of implementing a new technology often you get permission. This way you keep evolving professionally and the company benefits from this. The DevOps team now uses tools like Terraform, Ansible and Packer to automate all deployments. It’s very inspiring and fun to use these new state of the art technologies and to see the positive effect of the delivered software.

Clay’s development team contains a group of highly skilled developers. The level of quality is very high. Everyone has the ambition to deliver quality and is always in search of ways to improve the software. This is very helpful for the DevOps team because this means a lot of people feel involved with the several DevOps processes we try to introduce and implement.

You can tell from this blog post being a DevOps at Clay is not your average 9 to 5 job, but it’s probably the most memorable, challenging job you’ll ever have!

With 30 ambitious Bricks we are disrupting the world of access control. Are you with us? Apply now!

*Our team is determined to reach above-mentioned goals. Nevertheless we ask you to take into consideration a situation could occur throughout the year, which might force us to take a different course of direction. Information regarding a certain turn of events can be found on this blog.