How software companies can end up in a Cloud hell, and how to fix it

The slippery slope

Custom software development services are in high demand today. Organizations are willing to pay considerable amounts of money for development of custom software, be it mobile applications relying on backend services in the Cloud, or tailoring of open source software to better fit their needs. This has provided the impetus for rapid growth of companies that provide software development services. Some of these companies simply rent developers to clients, others work on customer projects handling both project management and development.

Development of software is the first, and most important step, in the software life cycle. However, things do not end there. Some clients may be unwilling to maintain the software that was developed for them, which leaves maintenance responsibility to the software company.

Software developers do not enjoy managing production deployments which areriddled with boring yet extremely important duties like ensuring that disaster recovery procedures work and that everything is monitored and backed up to a sufficient degree. Historically that has been the turf of system administrators.

It is also likely that the cash cow for a software company is developing software, not maintenance of production systems. Unless, of course, the company had the foresight to factor in the latter to the contracts they made with their clients.

The production maintenance responsibility can end up being a huge burden if client preferences are followed to the letter: with a large number of clients the software company can end up responsible for a huge mix of technologies. For example, one production application may be deployed in AWS on top of Fargate, another in Google Cloud on top of Kubernetes and yet another in Azure on virtual machines with autoscalers, load balancers and managed databases. This is very problematic, because the feature set of each of the three public Clouds is huge:

While these Cloud share many similarities, most of their services are different enough to cause lots of maintenance headaches if managed to distribute your development and production workloads across them and did it all manually.

Cloud spaghetti

Cloud spaghetti (photo: Ksenia Chernaya, pexels.com)

Developers usually do not need DevOps or infrastructure code tools in their line of work. A natural consequence of this is that they don’t know those tools well, which means that they tend to lean towards configuring production environment manually. Containers are an exception to this rule: they are widely used by developers and force them to write infrastructure code - typically Dockerfiles. However, even containers depend on correct setup of their runtime environment, be it Amazon ECS, managed Kubernetes on Digital Ocean or Azure Container Instances. Again not something you need to be worried as a developer if you run your code in Docker Compose.

This slippery slope can lead to a situation where each production deployment depends on a single developer and others struggle to help him or her out because of technology sprawl caused by lack of standardization.

It should also be noted that container images created for development purposes are not necessarily suitable for a production environment. In production you need security hardenings, different configurations and often have to add things like sidecar containers for handling logging, monitoring and backups.

Hiring a full-stack DevOps messiah?

Once a software development company finds itself in the situation like above the natural inclination is to hire their way out of it - in other words get somebody to clean up the mess. Depending on the depth of the pit the company has dug itself and the level of realism they have they may be looking for a DevOps messiah. Such a person would be a full-stack software developer who also happens to be an expert in AWS, Azure, Google Cloud, Terraform, Kubernetes, Docker, Ansible, Linux, Lean, Scrum, Kanban and DevOps. The idea is that this person would be a developer who could, on the side, manage all the DevOps tools and practices. The problem is that such people do not exist. One person may know something about all of these, but being proficient in all of them something else entirely. To give some perspective of what skills a DevOps needs:

DevOps roadmap by TEL4VN

A super-senior DevOps who began his or her career in the golden age of baremetal servers, lived through the age of the virtual machines and classic configuration management and now walks naturally in a world filled with containers, Clouds and serverless, would be able to cope with all the DevOps things, but such a person would definitely not fill the boots of a full-stack developer.

On the flipside a good and experienced full-stack developer is rarely if ever a good DevOps. You learn by doing, and if your doing is split across too many fields you never become an expert in any.

Even if people in this messiah category were available, what are the chances that they would be available on the market looking for a job?

Hiring an in-house DevOps team?

So, the best thing the software company can do is to step back and take a reality check. What they’re really looking for is a DevOps team that can work with their own developers to setup up development and testing pipelines and to maintain a stable production infrastructure.

The challenge with both approaches is that DevOps skills, too, are in high demand. That’s probably why CEOs and CTOs get bombarded by “partnership” proposals from DevOps outsourcing companies that have people from Ukraine, Poland, Romania and India on their payroll. Their prices vary, but in 2021 prices for outsourced Ukrainian DevOps are in this ballbark:

  • Junior DevOps (30€/hour or 4800€/month)

  • Senior DevOps (40€/hour or 6400€/month)

  • Super-senior DevOps (60€/hour or 9600€/month)

You can get a junior DevOps from India a bit cheaper, starting at 20€/hour, but the cultural differences to Europe and U.S. are big which may cause challenges down the line for the unprepared.

Hiring experienced, local DevOps in most western European countries is challenging, which better explains the steady outbound marketing push from outsourcing companies in Eastern Europe than the difference in cost.

If the software company felt they had a need for a DevOps messiah then hiring one or two junior DevOps is not a proper solution. As the word “junior” implies such a person is still relatively inexperienced in what they do. Inexperienced people are generally insecure and those that are not can easily become a liability especially if they have what we call “a cowboy mentality”: shoot from the hip instead of aiming first. Cowboy or not, a junior DevOps can only be expected to solve junior-level challenges.

A much better plan is to hire a super senior DevOps and possibly one or two junior DevOps he can lead. It won’t be cheap, but it will get the job done.

Hiring external DevOps contractors?

This article was written by us, and as we sell DevOps services we can’t be considered impartial. That said, we have decades of experience working with developers (who we love), system admins (the grumpy bearded guys) and DevOps (hipster sysadmins who install everything they read about on Hacker News), so we have a fair amount of experience to back up our claims.

The benefit of hiring contractors instead of hiring employees is that you don’t need to commit to hiring them full-time and you can get senior guys without having to have piles of money next to you.