A few years ago, the cloud was the magical solution for endless storage and accessibility. A decade later, the magic has dissipated and the cloud is everywhere. Everyone, from smartphone users to bulk data consumers, is now using the cloud. The rise of managed services testifies that cloud solutions have reached maturity. The advantages for developers and clients are immense. Infrastructure and technology are ready for use from the start, so that development can fully focus on functionality and smart combinations of existing platforms.
Article from Objective 31, 2019
The evolution of the cloud
The story of the cloud is really the story of an expanding infrastructure. In the past, a file server used to be a physical computer buzzing away somewhere in a basement. When computers became more powerful, virtualization emerged: multiple virtual servers could be hosted on a single physical machine, without any noticeable difference for users. There was still a machine buzzing away in the basement, until growing bandwidth and faster data transport made it possible to host virtual servers at long distance on computers in data centers. Setting up and managing a technical infrastructure yourself, like the one in the basement, was not necessary anymore. The cloud offered unlimited storage capacity and back-up facilities and was always accessible, from any location that had an internet connection. And that was only the beginning.
As a technology company, we work with modern cloud technology mainly in the form of managed services. Large companies like Amazon, Google, and Microsoft offer complete services without any prior investment, like ready-to-use webservers or database servers that can be put into operation immediately; the client can choose the platform. Or – and this involves a closer look at the technology behind it – a container service for the use of Docker containers with Kubernetes; a technology that allows applications to run independently of platforms. The managed services and platforms portfolio is rapidly growing.
Network in the cloud
Managed services are like an assembly kit for setting up a complete infrastructure. This makes it possible to quickly build entire networks in the cloud, including fully equipped virtual servers, firewalls, load balancers etc. This ‘infrastructure as a service’ can even be placed in a particular geographical location of choice. The three large cloud providers offer availability zones for this: data centers in specific locations. Thus Amazon has data centers in places such as Frankfurt, São Paulo, and Tokyo. By choosing a specific data center, the client can be certain that the virtual infrastructure runs on machines that are physically located in the chosen location. This is an important option given the fact that national and European legislation affects certain applications.
While on the one hand new concepts like managed services and availability zones are being developed, on the other hand the ‘old’ advantages of the cloud have lost none of their relevance: ultimate scalability in storage capacity, broad online availability, and high reliability. Although even the cloud cannot guarantee 100 % uptime, choosing the right options can certainly approximate this figure. But ultimately, even a cloud consists of racks with computers.
In addition to flexibility, working in the cloud also offers technological advantages. Cloud centers have usually been built close to the internet backbone, ensuring high data speeds and highly reliable connections. Moreover, developers no longer have to worry about things like power supply, energy demand, and cooling. The cloud provider deals with all that.
Cloud-independent working in the cloud
Clients choose solutions in the cloud for various reasons. One of our clients, Energie Data Services Nederland (EDSN), in a tender for the development of a series of IT facilities, stated that they were looking for a partner whose core values were ‘anticipation’ and ‘innovation’. In addition, this partner would have to know the balance between maintaining facilities at a sufficient level and responding to the right changes. Of course scalability also played a role: it was not known yet how many applications would be developed and how much data would need to be stored. All these demands in conjunction with each other made us realize that our solution would have to be based on cloud technology.
‘Vendor lock-in’ is always a risk when you are dealing with large providers like Amazon, Google, or Microsoft. But Technolution highly values supplier independence, and in our solution for EDSN we therefore chose a ‘cloud-agnostic’ solution with Amazon Web Services. This may seem like an impossible task. And yet it is possible: setting up a complete infrastructure in the Amazon cloud (or in that of any of the other big providers) which can in principle be transferred to another cloud provider or to your own infrastructure. We say ‘in principle’, because transferring a complex infrastructure to another environment is always a daunting task. If only because you are often looking at huge data quantities, not something you can just pick up and put down again in a different place. Cloud-agnostic development primarily means: working in the cloud with platform components that are based as much as possible on open source components. Amazon for instance offers the database service Aurora, which is based on the open source standard MySQL. This means that an export of the database can be read into another MySQL implementation with minimal adjustments.
Infrastructure as code
Having a virtual infrastructure has clear advantages over working with an office filled with buzzing hardware: as soon as it has outlived its usefulness, it can simply be discontinued. The opposite is also true: the current state of the art makes it possible to set up a complete infrastructure in the cloud with a few simple actions. This can even be almost entirely automated in scripts, which define how the infrastructure should be built. For instance: a Kubernetes cluster of four devices, an Active Directory Server, a load balancer, and a highly available Database service, including network routing and firewalling. The script is run, and thirty minutes later the entire infrastructure is ready for use. The same job, if done with ‘real’ devices and software, would take days to complete.
This method, called Infrastructure as code, makes it very easy to quickly set up a test environment and carry out pilots in an isolated setting. Does the system tested meet all the requirements? Then you just need to make a clone of the infrastructure including all installed applications, and users are ready to go. We have successfully used this approach within the developments for EDSN. As a result we have full control over the technical environment, so that we can focus entirely on developing the applications.
Three main advantages of the cloud
• Scalability and flexibility; storage, infrastructure, and platforms can be rapidly scaled up and down.
• No separate infrastructure required.
• Stability and robustness; large data centers offer automatic back-up and recovery of the infrastructure. The latest technology is always available, as cloud services keep pace with technological developments.
Areas of concern
The advantages of the cloud are considerable. Of course there are also areas of concern. One of these is cost. Scaling cloud services up or down is often a matter of ticking or unticking a box, and it is easy to forget that this flexibility has financial consequences. Each additional cloud option that is ticked incurs extra cost. Creating a temporary infrastructure for a pilot or a test can make a big difference in monthly subscription costs. Financial management of larger projects is therefore difficult to predict. It is crucially important to keep track of which cloud options are needed when, and to ensure that the available options are used as efficiently as possible. All the more so when the solution is being realized with bottom-line responsibility, as is the case with Technolution in the EDSN project. Of course the cloud providers are also aware of this aspect, and they are offering smart solutions like spot services: cheaper cloud services that are available only when the cloud center has overcapacity. This could be a solution for tasks that do not have to be performed immediately.
Another area of concern is technological. Cloud systems require an internet connection. That can be a challenge for the integration with certain ‘real world’ systems, like for instance smart energy meters that cannot or may not be directly approached via the internet. Or operating traffic light controllers through traffic management systems. Solutions can always be found for situations like these, for example communication through a dedicated system, but these can be surprisingly complex.
The future of the cloud
The technological possibilities of the cloud will only increase over the coming years. One recent development in the field for example is serverless computing: bits of software that can be executed without requiring a specific infrastructure definition. The software floats in the cloud, as it were, and responds to predefined trigger values and events. These bits of software (or services) of course function in a wider context, and it is up to the developer to decide how the whole complex of services comes together.
This also shows why thorough technological expertise is and will remain indispensable when it comes to developing systems in the cloud. Seen from a helicopter perspective, the cloud may seem like a simple assembly kit. But issues such as the efficient design of infrastructures, developing and configuring applications, managed services, and optimal use of new developments such as serverless computing still require deep knowledge of technology.