Cloud Computing: Develop the skills to automate • Celtic3d

For several years, I have been experimenting with 3d modelling. This has been as a pastime alongside my "day job" of managing IT services in a large company. These started off as distinct and separate activities, but recently have been on a converging path.

I am an evangelist for cloud computing. I am also enthusiastic about is possible with a bit of applied know-how. In the beginning, much of the motivation behind Celtic3d was to put my money where my mouth is and prove-out some of the theoretical advantages of cloud by learning and experiencing the practicalities for myself.

Cloud Computing

There has been a trend in IT over the last decade to move to virtual systems. It is now unusual for companies to have their own IT servers tucked away in a computer room, and it is the norm to make use of the managed hosting providers in the market. But things have not stopped there. From virtualization came "cloud".

Defining cloud is still sometimes contentious. To be clear: When I refer to "cloud", I am referring to IT services that are: elastic (the capacity you use can grow and shrink); on-demand (you only pay for the capacity you use); accessed over the network using standard techniques (usually over the internet); and multi-tenanted (shared with other customers). {Ref: The NIST Definition of Cloud Computing, 2011 }

For the latter point, you may ask, why is sharing facilities with other customers a benefit to me? The answer is simply to look at who is paying for the capacity you don't use. Also look at how big the customer base is driving competition on price and functionality. If your supplier can't charge someone else for capacity you are not using, you can bet your bottom dollar you will end up paying for that capacity somehow.

Within my day-job we poured over what Cloud meant to our IT environment and how it might change our approaches to managing and consuming IT Services. I am certain most companies will have had similar ponderings. A few points seem clear to me:

Don't bother designing something that already exists in the market, unless you think you can do it better and cheaper yourself (which is hardly ever the case).
Private Cloud is a misnomer. Systems for your exclusive use are neither elastic nor on-demand, nor multi-tenanted. Private cloud=virtualization, which is no bad thing, but it is not cloud.
IT capacity is no longer a barrier to entry. There is no need for multi-million dollar investment to build your own data centre. Dial up what you need whenever you need it.
Cloud can improve agility in being able to develop and deploy systems quickly. More ideas can be explored and then abandoned or expanded depending on results.
Everything can be automated, so automate everything. The payback in terms of being able to spin up and adjust resources automatically in response to demand, in response to failures, in response to incidents is potentially huge.

But there are some gotchas.

Security is important. Of course you need to secure your systems and data. You also need to address perceived risks. You may well need to provide formal assurance to external agencies. All these points are solvable, and there is a case to be made that cloud can often be more secure than home-grown measures, but there are no shortcuts to doing the groundwork.
There are lots of rules and ways of doing things in IT. These exist (for the most part) for very good reasons. The processes and practices to manage IT have evolved over many years. While you can't completely throw out the rulebook, existing practices can be a major obstacle to adopting new approaches. Often the solution is to go back to first principles. For example, if there is a rule that a server cannot be deployed before some approval is obtained, ask - is that a financial control and can that control be satisfied through some other means, or is it a security control and can that be satisfied through other means? Be prepared to ask why.
There is much, much more to cloud than virtual servers. In recent years the functionality available to developers has exploded. This goes way beyond provisioning infrastructure to things like social modules that can be dropped into your solution, CDN networks for increasing responsiveness, e-mail capabilities for applications to plugin into, analytics, media file transcoding on-demand, a raft of pre-built solutions to add firewalls or e-commerce payment gateways to your solution. The list is ever-growing.
You need the right skills and competencies to manage cloud. Simple to say, but difficult to address.

Infrastructure as Code

In cloud you can write scripts to start a server. But why stop there? Once the server is launched, have the script do an update to the latest patch level. Then continue and have the script install an application, and then apply configuration.

This is where things get really interesting. Once you have figured out how to script the full launch and configuration process other things become possible. Rather than trying to diagnose and fix failing components, it may well be quicker and cheaper to delete them and run a script to launch a replacement. Creating test and development instances becomes fast, easy and cheap.

Of course there are some architectural patterns important to making this possible. You need to separate your data from the servers. You need to spread things out into smaller components and services that link together. Rather than big monolithic systems with everything on the same box, think small and modular.

Skills and Competencies

From my perspective, IT professionals are often split into two camps, the infrastructure crowd comprising the network folks, the sys admins, the database admins the ITIL process folks and so on, separate and distinct from the developers, designers, analysts, coders etc.

If you excuse some broad generalisations that does disservice to both IT Operations and Software Engineering:

On the Infrastructure side:

Expert at deploying, patching and maintaining systems.
Trusted to maintain backup and handle restoration.
Focused on system availability and performance.
Guardians of security rules for network, server, data, physical access and regulatory compliance.

On the Developer side:

Expert at developing solutions & solving business problems.
Masters of version control and code management.
Conversant in multiple application languages and data formats
Familiar with testing and code deployment best practices.

To be successful in cloud, you need to share and combine best practice from both sides.

Once you master the basics, and have scripted your launch processes, it is possible to treat your infrastructure deployments as code. This is where some of the key skills and practices from the Developer community become critical. Those scripts need a code management system, they need version control, bug tracking and practices not usually familiar to infrastructure folks. Conversely, those scripted components need to be created within virtual subnets, have security groups and firewall rules defined, connect to backend databases and respect data policies. Topics the Infrastructure folks are expert at.

Combined, these fields are labeled DevOps. The concept is simple. The implementation, difficult.

A Terrible Truth

A couple of years back, despite being seen as the go-to guy for Cloud in my company, I realized a terrible truth. Despite decades of developing a pretty successful career within IT, most of my skills lay in the infrastructure sphere. I was beginning to understand Cloud enough to see where my knowledge was lacking. What coding skills I once had were by now, rusty and inadequate. If I am brutally honest with myself, even my infrastructure skills could do with a refresher, neglected as I climbed through the supervisory and then management rungs in my organisation. I remember being told early in my IT career that I needed to choose between being a technical guy, or stepping up to supervisory role, trying to be both was discouraged. But here I was, trying to sell a message within my organisation that we needed to develop skills and competencies for cloud and forge links with the Developers, while only having a rough idea, based on experience decades ago, of what I was talking about. I needed to up my game.

Celtic3d

So that's where this website started. As a hands-on laboratory to try out some of my ideas and concepts for cloud.

I had seen dozens of cloud demos where someone dazzles the audience by typing on their keyboard and two minutes later point to a new server being created "in the cloud". And that is neat, but then what? How does the app then get installed and configured? How is production managed? How can I get a development copy of the production system? Where best to put your data? I wanted to follow the process through, including a proper development -> test -> deploy pipeline and also figure out how to link all the components together.

As a side benefit, along the way I brushed up on my HTML5 and CSS and resurrected some long forgotten scripting skills with Python and learned JavaScript and PHP. I now know how to set up Apache and a WordPress environment and secure it, install SSL certificates and a bunch of stuff I had not done in years. I learned how to use GIT to manage versions for launch scripts and CloudFormation templates. I have also been experimenting with Python scripting in Blender to automate some of the 3d modelling tasks on the fly.

As you might glean from this list, I lean towards open-source solutions: Linux; WordPress; Blender; Python; GIT. It is not just that they are free to use. They are very robust and very good at what they do. Getting used to the command-line interface and JSON or XML file formats most of these employ makes it easy to work with similar interfaces in Amazon Web Services (AWS). Other cloud philosophies are available.

Bringing my interest in 3d design back into the picture, I think it is perfectly possible to build a virtual factory in the cloud, where customers shop for real products that are customized and manufactured to order and shipped anywhere in the world. All fully automated so that the effort on my part is spent creating designs rather than configuring servers and taking products to the post office. That is what I have set out to do.

Design Principles

We have built Celtic3d on these design principles:

The public site should be managed to production standards with all the usual and required safeguards around customer privacy and data.
The site should be able to complete any order automatically and without any manual steps.
An architecture of lots of small, loosely coupled components.
The site should maintain itself with automated patches and updates and automatic data feeds between systems.
The site can be scaled quickly as load grows.
Running costs kept under a tight reign and proportionate to demand.

Conclusion

Continuous training will be familiar to anyone working in IT but even by IT standards it takes a significant re-investment in core skills to get fighting fit for Cloud.

Although I have been working in IT for decades, Cloud computing done right requires a fundamental reassessment of how IT is built, deployed and managed. Building my own e-commerce site in my spare time might seem an extreme method of carrying out that reassessment, but, in my experience, it was necessary to gain insights into how all the different components fit together to form an overall solution and how to apply those insights to real-word examples.

The experience also reaffirmed my view that it is better to use a solution someone else has built rather than try to build my own. Yes, I could build a website from scratch with user login and sanitizing functions for user input and so on, but why spend months doing that when WordPress can do 90% of what I need straight away and still gives me full flexibility to adapt and amend to meet my exact needs? I get the benefit of security updates published regularly and applied automatically. I can spend that time doing other stuff.

It is in my nature to want to pick things apart, to understand through trying it out. I'm not suggesting everyone needs to build their own website to understand what is different about cloud. I can only tell you that it helps.

Note

The views expressed in this article are personal and do not represent the views of any current or former employer.

If you found this article interesting, please like and share.