Adding the check to synthetic.xml

First, we will have to tell Deployit that the new condition exists. We do this by adding properties to existing Deployit types. Deployment conditions in Deployit are defined by pairs of properties, one on the Environment and one on a DeploymentPackage (a version of an application).

To introduce the property into the Deployit's type system, we have to edit the synthetic.xml file on the Deployit Server. The synthetic.xml file can be located in $DEPLOYIT_HOME/conf. For more information about the Deployit type system, please refer to the Reference Manual and the Customization Manual

First we add the property to the Environment type. Locate the definition of udm.Environment and add the property:

<type-modification type="udm.Environment">
  ...
  <property name="requiresSignedOffByReleaseManager" kind="boolean"
    required="false" category="Deployment Checklist" />
  ...

This means that for any Environment there will be an additional boolean property. The property has to start with the prefix requires in order to be picked up by the Release Dashboard.

The next thing we have to do in the synthetic.xml is to tell Deployit that deployment packages will have a property that can satisfy the check specified above. Note that each version of an Application will have different sets of properties checked during its voyage through the deployment pipeline. We therefore set the property on the udm.Version type.

Locate the type udm.Version and add the property. The property name is based on the property set on the Environment, but with the requires prefix replaced by satisfies.

<type-modification type="udm.Version">
  ...
  <property name="satisfiesSignedOffByReleaseManager" kind="boolean"
    required="false" category="Deployment Checklist"
    label="Signed off by Release Manager" />
  ...

What we just did: for each individual deployment package we add the possibility to satisfy the constraints imposed by the requires property of the Environment.

After saving the file, restart the Deployit Server for the changes to take effect.

Configuring the Environment

Now that we have added the check, we can enable it for an environment. In this case, we will enable it only for the Production Environment, but of course you could enable it for any environment.

Once a check has been enabled on an environment, only deployment packages that satisfy the requirements can be deployed to this environment. Deployments that do not satisfy the check are not allowed.

We can easily configure the Environment from the UI. After restarting the Deployit Server, reload the UI in the browser. Got to the Repository tab, and find the Production Environment in the Environments tree. Double-click on it to open its editor and click on the "Deployment Checklist" tab. The new property will not be enabled, meaning that this check is not active for the Production Environment. Tick the box to enable it and save.

Now go to the Release Dashboard and you will see that the condition has been added

Setting permissions

Here's a summary of the relevant security settings for the Release Dashboard:

• The repo#edit permission on the DeploymentPackage is used to check if a user is allowed to set or unset checks in the Deployment Checklist.
• When viewing the deployment pipeline, only the environments are shown if the user has readpermission on it. For example, if user 'developer' has access to the Development and Testing environments, she will only see those environments in the Release Dashboard and not the others.
• Regular deployment permissions (deploy#initial, deploy#upgrade) apply when a deployment is initiated from the Release Dashboard.

Last week was our most recent Tech Rally and here are some of the things we worked on.

Easy way to incorporate scripts in a deployment

Many customers want to customize Deployit to fit their environment. Although this is possible, sometimes users just want to include a particular script at a certain place in their deployment plan, without creating new custom CIs. We prototyped a script contributor which allows user to provide a script and the conditions under which to trigger it and it will be included in the deployment plan automatically.

Record command line scripts using the GUI

To make it easier to write command line scripts for Deployit, add the ability to record a GUI session and translate the commands into a CLI script. The user can start/stop recording and copy the resulting CLI script for further editing or inclusion in a plugin.

Here is a screenshot of this feature in action:

Customize Deployit using the GUI

Editing an XML file to customize Deployit can be intimidating for new users. Make it possible to customize Deployit (adding new types, changing existing types, changing scripts) directly from the GUI. Update the running Deployit system with these changes so they are available immediately without a server restart.

Deployment to a virtual (cloud) environment

THe idea is to allow users to deploy to a cloud environment from Deployit. The target environment is provisioned on the fly during the deployment. We prototyped an approach that allows the user to create a representation of the middleware in Deployit and the environment that contains the virtual middleware takes care of provisioning. To test this idea, we created an environment based on Vagrant and extended the hosts in Deployit with a Vagrant VM id. The VM that Vagrant instantiates uses Puppet to properly install and configure the middleware. Once this is done, the deployment continues with installing the middleware.

Here is a screenshot of this deployment in action:

Alternative Deployit GUI testing tool

We're always on the lookout for improvements to our development and testing tools. In this case, the question was whether using a combination of free, open-source tools could replace the RIATest testing tool we use for the Deployit GUI. We found that the RIATest tool, while it has it's limitations, does make it a lot easier to test the Deployit GUI. There would have to be a lot of custom work to get the open-source tools to this level. If only RIATest would improve their scripting language!

All in all, it was a fun day of hacking and creativity! Really cool to see what we can accomplish in just one day!

In this example, we'll be setting up security roles using the GUI. The example environment has two applications, OnlineOrders and SiteSearch that are both deployed to a test server before going to production. There are two different teams developing and deploying the applications. One team can't see the other's team application. Moreover, developers can only deploy to the test environment. Deployers can deploy both to the test environment and to production.

We'll set up different security roles for the different users in the organization.

This example assumes that there are already users and groups present in the system, for example by way of LDAP integration. For more detailed information on Deployit Security (including LDAP integration), please refer to the Deployit Security Manual

Overview

Here's an overview of the environment for this example.

We have two applicatons: OnlineOrders and SiteSearch. They're maintained by two different teams and deployed independently.

Both applications are deployed to the same environments: The Test Environment, containing one server called testserver and the Production Environment, that contains two physical servers, server1 and server2.

Roles

At the current state, no roles have been implemented.

This is the list of roles we want to have in the system.

• OnlineOrders developer - May update the OnlineOrders application.
• SiteSearch developer - May update the SiteSearch application.
• Lead developer - May update and install any application on the Test Environment
• OnlineOrders Deployer - May update OnlineOrders application in Test and Production
• SiteSearch Deployer - May update SiteSearch application in Test and Production
• Lead Deployer - May install any application on any environment

To add roles, log in as the a user with admin rights and got the Admin tab. Under the Admin tab, click on the Roles tab. This is the screen where you add, modify and delete roles. To add a role, click on the green plus on the right-hand side of the screen.

Enter the name of the role in the first column. The role is used within Deployit only. In the second column, 'Principals', you can specify a list of users and groups. If you have configured Deployit to point to your LDAP directory, you can add users and groups from LDAP here. To enter more than one principal, use a comma-separated list.

Hit Save to persist the changes. Note that the UI will display the roles in alphabetical order.

Directories

The following is important enough to put in italics:

In Deployit, you can only set security permissions on directories.

That means it's not possible to set permissions on individual CI nodes, like Packages or Environments. In practice you will need to impose a folder structure on your repository in order to set fine-grained permissions. The advantage here is that security will be explicit and localized in one place. Note that the root nodes in the repository ('Applications', 'Environments', etc.) are a special kind of folder and also allow setting of permissions.

Now let's add some directories. Go to the Repository browser, right-click on Applications and choose New > core > core.Directory. Name the new directory "Order". Now move the OnlineOrder package into the newly created Order directory by way of drag-and-drop. Following this procedure we create the following structure:

Assigning Roles

To assign roles, right-click on a Directory and choose Permissions. Let's start by assigning roles to the Applications/Orders directory.

In the resulting screen, we add roles for Lead Developer, Lead Deployer, OnlineOrders Developer and OnlineOrders Deployer.

We assign the roles as follows.

This means that both the Lead Developer and the Lead Deployer are allowed to add and remove applications in the Orders directory, but regular developers and deployers assigned to the project may only upgrade an existing deployment, say from OnlineOrders 1.0 to 2.0.

We can do a similar setup for the Search directory, substituting the OnlineOrders Developer and Deployer roles with the corresponding SiteSearch roles.

Note that if a role is not added to this screen, it effectively has no permissions. To remove a role, simply untick all its permissions and hit Save.

Now we'll set permissions to the Test environment. This environment is can be accessed by both development and ops teams. Simply add all roles and grant all permissions to the Lead Developers and Deployers. Regular developers / deployers have more restricted access.

For production, we only want deployers to have access. Only the lead deployer may alter the environment from Deployit.

Refining Security Levels

In our current setup, the Test and Production environments are shared for the OnlineOrders and SiteSearch applications. That means that the OnlineOrders deployers can see the SiteSearch application and vice versa. To avoid this, we need to split the environments into application-specific environments. So the Production environment will be split into an 'Online Order Production environment' and 'SiteSearch Production environment'. Note that they will still refer to the same physical servers. But, by splitting them up, you can effectively hide them from different sets of users in Deployit. To do so, we will also need another Directory layer, that splits the applications under the Environment node.

Here's the new setup:

The permissions for Environment/Orders/Production look like:

There are several important issues to note here.

NOTE 1 - Security setting on a lower level overwrite all permissions from a higher level. So the settings on the Orders/Production directory is all there is for the OrderSearch Production Environment. There is no inheritance from higher levels, combining settings from various directories.

NOTE 2 - If there are no permissions set on a directory, the permission settings from the parent are taken (recursively). So if you have a deep hierarchy of nested directories, but you don't set any permissions on them, Deployit will take the permissions set on the root node, Environments for example.

NOTE 3 - All directories higher up in a hierarchy need to provide read permission for the roles defined in the lowest directory. Otherwise the permissions itself can't be read! This analogous to file permissions on unix directories. On unix, you also need read permissions on (all) parent directories, in order to list its contents. For example, here's the permissions table of Environment/Orders

Restricted access in the UI

After setting up the security roles, users will have restricted access to the UI. For example, this is what an OnlineOrder developer will see when logging in:

Note that the SiteSearch application and environments are missing, as well as the Production environment for OnlineOrders. The Admin tab is also inaccessible.

This concludes our Security Setup example. For more information, please refer to the Deployit Security Manual

Our approach to software development has changed in the last few years. IT professionals and software developers are working more closely together than ever before. The DevOps trend also extends to an acknowledgement that automation is a key factor in reducing costs and increasing release speeds.

Striving to be cost effective is a constant for any business but the increased focus on speed of deployment is a product of the growth of the Cloud and Agile development methodology. The bottom line is – the faster new features, fixes and improvements reach the customer, the greater their satisfaction. The same principle applies in an enterprise environment – the faster the latest version of an application reaches users, the more productive they can be.

Both environments require scalability. As your product offerings grow and branch out, the delivery method must be capable of handling the changes. In the enterprise environment a large portfolio of software applications is the norm and any deployment solution must be able to scale.

Automated app deployment can reduce costs, increase speed and scale as needed, but before we take a look at the solution let’s discuss the problem.

App Deployment Nightmares

There are a lot of potential problems in deploying applications manually. Not least of which is the time it can consume for developers and IT support. Configuration is the first challenge. Where is the application stored? Which version should be installed? Where should it be installed? Where is the configuration file and how is it applied?

Security is another issue. What permissions are required to install and run the application? How much access do users have?

Manual steps also cost time and money and they introduce the potential for human error. You may have to copy a specific file to a specific location or rename a file. Even running separate scripts for different environments and projects can be a time drain and every manual process required increases risk.

Automation Headaches

Automated app deployment has its own set of problems. If you have someone create custom deployment scripts then you can bypass some of the problems but that will only work for one app deployment. It also takes time and it means you are relying on that one specialist to update the scripts as required. Scripts can be very complex and they aren’t standardized, especially if different developers write different scripts.

If you have a series of scripts then you need a process in place and instructions on how to run them correctly. They will also need to be edited frequently to cater for any changes. A monolithic script might be able to cover the entire deployment process so that you have just one script to run but what happens when you have to deploy to a different environment? What about partial upgrades? Any changes will necessitate a modification of the script.

Either case requires specialist support to run and adapt and results in a system that won’t work for a different app deployment. If you need to rollback for any reason it will be a major headache. So what is the solution?

Intelligent ARA

Looking to push Application Release Automation to the next level requires some clever tools. In order to break the bottleneck you need a system that is truly automated, standardized and capable of handling multiple application deployments across projects. The aim is to avoid reliance on specialists writing scripts.

The advantages can be impressive. According to a Gartner report by Ronni J. Colville, ARA tools help to minimize the risk of human error and add reliability. They also provide transparency for all of the stakeholders involved and, more importantly, limit downtime and ensure faster delivery to the end user. That visibility throughout the process can be a real boost to DevOps as both ends of the operation can see the process in action.

Continuous Deployment

There’s one step beyond an intelligent app deployment system and that’s the idea of rolling out software updates as soon as possible to the live server through a continuous integration system. Realistically this requires you to have some kind of continuous automated testing system in place so you can reduce the risk of introducing problems. On the other hand it means you can push out extremely frequent updates, dispense entirely with any manual tasks associated with release and forget about the last minute rush to squeeze in features before the build deadline which so often results in bugs.

A fully automated system capable of planning and adjusting as necessary to serve various environments makes continuous deployment a potential reality. Provided you have the ability to track the process, rollback and uninstall where necessary, you can minimize the risks. For true commitment to an agile development methodology this is a natural endpoint.

Bottleneck at the Funnel

As software development has advanced a number of new trends and ideas are helping developers to get their product to market faster. It is always more cost effective to solve problems as early as possible in the process and this simple truth has transformed our approach to development and testing.

The deployment end of the funnel has remained as a bottleneck sapping up the time and energy of talented software engineers and IT professionals. It is still often a potential source of human error and when it is badly handled the results can be devastating in terms of downtime or dissatisfied customers.

The same logic being applied to software development needs to carry over into the deployment phase. Ultimately by building a complete pipeline from development, through quality assurance, to deployment, where automation and standardization is the ideal, we can save money, deliver as fast as possible and scale as needed. It just makes sense.

As part of those initiatives, many of our customers are also virtualizing their infrastructure. The functionality that makes Deployit ideal for deploying new releases also make it a perfect companion to an on-demand infrastructure strategy. When spikes in demand for applications hit, virtualized infrastructure makes it possible to scale up quickly and automatically. But this infrastructure is not terribly useful without an application running on it. Deployit ensures that the newly provisioned servers run the right version of the desired application (configuring loadbalancers, static HTML, Java or .NET applications and databases) and join in shouldering the increased load.

Deployit and Puppet

Provisioning and deployment are complementary processes that, together, enable on-demand application scaling. There are two ways to integrate Deployit and Puppet to make this happen.

Chronologically, provisioning has to take place before deployment can. The deployment automation solution must become aware of the new middleware before an application can be deployed to it. In our case, when a new host is started and Puppet is run to provision it, Puppet drives the process and informs Deployit when it is done. This type of integration is the topic of this blog.

You could consider Puppet to deploy your applications as well. However, there are a couple of things to consider. First, deployments to WebSphere, WebLogic or JBoss all have their own peculiarities. Puppet does not have any middleware platform intelligence out of the box. Second, Puppet does not manage cross-node dependencies so when scaling up an entire stack (say web server, application server and database) Puppet can not be used to perform a deployment when the entire stack is provisioned. Finally, deployments to an environment require knowledge of that environment that goes beyond the node being provisioned. For instance, a deployment of a website on a new virtual server may require reconfiguring the environment's loadbalancer. Puppet does not have this type of overview.

Scaling up in these scenarios can best be driven from the application layer. The next blog describes how Deployit can drive deployments to virtual environment with Puppet to provide an overall, end-to-end deployment service.

Driving Deployit from Puppet

Puppet supports a flexible module system that allows providers to plug in additional functionality. This functionality can then be invoked from Puppet manifests, descriptions of the configuration newly provisioned node should have. To drive Deployit from Puppet, we created a Puppet Deployit module that can be found in Deployit's community github repository. It adds resources to Puppet that can be used from a manifest to interact with Deployit. The Jython Deployit command line interface (CLI) is installed on the node to communicate with the Deployit server.

The main use case our customers have is registering middleware that is provisioned through Puppet in Deployit to make it ready for deployment. In the following example, a Puppet manifest is shown that provisions a node with an Apache web server and creates a representation of the server in a central Deployit installation.

The Puppet manifest consists of three main sections. The first section installs the latest version of the Apache webserver. The second section installs the Deployit CLI. The final section registers the provisioned webserver with Deployit. We will examine this last section in detail.

The resource deployit::ci represents a CI (Configuration Item) that can be created in Deployit. Both a host CI (the server that Apache is running on) and the Apache web server itself must be created. The id, type and properties of the CI are specified in the manifest. Each of these values is the same for every provisioned webserver. Note the use of the configuration value $::ipaddress_eth1. This facter fact is replaced at runtime with the IP address of the virtual machine. It's use makes it possible to reuse the exact same manifest for every Apache instance provisioned. The apache2-server CI is also added to the TEST-Website environment so that it can be targeted for deployment.

When the webserver node is fully provisioned, this is how the CIs look in the Deployit Repository Browser:

Puppet-provisioned webserver in Deployit

If we have a Website application version 1.0 loaded in Deployit, we can deploy it to the TEST-Website environment now, with the HTML in our website automatically targeted to the newly provisioned server:

Deployment of Website/1.0 to Puppet-provisioned webserver

Conclusion

Deployit and Puppet make a great team to achieve effortless scaling of your application. Using the Deployit Puppet module, Puppet provisioned nodes can register themselves with Deployit to make them ready for deployments.

Stay tuned for the next blog which will show how Deployit can drive Puppet to provision a completely new environment and instantly deploy your application on it.

I was recently setting up a Vagrant³ environment to demonstrate Deployit's Puppet module, which automatically registers new Puppet-provisioned middleware with your deployment automation platform to enable application-tier deployments to it, and ended up wrestling for quite some time with a tricky VirtualBox problem.

The issue in question has been around for over two years now, and relates to VirtualBox's DHCP server sometimes, under as-yet-undetermined circumstances, failing to allocate an IP address to the NAT interface.
Since all Vagrant-managed images get a NAT interface by default⁴, this is more than a little inconvenient: Vagrant simply hangs during the VM configuration phase.

Since the problem doesn't occur deterministically, one way to work around this issue is simply to avoid having to reboot the image: play the "NAT lottery" until you get lucky by killing the VBoxManage process if the image is hanging and trying vagrant up again, then run vagrant suspend rather than vagrant halt and you can resume the images when you need them.

That will work, but I wasn't particularly happy with this approach because, aside from me not liking the idea of repeatedly killing hypervisor processes (I'm somewhat of a pacifist in this regard ), it effectively "cripples" Vagrant: the ease with which you can start, stop and re-configure images is precisely one of the things that makes Vagrant so useful!

One of the things I quickly discovered is that, if you start a Vagrant-created image via the VirtualBox UI and it experiences the problem⁵, cycling the NAT adapter with ifdown eth0 && ifup eth0 fixes things: this second time, it is able to pick up an IP address from the DHCP server.

Unfortunately, this does't get you far with an image that Vagrant itself is trying to start: Vagrant creates headless sessions, so you can't actually access them through the VirtualBox UI until you've killed Vagrant and the VBoxHeadless process it starts.

Luckily, VirtualBox allows you to execute commands on the guest OS without having to use the UI, via VBoxManage's guestcontrol command. So when Vagrant was again hanging while waiting to connect to the image, the first thing I tried was

/path/to/vboxmanage guestcontrol my-vagrant-image exec "/usr/bin/sudo" --username vagrant --password vagrant --verbose --wait-stdout ifdown eth0
/path/to/vboxmanage guestcontrol my-vagrant-image exec "/usr/bin/sudo" --username vagrant --password vagrant --verbose --wait-stdout ifup eth0


That did, as hoped, allow the NAT adapter to pick up an IP address. Unfortunately, it also confused Vagrant, which (presumably thinking that the image had gone offline) quit.

Happily, you don't have to bring down the adapter to request a new IP address: dhclient will do just as well. And indeed

/path/to/vboxmanage guestcontrol my-vagrant-image exec "/usr/bin/sudo" --username vagrant --password vagrant --verbose --wait-stdout dhclient


works: the NAT adapter picks up an IP address and, after a few seconds, Vagrant continues with the image configuration.

Something to hopefully help out even if it indeed takes another couple of years to get to bottom of the actual issue

Here is a sneak peak at some of our upcoming features:

.Net Plugin

Deployit now seamlessly integrates with .Net in addition to all other middleware platforms in your stack. Follow this link to a full list of platforms supported by Deployit - http://www.xebialabs.com/platform_support .

With Deployit , Enterprises can implement one deployment Best Practice for all deployments company-wide.

Pipeline Dashboard

The Deployit reporting has been further enhanced to include a pipeline of application releases. This report gives real-time insight into an applications progress through the release pipeline. This report provides useful data points regarding the overall health of your deployments and will allow a side by side comparison of application releases across a number of variables.

Enhanced Security

Deployit has leading security already built into the product, e.g. Role-based login.  With the latest version of the Deployit product all security, login credentials, etc, can be managed from within the GUI. This makes Deployit's security as usable as Deployit the platform and continues the justification that Deployit is an Enterprise product built for Enterprise teams.

More soon!!!

Author of Configuration Management Best Practices, Practical Methods that Work in the Real World

Paper Excerpt:
Agile Process Maturity is essential for successful rapid iterative development and deployment automation is a key ingredient. Technology teams are finding that Agile practices help them to deliver working software that meets or exceeds their customer's expectations. The goal for every company is to establish practices that help all of their projects (and teams) succeed and that is exactly where Agile Process Maturity can help . These successes would be impossible without Agile CM's focus on rapid iterative development. Individuals and interactions may be more important than processes and tools, but you won't achieve success without automated build, package and deployment and that is exactly where wisely-chosen tools and process add value. This paper will describe what you need to consider to establish Agile process that are repeatable and able to scale to meet the needs of your organization.

To download the paper - click here

Webinar: March 22nd, 12pm est

Presenter:
Michael Azoff, Principal Analyst at Ovum's Software Solutions Group.
Andrew Phillips, VP of Products at XebiaLabs

Devops has grown from within the operations community to embrace Agile principles and values and to deal with the increasing challenges that operations face. At the same time, Agile developers push the boundaries further with continuous delivery from source code change to production.

In this webinar, Michael Azoff will explore the challenges and opportunities that are changing the nature of application development and delivery, enabling businesses to respond to changing market needs, in minutes.

Andrew Phillips, XebiaLabs, will deliver a deployment automation market analysis and deployment automation product showcase. He will give you a greater understanding of how to integrate deployment automation into your evolving DevOps and Agile Strategies.

To Sign Up - http://www.xebialabs.com/Join_Webinar

Guest Post: Bob Aiello, Author of Configuration Management Best Practices, Practical Methods that Work in the Real World

Ergonomics is the study of making work environments more efficient, usually for the purpose of avoiding injuries and improving productivity. Another important aspect of ergonomics is avoiding mistakes. When looking at the controls in the cockpit of a plane, the untrained observer can easily see that engineers have designed the controls to greatly minimize the possibility of a human error. So it is puzzling to observe that some software engineers create confusing, incomplete and contradictory installation instructions that often lead to costly mistakes and rework. Technology professionals pride themselves on being smart and creative. I can recall many conversations where my colleagues tried to impress each other with their ability to handle complex concepts that others could not possibly follow. I am not like that. I prefer to create processes and software automation that ensures that a junior software engineer could implement the release even at 2 am (half sleepy) after drinking a couple of beers. Building in such user-friendliness requires some creative software ergonomics.

Some computer languages are rather confusing. Many Perl programmers make use of complex (powerful) regular expressions that can manipulate and process large amounts of data. But then again, trying to read (and understand) some Perl regular expressions is almost impossible. I am not attacking (or against the use of) Perl, although I will admit my own bias in favor of Ruby as a scripting language. I am not alone in this regard as many deployment engineers utilize, and even some deployment frameworks are being written in, Ruby. A deployment framework is a set of tools and processes designed to help manage the entire deployment process. This often includes taking the existing deployment scripts and placing them into the deployment frameworks, usually enjoying the features of a dashboard showing the status of each step of the release.

This is where deployment frameworks and ergonomics can share some synergy. Build, package and deployment automation should be created with a focus on ergonomics. This means that your procedures, including scripts, should be easy to use and resistant to the errors frequently caused during human interaction. That sounds simple, but how specifically does one implement ergonomics in a script? I write my scripts to including testing (often called verification and validation) into each step. This approach can help the operator verify that the correct results are being achieved. Now I might just loose some of my gunslinger reputation by making my scripts easy to use, but this focus on clarity also means that we are able to release code more often and be completely error-free-. Another tip is to allow the available tools to do the work for you. Some Deployment frameworks have the built in intelligence to work with application servers (e.g. WebSphere, WebLogic, Jboss and Tomcat). A few even sport advanced features such as algorithmic based release automation. Do remember that this is an iterative process and may take you a few release cycles before your scripts operate without any required human intervention. In fact, in the beginning you should have specific places for your operator to pause, read the messages on the screen and then press enter when they are confident that the script is working correctly.

Using the concepts learned from ergonomics will help you implement deployment automation that is fast, reliable and completely error-free!