Installing ang configuring Jenkins using Chef - Part 2 - Introducing Vagrant

In the first part of this series we created a Chef cookbook that installs Jenkins, configure it and creates a new Job.
Now we have to execute it.
The simplest thing we can do is directly launching Chef on the machine we are working with:

sudo chef-client --local-mode --runlist 'recipe[jenkins_server]'

This installs Jenkins on the same machine. If we need to install Jenkins on another machine we could move our cookbook on a Chef server and then our target machine can grab it from the server and execute it locally.

But today I want to show you a third way that I find very effective for development environments: installing on a virtual machine managed by Vagrant.

Vagrant is a tool that allows you to easily create and configure virtual machines in a reproducible way. Vagrant does not run the virtual machine itself but instead stands on the shoulders of giants like VirtualBox, VMware, AWS, etc.

If we install Vagrant (and a virtualization tool, for example VirtualBox) we only have to create a little script like:

Vagrant.configure(2) do |config|
  config.vm.box = "hashicorp/precise32"
end

and call it "Vagrantfile". When you launch the command

vagrant up

Vagrant will:

• download from its repository the VM you requested, in our case an Ubuntu 12.04 (Precise Pangolin) 32 bit VM
• identify your virtualization tool (VirtualBox?)
• import the VM in your virtualization tool
• launch the VM

The VM (or 'box, as Vagrant calls it) is downloaded only the first time. Successive uses of the same box does not require a new download.

To stop the VM the command is:

vagrant halt

The VM will remain available for successive restart with vagrant up. If we want to completely remove the VM use

vagrant destroy

We can put in our Vagrantfile some network configuration, for example with

Vagrant.configure(2) do |config|
  config.vm.box = "hashicorp/precise32"
  config.vm.network "public_network", ip: "192.168.0.50"
end

we are saying Vagrant that we want our VM published on our local network with the given IP address. In this way we can access our VM with ssh (user: vagrant pwd: vagrant).

To launch Jenkins on our VM, first install the vagrant berkshelf plugin.

vagrant plugin install vagrant-berkshelf

In this way Vagrant will be able to automatically download the cookbooks from the Chef supermarket.

Then modify our Vagrantfile in this way:

Vagrant.configure(2) do |config|
  config.vm.box = "hashicorp/precise32"
  config.vm.network "public_network", ip: "192.168.0.52"

  config.berkshelf.enabled = true

  config.vm.provision "chef_solo" do |chef|
    chef.add_recipe "tomcat_server"
  end
end

and in the same directory put a file called Berksfile containing

source 'https://supermarket.chef.io'

cookbook 'jenkins_server', path: './cookbooks/jenkins_server'

Now copy your cookbook in the directory specified in the Berksfile and you are able to launch a VM with Jenkins installed and configured.

Notice that all the files we have created until now are text files contained in a single directory. You can copy this directory wherever you want and, supposed you installed Vagrant and a Virtualization tool, you can launch a full configured Jenkins machine in minutes.

You can even put it on a versioning server (Git, Subversion, etc.) so you can maintain and branch it whenever you want.

The directory contains only a few kbytes of text files. Imagine you want to do the same thing using an exported VM. The resulting file will weight a lot of Gbytes. And if you need different versions you have to maitain a lot of this heavyweight files!

It's all for now. In the next part we will talk about how the solve the problems we left behind in the first part.

Installing and configuring Jenkins using Chef

Installing Jenkins with Chef is really simple. There is a Jenkins cookbook in the Chef Supermarket so it's a matter of including a couple of recipes from it.

But installation is only the first part of the work. We also need configuration, job creation, etc. The same Jenkins cookbook offers a set of resources that help us but if you've always configured Jenkins using the web interface (like me) it's not so obvious how to use them.

So I started some experiments and finally I obtained a full working job. And this is what I did...

The goal was to have a working Maven job that downloads the sources from a Subversion repository and build the project.

Doing this manually requires the following steps:

• install Jenkins
• configure JDK installation
• configure Maven installation
• create the job

We want to do all this automatically using Chef so, after creating a new Chef cookbook with

chef generate cookbook cookbooks/jenkins_server 

the first thing to do is adding a dependency in metadata.rb so Berkshelf can automatically download the Jenkins cookbook and all its dependencies from the supermarket

depends 'jenkins', '~> 2.2.2'

Then we need to write our recipe, so in recipes/default.rb we add:

include_recipe 'jenkins::java'
include_recipe 'jenkins::master'

These two recipes, included from the Jenkins cookbook, install the JDK and Jenkins itself. The Jenkins service is started too.

Then we have to start configuration. It can be done in multiple ways but I chose to directly manipulate jenkins config files. Probably using a groovy script is more elegant but it requires more study!

template "#{node['jenkins']['master']['home']}/config.xml" do
  source "config.xml.erb"
end

These 3 lines simply copy the config.xml file from the templates cookbooks directory in the right position.

To obtain a template config.xml simply copy it from an existing Jenkins installation and remove all the unnecessary part.

I got the following:

<?xml version='1.0' encoding='UTF-8'?>
<hudson>
  <disabledAdministrativeMonitors/>
  <version>1.612</version>
  <numExecutors>2</numExecutors>
  <mode>NORMAL</mode>
  <useSecurity>true</useSecurity>
  <authorizationStrategy class="hudson.security.AuthorizationStrategy$Unsecured"/>
  <securityRealm class="hudson.security.SecurityRealm$None"/>
  <disableRememberMe>false</disableRememberMe>
  <projectNamingStrategy class="jenkins.model.ProjectNamingStrategy$DefaultProjectNamingStrategy"/>
  <workspaceDir>${ITEM_ROOTDIR}/workspace</workspaceDir>
  <buildsDir>${ITEM_ROOTDIR}/builds</buildsDir>
  <systemMessage>This is jenkins</systemMessage>
  <jdks>
    <jdk>
      <name>Default JDK</name>
      <home>/usr/lib/jvm/java-1.7.0-openjdk-i386</home>
      <properties/>
    </jdk>
  </jdks>
  <viewsTabBar class="hudson.views.DefaultViewsTabBar"/>
  <myViewsTabBar class="hudson.views.DefaultMyViewsTabBar"/>
  <clouds/>
  <quietPeriod>5</quietPeriod>
  <scmCheckoutRetryCount>0</scmCheckoutRetryCount>
  <views>
    <hudson.model.AllView>
      <owner class="hudson" reference="../../.."/>
      <name>All</name>
      <filterExecutors>false</filterExecutors>
      <filterQueue>false</filterQueue>
      <properties class="hudson.model.View$PropertyList"/>
    </hudson.model.AllView>
  </views>
  <primaryView>All</primaryView>
  <slaveAgentPort>0</slaveAgentPort>
  <label></label>
  <nodeProperties/>
  <globalNodeProperties/>
</hudson>

As you can see there are only two things I had to customize:

• the JDK installation
• the default view

I used the same JDK that run Jenkins without installing a new one. The JDK installation has a problem here. I had to hard wire the path of the JDK dir. In a subsequent post we'll see how to solve this problem.

And now we have to install Maven. We can let Jenkins install it for us putting the right file in the right position with:

template "#{node['jenkins']['master']['home']}/hudson.tasks.Maven.xml" do
 source 'hudson.tasks.Maven.xml.erb'
end

The hudson.tasks.Maven.xml.erb we should put in our templates dir is:

<?xml version='1.0' encoding='UTF-8'?>
<hudson.tasks.Maven_-DescriptorImpl>
  <installations>
    <hudson.tasks.Maven_-MavenInstallation>
      <name>Default Maven</name>
      <properties>
        <hudson.tools.InstallSourceProperty>
          <installers>
            <hudson.tasks.Maven_-MavenInstaller>
              <id>3.2.2</id>
            </hudson.tasks.Maven_-MavenInstaller>
          </installers>
        </hudson.tools.InstallSourceProperty>
      </properties>
    </hudson.tasks.Maven_-MavenInstallation>
  </installations>
</hudson.tasks.Maven_-DescriptorImpl>

To have a flexible cookbook we can substitute the Maven version with an attribute that can be customized using a Chef role or environment but now I want to keep the whole thing as simple as possible.

After the configuration files are copied in the right place Jenkins has to reload the configuration, so:

jenkins_command 'reload-configuration'

The last thing to do is job creation. In the Jenkins cookbook there is a resource for that but it needs a job configuration file that must exist on the target node.

As suggested by the Jenkins plugin documentation we can put it in the templates and then copy it in Chef's file cache path with:

job_config = File.join(Chef::Config[:file_cache_path], 'job-config.xml')

template job_config do
  source 'webapp-job-config.xml.erb'
end

The job config file should look like this:

<?xml version='1.0' encoding='UTF-8'?>
<maven2-moduleset plugin="maven-plugin@2.7.1">
  <actions/>
  <description></description>
  <keepDependencies>false</keepDependencies>
  <properties/>
  <scm class="hudson.scm.SubversionSCM" plugin="subversion@1.54">
    <locations>
      <hudson.scm.SubversionSCM_-ModuleLocation>
        <remote>http://192.168.0.50/svn/repo/webapp/trunk</remote>
        <local>.</local>
        <depthOption>infinity</depthOption>
        <ignoreExternalsOption>false</ignoreExternalsOption>
      </hudson.scm.SubversionSCM_-ModuleLocation>
    </locations>
    <excludedRegions></excludedRegions>
    <includedRegions></includedRegions>
    <excludedUsers></excludedUsers>
    <excludedRevprop></excludedRevprop>
    <excludedCommitMessages></excludedCommitMessages>
    <workspaceUpdater class="hudson.scm.subversion.CheckoutUpdater"/>
    <ignoreDirPropChanges>false</ignoreDirPropChanges>
    <filterChangelog>false</filterChangelog>
  </scm>
  <canRoam>true</canRoam>
  <disabled>false</disabled>
  <blockBuildWhenDownstreamBuilding>false</blockBuildWhenDownstreamBuilding>
  <blockBuildWhenUpstreamBuilding>false</blockBuildWhenUpstreamBuilding>
  <triggers/>
  <concurrentBuild>false</concurrentBuild>
  <aggregatorStyleBuild>true</aggregatorStyleBuild>
  <incrementalBuild>false</incrementalBuild>
  <ignoreUpstremChanges>false</ignoreUpstremChanges>
  <archivingDisabled>false</archivingDisabled>
  <siteArchivingDisabled>false</siteArchivingDisabled>
  <fingerprintingDisabled>false</fingerprintingDisabled>
  <resolveDependencies>false</resolveDependencies>
  <processPlugins>false</processPlugins>
  <mavenValidationLevel>-1</mavenValidationLevel>
  <runHeadless>false</runHeadless>
  <disableTriggerDownstreamProjects>false</disableTriggerDownstreamProjects>
  <blockTriggerWhenBuilding>true</blockTriggerWhenBuilding>
  <settings class="jenkins.mvn.DefaultSettingsProvider"/>
  <globalSettings class="jenkins.mvn.DefaultGlobalSettingsProvider"/>
  <reporters/>
  <publishers/>
  <buildWrappers/>
  <prebuilders/>
  <postbuilders/>
  <runPostStepsIfResult>
    <name>FAILURE</name>
    <ordinal>2</ordinal>
    <color>RED</color>
    <completeBuild>true</completeBuild>
  </runPostStepsIfResult>
</maven2-moduleset>

Again, there are a lot of things we can do to have a flexible cookbook like the parametrization of the Subversion URL.

At last

jenkins_job 'Webapp' do
 config job_config
end

will create the job.

Now we only need to launch Chef to have a Jenkins machine configured and ready to launch the job.

In the next part we can launch Chef using Vagrant.

In the meanwhile, if you want to download the cookbook you can find it in this Github repo in the cookbooks/jenkins_server directory.