Infrastructure
testing with Jenkins,
Puppet and Vagrant

Carlos Sanchez
@csanchez
http://csanchez.org
http://maestrodev.com
@csanchez

ASF
Member

Apache
Maven

Eclipse
Foundation

csanchez.org
maestrodev.com
How we got here
Agile

planning
iterative development
continuous integration
release soon, release often
Fear of change
Risky deployments
It works on my machine!
Siloisation
Dev Change vs. Ops stability
OPs requirements

Operating System
config files
packages installed
multi stage
configurations
dev
QA
pre-production
production
Deployment

How do I deploy
this?
documentation
manual steps
prone to errors
Cloud

How do I deploy
this?
to hundreds of
servers
DevOps
DevQaOps ?
DEV

QA

OPS
QA

DEV

OPS
Specs
Packages
Versions

DEV

PROD
DEV

QA

Testability
Metrics
Logs
Security
updates

DEV

PROD
is not about
the tools
but

how can I
implement
IT
Tools can
enable
change in
behavior and
eventually
change
culture
Patrick Debois
everyone is
intelligent
enough
every tool is
cloud enabled
every tool is
DevOps(tm)
3 key concepts
Continuous
Delivery
Continuous
delivery
Infrastructure as
Code
it’s all been invented, now it’s
standardized
manifests
ruby-like
ERB templates
exec { "maven-untar":
command => "tar xf /tmp/x.tgz",
cwd => "/opt",
creates => "/opt/apache-maven-${version}",
path => ["/bin"],
} ->
file { "/usr/bin/mvn":
ensure => link,
target => "/opt/apache-maven-${version}/bin/mvn",
}
file { "/usr/local/bin/mvn":
ensure => absent,
require => Exec["maven-untar"],
}
file { "$home/.mavenrc":
mode => "0600",
owner => $user,
content => template("maven/mavenrc.erb"),
require => User[$user],
}
package { 'openssh-server':
ensure => present,
}

infrastructure
IS code
service { 'ntp':
name
=> 'ntpd',
ensure
=> running,
}

declarative model
state vs process
no scripting
Follow
development
best
practices
tagging
branching
releasing
dev, QA, production

new
solutions
new
challenges
Self
servicing
Infrastructure
always
available
virtualization & cloud
empower developers
reduce time-to-market
devs buy-in
With great power
comes great
responsibility
Vagrant
empower
developers
dev-ops
collaboration
automation
Vagrant
Vagrant
Virtual and cloud automation
VirtualBox
VMWare Fusion
AWS
Rackspace

Easy Puppet and Chef provisioning
Keep VM configuration for different projects
Share boxes and configuration files across teams
base box + configuration files
Vagrant base boxes

www.vagrantbox.es
puppet-vagrant-boxes.puppetlabs.com
anywhere! just (big) files
using Vagrant

$ gem install vagrant
$ vagrant box add centos-6.0-x86_64 
http://dl.dropbox.com/u/1627760/centos-6.0-x86_64.box

$
$
$
$
$
$

vagrant
vagrant
vagrant
vagrant
vagrant
vagrant

init myproject
up
ssh
suspend
resume
destroy
Vagrant
Vagrant.configure("2") do |config|
# Every Vagrant virtual environment requires a box to build off of.
config.vm.box = "CentOS-6.4-x86_64-minimal"
config.vm.box_url = "https://.../CentOS-6.4-x86_64-minimal.box"
# web server
config.vm.define :www do |config|
config.vm.hostname = "www.acme.local"
config.vm.network "forwarded_port", guest: 80, host: 10080
config.vm.network "private_network", ip: "192.168.33.12"
end
config.vm.provision :puppet do |puppet|
puppet.module_path = "modules"
puppet.manifest_file = "site.pp"
end
end
Geppetto
http://cloudsmith.github.com/geppetto
Maven and Puppet
What am I doing to automate deployment

Ant tasks plugin
ssh commands
Assembly plugin
Cargo
Capistrano
What can I do to automate deployment

Handle full deployment including infrastructure
not just webapp deployment

Help Ops with clear, automated manifests
Ability to reproduce production environments
in local box using Vagrant / VirtualBox / VMWare

Use the right tool for the right job
Maven-Puppet module
A Maven Puppet module
https://github.com/maestrodev/puppet-maven
fetches Maven artifacts from the repo
manages them with Puppet
no more extra packaging
Installing Maven

$repo1 = {
id => "myrepo",
username => "myuser",
password => "mypassword",
url => "http://repo.acme.com",
}
# Install Maven
class { "maven::maven":
version => "2.2.1",
} ->
# Create a settings.xml with the repo credentials
class { "maven::settings" :
servers => [$repo1],
}
New Maven type

maven { "/tmp/maven-core-2.2.1.jar":
id => "org.apache.maven:maven-core:jar:2.2.1",
repos => ["http://repo1.maven.apache.org/maven2",
"http://mirrors.ibiblio.org/pub/mirrors/maven2"],
}
New Maven type

maven { "/tmp/maven-core-2.2.1.jar":
groupId => "org.apache.maven",
artifactId => "maven-core",
version => "2.2.1",
packaging => "jar",
repos => ["http://repo1.maven.apache.org/maven2",
"http://mirrors.ibiblio.org/pub/mirrors/maven2"],
}
Examples
Infrastructure
www
httpd
Jenkins
Archiva

QA
httpd,
tomcat,
postgres

tomcat1
tomcat
db
postgres
Tomcat cluster + postgres
httpd
www.acme.com
tomcat
tomcat1.acme.com

tomcat
tomcat2.acme.com
postgres
db.acme.com

...
Continuous Delivery
commit

git repo

post commit hook
jenkins

developer

build

db

production

tomcat*
www

QA vm

binary
repository
integration
testing
Puppet Modules required
$ bundle install && librarian-puppet install
forge 'http://forge.puppetlabs.com'
mod 'puppetlabs/java', '>=1.0.0'
mod 'puppetlabs/apache', '>=0.9.0'
mod 'puppetlabs/postgresql', '>=3.0.0'
mod 'puppetlabs/firewall'
mod 'camptocamp/tomcat',
:git => 'https://github.com/carlossg/puppet-tomcat.git',
:ref => 'setclasspath'
mod 'maestrodev/maven', '>=1.0.0'
mod 'stahnma/epel', '>=0.0.2'
mod 'maestrodev/avahi', '>=1.0.0'
mod 'acme', :path => 'mymodules/acme'
mymodules/acme/manifests/db_node.pp
class 'acme::db_node' {
class { 'postgresql::server':
ip_mask_allow_all_users => '192.168.0.0/0',
listen_addresses
=> '*',
postgres_password
=> 'postgres',
} ->
postgresql::server::db { 'appfuse':
user
=> 'appfuse',
password => 'appfuse',
grant
=> 'all',
}
firewall
proto
port
action
}
}

{ '100 allow postgres':
=> 'tcp',
=> '5432',
=> 'accept',
mymodules/acme/manifests/tomcat_node.pp I
class acme::tomcat_node(
$db_host = 'db.local',
$repo = 'http://carlos-mbook-pro.local:8000/repository/all/',
$appfuse_version = '2.2.2-SNAPSHOT',
$service = 'tomcat-appfuse',
$app_name = 'appfuse',
$tomcat_root = '/srv/tomcat') {
# install java
class { 'java': }
# install tomcat
class { 'tomcat': } ->
# create a tomcat server instance for appfuse
# It allows having multiple independent tomcat servers in different ports
tomcat::instance { 'appfuse':
ensure
=> present,
http_port => 8080,
}
# where the war needs to be installed
$webapp = "${tomcat_root}/${app_name}/webapps/ROOT"
mymodules/acme/manifests/tomcat_node.pp II

# install maven and download appfuse war file from our archiva instance
class { 'wget': } ->
class { 'maven::maven' :
version => '3.0.4',
} ->
# clean up to deploy the next snapshot
exec { "rm -rf ${webapp}*": } ->
maven { "${webapp}.war":
id
=> "org.appfuse:appfuse-spring:${appfuse_version}:war",
repos
=> [$repo],
require => File["${tomcat_root}/${app_name}/webapps"],
notify => Service[$service],
} ->
firewall
proto
port
action
}
}

{ '100 allow tomcat':
=> 'tcp',
=> '8080',
=> 'accept',
mymodules/acme/manifests/www_node.pp
class acme::www_node($tomcat_host = 'tomcat1.local') {
class { 'apache': }
class { 'apache::mod::proxy_http': }
# create a virtualhost that will proxy the tomcat server
apache::vhost { "${::hostname}.local":
port
=> 80,
docroot
=> '/var/www',
proxy_dest => "http://${tomcat_host}:8080",
}
firewall { '100 allow apache':
proto
=> 'tcp',
port
=> '80',
action
=> 'accept',
}
}
manifests/site.pp

import 'nodes/*.pp'
node ‘parent’ {
class {'epel': } ->
class {'avahi':
firewall => true,
}
}
manifests/nodes/tomcat.pp

# tomcat1.acme.com, tomcat2.acme.com,...
node /tomcatd..*/ inherits ‘parent’ {
file {'/etc/motd':
content => ”tomcat server: ${::hostname}n”,
}
class {'acme::tomcat_node'}
}
manifests/nodes/qa.pp

node /qa..*/ inherits ‘parent’ {
class {'acme::db_node': }
class {'acme::tomcat_node':
db_host => 'localhost',
}
class {'acme::www_node':
tomcat_host => 'localhost',
}
}
Infrastructure unit
testing
rspec-puppet

Unit testing your puppet manifests
Ensuring packages, config files, services,...
spec/hosts/db_spec.pp

require 'rspec-puppet'
describe 'db.acme.com' do
let(:facts) { {
:osfamily => 'RedHat',
:operatingsystem => 'CentOS',
:operatingsystemrelease => ‘6.3’} }
it { should contain_class('postgresql::server') }
end
spec/hosts/www_spec.pp

require 'rspec-puppet'
describe 'www.acme.com' do
let(:facts) { {
:osfamily => 'RedHat',
:operatingsystem => 'CentOS',
:operatingsystemrelease => ‘6.3’} }
it { should contain_package('httpd') }
end
Example code and slides

Available at
http://slideshare.csanchez.org
http://github.csanchez.org
http://blog.csanchez.org
Danke!
http://csanchez.org
http://maestrodev.com

csanchez@maestrodev.com
carlos@apache.org
@csanchez
Photo Credits
Brick wall - Luis Argerich
http://www.flickr.com/photos/lrargerich/4353397797/

Agile vs. Iterative flow - Christopher Little
http://en.wikipedia.org/wiki/File:Agile-vs-iterative-flow.jpg

DevOps - Rajiv.Pant
http://en.wikipedia.org/wiki/File:Devops.png

Pimientos de Padron - Howard Walfish
http://www.flickr.com/photos/h-bomb/4868400647/

Compiling - XKCD
http://xkcd.com/303/

Printer in 1568 - Meggs, Philip B
http://en.wikipedia.org/wiki/File:Printer_in_1568-ce.png

Relativity - M. C. Escher
http://en.wikipedia.org/wiki/File:Escher%27s_Relativity.jpg

Teacher and class - Herald Post
http://www.flickr.com/photos/heraldpost/5169295832/

Infrastructure testing with Jenkins, Puppet and Vagrant - Agile Testing Days 2013