Virtual testing &
development*
* batteries included
$ git clone git://server.com/project.git
$ run
Ideal world
$ git clone git://server.com/project.git
Real world
$ git clone git://server.com/project.git
... read help
$ mkdir, cp, vim config.conf, configure, make, make
install
... ask guru
$ /project/hidden/blackmagic --fix_that --fix_this --
undocumented_parameter=MAGIC_CONST_42
... %^ing @#$@#%@!!!!
Real world
$ git clone git://server.com/project.git
... read help
$ mkdir, cp, vim config.conf, configure, make, make
install
... ask guru
$ /project/hidden/blackmagic --fix_that --fix_this --
undocumented_parameter=MAGIC_CONST_42
... %^ing @#$@#%@!!!!
$ run
Real world
Why?
Operating System
User Space
Music
WebServer
DBAppServer
IM
Editor
Browser
Problems
• No isolation (oh, wrong binary on
path...?)
• Not repeatable (that README ain’t
gonna run itself!)
• No guarantees (“..but it works on my
computer!”)
Solution?
Operating System
Music
WebServer DBAppServer
IM Editor
BrowserUser Space
Virtualized OS
Benefits
• Isolation (project and version specific
files only)
• Repeatable (snapshots, copies,
automation)
• Guarantees (same OS different day)
Business benefits
• Reduce costs (less hardware, lower energy
consumption)
• Backup and data protection
• Application availability
• Less time to respond to changing business
needs
• Less time spent on routine tasks
• Better IT department perception
Time spend on routine tasks
according to VMware small & medium business survey
Why haven’t we been doing
this for a long time?
• Big companies have been
• Amazon EC2
• Microsoft
• Google
• Only recently possible on local machines
• Cheat hardware
• Cheap RAM
• Desktop virtualization API
Ok, let’s go virtual.
Now what?
Vagrant is a tool for building and
distributing virtualized development
environments (vagrantup.com)
VirtualBox is cross-platform virtualization software
that works on all the major platforms
(virtualbox.org)
High level overview
• Describe environment with Vagrantfiles
• Command line interface to manage VM lifecycle
• Automate provisioning and configuration
• Share folder with host OS via NFS
• Provide SSH access to guest OS
• Network multiple VMs
• Package and distribute VMs
Vagrant boxes
• Box = pre-packaged VM environment (~OS
image)
• A number of ready to use boxes @ http://
vagrantbox.es
• Ubuntu 32 / 64
• CentOS
• Solaris
• RH
Vagrantfile
• Vagrantfile is to Vagrant as Makefile is to
Make
• Describes VM in code
• One Vagrantfile per project (can contain
multiple VMs)
• Commit it to version control
• Actually a Ruby code - easy to extent
Vagrantfile
Vagrant::Config.run do |config|
config.vm.box = “base_ubuntu"
end
Command line
$ vagrant help
vagrant init
vagrant up
vagrant halt
vagrant reload
vagrant suspend
vagrant resume
vagrant destroy
vagrant provision
vagrant package
…
• Isolation (project and version
specific files only)
• Repeatable (snapshots, copies,
automation)
• Guarantees (same OS different day)
Chef is an open-source systems
integration framework built specifically
for automating the cloud (opscode.com/
chef)
Puppet is a system for automating
system administration tasks, works
on most UNIX-like operating
systems (puppetlabs.com)
How Chef/Puppet works
• You create recipes
• Recipes describe desired system state
• Then you manage your server park
• Safely
• Repeatedly
• In automated way
Provisioning in Vagrant
• Use Chef, Puppet, bash scripts, etc. to
manage VM configuration
• Repeatable!
• Use same tools for production - Chef,
Puppet
Vagrant + Chef
Vagrant::Config.run do |config|
config.vm.box = “base_ubuntu_box"
config.vm.provision :chef_solo do |chef|
chef.recipe_url = "http://server/cookbooks.tar.gz"
chef.add_recipe(“aptitude")
chef.add_recipe(“apache2")
chef.add_recipe(“mysql-server")
chef.add_recipe(“python27")
chef.add_recipe(“pip")
end
end
Networking
• Assign IP to VM in Vagrantfile
• Forward ports
• Access VM from your browser
Networking
Vagrant::Config.run do |config|
config.vm.box = "lucid32"
config.vm.network("10.20.30.49")
config.vm.forward_port("web", 80, 4567)
end
• Isolation (project and version
specific files only)
• Repeatable (snapshots, copies,
automation)
• Guarantees (same OS different day)
Packaging
• When environment is stable
• Package and distribute prepared VMs
• Minimize setup time
• Isolation (project and version
specific files only)
• Repeatable (snapshots, copies,
automation)
• Guarantees (same OS different day)
Beyond that
• Multi-VM Vagrantfile
• DB
• AppServer
• WebServer
• Use vagrant as library
• Call API functions to do the tasks from
Ruby code
• Run custom SSH commands
Demo
http://youtu.be/O3-MNsowgHc
Demo
github.com/dreamiurg/timetest
•Django application
•Apache2 + MySQL on single VM
•Additional Python packages like used (South, annoying)
•DB migrations handled by South
•Deployment process
• Vagrant is used to create VM, install required packages and
configure services
• Fabric (fabfile.org) is used to populate DB, set up apache2 conf
and install Python modules
Prerequisites
$ <install python>
$ <install ruby>
$ <install virtualbox>
$ gem install vagrant
$ git clone http://server/project.git
$ cd project
Demo: create VM, deploy project
00:00:00 $ vagrant box add base http://server/base.box
... base box for VM is copied to your computer
00:02:00 $ vagrant up
... vagrant creates VM from base box, installs and configures
required packages – apt, apache2, mysql, python, git, …
00:05:00 $ fab deploy
... DB is populated with initial data, apache config updated, apache
restarted
00:07:00 $ curl http://localhost:8080/
... site is up and running.
Real world
Our experience
• 10+ projects during last 18 months
• From small (single webserver, no DB) to
medium (several webservers + load balancer,
master/slave DB replication)
• New developer start time reduced from 2-3
days to 1 hr on average
• Number of deployment/environment defects
reduced by 75%
Links
• http://vagrantup.com
• http://www.vagrantbox.es
• http://fabfile.org
• http://opscode.com
• http://puppetlabs.com
• http://github.com/dreamiurg/timetest
• http://www.vmware.com/files/pdf/VMware-SMB-
Survey.pdf
Dmitry Guyvoronsky
http://about.me/dreamiurg
dmitry.guyvoronsky@gmail.com
@dreamiurg

Virtualization with Vagrant (ua.pycon 2011)

  • 1.
  • 2.
    $ git clonegit://server.com/project.git $ run Ideal world
  • 4.
    $ git clonegit://server.com/project.git Real world
  • 5.
    $ git clonegit://server.com/project.git ... read help $ mkdir, cp, vim config.conf, configure, make, make install ... ask guru $ /project/hidden/blackmagic --fix_that --fix_this -- undocumented_parameter=MAGIC_CONST_42 ... %^ing @#$@#%@!!!! Real world
  • 6.
    $ git clonegit://server.com/project.git ... read help $ mkdir, cp, vim config.conf, configure, make, make install ... ask guru $ /project/hidden/blackmagic --fix_that --fix_this -- undocumented_parameter=MAGIC_CONST_42 ... %^ing @#$@#%@!!!! $ run Real world
  • 8.
  • 9.
  • 10.
    Problems • No isolation(oh, wrong binary on path...?) • Not repeatable (that README ain’t gonna run itself!) • No guarantees (“..but it works on my computer!”)
  • 11.
  • 13.
    Operating System Music WebServer DBAppServer IMEditor BrowserUser Space Virtualized OS
  • 14.
    Benefits • Isolation (projectand version specific files only) • Repeatable (snapshots, copies, automation) • Guarantees (same OS different day)
  • 15.
    Business benefits • Reducecosts (less hardware, lower energy consumption) • Backup and data protection • Application availability • Less time to respond to changing business needs • Less time spent on routine tasks • Better IT department perception
  • 16.
    Time spend onroutine tasks according to VMware small & medium business survey
  • 17.
    Why haven’t webeen doing this for a long time? • Big companies have been • Amazon EC2 • Microsoft • Google • Only recently possible on local machines • Cheat hardware • Cheap RAM • Desktop virtualization API
  • 18.
    Ok, let’s govirtual. Now what?
  • 19.
    Vagrant is atool for building and distributing virtualized development environments (vagrantup.com)
  • 20.
    VirtualBox is cross-platformvirtualization software that works on all the major platforms (virtualbox.org)
  • 21.
    High level overview •Describe environment with Vagrantfiles • Command line interface to manage VM lifecycle • Automate provisioning and configuration • Share folder with host OS via NFS • Provide SSH access to guest OS • Network multiple VMs • Package and distribute VMs
  • 22.
    Vagrant boxes • Box= pre-packaged VM environment (~OS image) • A number of ready to use boxes @ http:// vagrantbox.es • Ubuntu 32 / 64 • CentOS • Solaris • RH
  • 23.
    Vagrantfile • Vagrantfile isto Vagrant as Makefile is to Make • Describes VM in code • One Vagrantfile per project (can contain multiple VMs) • Commit it to version control • Actually a Ruby code - easy to extent
  • 24.
  • 25.
    Command line $ vagranthelp vagrant init vagrant up vagrant halt vagrant reload vagrant suspend vagrant resume vagrant destroy vagrant provision vagrant package …
  • 27.
    • Isolation (projectand version specific files only) • Repeatable (snapshots, copies, automation) • Guarantees (same OS different day)
  • 28.
    Chef is anopen-source systems integration framework built specifically for automating the cloud (opscode.com/ chef)
  • 29.
    Puppet is asystem for automating system administration tasks, works on most UNIX-like operating systems (puppetlabs.com)
  • 30.
    How Chef/Puppet works •You create recipes • Recipes describe desired system state • Then you manage your server park • Safely • Repeatedly • In automated way
  • 32.
    Provisioning in Vagrant •Use Chef, Puppet, bash scripts, etc. to manage VM configuration • Repeatable! • Use same tools for production - Chef, Puppet
  • 33.
    Vagrant + Chef Vagrant::Config.rundo |config| config.vm.box = “base_ubuntu_box" config.vm.provision :chef_solo do |chef| chef.recipe_url = "http://server/cookbooks.tar.gz" chef.add_recipe(“aptitude") chef.add_recipe(“apache2") chef.add_recipe(“mysql-server") chef.add_recipe(“python27") chef.add_recipe(“pip") end end
  • 34.
    Networking • Assign IPto VM in Vagrantfile • Forward ports • Access VM from your browser
  • 35.
    Networking Vagrant::Config.run do |config| config.vm.box= "lucid32" config.vm.network("10.20.30.49") config.vm.forward_port("web", 80, 4567) end
  • 36.
    • Isolation (projectand version specific files only) • Repeatable (snapshots, copies, automation) • Guarantees (same OS different day)
  • 37.
    Packaging • When environmentis stable • Package and distribute prepared VMs • Minimize setup time
  • 39.
    • Isolation (projectand version specific files only) • Repeatable (snapshots, copies, automation) • Guarantees (same OS different day)
  • 40.
    Beyond that • Multi-VMVagrantfile • DB • AppServer • WebServer • Use vagrant as library • Call API functions to do the tasks from Ruby code • Run custom SSH commands
  • 41.
  • 42.
    Demo github.com/dreamiurg/timetest •Django application •Apache2 +MySQL on single VM •Additional Python packages like used (South, annoying) •DB migrations handled by South •Deployment process • Vagrant is used to create VM, install required packages and configure services • Fabric (fabfile.org) is used to populate DB, set up apache2 conf and install Python modules
  • 43.
    Prerequisites $ <install python> $<install ruby> $ <install virtualbox> $ gem install vagrant $ git clone http://server/project.git $ cd project
  • 44.
    Demo: create VM,deploy project 00:00:00 $ vagrant box add base http://server/base.box ... base box for VM is copied to your computer 00:02:00 $ vagrant up ... vagrant creates VM from base box, installs and configures required packages – apt, apache2, mysql, python, git, … 00:05:00 $ fab deploy ... DB is populated with initial data, apache config updated, apache restarted 00:07:00 $ curl http://localhost:8080/ ... site is up and running.
  • 45.
  • 46.
    Our experience • 10+projects during last 18 months • From small (single webserver, no DB) to medium (several webservers + load balancer, master/slave DB replication) • New developer start time reduced from 2-3 days to 1 hr on average • Number of deployment/environment defects reduced by 75%
  • 48.
    Links • http://vagrantup.com • http://www.vagrantbox.es •http://fabfile.org • http://opscode.com • http://puppetlabs.com • http://github.com/dreamiurg/timetest • http://www.vmware.com/files/pdf/VMware-SMB- Survey.pdf
  • 49.