Vagrant to-aws-flow

Vagrant to-aws-flow

• 1.
  Vagrant to AWSFlow Delivered by Colin Johnson, BlueChipTek (@bluechiptek)
• 2.
  Who am I? •DevOps Engineer and Occasional Developer • Certifications: • AWS Certified Solutions Architect – Associate Level (Number 224) • AWS Certified SysOps Administrator – Associate Level (Number 2564) • Worked with a number of Bay Area companies: inPowered, Jive Software, Lyft, NetShelter, Wingz • Contributed VPC and Testing Support for boto_* Modules in SaltStack Project
• 3.
  Goals for thisPresentation • Understand benefits and pain points of cloud, local and vagrant based development • Describe a development flow that combines vagrant and AWS to create a: • consistent environments for all developers • consistent environment from development to production • help organizations move fast – if they are not already doing this • integrate nearly flawlessly with AWS • Ease Development <-> Production Software Deployment
• 4.
  Why Vagrant toAWS Flow? • Benefits for Everyone: • Really Fast. Minutes to get a development environment setup. • Really Fast. No lag to test code changes. • Really Consistent. More similar to production than traditional “local” development. • Easy. Developers just type “vagrant ssh” • Developers • Develop Offline • Provision Infrastructure on Own • DevOps / Operations • No need to be involved in development or environment setup. • QA • Test Offline. • Automate Testing.
• 5.
  What does Vagrantdo? • Simple Use Case: • Allows local development. • Offline? No Problem. • Ensures consistency. • You use a Mac, your co-worker uses Windows but you want to run your app on Ubuntu? No Problem. • Advanced Use Cases: • Allows local development of an entire “dev” or “qa” application. • Your application depends on MySQL, redis and an application developed by another group as well? No problem. Run them all locally. • Run tests. • Need to run tests quickly but don’t want to automate infrastructure? No problem. Vagrant up. Test. Vagrant destroy.
• 6.
  Experts at inPowered •Problem - developers loved AWS, but wanted to develop locally. • Developers couldn’t develop offline • Developers had to “share” a single Development environment • Developers were developing on OS X whereas production was Amazon Linux • Developers were struggling to access Redis/MySQL and other dependencies • Solution: • Vagrant + AWS Flow: • Develop in Vagrant, Ship to AWS! • Ubuntu 15.04 for both Vagrant and AWS • Same configuration management for both Vagrant and AWS • Vagrant: • offline development • per developer environments • all dependencies were run or “mocked” locally
• 7.
  Salt boto_* ModuleDevelopment at Lyft • Project: • adding VPC support and testing to salt’s boto_* modules (https://github.com/saltstack/salt/) • Problems: • Needed to do testing of AWS automation (approximately 200+ tests), but building test environment load balancers, security groups, etc was taking too long as did calls • Preferred testing locally • Preferred testing with multiple Operating Systems • Solution: • Two Vagrant instances for Running Salt – Ubuntu 12.04 and 14.04 – fresh environment for every test • Moto* for mocking AWS calls (https://github.com/spulec/moto) • Tests could be run by simply typing “vagrant up” and results could be generated and reviewed
• 8.
  Vagrant Development vsCloud Development • One Environment for each Developer on Cloud Platform • Administration burden – DevOps either has to Build or Automate • I used to spend 10 hours/week setting up environments for new Developers • Inefficient Spend • Connectivity and Security requirements extends to uncontrolled “Development” environments • Shared Development on Cloud Platform • Sharing? You must be kidding. • Who “owns” the server? Your feature, my configuration change and someone else’s change are all mixed together. Testing is going to be so fun! • AWS / Cloud Development does typically offer all dependencies and services.
• 9.
  Vagrant Development vsCloud Development: Act I – Internet Access Required • Developer 1: • Going to be on vacation for the next week. Probably limited Internet access. • Manager: • No problem. Can you get the new “every metric for every user action feature” shipped on Monday? • Developer 1: • I’ll try, but I can’t really test my code in without Internet.
• 10.
  Vagrant Development vsCloud Development: Act II – Access and Management Headaches • DevOps Engineer: • It looks like port 22 and port 80 in all our development environments are open. Google just crawled all our development sites. • Manager: • Do you know why all the ports are open? • DevOps Engineer: • The docs for the webserver project say “to ssh in or do web testing, go to the security group and open ports 22 and 80 to 0.0.0.0/0” • Manager: • Do you know why? • DevOps Engineer: • I think the Developers get tired of waiting for VPN credentials to be issued.
• 11.
  Vagrant Development vsLocal Development • Match local and production environments • Local environment (OS X, Windows or *nix) may not match production • Faster Environment Standup • Run same configuration management as production! • Develop Offline / Complete infrastructure available • Run your app and all dependencies
• 12.
  Vagrant Dev vsLocal Dev: Act I – Dev/Prod Inconsistency • Developer: • It worked on my machine. • DevOps: • So, it worked on OS X 10.9.2, nginx preview release 1.9.11 and MariaDB. • Developer: • Yes. I also had to install SciPy and NumPy. • DevOps: • Anything else? • Developer: • I think that is it. Make sure to use SciPy 0.17.
• 13.
  Vagrant Development vsLocal Development Act II – Local Development Inconsistency • Developer 1: • It worked on my machine. OS X 10.9.2 with nginx 1.9.11 preview release… • Developer 2: • I’m running Windows. 10. • Developer 1: • Damn. Get a Mac. Everyone else uses them. • Developer 3: • You should use Linux. Same as production. Except for MariaDB instead of MySQL. But I don’t think that’s a problem.
• 14.
  Vagrant Development vsLocal Development Act III – Running the Application • Developer 1: • I’ve been trying to get nginx 1.9.11 preview installed on my Mac for the last four hours • Developer 2: • Use homebrew. I found an article online somewhere. • Developer 2: • Here’s a link: http://www.robertmulley.com/tutorial/nginx-install-and-setup- mac-os-x-yosemite/ • Developer 1: • Link me for MongoDB, InfluxDB and Redis installs as well :-)
• 15.
  Vagrant to AWSFlow: • Standup Development in Minutes • Easy ssh Access • Easy file access • Run local mocks • No sharing environments!
• 16.
  Vagrant + AWS:How To • Use same OS in Vagrant as AWS • Separate and Make Modular Each “Concern” • Infrastructure Provisioning • Configuration Management • Software Deployment • Avoid or Be Aware of Platform Specific Technologies • Vagrant synced_folder = not available in AWS – you can still use for software deployment • AWS IAM Profile = not available in Vagrant/Virtualbox – you can still use for software deployment
• 17.
  Vagrant Setup • InfrastructureProvisioning: • Run “vagrant up” • Uses a Vagrantfile in root directory of project • Vagrantfile defines: • CPU, Memory • Operating System – use same as AWS! • A “Provisioner” – which runs Configuration Management • Configuration Management (aka “Provisioner”) • I prefer shell as “glue code” • Use same Configuration Management as AWS - Ansible, Chef, Puppet, Salt, etc. • Software Deployment • In Vagrant, use a “synced directory”– then changes made by developers are manifest in real-time near real-time
• 18.
  AWS Setup • InfrastructureProvisioning: • Run “cfn-create-stack” • Creates an Auto Scaling Group and Launch Configuration with user-data • Use same OS as Vagrant! • Launch Configuration calls Configuration Management • Configuration Management (aka “Provisioner”) • I prefer shell as “glue code” • Use same Configuration Management as Vagrant - Ansible, Chef, Puppet, Salt, etc. • Software Deployment • Use whatever you’d like…
• 19.
  Vagrant to AWSFlow - Diagram Whatever you’d like… - Ansible - Chef - Other - Puppet - SaltStack - Shell Script(s)? Just run same thing on both AWS and Vagrant! Vagrant AWS / CloudFormation Creates Instance(s) Software Deployment Infrastructure Provisioning Configuration Management Vagrant - “sync_dir” AWS Whatever you’d like: - AWS Code Deploy - Fabric - Git Push - Artifact and Sync
• 20.
  Demo In the next15 minutes we will: 1. Standup local development infrastructure on Ubuntu. 1. And look at a Vagrantfile. 2. Standup identical* production infrastructure in AWS. 1. And look at a CloudFormation file. 3. Deploy code to AWS and run the same code in Dev/Prod. 1. The deployment tool is a bit of a hack, btw…