OSU DevOps Bootcamp Documentation Release 0.0.1 OSU OSL OSU LUG October 11, 2014 Contents 1 2 Contents: 1.1 Get Involved . . . . . . . . . . . . . . . . . . . . . . . 1.2 Policies . . . . . . . . . . . . . . . . . . . . . . . . . . 1.3 Curriculum . . . . . . . . . . . . . . . . . . . . . . . . 1.4 BootCamp Guide . . . . . . . . . . . . . . . . . . . . . 1.5 Recommended Resources . . . . . . . . . . . . . . . . 1.6 FAQ . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.7 What is DevOps? . . . . . . . . . . . . . . . . . . . . . 1.8 Purpose of DevOps BootCamp . . . . . . . . . . . . . . 1.9 Vagrant . . . . . . . . . . . . . . . . . . . . . . . . . . 1.10 DevOps Bootcamp Introduction . . . . . . . . . . . . . 1.11 Lesson 1: The Very Basics . . . . . . . . . . . . . . . . 1.12 Lesson 2: Single System Fundamentals . . . . . . . . . 1.13 Lesson 2: Homework . . . . . . . . . . . . . . . . . . . 1.14 Lesson 3: Editors & Version Control . . . . . . . . . . 1.15 Lesson 3: Intro to Git . . . . . . . . . . . . . . . . . . 1.16 Lesson 3: GitHub . . . . . . . . . . . . . . . . . . . . 1.17 Lesson 4: Scripting, Troubleshooting, & Workflow . . . 1.18 Lesson 5: Web Applications . . . . . . . . . . . . . . . 1.19 Lesson 6: Boot and the Filesystem Hierarchy . . . . . . 1.20 Lesson 7: Databases . . . . . . . . . . . . . . . . . . . 1.21 Lesson 8: Security & Authentication . . . . . . . . . . 1.22 Lesson 8: Web application security . . . . . . . . . . . 1.23 Lesson 9: Networking . . . . . . . . . . . . . . . . . . 1.24 Lesson 10: Networking part 2 . . . . . . . . . . . . . . 1.25 Lesson 11: Devops & Configuration Management Intro 1.26 Lesson 12: Configuration Management . . . . . . . . . 1.27 Lesson 13: Contributing to Open Source . . . . . . . . 1.28 Lesson 14: Review . . . . . . . . . . . . . . . . . . . . 1.29 Lesson 16: Email . . . . . . . . . . . . . . . . . . . . . Indices and tables Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 3 3 5 7 7 8 9 9 10 12 15 28 39 39 46 50 54 63 68 81 88 100 104 111 125 134 142 147 149 155 157 i ii OSU DevOps Bootcamp Documentation, Release 0.0.1 The OSL is kicking off DevOps BootCamp this year with the brand new DevOps DayCamp! DevOps DayCamp is a dual-track day with one track to help inexperienced attendees get started with DevOps, as well as a second track comprised of a hands-on hackathon with educational sessions throughout the day for the more advanced DevOps crowd. Advanced sessions will be given by industry professionals and will include Ansible, Travis CI and Docker. For more information, checkout http://devopsbootcamp.osuosl.org/daycamp/. DevOps BootCamp is an OSU Open Source Lab and OSU Linux Users Group program that aims to boost our impact and outreach while shrinking the skills gap for OSU students interested in DevOps. The OSU Open Source Lab provides quality hosting for more than 160 open source projects. Along with delivering over 460 terabytes of open source data a month, the OSL employs and mentors students to be expert systems administrators and software developers ready to enter the industry. The OSL encounters more demand for qualified students than the employees graduating each year can supply. We’d like to give back to OSU and the wider open source community by training interested students from all disciplines in the real-world skills necessary for success as software developers and systems engineers. To this end, we’ll be offering an extracurricular training program with 2 to 3 sessions per month through the school year, in which current OSL students and software professionals will teach all topics that are essential for success. Although hands-on labs will be offered later in the program, no initial familiarity with software development or Linux will be assumed. This program is based on the PSU Braindump, where for 21 years they have been turning PSU students into sysadmins who have rapidly been flowing into the growing number of startups in Portland. The DevOps BootCamp program will be dynamically shaping itself based on the feedback we get from interested students. Contents 1 OSU DevOps Bootcamp Documentation, Release 0.0.1 2 Contents CHAPTER 1 Contents: 1.1 Get Involved 1.1.1 Mailing list Join the mailing list for updates. 1.1.2 IRC Join us on irc.freenode.net in #devopsbootcamp (students will be setting up an IRC network for the program in a later lesson). 1.1.3 Website & Curriculum If you’d like to help edit this site, email devopsbootcamp or ping anyone in #devopsbootcamp on Freenode with your github username to get access to the web site repo. You’ll also want to learn the ReStructured Text markup language to edit the site, if you don’t already know it. 1.1.4 Meetings We typically meet weekly in KEC 1007 or 1005 depending on the room schedules. Please check back later for when our first meeting in the Fall will be. Recurring meeting times following the first meeting will be decided based on the availability of those who show up for the first meeting. Last year we met on Thursdays from 6-8pm. 1.2 Policies 1.2.1 “The Deal” You get: • Mentorship from students and professionals with advanced skills in software development and systems administration • Professional connections in the software industry 3 OSU DevOps Bootcamp Documentation, Release 0.0.1 • A welcoming environment to start learning in if you’ve always wanted to learn about software development and systems administration but have never been sure where to start • An opportunity to fill in any gaps in your knowledge if you’re a self-taught coder or sysadmin • The skills to build and deploy open source software or to contribute to existing projects Why we’re offering this: • The OSL gets a larger pool of candidates to recommend to companies interested in recruiting OSL students • The OSL gets to work with a wider variety of students, which fits better with its new status as part of the school of EECS • The open source community gets more contributors to its projects 1.2.2 Attendance We will schedule BootCamp meetings for a time that works for a majority of attendees. Each meeting’s curriculum will build on what you learned the previous session, so if you miss a meeting you’re responsible for studying the basics of its content on your own. To facilitate this, we will provide resources for independent study about each topic that we cover. BootCamp leaders will be available at times outside of the regular meetings to help answer any questions about the training program’s content. We will not spend class time reviewing material for those who skip a lecture. If you attend a lesson and don’t understand something from it, you’re welcome to ask in class, because others very likely have the same concern. But if you want to make up a class you skipped, it’s disrespectful to those who attended to spend their class time on questions which you could resolve on your own time. 1.2.3 Laptops As the course progresses, you will need a laptop. We hope and recommend that you will decide to set up your laptop to dual-boot to Linux as the course progresses, but but it’s not required. If you don’t own a laptop and are an OSU student you can check out a laptop from the OSU Library for at least 24 hours at a time. Realistically, as long as it’s new enough to boot from USB and connect to wireless networks, your laptop’s specifications don’t matter much. If it’s so new that its UEFI configuration prevents you from dual-booting with Linux, it will be powerful enough to run virtual machines. If it’s old enough to be unable to run VMs but still has wireless connectivity, we’ll teach you how to ssh to a remote server to perform more computationally intensive tasks. If you are not an OSU student and do not have access to a working laptop, contact the DevOps BootCamp organizers and we’ll see whether we can arrange to loan you something for meetings. 1.2.4 Target Audience Our goal is to make the DevOps BootCamp program accessible to students and community members from all different backgrounds. You have to want to learn, be willing to ask questions whenever you don’t understand something, and be open to making time to play with the cool tools/toys which we’ll be teaching you about. Fluency in English is strongly recommended, because it’s the language in which the course will be taught as well as the language of almost all open source technical documentation and code comments. 4 Chapter 1. Contents: OSU DevOps Bootcamp Documentation, Release 0.0.1 1.3 Curriculum The curriculum will be constantly evolving as we develop this course. The first unit will be an overview of what students will need to know in order to understand the deeper technical concepts which will be covered later. Here’s what we’re hoping to mentor and teach students about: 1. Linux Basics 2. Basic System Administration 3. Basic FOSS Development Methodologies 4. Base infrastructure services for any organization (DNS, Email, etc) 5. Building a mock infrastructure for a mock company from top to bottom Note: Note that the curriculum for future weeks is constantly under development. Lesson titles are included to provide an overview of what we’ll cover, but specific content is likely to get moved around between lessons as we refine the curriculum. Slide content will be linked here as soon as it’s drafted, but only finalized 1-2 weeks before the lessons when it’s presented. 1.3.1 Lesson 1: The Very Basics Shell, virtualbox+vagrant, IRC. • 6pm 11/7/2013, KEC1007. 34 people attended. • Lesson 1 Video • Lesson 1 Slides 1.3.2 Lesson 2: Single System Fundamentals File permissions, users, groups, and package management • 6pm 11/14/2013, KEC1001. 27 people showed up. • Lesson 2 Video • Lesson 2 Slides 1.3.3 Lesson 3: Editors & Git Vim, Emacs, and Git for version control • 6pm 11/21/2014, BEXL. 21 people showed up. • Part 1, text editors video • Part 2, version control video • Lesson 3 Slides 1.3. Curriculum 5 OSU DevOps Bootcamp Documentation, Release 0.0.1 1.3.4 Lesson 4: Scripting & Troubleshooting Python and Bash. Overview of troubleshooting/debugging skills. • 6pm 01/9/2014, KEC1007. • Scripting and Troubleshooting Video • Lesson 4 Slides 1.3.5 Lesson 5: Services & Deploying a Web App • 6pm 1/16/2014, KEC1007 • Services & Web App video 1/2 • Services & Web App video 2/2 • Lesson 5 Slides 1.3.6 Lesson 6: Boot Process & Filesystem Hierarchy • 6pm 1/30/3014 – 14 attendees; time conflicts with Intel and Reddit founder • Boot & Filesystem Video • Lesson 6 slides 1.3.7 Lesson 7: Databases • 6pm 2/13/2014 (moved from 2/6 due to snow) • Lesson 7 slides 1.3.8 Review of VM and App Setup • 6pm 2/20/2014 • No video, hands-on... 14 people showed up • Review Slides 1.3.9 Lesson 8: Security & Authentication • 6pm 2/27/2014 – 22 people present • Lesson 8 video • Lesson 8 slides 1.3.10 Lesson 9: Networking overview • 6pm 4/10/2014 • Lesson 9 video • Lesson 9 slides 6 Chapter 1. Contents: OSU DevOps Bootcamp Documentation, Release 0.0.1 1.3.11 Lesson 10: DNS • 6pm 5/1/2014 • Lesson 10 video • Lesson 10 slides 1.3.12 Lesson 11: Automation and DevOps • 6pm 5/8/2014 • Lesson 11 video • Lesson 11 slides 1.3.13 Lesson 12: Configuration Management • 6pm 5/15/2014 – 8 students, 10 people total • Lesson 12 video • Lesson 12 slides 1.3.14 Lesson 13: Open Source & Hands-On • 6pm 5/22/2014 • Lesson 13 Video • Lesson 13 slides 1.4 BootCamp Guide This is the DevOps BootCamp Guide. It will cover the various topics during the sessions in more detail. • Vagrant 1.4.1 Administrative Stuff • link for finding rooms 1.5 Recommended Resources 1.5.1 Scripting • The Advanced Bash Scripting Guide • Learn Python The Hard Way 1.4. BootCamp Guide 7 OSU DevOps Bootcamp Documentation, Release 0.0.1 1.5.2 Systems Administration Tutorials • LinuxTraining – This is the resource which the OSL uses to quickly train systems administrators. The whole book is here. • OpsSchool is working on a tutorial somewhat like DevOps BootCamp, but it’s currently incomplete. Consider contributing to it! • Linux Command Line Tips: Become a Master is a quick start guide to get you going with bash 1.5.3 Open Source The following resources are used as textbooks in Carlos Jensen’s CS419 Open Source class: • The Cathedral and the Bazaar (Eric S. Raymond) • Producing Open Source Software (Karl Fogel) • Open Advice (Lydia Pintscher) 1.5.4 Culture • The Jargon File • ESR’s guide to asking good questions • Geek Feminism and Systers are worth looking into if you enjoy discussing diversity or IT’s lack thereof. 1.6 FAQ Email questions to [email protected] if they aren’t answered here! 1.6.1 Is this a for-credit class? No, although the curriculum is based on CS312, Linux System Administration, which hasn’t been offered for several years. 1.6.2 What does it cost? Financially, it’s free. However, to benefit from the hands-on learning opportunities that we’ll be offering, you should plan to commit 5 or more hours per week to attending lectures and playing with the tools and technologies that the lectures will discuss. 1.6.3 I don’t know about computers! Can I still join? YES! If you already knew all this stuff, you’d be teaching rather than studying it. If you think you don’t know enough, try anyway – in the worst case you’ll discover that your self-assessment was correct, and in the best case you’ll become a successful open source contributor. 8 Chapter 1. Contents: OSU DevOps Bootcamp Documentation, Release 0.0.1 1.6.4 When will meetings be? Weekday evenings, with a day of the week being chosen during the first meeting. We try to choose a day that works best for most students but it’s always difficult to find something that works for everyone. For example, last year we had meetings on Thursdays. 1.6.5 Is this only for students? No, anyone from the community can participate. Parts of the earlier lessons will assume access to student resources, but we can work with non-students on an individual basis to get them access to equivalent tools. Community members can use the Visitor wifi network when on campus. 1.6.6 What’s the difference between BootCamp and regular LUG meetings? Regular LUG meetings are usually targeted toward an intermediate to expert audience, and make no attempt to comprehensively teach a set of skills across multiple meetings. We might teach about version control one week, privacy tools the next, and a new open source project by a local company the next. DevOps BootCamp will provide continuity and comprehensive open source education that’s outside the scope of regular LUG meetings. As you progress through Bootcamp, you’ll understand more and more of the technical content of LUG talks. 1.6.7 I wasn’t able to participate during Fall term. Can I still join? Yes! Please read through lessons 1-3 (linked from the curriculum page) and ask in IRC if you have trouble following the instructions for them. Catch up on that content and you’ll be ready to join in Winter term. 1.6.8 I have a question? Ask it on IRC if you have that set up, or email [email protected] or the mailing list. 1.7 What is DevOps? DevOps (a portmanteau of software development and server operations) represents the integration between software development and operations engineering that has become increasingly necessary with the emergence of cloud computing. In essense, the jobs are no longer mutually exclusive- developers need more operations-based knowledge in order to understand how their application will be deployed, test locally and preemptively address potential security risks, while systems administrators need to know developer-based materials in order to design infrastructure optimally to fit an application’s needs without wasting capacity and to troubleshoot issues with the increasingly complex software cycle. Additionally, site reliability engineering and many modern security roles require a background with a balance between development and operations. 1.8 Purpose of DevOps BootCamp 1.8.1 From front page: DevOps BootCamp is an OSU Open Source Lab and OSU Linux Users Group program aimed at boosting our impact and outreach while shrinking the skills gap for OSU students interested in DevOps. 1.7. What is DevOps? 9 OSU DevOps Bootcamp Documentation, Release 0.0.1 1.8.2 From FAQ: DevOps BootCamp will provide continuity and comprehensive open source education that’s outside the scope of regular LUG meetings. 1.8.3 “The Deal” Students get: • Mentorship from students and professionals with advanced skills in software development and systems administration • Professional connections in the software industry • A welcoming environment to start learning in, if you’ve always wanted to learn about software development and systems administration but never been sure where to start • An opportunity to fill in any gaps in your knowledge, if you’re a self-taught coder or sysadmin • The skills to build and deploy open source software, or contribute to existing projects Why we’re offering this: • The OSL gets a larger pool of candidates to recommend to companies interested in recruiting OSL students • The OSL gets to work with a wider variety of students, which fits better with its new status as part of the school of EECS • The open source community gets more contributors to its projects 1.8.4 Target Audience Our goal is to make the DevOps BootCamp program accessible to students and community members from all different backgrounds. You have to want to learn, and be willing to ask questions whenever you don’t understand something, and be open to making time to play with the cool tools/toys which we’ll be teaching you about. 1.9 Vagrant 1.9.1 Trying Linux as a virtual machine with Vagrant In order to create a stable learning environment, we’re going to have everyone use a Devop tool called Vagrant. Vagrant is typically used to help assist both developers and system engineers in ensuring that their application and system deployments work predictably. For our purposes we’re going to use it as an easy way for new people to get to a Linux prompt quickly with no fear of breaking their system. Vagrant basically interfaces with hypervisors such as VirtualBox with a unified command line interface. This makes it portable between Mac, Windows and even Linux! Vagrant itself is just a collection of ruby scripts while Virtualbox does all the virtual machine magic. We will go in more depth with how to install Linux manually, but for now we’ve done all the hard work and have create pre-made Linux images. 10 Chapter 1. Contents: OSU DevOps Bootcamp Documentation, Release 0.0.1 1.9.2 Installing Virtualbox VirtualBox is a free and open source virtual machine manager that you can install on a variety of operating system platforms such as Mac and Windows. 1. Go to the VirtualBox download page and download the latest copy of VirtualBox. 2. Go through the installer and use the default settings when prompted. 3. Once completed, see if you can open up the VirtualBox Manager 4. If it asks you to download the latest extensions, go ahead and do that. 1.9.3 Installing Vagrant Vagrant is a free and open source tool used to build complete devops environments. 1. Go to the Vagrant download page and choose the newest version available. 2. Choose all the defaults during the install if it asks you any questions 1.9.4 Testing Vagrant 1. If you’re on a Mac, open up the Terminal. If you’re on windows, open up a DOS prompt (cmd.exe). 2. Type: vagrant status 3. If there are no errors then Vagrant was installed correctly! 1.9.5 Cloning the Vagrant Repo We keep all of our vagrant configuration in a git version controlled repository. If you don’t have git installed, please go download git and install it. Once you have downloaded and installed git, go to a folder where you want to keep the repository and type the following: git clone https://github.com/DevOpsBootcamp/devopsbootcamp-vagrant.git If installing git is too difficult, you can also download a zip file containing the repository. We will teach you more about git later in the year, so don’t worry! 1.9.6 Using Vagrant Now that we have the entire environment working, lets get to playing with Linux! Open the terminal and get to the directory where the devopsbootcamp-vagrant repo is at. You can run the following commands: # Start the VM vagrant up # SSH into the VM vagrant ssh # Stop and halt the VM vagrant halt 1.9. Vagrant 11 OSU DevOps Bootcamp Documentation, Release 0.0.1 # Destroy and remove the VM vagrant destroy Also check out the Vagrant Documentation for more information. You can also always type -h to find out more information about a command. 1.9.7 Troubleshooting Depending on how your laptop or computer is confused in the BIOS, you may or may not run into issues getting Vagrant and VirtualBox to work properly. Booting the VM takes forever and never completes The most common problem is when you type vagrant up the system just waits and waits and never finishes the boot process. The most likely cause is because your system doesn’t have hardware virtualization enabled in the BIOS. You can either enable the feature or you can disable the requirement in the Vagrantfile. 1. Enabling Hardware Virtualization Reboot your machine and press the appropriate buttons to get into the BIOS or Setup screen. This is usually done by hitting a combination of the escape or function keys. Once you get into the BIOS, find a screen that has some options for the CPU. Each BIOS is different but you are basically trying to find a feature called Hardware Virtualization. Once you find it, go ahead and enable it and reboot your system. 2. Disabling HW Virtualization in Vagrant Open the Vagrantfile with your favorite editor of choice. Find the line that says hwvirtex and remove the # from the front of the line. Now run vagrant destroy -f and then vagrant up again. This should boot the VM properly now. 1.10 DevOps Bootcamp Introduction OSU OSL & OSU LUG http://devopsbootcamp.osuosl.org 1.10.1 What’s DevOps? Software Development + Operations Engineering 1.10.2 What’s BootCamp? • New this year • New this year • Free education program – Mentorship – Apprenticeship 12 Chapter 1. Contents: OSU DevOps Bootcamp Documentation, Release 0.0.1 – Hands-on training • NOT: – OSU class for credit – student job 1.10.3 Why are we doing this? • OSL’s move to EECS • Bridge the “Skills Gap” • Demand from companies for students • Build community 1.10.4 What you’ll do: • Learn: – Linux systems – Networking – Software development – ...and the other skills you’ll need • Build: 1.10. DevOps Bootcamp Introduction 13 OSU DevOps Bootcamp Documentation, Release 0.0.1 – Duplicate internet infrastructure – Graduate to helping with OSL tasks * work with real open source projects * solve real problems 1.10.5 Can you do it? • Probably! • No background knowledge needed • Time commitment – Attend lectures – Play with what we teach you – You get out what you put in • If in doubt. . . try anyway! 1.10.6 Career Paths: • Hands-on learning & OSL reputation: – Internships – Industry connections – Start career with mid-level/high-level experience • Justin – community experience builds skills, professional relationships – Linux skills helped in grad school * skills learned there helped land first sysadmin job – having your own �playground’ to experiment with 1.10.7 OSL Hiring: • Students need basic skills to join – Systems engineers – Developers – Media • Typically 2-3 years employment • Alumni – Google, Rackspace, Intel, Microsoft, Mozilla, Facebook 14 Chapter 1. Contents: OSU DevOps Bootcamp Documentation, Release 0.0.1 1.10.8 What next? • Find our site (http://devopsbootcamp.osuosl.org) • Fill out registration with times available • Join mailing list • Tell us about yourself – Who are you – What do you want to learn here – Any background in devops? 1.11 Lesson 1: The Very Basics 1.11.1 Today’s Agenda • Getting (to) Linux • The Terminal & Shell – Scripts, file paths, special characters • Productivity tricks – Getting help 1.11. Lesson 1: The Very Basics 15 OSU DevOps Bootcamp Documentation, Release 0.0.1 • IRC – Vocabulary – Get connected – Etiquette 1.11.2 A note about notation • Variables – $varname – <varname> • Shell prompt – $ – �literal stuff in backticks� • foo, bar, baz, username, etc. 1.11.3 How to get (to) Linux • How many have it already installed? • Install VM or dual-boot • When stuck on Windows, use PuTTy: • Students: ssh <onidusername>@shell.onid.oregonstate.edu • flip{1-3} are Engineering servers; less reliable 1.11.4 Trying Linux on a Virtual Machine Virtual machines act as a full system on a physical machine • Hypervisors: – VirtualBox (free) 16 Chapter 1. Contents: OSU DevOps Bootcamp Documentation, Release 0.0.1 1.11. Lesson 1: The Very Basics 17 OSU DevOps Bootcamp Documentation, Release 0.0.1 – VMWare (mostly free) – KVM (Linux only hosts) – Parallels • Public Cloud Virtual Machines – Amazon EC2, Rackspace Cloud, Google Compute Engine, etc • Easy way to test without breaking your machine! 1.11.5 Installing Linux on Virtualbox Note: Try other distributions if you like to see what’s different. Debian is a great next step to try out. 1. Download and install: https://www.virtualbox.org/wiki/Downloads 2. Grab the latest minimal ISO: http://centos.osuosl.org/6/isos/x86_64/ 3. Create VM (a) New -> Name “CentOS” -> Default Ram -> Default Disk settings (b) Settings -> Storage -> Empty -> CD/DVD Drive -> Select ISO (c) Start -> press enter -> Skip media check 4. \o/ 1.11.6 Vagrant & VirtualBox Note: We’re using CentOS as our base image for now but will use Debian later. You can see the gui by uncommenting the line in the Vagrantfile. • Vagrant is a tool used with Virtualbox (and other) platforms • Make a reproducible pre-installed Linux environment • Download and install: http://www.vagrantup.com/ • Clone our repo, start and access the vm: # clone git clone https://github.com/DevOpsBootcamp/devopsbootcamp-vagrant.git # start up cd devopsbootcamp-vagrant vagrant up # access vm vagrant ssh 1.11.7 Vagrant cheat sheet Note: We’ll get into more detail later in how you can access ports on your VMs and other use cases. 18 Chapter 1. Contents: OSU DevOps Bootcamp Documentation, Release 0.0.1 # start vagrant up # stop vagrant halt # destroy (remove vm) vagrant destroy # ssh to the vm vagrant ssh Also check out the Vagrant Documentation for more information. 1.11.8 The Terminal • Used to mean the keyboard+monitor – Now that’s a crash cart • Terminal emulator • Shell: Use bash; others include csh, zsh, tsch – ~/.bashrc 1.11.9 Basic Shell Commands Note: Explain architecture built in commands vs. external binaries Demo commands Directory movement and file manipulation: Cd, pwd, ls, rm, mv, touch User info id, whoami, w 1.11. Lesson 1: The Very Basics 19 OSU DevOps Bootcamp Documentation, Release 0.0.1 20 Chapter 1. Contents: OSU DevOps Bootcamp Documentation, Release 0.0.1 Pipes redirection (pipe.txt, redirect.txt) Special variables $?, $$ (pid.sh), !!, !*, !$ • ls, cd, cat, echo • invoke/call an installed program • get help: man <program> test@x230 ~ $ tree . -- Documents | -- Code | | -- scripts | | -- test.sh | -- School | -- Work -- Pictures -- manatee.gif -- turtle.png 6 directories, 5 files 1.11.10 Invoking a script Note: Permissions discussed later. $ ls -l $ chmod +x $filename Arguments are extra information that you pass to a script or program when you call it. They tell it in more detail what you want to do. $ ls -a -l $ ls -al 1.11. Lesson 1: The Very Basics 21 OSU DevOps Bootcamp Documentation, Release 0.0.1 Why pass arguments on the command line rather than having an interactive mode? 1.11.11 File Paths • . means current directory • .. means parent directory • Tilde (~) means your homedir (/home/$username) • / separates directories (not \) • / is root directory, so ~ expands to /home/$username/ • current path appears in your prompt: I’m logged in as the user test on the machine named x230 test@x230 ~ $ ls Documents Pictures test@x230 ~ $ cd Documents/ test@x230 ~/Documents $ ls Code School Work test@x230 ~/Documents $ Note: root directory is not to be confused with a home directory for the root account 1.11.12 Special Characters • escape with \ to use them literally • # means a comment • ; allows multiple commands per line • !, ?, *, &&, & • Regular expressions (we’ll learn more later) 22 Chapter 1. Contents: OSU DevOps Bootcamp Documentation, Release 0.0.1 1.11.13 Type less • Reverse-i-search – ctrl+r then type command • aliases – ~/.bashrc • Tab completion Automation > Typing > Mouse 1.11.14 Help, get me out of here! • ctrl+c kills/quits • ctrl+d sends EOF (end-of-file) – also means logout • :q gets you out of Vi derivatives and man pages – esc - esc - :q if you changed modes • read what’s on your screen; it’ll help you 1.11.15 Knowledge Check 1.11. Lesson 1: The Very Basics 23 OSU DevOps Bootcamp Documentation, Release 0.0.1 test@x230 ~ $ tree . -- Documents | -- Code | | -- scripts | | -- test.sh | -- School | -- Work -- Pictures -- manatee.gif -- turtle.png 6 directories, 5 files • What user am I logged in as? • What command did I just run? • What is my current directory when I run that command? 1.11.16 More about Man Pages • the manual (rtfm): $ man <program> $ man man • use /phrase to search for phrase in the document; n for next match • else: $ <program> --help 1.11.17 Documentation Man pages, blogs you find by Googling, StackOverflow • Contribute to community – Correct it if it’s wrong – Remind them what newbies don’t know – Write your own • For your future self as well • Start now 1.11.18 Asking for help It’s okay to ask. 1. What should be happening? 2. What’s actually happening? 3. Google it 24 Chapter 1. Contents: OSU DevOps Bootcamp Documentation, Release 0.0.1 4. Skim the manuals of each component 5. Identify a friend, mentor, or IRC channel who could help 6. When they’re not busy, give them a quick synopsis of points 1 and 2, mentioning what possibilities you’ve ruled out by searching. Contributions = expertise + time Don’t waste experts’ time, but do build your expertise. 1.11.19 IRC • Internet Relay Chat • Very old (RFC 1459 May 1993) • Works on everything (no GUI needed) • The people you want to listen to are there 1.11.20 A Client Note: Switche to a terminal and show example Use irssi in screen # This step is optional, but persistent IRC is cool $ ssh <username>@<preferred shell host> # start Screen $ screen -S irc # start your client $ irssi # after ending ssh session, to get back: $ ssh <username>@<preferred shell host> $ screen -dr IRC 1.11.21 Networks /connect irc.freenode.net /nick <myawesomenickname> /msg nickserv register <password> <email> /nick <myawesomenickname> /msg nickserv identify <password> 1.11. Lesson 1: The Very Basics 25 OSU DevOps Bootcamp Documentation, Release 0.0.1 1.11.22 Channels /join #osu-lug /join #devopsbootcamp /list • tells all channels on network • Don’t do this on Freenode! /topic tells you the current channel’s topic /names tells you who’s here 1.11.23 Commands • take action with /me does thing� • everything else starting with / is a command /say $thing /join, /part, /whois <nick>, /msg, /help <command> Note that nothing shows up in the channel when you run a /whois command; it shows up either in your status buffer or your conversation with the person. 12:04 -!- _test_ [[email protected]] 12:04 -!- ircname : Example User 12:04 -!- channels : #ExampleChannel 12:04 -!- server : moorcock.freenode.net [TX, USA] 12:04 -!- hostname : c-50-137-46-63.hsd1.or.comcast.net 50.137.46.63 12:04 -!- idle : 0 days 0 hours 2 mins 38 secs [signon: Wed Nov 6 12:00:30 2013] 12:04 -!- End of WHOIS 1.11.24 Useful tricks • Tab-complete works on nicknames. use it. • Highlight when people say your name • Symbols are not part of names; they mark status in channel • Logging (expect it); �/set autolog on� • chanserv and nickserv are good bots to know – hamper is also a bot 1.11.25 Screen & Irssi Hints • Paste with ctrl+shift+v – PuTTY defaults to right-click to paste • to get back, screen -dr IRC • Can you use man screen to find out what the d and r flags mean? 26 Chapter 1. Contents: OSU DevOps Bootcamp Documentation, Release 0.0.1 SCREEN(1) SCREEN(1) NAME screen - screen manager with VT100/ANSI terminal emulation SYNOPSIS screen [ -options ] [ cmd [ args ] ] screen -r [[pid.]tty[.host]] screen -r sessionowner/[[pid.]tty[.host]] Manual page screen(1) line 1 (press h for help or q to quit) 1.11.26 Etiquette • Lurk more • Don’t ask to ask – Lure help out of hiding with tasty details of problem • Show that you’re worth helping • Read the topic – /topic – Output only shows up in your channel, not to everyone else • Pastebin code • Choose your nick carefully 1.11.27 Terminology • ping/pong • flapping • tail • hat • nick • netsplit • kick/ban/k-line • common emotes 1.11. Lesson 1: The Very Basics 27 OSU DevOps Bootcamp Documentation, Release 0.0.1 – o/ AND \o high fives – /me & means afk 1.11.28 Review • What’s Linux? • How do you open a terminal emulator? – this varies between window managers • I have the script test.py. How do I run it?? • How do you list all the files in the current directory? • Give 2 ways to change directory to your home directory. • How do you start an irc client? – How often should you need to start your IRC client? • How do you reconnect to a screen session? • Give an example of something which you should not do in IRC 1.12 Lesson 2: Single System Fundamentals Or, how to be a power user. 1.12.1 Housekeeping 10 minutes to make sure everyone’s Virtualbox instances are up and running Your opinions: • Want to meet during dead week vs. have last meeting of term next week? • Start wk1 of winter term? 28 Chapter 1. Contents: OSU DevOps Bootcamp Documentation, Release 0.0.1 1.12.2 Today’s Topics • What are files? – Permissions/users • What are user accounts? – User management – Root vs. normal – Groups • What are packages? – How to use package managers – Differences between Linux Distributions – Other package managers 1.12.3 What are users? • You, right now $ $ $ $ whoami who w id # # # # your username who is logged in? who is here and what are they doing? user ID, group ID, and groups you’re in • Not just people: Apache, Mailman, ntp 1.12.4 Users have • Username • UID • Group • Usually (but not always) password • Usually (but not always) home directory 1.12.5 Managing users $ # $ $ $ cat /etc/passwd username:x:UID:GID:GECOS:homedir:shell useradd $USER # vs adduser, the friendly Ubuntu version userdel $USER passwd # # $ $ GECOS: full name, office number and building, office phone extension, home phone number (General Electric Comprehensive Operating System) chfn # change GECOS information; only works sometimes finger # tells you someone’s GECOS info 1.12. Lesson 2: Single System Fundamentals 29 OSU DevOps Bootcamp Documentation, Release 0.0.1 1.12.6 Passwords • /etc/shadow, not /etc/passwd test@x230 ~ $ ls -l /etc/ | grep shadow -rw-r----- 1 root shadow 1503 Nov 12 17:37 shadow $ sudo su $ cat /etc/shadow daemon:*:15630:0:99999:7::: bin:*:15630:0:99999:7::: sys:*:15630:0:99999:7::: mail:*:15630:0:99999:7::: # name:hash:time last changed: min days between changes: max days # between changes:days to wait before expiry or disabling:day of # account expiry $ chage # change when a user’s password expires 1.12.7 Root/Superuser • UID 0 • sudo 1.12.8 Acting as another user $ $ $ $ su $USER su sudo su sudo su $USER # # # # become user, with become root, with use user password become $USER with THEIR password root’s password instead of root’s your password If someone has permissions errors: • Check that they or their group owns the files • Check that they have the flag +x to execute 1.12.9 What are groups? • Manage permissions for groups of users 30 Chapter 1. Contents: OSU DevOps Bootcamp Documentation, Release 0.0.1 1.12. Lesson 2: Single System Fundamentals 31 OSU DevOps Bootcamp Documentation, Release 0.0.1 $ $ $ $ groupadd usermod groupmod cat /etc/group root:x:0: daemon:x:1: bin:x:2: sys:x:3: adm:x:4: tty:x:5: # group name:password or placeholder:GID:member,member,member 1.12.10 Hands-On: Users and Groups Note: To give yourself sudo powers do the following: 1. Add your user to the wheel group using gpasswd. 2. As the root user, use visudo and uncomment this line: %wheel ALL=(ALL) ALL 3. Save the file and now you should have sudo! We’ll cover sudo in more depth at a later time. • Create a user on your system for yourself, with your preferred username • Give your user sudo powers • Use use su to get into your user account • Change your password • Create a directory called bootcamp in your home directory • Create a group called devops 1.12.11 What are files? • Nearly everything • Files have: – Owner – Permissions – inode – Size – Filename test@x230 ~ $ ls -il total 8 2884381 drwxrwxr-x 5 test test 4096 Nov 6 11:46 Documents 2629156 -rw-rw-r-- 1 test test 0 Nov 13 14:09 file.txt 2884382 drwxrwxr-x 2 test test 4096 Nov 6 13:22 Pictures 32 Chapter 1. Contents: OSU DevOps Bootcamp Documentation, Release 0.0.1 1.12.12 File extensions • .jpg, .txt, .doc • Really more of a recommendation – File contains information about its encoding $ file $FILENAME # tells you about the filetype test@x230 ~ $ file file.txt file.txt: ASCII text test@x230 ~ $ file squirrel.jpg squirrel.jpg: JPEG image data, JFIF standard 1.01 1.12.13 ls -l • First bit: type • Next 3: user • Next 3: group • Next 3: world • user & group $ ls -l drwxrwxr-x 5 test test 4096 Nov 6 11:46 Documents -rw-rw-r-- 1 test test 0 Nov 13 14:09 file.txt drwxrwxr-x 2 test test 4096 Nov 6 13:22 Pictures chmod and octal permissions +=====+========+=======+ | rwx | Binary | Octal | +=====+========+=======+ | --- | 000 | 0 | | --x | 001 | 1 | | -w- | 010 | 2 | | -wx | 011 | 3 | | r-- | 100 | 4 | | r-x | 101 | 5 | | rw- | 110 | 6 | | rwx | 111 | 7 | +=====+========+=======+ • u, g, o for user, group, other • -, +, = for remove, add, set • r, w, x for read, write, execute chown, chgrp user & group 1.12. Lesson 2: Single System Fundamentals 33 OSU DevOps Bootcamp Documentation, Release 0.0.1 # Change the owner of myfile to "root". $ chown root myfile # Likewise, but also change its group to "staff". $ chown root:staff myfile # Change the owner of /mydir and subfiles to "root". $ chown -hR root /mydir # Make the group devops own the bootcamp dir $ chgrp -R devops /home/$yourusername/bootcamp 1.12.14 Types of files drwxrwxr-x 5 test test 4096 Nov 6 11:46 Documents -rw-rw-r-1 test test 0 Nov 13 14:09 file.txt drwxrwxr-x 2 test test 4096 Nov 6 13:22 Pictures ---------------- ------- -------- ------------ ------------| | | | | | | | | | | File Name | | | | +--- Modification Time | | | +------------Size (in bytes) | | +----------------------Group | +-------------------------------Owner +---------------------------------------------File Permissions - is a normal file d is a directory b is a block device 1.12.15 ACLs • Access control lists • Not recommended; hard to maintain • Typically how other OSes manage permissions • Support depends on OS and filesystem 1.12.16 Hands-On: Files and Permissions $ touch foo # create empty file called foo • As root, create a file in /home/$yourusername/bootcamp • Who can do what to the file? • Make the devops group own the file • Make a file called allperms and give user, group, and world +rwx • Make more files and practice changing their permissions 34 Chapter 1. Contents: OSU DevOps Bootcamp Documentation, Release 0.0.1 1.12.17 Package Management Take care of installation and removal of software Core Functionality: • Install, Upgrade & uninstall packages easily • Resolve package dependencies • Install packages from a central repository • Search for information on installed packages and files • Pre-built binaries (usually) Popular Linux Package Managers • .deb / APT (used by Debian, Ubuntu) • .rpm / YUM (used by RedHat, CentOS, Fedora) 1.12.18 RPM & yum (RedHat, CentOS, Fedora) RPM Binary file format which includes metadata about the package and the application binaries as well. Yum RPM package manager used to query a central repository and resolve RPM package dependencies. Yum Commands (Redhat, CentOS, Fedora) We’ll use the tree package as an example below. # Searching for a package $ yum search tree # Information about a package $ yum info tree 1.12. Lesson 2: Single System Fundamentals 35 OSU DevOps Bootcamp Documentation, Release 0.0.1 # Installing a package $ yum install tree # Upgrade all packages to a newer version $ yum upgrade # Uninstalling a package $ yum remove tree # Cleaning the RPM database $ yum clean all RPM Commands Low level package management. No dependency checking or central repository. # Install an RPM file $ rpm -i tree-1.5.3-2.el6.x86_64.rpm # Upgrade an RPM file $ rpm -Uvh tree-1.5.3-3.el6.x86_64.rpm # Uninstall an RPM package $ rpm -e tree # Querying the RPM database $ rpm -qa tree # Listing all files in an RPM package $ rpm -ql tree 1.12.19 DPKG & Apt (Debian, Ubuntu) Deb Binary file format which includes metadata about the package and the application binaries as well. DPKG Low level package installer for the .deb file format. Does no package dependency resolution. Apt DPKG package manager used to query a central repository and resolve Deb package dependencies. Considered mostly a front-end to dpkg. 36 Chapter 1. Contents: OSU DevOps Bootcamp Documentation, Release 0.0.1 Apt Commands (Debian, Ubuntu) Note: You can also use aptitude as a front-end to dpkg instead of apt-get. # Update package cache database $ apt-get update # Searching for a package $ apt-cache search tree # Information about a package $ apt-cache show tree # Installing a package $ apt-get install tree # Upgrade all packages to a newer version $ apt-get upgrade $ apt-get dist-upgrade # Uninstalling a package $ apt-get remove tree $ apt-get purge tree Dpkg Commands Low level package management. No dependency checking or central repository. # Install or upgrade a DEB file $ dpkg -i tree_1.6.0-1_amd64.deb # Removing a DEB package $ dpkg -r tree # Purging a DEB package $ dpkg -P tree # Querying the DPKG database $ dpkg-query -l tree # Listing all files in a DEB package $ dpkg-query -L tree 1.12.20 Language-specific Package Managers • Languages sometimes have their own package management suite • Can be useful for using newer versions of packages • Examples – pip (Python) – rubygems (Ruby) – CPAN (Perl) 1.12. Lesson 2: Single System Fundamentals 37 OSU DevOps Bootcamp Documentation, Release 0.0.1 – cabal (Haskell) – npm (NodeJS) – ... and so on forever ... 1.12.21 Other Package Managers They each fill a specific niche and have their own pros and cons. • Portage (Gentoo) – Source based package installer • pacman (Arch Linux) • ZYpp / Zypper (SUSE) – Yet another RPM package manager 1.12.22 Installing from source • Download source tarball, run build scripts and install in a local directory. • RPM/DEB packages do this for you • Not for the faint of heart ... Not recommended! • Using grep as an example $ $ $ $ $ $ wget http://mirrors.kernel.org/gnu/grep/grep-2.15.tar.xz tar -Jxvf grep-2.15.tar.xz cd grep-2.15 ./configure make make install 1.12.23 Hands-on: Package Management • Install the git package • Query the RPM/APT database for installed packages • List the files in an installed package • Remove the git package 1.12.24 Questions: • read example output of ls -al • read output of yum or aptitude search • install a package on their VM/partition (Vim, Git) – explain what dependencies it also installed 38 Chapter 1. Contents: OSU DevOps Bootcamp Documentation, Release 0.0.1 1.13 Lesson 2: Homework 1. Why is a salt used when storing encrypted passwords in /etc/shadow? 2. What portion of the MD5 hash ’$1$xxUwcovy$JfV9i7j9H/NFA3RBCrVHN.’ is the salt? 3. What UID is used for the root user? 4. Where is a user’s primary (“default”) group defined? Specifically which file and which “field”. 5. Add a user foobar to your system. Use useradd to add the user and to create their home directory containing files from /etc/skel. Show the user’s entry in /etc/passwd as well as the full useradd command needed. 6. Create a group bootcamp on your system. Show the command used (not editing files by hand). 7. Assume the user foobar belongs to multiple groups. Add foobar to the bootcamp group by using a system command (not editing files by hand) without changing any of their other groups. Show the command used. 8. What chmod command (using octal mode) would you use to allow owner read and write access and group read access (and no other permissions!) to a file foo? Using chmod without octal mode, how would you do the same? 9. What does it mean for a binary to be setuid? is setuid a potential security risk? Why is this important for tools such as passwd? 10. You have the following foo directory drwxr-xr-x 7 lance bootcamp 4096 Mar 31 09:15 foo What chmod command can you run to ensure files created inside that directory will default to having bootcamp group ownership? Assume the user creating the files is in the bootcamp group. 1.14 Lesson 3: Editors & Version Control 1.14.1 Check in from last week Is anyone having problems with package management? Install the packages named git, vim, and emacs 1.14.2 What we’ll discuss • What’s a text editor? • Features of an editor • Version control: Git! 1.14.3 How do you edit files? 1.13. Lesson 2: Homework 39 OSU DevOps Bootcamp Documentation, Release 0.0.1 1.14.4 No. Never, Ever edit code in Word/LibreOffice • Autocorrect is your worst enemy • Syntax highlighting is nice • You are editing plain text, not �documents’, i.e. .doc, .rtf, etc 1.14.5 Then what should you use? • Text editors – Command-line – available in your package manager • For coding, an IDE can be helpful 1.14.6 Integrated Development Environments • Entire toolchain in one place • Usually graphical 1.14.7 Sysadmin tools • Frequently work with remote machines over a ssh connection – xforwarding can sort of give you a GUI – Not all remote machines have a desktop environment • Faster to edit file in place than transfer it down then back up – Frequent small changes when testing or debugging • Broken machines often have only terminal via crash cart Editor Requirements • Small program, runs in terminal • Preferably installed by default on most systems 1.14.8 Features of an editor • Needs to be installed on your system • Create files • Move around within the file • Add or delete text • Easily automate tedious tasks 40 Chapter 1. Contents: OSU DevOps Bootcamp Documentation, Release 0.0.1 1.14.9 Text Editors Note: Quick intro/history: ed editor Pros: low-bandwidth, installed pretty much everywhere, very fast and powerful for complicated and repetitive tasks Cons: Steep learning curve, different “modes” can be confusing at first Sublime and other desktop editors: nice for serious programming, but learn the basics of simple text editors even if you want to be a developer, you won’t always be able to edit your code on your own desktop • ed -> Vi -> Vim • Stallman -> lisp -> emacs Avoid Pico/Nano, Notepad++, SublimeText 1.14.10 Emacs 1.14. Lesson 3: Editors & Version Control 41 OSU DevOps Bootcamp Documentation, Release 0.0.1 Note: Originally released 1976 name from Editor MACros for TECO editor, originally Tape Editor and COrrector at MIT in the �60s But, along the way, I wrote a text editor, Emacs. The interesting idea about Emacs was that it had a programming language, and the user’s editing commands would be written in that interpreted programming language, so that you could load new commands into your editor while you were editing. You could edit the programs you were using and then go on editing with them. – Richard Stallman, http://www.gnu.org/gnu/rms-lisp.html 1.14.11 Vim Note: originally written for Amiga systems (Commodore PCs), 1988 vim released 1991 vimscript, Lua (as of Vim 7.3), Perl, Python, Racket, Ruby, Tcl (tool command language). vi written by Bill Joy in 1976, visual mode for line editor called ex line editors are from age of teleprinters, no cursors • Available almost everywhere • Lightweight • Design decisions explained in http://docs.freebsd.org/44doc/usd/12.vi/paper.html • Modal editor (command, insert, visual) 1.14.12 How to choose • What can the people around you help with? • Try both; choose one and get good at it • Have a good answer when people ask why you made that choice – “Because it’s familiar” is tolerated – “Because I was initially taught it” is common but accepted (honesty) 42 Chapter 1. Contents: OSU DevOps Bootcamp Documentation, Release 0.0.1 – “Because $usecase” provokes argument but more respected – “Because I tried both and picked this one” is rare but good • Your use case as a sysadmin or developer 1.14.13 Modes How to tell? -- INSERT --- VISUAL -- 144,1 144,77 36% 36% 1.14.14 Commands Note: Moving around in a file Search / replace Text manipulation, ie: cw, dw, c$, yy / p, x, . 1.14.15 Moving Around h j k l 0 move move move move move one character to the left. down one line. up one line. one character to the right. to the beginning of the line. 1.14. Lesson 3: Editors & Version Control 43 OSU DevOps Bootcamp Documentation, Release 0.0.1 $ move to the end of the line. w move forward one word. b move backward one word. G move to the end of the file. gg move to the beginning of the file. . move to the last edit. 1.14.16 Configuration / customization Note: there are many many options and pre-existing packages to make editing nice for sysadmins and developers • .vimrc • :set Some sets of Vim plugins and configurations are available • https://github.com/astrails/dotvim • https://github.com/carlhuda/janus Use them for research on what’s available to improve dev productivity 1.14.17 Learning Resources • $ vimtutor • http://vim-adventures.com/ 1.14.18 Regular expressions You should know basic substitution: :%s/foo/bar/g On IRC, Hamper does rudimentary regex in the form s/foo/bar/ applying only to the most recent comment. This is not shell globbing Resources for learning • RegExr - an interactive Regular Expression editor and debugger • Regular-Expressions.info - Tutorials and general information 1.14.19 Editor questions? • Open an editor, find a cheat sheet, try to add some text • Modify the text: “disemvowel” it $ vim testvim.txt <i> Hello world! <esc> :%s/[aeiou]//g 44 $ emacs testemacs.txt Hello world! <esc> < <alt + x> Chapter 1. Contents: OSU DevOps Bootcamp Documentation, Release 0.0.1 1.14. Lesson 3: Editors & Version Control 45 OSU DevOps Bootcamp Documentation, Release 0.0.1 :wq replace-regexp [aeiou] <enter> <ctrl + x> <ctrl + s> <ctrl + x> <ctrl + c> 1.15 Lesson 3: Intro to Git 1.15.1 Version Control is Hard 1.15.2 Why Bother? 1.15.3 Better Options: Version Control Note: Collaboration with multiple developers is important to mention • Commit = Snapshot of part of your project’s state • Centralized (SVN, CVS) vs. Decentralized (Git, hg) • We’ll look at Git today – Easier to learn other VCS from Git – Widely used in the open source world 1.15.4 Git git noun. Brit.informal. 1. an unpleasant or contemptible person. 1.15.5 Using Git Locally # This initializes a git repo. Use �man git-init� for more info. $ git init # This puts <filename> into the staging area. It isn’t committed yet. # �git diff� to see what changes aren’t yet in staging. $ git add <filename> Use # This actually makes the commit. Use �git status� to see what’s in staging # but not yet committed. Use �git show� or �git log� to see recent commits. $ git commit -m "I did a thing!" • Undo things? the git book explains well • Did I remember to commit that? $ git status • What commits have I made lately? $ git log 46 Chapter 1. Contents: OSU DevOps Bootcamp Documentation, Release 0.0.1 Figure 1.1: Image from XKCD Figure 1.2: Image from PhD Comics 1.15. Lesson 3: Intro to Git 47 OSU DevOps Bootcamp Documentation, Release 0.0.1 48 Chapter 1. Contents: OSU DevOps Bootcamp Documentation, Release 0.0.1 1.15.6 More on commits Your work goes from unstaged to staging area with git add. $ git config --global user.name ’Your Name’ $ git config --global user.email [email protected] • Everything in staging gets wrapped up into an object that contains – changes – timestamp – author info – parent commit hash • These live in .git/ in your project directory • Commits go to other locations with git push 1.15.7 What Not To Do Figure 1.3: github/ image from http://arstechnica.com/security/2013/01/psa-dont-upload-your-important-passwords-to- • Don’t delete the .git files Note: If you kill them, git loses its memory :( • Redundant copies of same work • “oops, undoing that” commits. – Use git commit --amend Note: Amending is fine as long as you haven’t pushed yet. It’s generally a bad idea to amend or rebase work that you’ve already shared with others, unless you really know what you’re doing. • Don’t wait too long between commits – Squashing them together later is easy Note: Commit every time you think you might want to return to the current state. You can revert back to any previous commit, but there is no way to magically add a commit in where you forgot to make one. 1.15. Lesson 3: Intro to Git 49 OSU DevOps Bootcamp Documentation, Release 0.0.1 • Don’t commit compiled/generated items Note: Mostly relevant to writing code, .gitignore allows you to avoid dealing with compiled binaries, generated output, log files, etc • Don’t commit secrets... Note: Yes, there are ways to sort of take them down off of GitHub, but somebody might have cloned your repo while it had the secrets in. Once someone has a piece of information, you can’t just take it away. 1.15.8 Daily workflow • Pull • Work • Add changes • Commit • Push Larger projects have more complex workflows Note: The picture is of the Git Flow branching model, and you’ll probably see it every single time anyone explains Git branching and merging to you. If you are working on a larger project or writing code, you’ll likely be using branches, this allows a project to keep many simultaneous code changes organized. 1.16 Lesson 3: GitHub • Manage permissions on repos • Back up your work • Social/gamification • Amazing documentation: http://help.github.com Note: GitHub serves a threefold purpose: It also has amazing documentation which you should all go read right now and consult whenever you’re the least bit confused. It’s like the Ubuntu forums in that it’s explained in a way the newbies can understand, but unlike them in that it’s all written by people who know what they’re doing. 1.16.1 Let’s Walk Through • Creating an account – Gravatar – How to read a profile 50 Chapter 1. Contents: OSU DevOps Bootcamp Documentation, Release 0.0.1 1.16. Lesson 3: GitHub 51 OSU DevOps Bootcamp Documentation, Release 0.0.1 52 Chapter 1. Contents: OSU DevOps Bootcamp Documentation, Release 0.0.1 Note: you just go to github.com and click the account creation links. To make a custom icon, go to gravatar.com and set up an account using the same email address as you used for github. The picture you upload on Gravatar will then show up for your github account. The most important thing about reading profiles is that not all of a person’s repos will display on the front page of their profile – to see them, got to the �repositories’ tab instead of �contributions’. • Creating SSH keys – ssh-keygen -t rsa • Uploading your SSH key • Creating a new repository • Fork somebody else’s repo • Edit files online • Submit a pull request 1.16.2 Help, Everythings’s Broken! Permission denied (publickey). fatal: The remote end hung up unexpectedly Solution ssh-add ~/.ssh/id-rsa or whatever key you have added on github To [email protected]:edunham/slides.git ! [rejected] master -> master (non-fast-forward) error: failed to push some refs to ’[email protected]:edunham/slides.git’ hint: Updates were rejected because the tip of your current branch is behind hint: its remote counterpart. Merge the remote changes (e.g. ’git pull’) hint: before pushing again. hint: See the ’Note about fast-forwards’ in ’git push --help’ for details. Solution To avoid a messy merge commit, git pull --rebase. 1.16.3 Learn More • http://git-scm.com/book • http://try.github.io/levels/1/challenges/1 1.16.4 Hands-On • Fork the devopsbootcamp dotfiles repo • Clone a copy of the repo to your VM and make a branch • Make a commit with a helpful commit message and push to your fork $ $ $ $ $ ssh-keygen -t rsa # make an SSH key and add it to your account git clone <url from sidebar of your fork> # clone the repo cd dotfiles # git commands only work in project directlry git checkout -b <yourname> # -b creates branch vim <filename> 1.16. Lesson 3: GitHub 53 OSU DevOps Bootcamp Documentation, Release 0.0.1 # # # $ git $ git $ git ’i’ to enter insert mode <esc> to get back to command mode :wq to save and quit add <filename> commit -m "please use a helpful commit message, not like this one" push 1.17 Lesson 4: Scripting, Troubleshooting, & Workflow Note: week 1, 1/9/2014 - log files - git bisect, git log, git blame - scientific method (small changes) 1.17.1 Agenda • Scripting – What’s a script? – What’s a scripting language? – Common scripting languages – Bash & Python examples • Troubleshooting 54 Chapter 1. Contents: OSU DevOps Bootcamp Documentation, Release 0.0.1 – Techniques – Examples • Workflow – Life cycle of a bug 1.17.2 Scripting 1.17.3 What’s a script? • Piece of code to automate a boring task • Explanation of how to do what you do 1.17.4 What’s a scripting language? Note: This is more programming language than scripting specific. • Compiled vs interpreted • Expresses logic and actions – Logic: loops, conditionals, statements – Actions: File I/O, print to console, interact with system utilities 1.17. Lesson 4: Scripting, Troubleshooting, & Workflow 55 OSU DevOps Bootcamp Documentation, Release 0.0.1 1.17.5 Common Scripting Languages • Bash • Python • Language affects the speed of development and performance of a task • Most tasks can be done using any language 1.17.6 Bash • Adds a bit of logic to automate your shell commands • Saves re-typing things • Powered by the tools you invoke from the shell • Scripts can be hard to read at first because they call to external utilities 1.17.7 Python 1.17.8 Why Python? • Easy to read • Easy to maintain • Quick to write • Lots of libraries 1.17.9 Troubleshooting 1.17.10 Informal method • Notice that something isn’t working right • Identify what should be happening – Define a success criterion (“it works if...”) 1.17.11 If it used to work • Determine what changed – Version control history (Git bisect) – Emails from the system? Logs? (Check for cron jobs or config mgmt) – Ask others who’ve been working on system • Use your own notes/documentation 56 Chapter 1. Contents: OSU DevOps Bootcamp Documentation, Release 0.0.1 1.17. Lesson 4: Scripting, Troubleshooting, & Workflow 57 OSU DevOps Bootcamp Documentation, Release 0.0.1 58 Chapter 1. Contents: OSU DevOps Bootcamp Documentation, Release 0.0.1 1.17. Lesson 4: Scripting, Troubleshooting, & Workflow 59 OSU DevOps Bootcamp Documentation, Release 0.0.1 60 Chapter 1. Contents: OSU DevOps Bootcamp Documentation, Release 0.0.1 1.17.12 If it’s never worked for you • Determine whether it’s possible at all • Find evidence of similar things working (code, blog posts, stackoverflow) • If there’s no evidence of anything like this working, you might be Doing It Wrong (tm) • If there’s documentation of something similar working: – Confirm that the docs are correct for the versions of things that you’re using – If they docs are wrong, fix them – If the docs appear right, figure out what differs between your code and the example • If there’s sample code, make sure you can run it – Your goal is minimum viable test case 1.17.13 After finding the problem • Did the docs tell you how to fix it? • If you can’t fix the problem, identify why not, and then fix that • Ask for help 1.17. Lesson 4: Scripting, Troubleshooting, & Workflow 61 OSU DevOps Bootcamp Documentation, Release 0.0.1 – Expert takes 5 minutes to answer a well-asked question – Newbie can waste hours 1.17.14 Formal method (from this) • Identify the problem • Establish a theory of probable cause (question the obvious) • Test the theory to determine the cause • Establish a plan of action to resolve the problem and implement the solution • Verify full system functionality and, if applicable, implement preventative measures • Document findings, actions, and outcomes 1.17.15 How to get help • Don’t ask to ask • Summarize what’s wrong • Summarize what you’ve tried and why it hasn’t worked • Make a specific request, politely • Pick the right place & time to ask 1.17.16 Documentation • Man pages • Wikis • Google (used wisely) – Assessing sites’ applicability and reliability * Who wrote it? * When? * Is the other content reliable? * Is feedback from others visible? If so, what does it say? 1.17.17 Sources of trouble When using something new • You probably misunderstood it. • Maybe their documentation was wrong. • If neither, then perhaps their code is wrong. • Submit a ticket or pull request to fix the docs or code 62 Chapter 1. Contents: OSU DevOps Bootcamp Documentation, Release 0.0.1 When something previously working breaks • Something changed • Someone updated something • Figure out who and why; document 1.17.18 Tickets • Ticket (often sysadmin) or Issue (often developer) • Ticket comes into tracking system, submitted by a user • Triage – Add details to tickets; consolidate duplicates – Contact submitter if more info needed – Add tags, milestones, priority, etc. • Ticket is assigned to someone, who fixes it • Someone else confirms that the fix works, then ticket is closed 1.17.19 Tickets vs. Issues • Workflow defined by tracker system – RT, Redmine, Chiliproject, GitHub issues, mailing lists • Issues/Bugs are developer work items which need to be included in a release of code • Tickets are sysadmin work items, often related to systems improvement or maintenance • Can’t log in because your account got reset: Ticket. • Can’t log in because the newest release of the software is incompatible with the old database format: Bug. 1.17.20 Some Examples • Trac • Chiliproject • RT • Bugzilla 1.18 Lesson 5: Web Applications 1.18.1 What is a web app? Note: • talk briefly about responsibilities of each component • talk briefly about http 1.18. Lesson 5: Web Applications 63 OSU DevOps Bootcamp Documentation, Release 0.0.1 • mention Virtual Environments (red box on diagram) 1.18.2 What is this Virtual Environment thing? Note: Discuss system packages vs local, mention rvm • Separates application packages from system packages • Choose versions that differ from the system versions • Maintain a list of required packages for easy install • Install modules as a user • Pip and Gem 64 Chapter 1. Contents: OSU DevOps Bootcamp Documentation, Release 0.0.1 1.18.3 Frameworks Ready-made building blocks take care of the tedious details: • Speaking HTTP • Connecting code to URLS • Talking to the database • Built-in development servers Popular frameworks: Django, CakePHP, Ruby on Rails, Flask Note: Talk about major differences between frameworks, especially Django and Flask 1.18.4 The SystemView app Display a filtered list of processes on the system Note: • explain basic idea of the app – reference code from lesson 4 1.18.5 Setting Up Update your vagrant file and vagrant up: 1.18. Lesson 5: Web Applications 65 OSU DevOps Bootcamp Documentation, Release 0.0.1 git pull vagrant up vagrant ssh Note: Changes to the vagrant file: forward port 5050 to 5000 on the vm 1.18.6 Setting Up Install packages sudo yum install python-virtualenv* sudo yum install git Create a virtualenv virtualenv systemview_venv And activate it source systemview_venv/bin/activate Note: • students probably already have git? • discuss what virtualenv actually does, what is in it • env variables, etc • they can put the virtualenv anywhere, discuss locations • discuss, but don’t use virtualenv tools (mkvirtualenv, use, etc) • explain what source does 1.18.7 Get the Code git clone [email protected]:DevOpsBootcamp/systemview.git Note: • break here for github account setup, key location (where are they checking code out from? Where is their key located?), etc • https://github.com/DevOpsBootcamp/systemview.git for anyone who can’t get their account/key working 1.18.8 Run the Code cd systemview python systemview.py 66 Chapter 1. Contents: OSU DevOps Bootcamp Documentation, Release 0.0.1 1.18.9 Fail Oops! Traceback (most recent call last): File "systemview.py", line 2, in <module> from flask import Flask, request, session, g, redirect, url_for, \ ImportError: No module named flask Note: talk about missing modules, we need to install them, this is what the venv is for 1.18.10 Pip A package manager for Python packages • Connects to PyPi, a vast repository of Python modules • Resolves dependencies, installs prerequisites • Can install packages from a list in a file 1.18.11 Install What’s Missing Make sure you are in your virtualenv, then: pip install flask pip install argparse Note: • How do you know if you are in the virtualenv? • can put requirements in requirements.txt for easy installation 1.18.12 Run and Test! python systemview.py -i 0.0.0.0 -d Now go to http://localhost:5050 Note: • talk about flags • go to terminal after this slide and talk about the code: – main module, templates, css, etc • Point out areas where bugs could be fixed or features added 1.18. Lesson 5: Web Applications 67 OSU DevOps Bootcamp Documentation, Release 0.0.1 1.18.13 Branch and Modify Create a Github issue for your changes https://github.com/DevOpsBootcamp/systemview/issues Create a branch for your changes git checkout -b my_name When you have made changes and everything works, push it up git push origin my_name Note: • talk about branching vs forking, get everyone working on a new feature or bug • use IRC handles for branch names to make sure you are unique and identifiable 1.18.14 Homework Add a feature or fix a bug, push your changes up. Github URL: https://github.com/DevOpsBootcamp/systemview Github issue tracker: https://github.com/DevOpsBootcamp/systemview/issues 1.19 Lesson 6: Boot and the Filesystem Hierarchy Note: • grub, filesystem stuff based roughly on Frostsnow’s talk • basics of kernel and differences between virtualization/physical (the picture that kevin draws) 1.19.1 The Linux Filesystem Hierarchy Note: Based on Wade’s talk https://github.com/clinew/presentation_filesystems/blob/master/presentation.tex What’s a filesystem? In computing, a file system is used to control how information is stored and retrieved. Without a file system, information placed in a storage area would be one large body of information with no way to tell where one piece of information stops and the next begins. • (http://en.wikipedia.org/wiki/Filesystem) 68 Chapter 1. Contents: OSU DevOps Bootcamp Documentation, Release 0.0.1 1.19.2 Filesystem can mean: • How the system’s files are arranged on the disk • How the disk actually holds the files – FAT and NTFS are old but Windows-compatible – ext3 is standard, ext4 is newer, xfs has fancier journaling * journaling tracks changes before write – sysadmins will encounter NFS and its competitors like Gluster Note: Moving from Windows? • Binaries, not executables. • Directories, not folders. • Read, not load. • Symbolic links, not shortcuts. • Write, not save. 1.19.3 The File System $ ls bin boot dev etc home initrd.img initrd.img.old lib lib64 lost+found media mnt 1.19. Lesson 6: Boot and the Filesystem Hierarchy opt proc root run sbin selinux srv sys tmp usr var vmlinuz 69 OSU DevOps Bootcamp Documentation, Release 0.0.1 1.19.4 Installed programs and utilities /bin /sbin /usr/bin /usr/sbin /usr/local/bin /usr/local/sbin • PATH environment variable $ echo $PATH /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin • which command $ which bash /bin/bash 1.19.5 User-Specific Data & Configuration • Data stored at /home/<username> – Desktop environment creates folders Documents, Pictures, Videos, etc. • Configurations in dotfiles within home (/.) • Lost+Found is not your desktop trash can – Lost blocks of the filesystem. – Usually not an issue. – If your desktop provides backups of deleted files, they’ll be somewhere in /home/<username>/ 1.19.6 Where are drives mounted? • Raw device appears under /dev. $ dmesg | tail [260930.208715] sdb: sdb1 [260930.320756] sd 6:0:0:0: >[sdb] Asking for cache data failed [260930.320765] sd 6:0:0:0: >[sdb] Assuming drive cache: write through [260930.320771] sd 6:0:0:0: >[sdb] Attached SCSI removable disk • USB filesystem under /media, main disk / • You can manually mount devices with mount – “Everything’s a file” – umount to unmount • /etc/fstab tells things where to mount • /etc/mtab shows where things are currently mounted 1.19.7 Space on drives • Use df to see disk free space. 70 Chapter 1. Contents: OSU DevOps Bootcamp Documentation, Release 0.0.1 $ df -h / Filesystem /dev/sda8 Size 73G Used Avail Use% Mounted on 29G 41G 42% / • Use du to see disk usage. $ du -sh /home/ 21G /home/ • Default output is in bytes, -h for human-readable output. 1.19.8 Three Tiers of Filesystem Hierarchy • /, essential for system booting and mounting /usr. • /usr, read-only system data for normal system operation. • /usr/local, locally-installed software. – Package managers usually install under / and /usr. 1.19.9 Common Directories Directory /bin /include /lib /sbin /boot /dev /etc /home /media /mnt Contents Binary files Header files for C/C++ programs Libraries Binary files for root (superuser) Files essential for booting kernel, initramfs Virtual filesystem, exports hardware devices System-wide configurations Individual users’ data Removable storage devices Like media – place to mount disks and things 1.19. Lesson 6: Boot and the Filesystem Hierarchy 71 OSU DevOps Bootcamp Documentation, Release 0.0.1 1.19.10 Common Directories Directory /opt /proc /root /run /sys /tmp /var /usr/share /usr/src Contents “Add-on application software packages” Virtual filesystem exporting system data homedir for root Volatile information accumulated since boot Virtual filesystem exporting kernel objects Temporary files Data which varies – logs, mail, etc. Architecture-independent, read-only data Kernel source code 1.19.11 /proc has lots of useful system information Which Linux kernel version are you running? $ cat /proc/version Linux version 3.5.0-17-generic (buildd@allspice) (gcc version 4.7.2 (Ubuntu/Linaro 4.7.2-2ubuntu1) ) #28-Ubuntu SMP Tue Oct 9 19:31:23 UTC 2012 Learn about system’s hardware $ less /proc/cpuinfo $ less /proc/meminfo Some parts of /proc can be written as well as read... $ echo 3 > /proc/sys/vm/drop_caches # drop caches 1.19.12 Commands for working with filesystems Creating filesystems $ mkfs Mounting filesystems $ # # # mount -t for type -o for options requires device path and mount point Loopback devices $ losetup $ /dev/loop* # makes it look like a device instead of a file 1.19.13 devfs sd* sr* /dev/null 72 Chapter 1. Contents: OSU DevOps Bootcamp Documentation, Release 0.0.1 /dev/random /dev/urandom /dev/zero 1.19.14 Blocks and dd • Block size is the size of chunks allocated for files • dd – Disk duplicator (or disk dump). * if=<path>, input file. * of=<path>, ooutput file. * bs=<size>, block size. * count=<size>, number of block to transfer. $ dd if=/dev/random of=/dev/sda # What will this do? 1.19.15 Filesystem Consistency • Metadata vs. data – Metadata is extra information the filesystem tracks about the file – Data is the file’s contents • Filesystem is consistent if all metadata is intact – fsck is FileSystem Consistency Check 1.19.16 More about Journaling • Filesystem consistency tool; protections against system freezes, power outages, etc. • Replaying the journal. • ext3’s three modes of journaling: – journal Data and metadata to journal. – ordered Data updates to filesystem, then metadata committed to journal. – writeback Metadata comitted to journal, possibly before data updates. 1.19.17 The Boot Process • Bootstrapping • Steps in the process • Boot loaders • Startup scripts • Boot levels 1.19. Lesson 6: Boot and the Filesystem Hierarchy 73 OSU DevOps Bootcamp Documentation, Release 0.0.1 1.19.18 Bootstrapping Note: kernel loaded into memory, initialization tasks, and available to users Init • kernel spawns init which is always PID 1 • controls the boot process • can be a simple script to a binary • Pull itself up by its own bootstraps • Automatic and manual booting • Driver Loading • Period of vulnerability – configuration errors – missing hardware – damaged filesystems • init – Always Process ID (PID) #1 – First process to start – Either a binary or can be a simple script (even a bash shell!) 1.19.19 Steps in boot process Note: Kernel • 1st stage – bootloader, 2nd, boot the kernel • boot from boot loader • load into memory • located in /boot/ on Linux Hardware config • locate & initialize hardware 74 Chapter 1. Contents: OSU DevOps Bootcamp Documentation, Release 0.0.1 • print out what it does System processes • init, kswapd, pdflush, etc • init only real process • Others look like processes for scheduling (appear as [kswapd] with ps) 1. Kernel initialization 2. Hardware configuration 3. System processes 4. Operator intervention (single-user) 5. Execution of start-up scripts 6. Multi-user operation 1.19.20 Booting Note: On hardware specific to UNIX (i.e. Sun) • firmware knows how to use devices • talk to the network • understand filesystems • all accessible via the commandline 1.19. Lesson 6: Boot and the Filesystem Hierarchy 75 OSU DevOps Bootcamp Documentation, Release 0.0.1 BIOS smarter than they used to be • Not standardized • Most servers support PXE • PCs vs Proprietary hardware – BIOS, UEFI, OpenBoot PROM, etc • BIOS – Basic Input/Output System – Very simple compared to OpenBoot PROM / UEFI – Select devices to boot from – MBR (first 512 bytes) • UEFI – Unified Extensible Firmware Interface – Successor to BIOS – Flexible pre-OS environment including network booting 1.19.21 Boot Loaders (Grub) Note: Grub • next generation PC boot loader • no need to “re-run grub” config updates • Grub config • disks are index based from zero • grub install commands • netboot, pretty, serial • device.map, grub.conf • robust with weird disk geometry • Grand Unified Bootloader • Dynamic fixes during booting • Can read the filesystem • Index based – (hd0,0) = sda1 • Grub “version 1” vs. “version 2” – Version 2 has more features, but more complicated – Latest Debian, Ubuntu and Fedora use v2 76 Chapter 1. Contents: OSU DevOps Bootcamp Documentation, Release 0.0.1 grub> root (hd0,0) grub> setup (hd0) grub> quit (Specify where your /boot partition resides) (Install GRUB in the MBR) (Exit the GRUB shell) grub-install 1.19.22 Single User Mode Note: Show on VM • enter grub, hit ESC, pick kernel, hit “e” for edit • use arrows Typically ask for root password • What is it used for? • Troubleshoot problems • Manual Filesystem Checks • Booting with bare services • Fix boot problems • Add “single” to kernel option • Solaris/BSD – boot -s 1.19.23 Startup Script Tasks Note: Verbose and print out description of what its doing. Old days were to manually adjust scripts, not anymore. Most are configurable now. 1.19. Lesson 6: Boot and the Filesystem Hierarchy 77 OSU DevOps Bootcamp Documentation, Release 0.0.1 • Setting up hostname & timezone • Checking disks with fsck • Mounting system’s disks • Configuring network interfaces • Starting up daemons & network services 1.19.24 System-V Boot Style Note: • System-V Most common today • Show system changing between different run levels. • Slightly different between Distros • Linux derived from System-V originally • Alternative init systems – systemd - Fedora 15+, Redhat 7+ and Debian* (dependency driven) – upstart - Ubuntu, Redhat 6 (event driven, faster boot times) Run levels: level 0 level 1 or S level 2 through 5 level 6 sys is completely down (halt) single-user mode multi-user levels reboot level 1.19.25 /etc/inittab Note: Look at inittab • Tells init what to do on each level • Starts getty (terminals, serial console) • Commands to be run or kept running • inittab not used with systemd or upstart 78 Chapter 1. Contents: OSU DevOps Bootcamp Documentation, Release 0.0.1 # The default runlevel. id:2:initdefault: # What to do in single-user mode. ~~:S:wait:/sbin/sulogin # What to do when CTRL-ALT-DEL is pressed. ca:12345:ctrlaltdel:/sbin/shutdown -t1 -a -r now # terminals 1:2345:respawn:/sbin/getty 38400 tty1 T0:23:respawn:/sbin/getty -L ttyS0 9600 vt100 1.19.26 init.d Scripts Note: sshd init script • case statement • functions • chkconfig • One script for one service/daemon • Start up services such as sshd, httpd, etc • Commands – start, stop, reload, restart • sshd init script $ service sshd status openssh-daemon (pid 1186) is running... $ service sshd restart Stopping sshd: Starting sshd: [ [ OK OK ] ] 1.19.27 Starting services on boot Note: Show sshd script show list, adding, removing, enabling, disabling • rclevel.d (rc0.d, rc1.d) • S = start, K = stop/kill • Numbers to set sequence (S55sshd) • chkconfig / update-rc.d – Easy way to enable/disable services in RH/Debian • Other distributions work differently 1.19. Lesson 6: Boot and the Filesystem Hierarchy 79 OSU DevOps Bootcamp Documentation, Release 0.0.1 $ chkconfig --list sshd sshd 0:off 1:off 2:on 3:on 4:on 5:on 6:off $ chkconfig sshd off $ chkconfig --list sshd sshd 0:off 1:off 2:off 3:off 4:off 5:off 6:off 1.19.28 Configuring init.d Scripts Note: show sendmail & network config examples for CentOS /etc/defaults seems to be more common between UNIX’s • /etc/sysconfig (RH) or /etc/defaults (Debian) • source Bash scripts • Daemon arguments • Networking settings • Other distributions are vastly different $ cat /etc/sysconfig/ntpd # Drop root to id ’ntp:ntp’ by default. OPTIONS="-u ntp:ntp -p /var/run/ntpd.pid -g" 1.19.29 Shutting Down Note: Modern systems are less touchy with hard resets, but still need to be careful. Only for emergencies. Shutdown -h • Not Windows, don’t reboot to fix issue • Can take a long time (i.e. servers) • Reboot only to – load new kernel – new hardware – system-wide configuration changes • shutdown, reboot, halt, init • wall - send system-wide message to all users $ wall hello world Broadcast message from root@devops-bootcamp (pts/0) (Fri Jan 31 00:40:29 2014): hello world 80 Chapter 1. Contents: OSU DevOps Bootcamp Documentation, Release 0.0.1 1.19.30 Homework 1.20 Lesson 7: Databases 1.20.1 Filesystems Review Questions • What might happen to a busy ext2 volume on power loss? • ext3? • ext4? Note: • ext2 may not be consistent upon restart • ext3 and ext 4 are not • but consistency only guarantees metadata is intact • What might happen to a busy ext2 volume on power loss? • ext3? • ext4? 1.20.2 But what about our poor data? • Possibly gone, like the wind • Or worse: Half completed writes! • General purpose operating systems, by design, don’t understand structured data 1.20.3 Enter the Database A database system’s fundamental goal is to provide consistent views of structured data using the tools the operating system makes available. Chief among them is fsync(2) Note: fsync instructs the operating system to flush all writes to disk before returning 1.20.4 Structure SQL databases are structured around Relational Algebra • Tables – Columns are fields – Rows define a relation between fields • A Primary key is a set of columns that uniquely identify rows in a table • A Foreign key is a column that matches the primary key of another table 1.20. Lesson 7: Databases 81 OSU DevOps Bootcamp Documentation, Release 0.0.1 1.20.5 Relational Algebra Visualized Note: joins are the principle use of relations. 1.20.6 Installing MySQL $ yum install mysql-server $ /sbin/service mysqld start $ /usr/bin/mysql_secure_installation 82 Chapter 1. Contents: OSU DevOps Bootcamp Documentation, Release 0.0.1 1.20.7 Managing MySQL $ /sbin/service mysqld status $ mysqladmin -p ping $ mysqladmin -p create nobel 1.20.8 Configuration • /etc/my.conf • The most important MySQL tuning rule: – almost always prefer InnoDB Note: we’re going to add: default_storage_engine = InnoDB 1.20.9 Users & Permissions $ sudo mysql -p mysql> CREATE USER ’vagrant’@’localhost’ IDENTIFIED BY ’password’; mysql> GRANT ALL PRIVILEGES ON nobel.* TO ’vagrant’@’localhost’ WITH GRANT OPTION; 1.20.10 Importing Data $ wget http://osl.io/nobel -O nobel.sql $ mysql -p nobel < nobel.sql $ mysql -p nobel SHOW TABLES; DESCRIBE nobel; 1.20.11 Basic Queries 4 basic operations on data: • SELECT • INSERT • UPDATE • DELETE 1.20.12 SELECT 1.20. Lesson 7: Databases 83 OSU DevOps Bootcamp Documentation, Release 0.0.1 SELECT yr, subject, winner FROM nobel WHERE yr = 1960; 1.20.13 Practice • Who won the prize for Medicine in 1952? • How many people were awarded the 1903 Nobel in Physics? • How many prizes were awarded to Linus Pauling? • How many people have won more than once? (Difficult) 1.20.14 INSERT INSERT VALUES (’2013’,’Literature’,’Herta Müller’) INTO nobel; Note: this data stops at 2008, so lets insert some 2009 awards 1.20.15 Practice In 2009: • Barack Obama won the Peace Prize • Elinor Ostrom and Oliver E. Williamson won the prize in Economics • http://en.wikipedia.org/wiki/List_of_Nobel_laureates 1.20.16 UPDATE UPDATE nobel SET winner=’Andrew Ryan’ WHERE subject=’Peace’ AND yr=’1951’; Note: obviously Andrew Ryan deserves the peace price for his work in the Rapture planned community 1.20.17 Practice • Brigid Tenenbaum Medicine prize in 1952 84 Chapter 1. Contents: OSU DevOps Bootcamp Documentation, Release 0.0.1 1.20.18 DELETE DELETE FROM nobel WHERE yr = 1989, subject = peace; Note: peace prizes can be controversial, and perhaps there’s a political interest in censoring our database? 1.20.19 Further Reading, Resources, etc. • Codd, E.F. (1970). “A Relational Model of Data for Large Shared Data Banks”. Communications of the ACM 13 (6): 377–387. • sqlzoo.net • CS 440: Database Management Systems 1.20.20 Hands-On: Make a Database • Create a new database mysql> create database systemview mysql> GRANT ALL PRIVILEGES ON systemview.* TO ’vagrant’@’localhost’ WITH GRANT OPTION; • Grant a user privileges on your new database Note: challenge them to do this based on the material in the last hour, maybe also demo the mysql console. Make sure everyone remembers the username and password for the next step. 1.20.21 Databases in Applications Applications love databases. • Application data - the information to be displayed and manipulated • User data - complex authentication and authorization • Logging, statistics, state and session data, etc... Note: All the various things an app might use a database for - note that the vast majority of web apps use them for something 1.20.22 Native SQL Most languages allow you to speak directly to a database Python: 1.20. Lesson 7: Databases 85 OSU DevOps Bootcamp Documentation, Release 0.0.1 #!/usr/bin/python import MySQLdb db = ("localhost","testuser","test123","nobel" ) cursor = db.cursor() cursor.execute("SELECT subject, yr, winner FROM nobel WHERE yr = 1960) data = cursor.fetchall() for winner in data: print "%s winner in %s: %s " % (winner[0], winner[1], winner[2]) db.close() Note: Note the plain SQL statement, recognizable from earlier. Point out the cumbersome nature of creating the connection, creating a cursor, sending the sql, getting data from the cursor (iterating over it if you want multiple results), etc. Similar interfaces exist for virtually all languages. 1.20.23 Introducing the ORM Object Relational Mapper • Maps an Object in an application to a database table or relationship • Talks SQL to the database, your favorite language to you • Lets you point to different databases with the same syntax • Intelligently manages transactions to the database Note: Make sure people know what you mean by “object”, mention possible difference between Postgres, sqlite, MySql, etc. Objects may map to one table, but might also incorporate relationships. ORMs also often optimize queries and manage transactions to make database queries as efficient as possible (like all other magic, though, sometimes this can backfire). 1.20.24 Life With a Python ORM Look, ma! No SQL! for subject, yr, winner in session.query(Nobel).filter_by(yr=1960): print "%s winner in %s: %s " % (subject, yr, winner) Much easier to read and understand, but requires some setting up first. Note: Of course we actually have to do a lot of setup work - setting up the model, engine, session, etc - but you do that once and can interact with the database as much as you want, without worrying about the cursor or connection. Note that we have no SQL in this statement, it is pythonic and has pythonic methods. The database table is now an object. 86 Chapter 1. Contents: OSU DevOps Bootcamp Documentation, Release 0.0.1 1.20.25 Setting Up the Magic - SqlAlchemy SqlAlchemy - a popular Python ORM, frequently used in Flask apps (like SystemView!). To use it, we’ll need to: • Import sqlalchemy • Create a “model” - a representation of our data in code • Create an “engine” and connect it to the database • Create a session to store the model instances and transactions Note: Model A object with all the properties, attributes, etc of our data, can also include code to manipulate that data in order to represent a specific view (i.e. automatically returning sorted results). It’s just a python class, instances are just python objects. Engine This handles the authentication with the database, it’s like the MySQLdb.connect above. Session An in-memory record of your changes to objects - all the orm objects you instantiate live int he session, and are only saved to the database when you say so. 1.20.26 Let’s Databasify Systemview Project: • Store search terms, then provide them as links on the search page, so you can just click the most common terms you search for. What else? Ideas? Note: Solicit ideas for another column or two, maybe number of times the term is used (easy incrementing example), or number of results from the least search. 1.20.27 Hands On • Install the following packages: sudo yum install python-devel sudo yum install mysql-devel • Check out systemview from GitHub (if you don’t have it already) git clone [email protected]:DevOpsBootcamp/systemview 1.20.28 Hands On (Cont...) • Switch to �save-search’ branch git checkout -tb save-search origin/save-search • Activate your virtualenv 1.20. Lesson 7: Databases 87 OSU DevOps Bootcamp Documentation, Release 0.0.1 source <path to virtualenv>/bin/activate • Install the requirements pip install -r requirements.txt Note: Talk about git branches again, explain tracking, git pull for people who already have it cloned, etc. Talk about the virtualenv, have people create a new one if they have lost the one they made last time. Talk about pip and what requirements.txt is all about - point out how easy it is to set up an app this way. Make sure requirements.txt contains sqlalchemy. DANGER! - people will need mysql-dev package! name varies by distribution, for centos it is libmysqlclient-dev 1.20.29 Goals • Connect the app to your new database • Add a new column • Save data to that column whenever someone searches • Fetch the data from that column and display it on the search page • challenge: limit the returned result to only 5 terms http://docs.sqlalchemy.org/en/rel_0_9/orm/tutorial.html Note: The code in the repo should have a simple model with one column, �term’, you can make a models.py, or just put it all in one file. If you separate them, talk about MVC. The code should start an sqlalchemy engine and session, save the search term normalized (lowercased, stripped), the column should be set to unique. Make sure the code handles the case of the term already existing in the database (when you add a count, increment the count when the term exists). You should probably initialize the db directly in the code, otherwise you’ll have to open up a python console, import the app and run the db update. 1.21 Lesson 8: Security & Authentication 1.21.1 What is security? se·cu·ri·ty sikyoorit/ noun • the state of being free from danger or threat. • the safety of a state or organization against criminal activity such as terrorism, theft, or espionage. • procedures followed or measures taken to ensure the safety of a state or organization. • the state of feeling safe, stable, and free from fear or anxiety. 88 Chapter 1. Contents: OSU DevOps Bootcamp Documentation, Release 0.0.1 1.21.2 Types of security • Physical security • Software security • Network security – Active vs. passive 1.21.3 Authentication vs. Authorization Authentication Are they who they say they are? Authorization Are they allowed access here? • Authentication requires proof of identity • Authorization requires authentication, plus permission from an authority 1.21. Lesson 8: Security & Authentication 89 OSU DevOps Bootcamp Documentation, Release 0.0.1 1.21.4 Identity Persistent vs. authoritative Imagine an identity thief who takes out lines of credit in their victim’s name then pays all the bills on time... • Is their identity persistent? • Is their identity authoritative? How about a project maintainer who never uses their real name online, but uses the same handle and email address across all sites? 1.21.5 Identity How about a project maintainer who loses the domain which was hosting their email, and thus changes addresses abruptly? If you’re a sysadmin who works with multiple projects, you will run into these concerns often. 1.21.6 Principle of Least Authority • User & Group management • ACLs • File permissions 1.21.7 System Security • Close ports • Firewalls • Process isolation • nmap vs. fail2ban 90 Chapter 1. Contents: OSU DevOps Bootcamp Documentation, Release 0.0.1 1.21.8 Other Attacks • Social engineering – Pretexting – Phishing – Baiting – Quid Pro Quo – Tailgaiting • Zero-Day vulnerabilities Note: Social engineering leverages those person-to-person skills us computer folks are so well known for not having. Can someone let me borrow their laptop for a minute? I just want to tell my mom I’ll be home late tonight... honest! Pretexting is when someone contacts you with a pretext. They sound like they’re authentic because they know something about you which they probably got off Facebook or something else. “Hi, I’m calling from Chase Bank and I noticed that your card might have been used at a location where fraud was committed. I have your name as Bob Smith and your date of birth as May third, 1992. Can you read me your card number and the three digits on the back?” Phishing is something we’ve all been warned about. We all know that eBay and Paypal aren’t going to email us asking for our bank account information. Just don’t fall for it and you’ll be okay. Baiting is a little more interesting. Ever walk down the street and notice a thumb drive or SD card on the ground? Hey – free thumb drive, right? Let’s just put it in the computer, what could go wrong? Plenty. If it’s too good to be true, it probably is. Quid pro quo is a trade – I’ll give you something if you give me something. Would you trade your password for a chocolate bar? According to the BBC, someone tried this outside a subway station in London in 2004, and more than seventy percent of people took that trade! Thirty-four percent gave it up for free. Don’t do that. I suspect many of you who live in the dorms have been told about tailgating. You’re unlocking the dorm door and someone comes up and says “hey, I forgot my key in my room, can you let me in” or maybe they’re wearing a Domino’s 1.21. Lesson 8: Security & Authentication 91 OSU DevOps Bootcamp Documentation, Release 0.0.1 uniform and are carrying a pizza. You’re a nice person, you want to help them. Don’t do it. And that leaves us with zero-day vulnerabilities. The term �zero day’ refers to the amount of time that the folks who write and maintain the code have had to fix it. If they don’t know about it, they can’t fix it. This is why it’s so important for us to report vulnerabilities when we discover them as was discussed earlier. 1.21.9 What to do if you discover a vulnerability First, test and document to verify that it exists. Then, disclose it privately to those responsible for fixing it Provide examples – it’s basically a bug report, but through private channels (not public tracker yet!) Give them time to release a patch before announcing it Some places have bug bounties 1.21.10 Passwords 1.21.11 Good Passwording • http://bash.org/?244321 1.21.12 Server Side • Rainbow Table • Hashing / salt • bcrypt/ scrypt 92 Chapter 1. Contents: OSU DevOps Bootcamp Documentation, Release 0.0.1 1.21. Lesson 8: Security & Authentication 93 OSU DevOps Bootcamp Documentation, Release 0.0.1 1.21.13 Password Managers • Password managers (LastPass, 1Password, KeepPass*) – Works with phones and other things • pass http://www.zx2c4.com/projects/password-store/ • vim -x passwords.txt • http://world.std.com/~reinhold/diceware.html Note: http://makezineblog.files.wordpress.com/2013/01/fractal-rainbow-table-runner-1.jpg We use passwords for everything we do online. Some (hopefully) semi random grouping of letters, numbers, and symbols which when combined with a username allow you to authenticate with a server or process. There are a couple common attacks on passwords, the most common of which is called a dictionary attack. This uses the fact that words are easier to remember than random characters, so it abuses human memory in order to greatly reduce the search space for passwords. pwgen + a password manager will help you have better passwords which you don’t even have to remember! DEMO Storing passwords on the server side is a whole other matter. One of the primary issues of concern is what happens if your server gets compromised. Lets say for instance that you just have a giant text file that has the form “username password” on each row. This would be super fast to to lookup users in, but if that file ends up in the wrong hands, you lose. A better option is to not store the passwords directly, but to store some representation of the password. This is where a hash comes in. Essentially a hash is a one way function that is fast to calculate, deterministic in output, but _very_ hard to reverse. If you store the hash of a password you can hash what they send you to verify who they are. Again we must consider what happens if our database was compromised. Since these hashes are deterministic and computing power is so cheap, we can precalculate what passwords correspond to what hashes, these precomputed files are called rainbow tables. To avoid the issue with rainbow tables we �salt’ our passwords. This adds a small random string to each password so that the search space for precomputing possible passwords becomes tera/petabytes large. Enough about passwords, we now move into more interesting things called keys! 94 Chapter 1. Contents: OSU DevOps Bootcamp Documentation, Release 0.0.1 1.21.14 Keys • Better than passwords • Symmetric vs Asymmetric • Diffie-Hellman / RSA 1.21.15 Key Exchange 1.21.16 RSA • Math • Math • More Math • Don’t be shy Note: Keys are password files. These can be used in place of a password for authentication and encryption. Symmetric keys essentially work like passwords. They are basically a one-time pad where both parties need to know the key to enable data to be stored and retrieved. Asymmetric keys work by encrypting with a public key (one everyone can see), but only being able to be decrypted by the private key (which you shouldn’t show anyone). 1.21. Lesson 8: Security & Authentication 95 OSU DevOps Bootcamp Documentation, Release 0.0.1 The fundamental problem with communication is that if you don’t have a preshared key between two users, everything you say is being listened to and presumably logged. Diffie Hellman key exchange is probably the most important result in cryptography. It allows two users to communicate in plaintext (non-encrypted) and trade their public keys in order to generate a shared secret so then they can communicate with encryption. RSA is an algorithm that follows Diffie-Hellman and is the most common way to do key exchange. 1.21.17 SSH • Password vs Keys • Passphrases • authorized_keys • Automation $ ssh -D 9999 [email protected] $ ssh -R 2222:localhost:22 freshblue.lake Note: ssh is secure shell and provides a shell to a unix machine over the �net by using RSA to encrypt communications between a client and server. Passwordless login, refuse connections without keys, tunneling. Commands at the end are run unecrypted. Passphrases work by adding a password to a key file. Add your friends public keys to authorized_keys so they don’t need a password to login. ssh-agent, .ssh/config, /etc/ssh/sshd_config DEMO Make ssh-keys, post to pastebin.osuosl.org 1.21.18 Brief History of Time (line of GPG) • P(retty)G(ood)P(rivacy) • Phil Zimmermann 1.21.19 GPG • E-mail privacy • Why you should use GPG • Why people don’t use GPG • Keys, signing, keyservers • Encryption 1.21.20 Ways to use GPG • Enigamail • mutt 96 Chapter 1. Contents: OSU DevOps Bootcamp Documentation, Release 0.0.1 1.21. Lesson 8: Security & Authentication 97 OSU DevOps Bootcamp Documentation, Release 0.0.1 • Command line $ gpg --encrypt manateessecrets.jpg.exe 1.21.21 Certificates and HTTP • Certificate Authorities • https • ssl/tls $ openssl req -new -x509 -key /etc/ssl/private/privkey.pem \ -out /etc/ssl/certs/cacert.pem -days 1095 1.21.22 Man in the Middle Note: • 650 CAs • Attacks on https/ssl • Future DEMO sslsniff 98 Chapter 1. Contents: OSU DevOps Bootcamp Documentation, Release 0.0.1 1.21.23 WiFi • wep • wpa • wpa2 • Wireshark – Demo Note: • Attacks • mschapv2 DEMO Wireshark 1.21.24 Crypto-wares • Files – Tarsnap, SpiderOak, rsync over ssh • Communications – VPN – TextSecure/ RedPhone – Tor – https everywhere • Security – Metasploit, BEEF – AirCrack, sslstrip 1.21.25 Math! • Primes • Number Theory • Fields • Elliptic Curves Note: DEMO rot13 1.21.26 One Last Thing • https://priv.ly ( proudly hosted at the OSL) • “I have nothing to hide“ • jeremykun.com 1.21. Lesson 8: Security & Authentication 99 OSU DevOps Bootcamp Documentation, Release 0.0.1 • thoughtcrime.org • https://www.schneier.com/ 1.22 Lesson 8: Web application security Figure 1.4: image source: https://info.cenzic.com/rs/cenzic/images/2013-vulnerability-summary_290x250.png 1.22.1 Web application security • Who needs to worry about web application security? – Everyone! 100 Chapter 1. Contents: OSU DevOps Bootcamp Documentation, Release 0.0.1 • What kinds of attacks are seen in the wild? – Many! • What can devops do about these attacks? – A lot! Note: Everyone needs to worry about web application security. You need to worry, because you’re learning how to write web applications. You want to avoid making decisions which could lead to exposing vulnerabilities and letting bad people use your service to do bad things. You also need to worry even if you’re not writing web applications, because you’re using web applications. The web is still a wild and wooly place, and the last line of defense for the user is their own common sense. What kinds of attacks are seen in the wild? The image shows a dizzying array of acronyms and shorthand but we’ll be going over those in a little more detail. And what can devops like us do about these attacks? Plenty – wait and see. 1.22.2 Code Injection • Attacks – SQL Injection – Cross-Site Scripting (XSS) – Cross-Site Request Forgery (CSRF) – Remote Code Execution • Defenses – Sanitize your inputs! http://bobby-tables.com/ https://docs.djangoproject.com/en/dev/ref/contrib/csrf/ http://guides.rubyonrails.org/security.html Note: These types of attacks consist of code that is introduced into the application causing unexpected behavior. This code can be introduced unintentionally by typical users who use quotes or ampersands in their inputs as well as intentionally by nefarious folks. The comic demonstrates a classic SQL injection attack. Bobby took advantage of the school’s software not properly sanitizing their inputs by including a command to drop the students table, causing the kind of chaos often seen in xkcd comics. Cross-site scripting works much the same way: someone posts a comment on a blog which includes Javascript which gets executed when you view the comment. When it is executed, it does something horrible like send them your cookie for that blog site. 1.22. Lesson 8: Web application security 101 OSU DevOps Bootcamp Documentation, Release 0.0.1 Cross-site request forgeries are similar but instead of Javascript you’ll see image links that really point to another site like your bank, hoping that your cookies will let them transfer money from your accounts to theirs. Remote site execution is what it sounds like: just like the SQL injection attack, but instead running a shell command on the web server. I think by now you all have enough experience with running commands on your virtual machines to know how bad that could be. Luckily, each of these threats can be addressed the same way: listen to Bobby’s mom and sanitize your inputs! There’s a web site dedicated to helping developers with SQL injection threats which I’ve listed above, but the same concepts apply to the other threats. Want to stop cross-site scripting? Don’t allow users to contribute arbitrary Javascript in comments. Want to stop cross site request forgeries? Make sure your GET requests are free of side-effects, and include tokens in your forms. As a bonus, Django will do that last bit for you if you let it – check out that second link up there for more details. That third link is the Rails security guide and provides advice on these issues as well as many others. 1.22.3 Web Server-Specific Attacks Figure 1.5: image source http://news.netcraft.com/wp-content/uploads/2014/02/apache-vulns1.png • Version-Based • Configuration-Based Note: There is a constant battle between developers and the bad guys – one side discovers vulnerabilities, the other side fixes them. One of the easiest things to do to keep the bad guys out is to use the most up-to-date version of your web server, regardless of whether it’s Apache or IIS or nginx. The graph above shows the most popular versions of Apache as of February 2014 according to Netcraft. Apache encourages admins to run the latest major release of the 2.4 stable branch, which is Apache 2.4.7. How many of those 102 Chapter 1. Contents: OSU DevOps Bootcamp Documentation, Release 0.0.1 releases do you see in that image? That’s right – none. Heck, two of the top fifteen are EOLed – they aren’t even receiving security updates any longer! This is bad. Don’t be like them. But it’s not enough to run the latest version. You should also make sure your configuration files are updated as well. Some default configurations will include accounts or passwords which can be guessed by hackers. Other times certain features will be enabled by default, which can introduce vulnerabilities you don’t expect even though you’re not using those features. Read the release notes when you update your software. Pay attention to details. They will. You should too. 1.22.4 Problems with Design and Implementation • Authentication and Authorization • Session Management • Information Leakage • Unauthorized Directory Access Note: The remaining threats facing the typical web developer come down to design and implementation problems. The fine points of authentication and authorization have been discussed already: make sure that all your actions are authorized by authenticated users and you should be okay. Also, don’t let your cookies have infinite lifetimes. Better to have your users occasionally log in again than let them be vulnerable to those cross-site attacks we covered before. Pro tip: PHP has a default setting for “session.cookie_lifetime” of zero, which means they never expire. If you’re using PHP, fix that. Information leakage is pretty sneaky. Let’s say your app allows users to request a password reset by entering their email addresses. If your app behaves differently when valid and invalid addresses are input, congratulations, you’re leaking information. Unauthorized directory access is a specialized form of information leakage – while it’s nice to let people know how to contact your staff, you might not want to let them download everyone’s email address and such. 1.22.5 What Not to Do: The Exercise 1.22.6 Getting Up to Date • ssh into your vagrant environment • change directory to your local systemview repo $ cd ~/projects/systemview • Make sure your local copy is up to date $ git pull • If you’ve modified code you’ll need to follow these instructions $ git stash save "some witty name about your work" $ git pull --rebase 1.22. Lesson 8: Web application security 103 OSU DevOps Bootcamp Documentation, Release 0.0.1 1.22.7 Let’s Check out Dean’s (not so) Awesome Code $ git checkout <not so awesome code branch goes here> 1.23 Lesson 9: Networking 1.23.1 Who is this talk for? Someone with little or no networking knowledge. ECE/CS 372 at OSU covers this content, more or less 1.23.2 What is a network? “a group or system of interconnected people or things“ To us, a network is: • Electronic devices • Sending signals over wire, fiber, or radio • Communicating data using a standardized protocol 1.23.3 What is a protocol? “A set of agreed upon rules for communication“ • Defines sequence & format of packets being sent 1.23.4 The OSI Model Open Systems Interconnection • Layers of abstraction Note: “Create a layer of easily localized functions so that the layer could be totally redesigned and its protocols changed in a major way... without changing the services expected from and provided to adjacent layers” 1.23.5 Layer 1: Physical Networking Hardware • Connector shapes • Wire, optical fiber, or radio signal specifications RS-232 104 Chapter 1. Contents: OSU DevOps Bootcamp Documentation, Release 0.0.1 1.23. Lesson 9: Networking 105 OSU DevOps Bootcamp Documentation, Release 0.0.1 1.23.6 Layer 2: Data Link MAC: Media Access Control • MAC address should be globally unique • ARP: Address Resolution Protocol (between layer 2 & 3) • NDP (neighbor discovery protocol) used in IPv6 • Flow control & error checking 1.23.7 Layer 3: Network Packet forwarding and routing Network and host addressing • IPv4 • IPv6 1.23.8 Layer 4: Transport Interact directly with program same-order delivery, reliability, flow control, and congestion avoidance TCP Transmission Control Protocol • used by HTTP, HTTPS, SMTP, POP3, IMAP, SSH, FTP, Telnet UDP User Datagram Protocol • No error checking built in • No retransmission delays • VoIP, media, games 1.23.9 Get your hands dirty In a linux terminal run:: ip a These will display information about your network interfaces. See also: ifconfig iwconfig 1.23.10 Example output: user@host:~$ ip a ... 2: eth2: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast state DOWN qlen 1000 link/ether 33:77:00:44:66:33 brd ff:ff:ff:ff:ff:ff 3: wlan1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000 106 Chapter 1. Contents: OSU DevOps Bootcamp Documentation, Release 0.0.1 link/ether 24:77:33:44:55:66 brd ff:ff:ff:ff:ff:ff inet 192.168.1.55/24 brd 192.168.1.255 scope global wlan1 inet6 fe80::2677:3ff:fed4:538c/64 scope link valid_lft forever preferred_lft forever 1.23.11 Netmask: Decimal IP Address 192.168.1.55 255.255.255.0 Binary IP Address 11000000.10101000.00000001.00110111 11111111.11111111.11111111.00000000 Part of address Network (Decimal) Network (Binary) Host (Decimal) Host (Binary) Corresponding address 192.168.1.0 11000000.10101000.00000001.00000000 0.0.0.55 00000000.00000000.00000000.00110111 Available Hosts: 192.168.1.[1-254] Broadcast address: 192.168.1.255 1.23.12 Netmask Example: Decimal IP Address 192.168.90.55 255.255.192.0 Binary IP Address 1.23.13 Netmask Example: Decimal IP Address 192.168.90.55 255.255.192.0 Binary IP Address 11000000.10101000.01011010.00110111 11111111.11111111.11000000.00000000 Part of address Network (Decimal) Network (Binary) Host (Decimal) Host (Binary) Corresponding address 192.168.64.0 0.0.26.55 1.23.14 Netmask Example: Decimal IP Address 192.168.90.55 255.255.192.0 Binary IP Address 11000000.10101000.01011010.00110111 11111111.11111111.11000000.00000000 Part of address Network (Decimal) Network (Binary) Host (Decimal) Host (Binary) Corresponding address 192.168.64.0 11000000.10101000.01000000.00000000 0.0.26.55 00000000.00000000.00011010.00110111 Available Hosts: 192.168.[64-127].[1-254] 1.23. Lesson 9: Networking 107 OSU DevOps Bootcamp Documentation, Release 0.0.1 Broadcast Address: 192.168.127.255 1.23.15 Routes user@host:~$ route Kernal IP routing table Destination Gateway default foo.osuosl link-local * 192.168.1.0 * Genmask 0.0.0.0 255.255.0.0 255.255.255.0 Flags UG U U Metric 0 1000 2 Ref 0 0 0 Use 0 0 0 Iface wlan1 wlan1 wlan1 user@host:~$ route -n Kernel IP routing table Destination Gateway 0.0.0.0 192.168.1.1 169.254.0.0 0.0.0.0 192.168.1.0 0.0.0.0 Genmask 0.0.0.0 255.255.0.0 255.255.255.0 Flags UG U U Metric 0 1000 2 Ref 0 0 0 Use 0 0 0 Iface wlan1 wlan1 wlan1 1.23.16 Bootstrapping What happens when your computer connects to a network? 1. Duplex and speed negotiation 2. Static or dynamic configuration is applied 1.23.17 Static Configuration Must in advance know: • IP Address • Netmask • Default Gateway • DNS Servers (optional in some cases) 1.23.18 Dynamic Configuration All of the statically defined parameters are retrieved over the network via DHCP But how do you communicate over the network without a network configuration? 1.23.19 Reserved IPv4 Addresses • 127.0.0.1 • 192.168.0.0 • 172.16.0.0 • 10.0.0.0 • 169.254.0.0 108 Chapter 1. Contents: OSU DevOps Bootcamp Documentation, Release 0.0.1 1.23.20 Public vs Private Address NAT Network Address Translation • lose end-to-end traceability • hides internal network topology • allows use of private IP’s over public internet • conserves limited public IP’s 1.23.21 Network Devices 1.23.22 Network Devices 1.23.23 Control Layer Connection oriented vs Connectionless 1.23.24 Collisions CSMA CA All Wireless networks use this Carrier Sense Multiple Access with Collisions Avoidance CSMA CD Carrier Sense Multiple Access with Collisions Detection Why is this important? http://articles.latimes.com/2007/aug/15/local/me-lax15 1.23. Lesson 9: Networking 109 OSU DevOps Bootcamp Documentation, Release 0.0.1 110 Chapter 1. Contents: OSU DevOps Bootcamp Documentation, Release 0.0.1 1.24 Lesson 10: Networking part 2 1.24.1 Agenda • Networking review • DNS • Web Server Setup 1.24.2 Networking Review 1.24.3 Networking Review 1.24.4 Networking Review 1.24.5 Networking Review 1.24.6 Networking Review 1.24.7 Intro to DNS I get the OSL’s home page when visiting osuosl.org 1.24. Lesson 10: Networking part 2 111 OSU DevOps Bootcamp Documentation, Release 0.0.1 112 Chapter 1. Contents: OSU DevOps Bootcamp Documentation, Release 0.0.1 1.24. Lesson 10: Networking part 2 113 OSU DevOps Bootcamp Documentation, Release 0.0.1 1.24.8 How do I know where the page came from? HOST(1) BIND9 HOST(1) NAME host - DNS lookup utility SYNOPSIS host [-aCdlnrsTwv] [-c class] [-N ndots] [-R number] [-t type] [-W wait] [-m flag] [-4] [-6] {name} [server] DESCRIPTION host is a simple utility for performing DNS lookups. It is normally used to convert names to IP addresses and vice versa. When no arguments or options are given, host prints a short summary of its command line arguments and options. 1.24.9 OSL $ host osuosl.org osuosl.org has address 140.211.15.183 How did it know? 1.24.10 DNS Domain Name System • Port 53 My laptop asked a DNS server how to get to osuosl.org. 114 Chapter 1. Contents: OSU DevOps Bootcamp Documentation, Release 0.0.1 1.24.11 Hierarchy of domains 1.24.12 Pop quiz • What’s 15 * 823? • 12345 1.24.13 Recursive DNS • Always gives real answer or error • Vulnerable to cache poisoning 1.24. Lesson 10: Networking part 2 115 OSU DevOps Bootcamp Documentation, Release 0.0.1 1.24.14 Non-recursive • Uses cache or referral • Includes all authoritative-only – root and top-level domain servers 1.24.15 Another Quiz • What’s 15 * 823? • How did you know so quickly? 1.24.16 Caching • You cached the answer. • DNS can be cached at routers, ISPs, and DNS servers to improve performance. – Negative caching: Remember fails • TTL 1.24.17 /etc/resolv.conf Configuration for BIND (Berkeley Internet Name Domain tool) • most common DNS resolver • current version is BIND 9 $ cat /etc/resolv.conf # Generated by resolvconf domain wireless.oregonstate.edu nameserver 128.193.15.13 nameserver 128.193.15.12 Can only handle recursive name servers, no referrals 1.24.18 /etc/hosts Used to skip looking up the DNS Useful for testing web sites Avoid malicious sites $ cat /etc/hosts 127.0.0.1 devops-bootcamp32.osuosl.org devops-bootcamp32 localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 116 Chapter 1. Contents: OSU DevOps Bootcamp Documentation, Release 0.0.1 1.24.19 Load Balancing Multiple IPs bound to a single hostname are returned in random order $ host google.com google.com has address google.com has address google.com has address google.com has address google.com has address google.com has address google.com has address google.com has address google.com has address google.com has address google.com has address 173.194.33.131 173.194.33.132 173.194.33.133 173.194.33.134 173.194.33.135 173.194.33.136 173.194.33.137 173.194.33.142 173.194.33.128 173.194.33.129 173.194.33.130 1.24.20 Records Note: rfc 1035 A hostname -> IPV4 address AAAA hostname -> IPV6 address CNAME like an alias, “Go look up this name’s record” PTR • Pointer to a canonical name, returns name and stops • Used in reverse DNS SOA • Start of Authority for a zone (such as osuosl.org) • Administrator contact info, timers, serial number MX Email (more on this next week) 1.24.21 Reverse DNS Note: http://support.simpledns.com/kb/a45/what-is-reverse-dns-and-do-i-need-it.aspx Reverse segments, then end with in-addr.arpa $ host osuosl.org # could also use dig osuosl.org has address 140.211.15.183 $ dig 183.15.211.140.in-addr.arpa ... ;; AUTHORITY SECTION: 15.211.140.in-addr.arpa. 10795 IN 1.24. Lesson 10: Networking part 2 SOA ns1.auth.osuosl.org. hostmaster.osuosl.org. 1398356710 300 90 117 OSU DevOps Bootcamp Documentation, Release 0.0.1 1.24.22 Web Apps: A Bit of Review • We created a python app called Systemview using the Flask framework • We tested Systemview by running Flask’s built-in webserver on the command line • Systemview ran on a special port we had to open up on the virtual machine 1.24.23 What We Want To Do • Install a production-quality web server on a standard port • Serve Systemview using that web server • Party 1.24.24 Why? • Flask’s web server is not robust or secure • We want to use standard ports for our web apps • We may want to run multiple apps on one server • Web server administration is cool 1.24.25 What is a Web Server Note: Webserver software, not hardware 1.24.26 Webservers Talk HTTP They don’t run code (well, they kinda do) • PHP, Python, Ruby, C don’t run in your browser • Separate servers (usually) run that code, and send the output of the code to the web server to send to your browser • Sometimes those separate servers are web server modules Note: Apache modules generally run in the apache process itself 1.24.27 A Digression: AJAX, JSON and APIs • Browsers render HTML/CSS (layout) • Browsers execute Javascipt (logic) • Javascript can dynamically update the layout • Javascript can handle user interaction • Javascript can call back to the server for more data 118 Chapter 1. Contents: OSU DevOps Bootcamp Documentation, Release 0.0.1 1.24. Lesson 10: Networking part 2 119 OSU DevOps Bootcamp Documentation, Release 0.0.1 • Javascript can process data • Javascript is Client Side Logic 1.24.28 AJAX • Asynchronous Javascript And XML • An http request initiated by Javascript • Javascript listens in the background • The app sends a response containing data • Javascript processes the data • Traditionally XML, but now mostly JSON Note: Lots of issues around security, javascript calling many servers, gathering data, calling servers outside the domain of the originating page, etc. Install RequestPolicy and NoScript, just to see who that web page is talking to while you read it. 1.24.29 JSON JavaScript Object Notation {"menu": { "id": "file", "value": "File", "popup": { "menuitem": [ {"value": "New", "onclick": "CreateNewDoc()"}, {"value": "Open", "onclick": "OpenDoc()"}, {"value": "Close", "onclick": "CloseDoc()"} ] } }} 1.24.30 XML The same text expressed as XML: <menu id="file" value="File"> <popup> <menuitem value="New" onclick="CreateNewDoc()" /> <menuitem value="Open" onclick="OpenDoc()" /> <menuitem value="Close" onclick="CloseDoc()" /> </popup> </menu> 1.24.31 APIs • When web apps talk to web apps • When javascript talks to a web app 120 Chapter 1. Contents: OSU DevOps Bootcamp Documentation, Release 0.0.1 • When curl talks to a web app They talk HTTP, using clearly defined GET or POST params to initiate actions on the remote application. curl http://graph.facebook.com/12345/friendlists curl https://api.github.com/users/osuosl/repos curl http://pub.sandbox.orcid.org/v1.1/0000-0001-7857-2795/orcid-bio Note: Take a look at the source of a web page, look at all the javascript! How much of it is talking to Google, to Facebook, etc? 1.24.32 Let’s Install a Web Server! yum install httpd 1.24.33 Apache What’s this httpd thing? “A patchy web server” - born of many patches to NCSA’s HTTPD (1995) • Venerable, tested, solid • Old, complex, slow (not really that slow) • Many modules for executing code • Many modules for all kinds of other things too 1.24.34 Let’s Serve Some Web Apache’s DocumentRoot is the default place where it will look for files to serve. It maps “/” in the URL to a location on disk http://localhost:8080/index.html ^ "/" is the DocumentRoot We’ll write some HTML in the DocumentRoot for Apache to serve. 1.24.35 But First, Config Files /etc/httpd/conf/httpd.conf DocumentRoot "/var/www/html" <Directory "/var/www/html"> Options Indexes FollowSymLinks AllowOverride None Order allow,deny Allow from all </Directory> Note: Just looking, we are not editing the configs here. Note the DocumentRoot and Directory 1.24. Lesson 10: Networking part 2 121 OSU DevOps Bootcamp Documentation, Release 0.0.1 1.24.36 Wait, What am I Writing Again? HTML: Hyper Text Markup Language Go to the DocumentRoot and create an html file: cd /var/www/html vim index.html <html> <head> <title>This is only a test!</title> </head> <body> <p>Nothing to see here, move along</p> </body> </html> Point your browser to: http://localhost:8080/index.html Note: HTML, is it code? Is it a language? Can you do logic with it? What happens if you forget the <html>? The browser does the rendering, the web server doesn’t care, it just sends the data along. HTTP Content-Type header says what kind of data. 1.24.37 Voila! • Apache receives a request for /index.html • It translates “/” into /var/www/html using the DocumentRoot directive • It looks in /var/www/html for the file “index.html“ • It finds your file and sends its contents, along with HTTP headers, back to your browser Note: Have a look at the page source. Edit the file, remove <html>, etc, look at source again. If time allows, use developer tools, firebug, etc to look at http headers 1.24.38 But I Want to Run Code! Let’s put some PHP code in the DocumentRoot: vim index.php <html> <head> <title>This is only a test!</title> </head> <body> <?php print "Hey, this is PHP!" ?> </body> </html> Then go to http://localhost:8080/index.php 122 Chapter 1. Contents: OSU DevOps Bootcamp Documentation, Release 0.0.1 1.24.39 What Went Wrong? Apache doesn’t know what PHP is, it needs a module to execute the PHP code and return data it can serve yum install php service httpd restart Note: Pop quiz - where do you look to find out what went wrong? Look at log files, talk about them, then look at page source. 1.24.40 Voila, Again. How does Apache know what to do with index.php? /etc/httpd/conf.d/php.conf <IfModule prefork.c> LoadModule php5_module modules/libphp5.so </IfModule> <IfModule worker.c> LoadModule php5_module modules/libphp5-zts.so </IfModule> AddHandler php5-script .php AddType text/html .php DirectoryIndex index.php Note: CentOS, and most distribution system packages put these conf files for modules in place for you. httpd.conf includes everything in conf.d - similar for Nginx 1.24.41 Ok, But I Want To Serve a Python App... There’s a module for that! (Actually several, but we are going to use this one) WSGI: Web Server Gateway Interface • Standardized interface for python apps to talk to web servers • Works with many different servers • Allows separation of python app and web server processes Note: talk about mod_python - runs python scripts directly, not bad for single scripts, but unwieldy for applications and frameworks. 1.24.42 Sounds Great, Let’s Go! yum install mod_wsgi Let’s clone the systemview app into a reasonable location while we are at it 1.24. Lesson 10: Networking part 2 123 OSU DevOps Bootcamp Documentation, Release 0.0.1 cd /var/www git clone https://github.com/DevOpsBootcamp/systemview.git cd systemview git checkout wsgi Note: Talk about the location - can be anywhere, but be consistent - /var/www is actually not in the web root, not accessible by default, don’t put things under the docroot! 1.24.43 Don’t Forget Virtualenv! (in the systemview/ directory) virtualenv --no-site-packages venv source venv/bin/activate pip install -r requirements.txt And lets make sure everything is owned by the web server: chown -R apache ../systemview Note: Web server user/group ownership is a major source of breakage - get cloning/pulling as the wrong user will change perms on files, possibly breaking things 1.24.44 What Makes an App WSGI? systemview.wsgi activate_this = ’/var/www/html/systemview/venv/bin/activate_this.py’ execfile(activate_this, dict(__file__=activate_this)) import sys sys.path.insert(0, ’/var/www/html/systemview’) from systemview import app as application 1.24.45 Configuring Apache for Systemview /etc/httpd/conf.d/systemview.conf WSGISocketPrefix /var/run WSGIDaemonProcess systemview user=apache group=apache threads=5 WSGIScriptAlias /systemview /var/www/systemview/systemview.wsgi <Directory /var/www/systemview> WSGIProcessGroup systemview WSGIApplicationGroup %{GLOBAL} Order deny,allow Allow from all </Directory> (Look for this in systemview/docs/systemview.conf) 124 Chapter 1. Contents: OSU DevOps Bootcamp Documentation, Release 0.0.1 Note: This will go into a vhost some day 1.24.46 Even More Voila http://localhost:8080/systemview There are a lot of steps to getting this app up, wouldn’t it be nice to automate this? Note: Future topics - configuration management and automated deploys, virtual hosts, best practices for app location, Nginx, UWSGI, PHP-FPM, etc 1.24.47 Homework • Deploy Systemview’s master branch with Apache (we merged the database code) • Read about Apache Virtualhosts • Install Nginx and UWSGI, deploy Systemview 1.25 Lesson 11: Devops & Configuration Management Intro 1.25.1 Session Summary • Devops introduction • Configuration management introduction 1.25.2 Devops Introduction What is it? “A software development method that stresses communication, collaboration and integration between software developers and information technology (IT) operations professionals.” [Wikipedia] 1.25.3 Definition of Devops • Software Engineering (Dev) • Technology Operations (Ops) • Quality Assurance (QA) Figure 1.6: Wikipedia (cc) 1.25. Lesson 11: Devops & Configuration Management Intro 125 OSU DevOps Bootcamp Documentation, Release 0.0.1 Figure 1.7: photo by http://thoriseador.deviantart.com/ (CC) 1.25.4 The old view “Dev” side being the “makers” “Ops” side being the “people that deal with the creation after its birth” This siloed environment has created much harm in the industry and the core reason behind creating Devops. Burn down those silos! 1.25.5 History of Devops • mid-2000s “Hey, our methodology of running systems seems to still be in a pretty primitive state despite years of effort. Let’s start talking about doing it better” • Velocity Conf 2008/2009 - increased presentations on “Agile System Administration“ • Agile 2008 Conf - “Agile Infrastructure” BOF – nobody showed up! • 2009 DevOpsDays in Ghent, Belgium - Patrick Debois 1.25.6 The Agile Approach • Iterative, incremental • Requirements change often thus need to be adaptive • Very short feedback loop and adaptation cycle • Quality focus Manifesto: • Individuals and interactions over processes and tools • Working software over comprehensive documentation • Customer collaboration over contract negotiation • Responding to change over following a plan That is, while there is value in the items on the right, we value the items on the left more. 126 Chapter 1. Contents: OSU DevOps Bootcamp Documentation, Release 0.0.1 1.25.7 Adapting Agile to Ops • Widening the principles towards infrastructure “Infrastructure as code” - i.e. configuration management • Integrating ops with dev, QA and product in the product teams • Continuous Integration “Give your developers a pager and put them on call” • Utilizing more specific metric and monitoring schemes 1.25.8 Better Tools enable Devops Explosion of new tools over the past few years: • Release tools (jenkins, travisci, etc) • Config Management (puppet, chef, ansible, cfengine) • Orchestration (zookeeper, noah, mesos) • Monitoring & Metrics (statsd, graphite, etc) • Virtualization & containerization (AWS, Openstack, vagrant, docker) 1.25.9 It’s not NoOps • Existing ops principles, processes and practices have not kept pace • Business & dev teams need more agility to keep up with competitors • Deep dev skill set + Deep ops skill set == awesomesauce • Ops people need to do a little dev • Dev people need to do a little ops 1.25.10 What is Devops Video 1.25.11 Devops Explained: No Horse Manure 1.25.12 Configuration Management What is it? “Configuration management is the process of standardizing resource configurations and enforcing their state across IT infrastructure in an automated yet agile manner.” [PuppetLabs] 1.25.13 History of CM • mid-1990s – “snowflake system”; few systems • Rise of Unix-like systems and commodity x86 hardware increased the need • CFEngine – First release 1993; v2 released in 2002 1.25. Lesson 11: Devops & Configuration Management Intro 127 OSU DevOps Bootcamp Documentation, Release 0.0.1 • mid-2000s through present – More agile CM systems emerged developed with the cloud in mind • 2008 – provisioning and management of individual systems were well-understood 1.25.14 Infrastructure as code • CM enables ops to define their infrastructure in code • Install packages, configure software, start/stop services • Ensure a state of a machine • Ensure policies and standards are in place • Provide history of changes for a system • Repeatable way of rebuild a system • Orchestrate a cluster of services together 1.25.15 CM Platforms • CFengine – Lightweight agent system. Manages configuration of a large number of computers using the client–server paradigm or stand-alone. • Puppet – Puppet consists of a custom declarative language to describe system configuration, distributed using the client–server paradigm. 1.25.16 CM Platforms (part 2) • Chef – Chef is a configuration management tool written in Ruby, and uses a pure Ruby DSL for writing configuration “recipes”. Also a client-server model. • Ansible – Combines multi-node deployment, ad-hoc task execution, and configuration management in one package. Utilizes SSH with little to no remote agents. 1.25.17 Puppet Example • Install apache and start the service • Puppet Domain Specific Language (DSL) package { "apache": name => "httpd", ensure => present, } 128 Chapter 1. Contents: OSU DevOps Bootcamp Documentation, Release 0.0.1 service { name ensure enable require } "apache": => "apache", => running, => true, => Package["apache"], 1.25.18 Chef Example • Install apache and start the service • Ruby code package "apache" do package_name "httpd" action :install end service "apache" do action [:enable, :start] end 1.25.19 CM Platform Comparison • CFEngine scales like mad, not very agile • Puppet – Uses a list of dependencies and figures out what order to run it in – The Puppet DSL can become a blocker and a problem, puppet also has scaling issues • Chef – Executes commands and scripts as they are listed with minimal amount of dependencies – Using ruby offers both its advantages and disadvantages • Each platform offers its own level of complexity 1.25.20 References • http://theagileadmin.com/what-is-devops/ • http://itrevolution.com/the-convergence-of-devops/ • http://en.wikipedia.org/wiki/DevOps • http://en.wikipedia.org/wiki/Agile_software_development • What is DevOps? - In Simple English (video) • DevOps Explained: No Horse Manure (video) 1.25. Lesson 11: Devops & Configuration Management Intro 129 OSU DevOps Bootcamp Documentation, Release 0.0.1 1.25.21 Traditional Development Workflow Scenario: Developer Mary Smith wants to deploy SystemView to a server administered by Ivan Bofh, a strict oldschool sysadmin. email conversation link 1.25.22 Email #1 >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> On April 3, 2013, at 4:22 PM, Mary Smith <[email protected]> wrote: Ops team, As discussed in the release schedule distributed by Mr. Bossman on 2/5, the development team is ready to deploy our flagship product SystemView this week. We will need Python 3.4 an Virtualenv on the production server, as well as a correctly configured Nginx vhost to direct users to the site. When we log into the production server to deploy the app’s code, we’ll need permission to write to /var/www and all of /etc for configuration reasons. Please also create the user and tables detailed in the attached spreadsheet on our MySql 5.7 database. Mary Smith Lead Developer, CruftWare SystemView product division 1.25.23 Email #2 >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> On April 5, 2013, at 9:15 AM, Ivan Bofh <[email protected]> wrote: Mary, Our production systems are standardized to CentOS 6, so Python is only supported up to version 2.6. The Python 2.6 version of virtualenv can be installed after you work with legal to file documentation of a full security audit of the package. Providing any account, let alone root, to developers on a production system is absolutely out of the question. Just document the app’s deployment process clearly and we’ll handle it. Ivan Bofh Senior Systems Engineer, CruftWare 1.25.24 Email #3 >>>> >>>> >>>> >>>> >>>> >>>> >>>> 130 On April 5, 2013, at 11:32 AM, Mary Smith <[email protected]> wrote: Ivan, That sounds like it will be simpler to just install the dependencies directly on the server instead of using virtualenv. I should be able to include this in the Jenkins configuration, as long as the CI users is running as root. Chapter 1. Contents: OSU DevOps Bootcamp Documentation, Release 0.0.1 >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> Speaking of which, the development team will need access to Jenkins or other continuous integration in order to automatically update the site when changes are pushed. Is mysql-dev installed yet? Also please confirm that the database is at systemview-prod.mysql57.cruftware.com. Mary Smith Lead Developer, CruftWare SystemView product division 1.25.25 Email #4 >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> On April 6, 2013, at 10:08 AM, Ivan Bofh <[email protected]> wrote: Mary, Why do you need to use Jenkins? I Googled it and it looks like a non-standard and immature implementation of some of CFEngine’s features. Just send me a CFEngine configuration file for the settings that you need. The updates can be done with an SVN post-commit hook. Due to administrative decisions that Mr. Bossman explained in a company-wide memo a couple of months ago, absolutely no dev libraries may be installed on production servers. Servers are for serving, not for compiling. Ivan Bofh Senior Systems Engineer, CruftWare 1.25.26 Email #5 >> >> >> >> >> >> >> >> >> >> >> >> >> >> On April 6, 2013, at 10:14 AM, Mary Smith <[email protected]> wrote: Do you at least have the Nginx vhost and uWsgi installation ready? Mary Smith Lead Developer, CruftWare SystemView product division On April 6, 2013, at 1:53 PM, Ivan Bofh <[email protected]> wrote: We don’t use Nginx or uWsgi. The specs should have said to convert the app to work with Apache and mod_wsgi for production deployment. Ivan Bofh Senior Systems Engineer, CruftWare 1.25.27 Email #6 > On April 6, 2013, at 2:37 PM, Mary Smith <[email protected]> wrote: > > Ivan, > > What’s the URL for the database? > 1.25. Lesson 11: Devops & Configuration Management Intro 131 OSU DevOps Bootcamp Documentation, Release 0.0.1 > Mary Smith > Lead Developer, CruftWare SystemView product division > On April 6, 2013, at 4:22 PM, Ivan Bofh <[email protected]> wrote: Mary, You’ll have to contact Sharon Negative ([email protected]), our DBA, and file a ticket to get the database access. She won’t be back from vacation for another 2 weeks so it might take awhile. Ivan Bofh Senior Systems Engineer, CruftWare 1.25.28 DevOps Workflow Scenario: DevOps-oriented developer Simon Johnson wants to deploy SystemView to a server administered by Ada Opdev, a DevOps-oriented sysadmin. irc conversation link 1.25.29 IRC #1 14:03 < JnomiS> AdaOpdev: hey, all the systemview tests are passing on python 3.4 14:04 <@AdaOpdev> yay! I’ll spin up a VM on the cluster with the production cookbook 14:04 < JnomiS> that’ll be at sysview23dev.internal.ourcorp, right? 14:05 <@AdaOpdev> actually we’re migrating over to a new test cluster 14:05 <@AdaOpdev> could you use sysview23.dev.ourcorp instead? 14:06 < JnomiS> sure 14:06 <@AdaOpdev> it’s set up for passwordless ssh login with your ldap account 14:07 < JnomiS> ok, awesome. 14:07 < JnomiS> thanks! 1.25.30 IRC #2 15:12 < JnomiS> hmm, I’ve been building mysql-python for the app with the mysql dev libraries, but it doesn’t look like you have those in production 15:22 < JnomiS> also, what database should i connect to from the dev instance? I’ve been using MySql 5.7 in testing 15:25 <@AdaOpdev> just get me a list of the names and databases you’ll need, and I’ll plug them into our MySql Chef cookbooks. 15:25 <@AdaOpdev> I checked the ORM docs, and you’re not actually using any features that changed between MySql 5.5 (which is what we’ve got in production right now) and 5.7 15:26 < JnomiS> okay, I’ll run the tests with mysql5.5 15:27 < JnomiS> everything seems to be working fine locally 132 Chapter 1. Contents: OSU DevOps Bootcamp Documentation, Release 0.0.1 1.25.31 IRC #3 16:00 < JnomiS> ugh, I totally forgot -- I’ll need to get root on the dev vm so I can install uwsgi and nginx 16:00 <@AdaOpdev> We actually manage UWsgi and Nginx through chef as well. Have you written cookbooks before? 16:02 < JnomiS> nope, I’ve always just used yours :) 16:05 <@AdaOpdev> http://reiddraper.com/first-chef-recipe/ and https://www.digitalocean.com/community/articles/how-to-create-simple-chef-cookbooks are a couple of good places to start 16:05 < JnomiS> thanks! 1.25.32 IRC #4 16:52 16:53 16:54 16:54 16:54 16:55 16:55 16:57 16:57 16:58 < JnomiS> do we have jenkins deployed anywhere yet? < JnomiS> I’d like to get continuous integration set up <@AdaOpdev> I’ve heard of Jenkins but not worked with it much < JnomiS> yeah, it’ll automate that deploy hook mess we used to have <@AdaOpdev> cool! <@AdaOpdev> let’s talk to our boss about getting an instance provisioned < JnomiS> okay, I’ll email him about it and cc you <@AdaOpdev> thanks <@AdaOpdev> for now let’s keep using Chef for everything we can < JnomiS> okay, sounds good. 1.25.33 Non-DevOps • Poor communication, territorialism (silos) • Development environment wildly different from production • Sysadmin averse to changes because environment is hard to test • Little willingness to cooperate or educate (trust/teamwork) • It can get even worse – More people in email thread = more confusion – Bikeshedding about top post vs bottom post – Mary is surprisingly clear about her exact requirements 1.25.34 DevOps • Developer tests on VM with same config as production • Realistic expectations about access and compatibility • More automation, configuration management • Sysadmin can debug the code and help developer • Developer and sysadmin help and educate one another (Chef, Jenkins) • Distributed tasks mean fewer choke points (single person who can block task) 1.25. Lesson 11: Devops & Configuration Management Intro 133 OSU DevOps Bootcamp Documentation, Release 0.0.1 1.26 Lesson 12: Configuration Management Note: starting off with cs312 content 1.26.1 Large site management • Configuration Management • Automation • Centralized • Standardization • History Note: • Automating most of what you do • Scaling the configuration management is key • Centralized so that a team can work together • History via SCM (source control management) 1.26.2 What config management does • Verify permissions for security/mgmt purposes • Distribute config files and scripts • Control batch jobs • Ensure packages are up to date • Look for file changes Note: Ensures that critical configs always are set like you want. No mucking around by a bad admin 1.26.3 Configuration Management • System-wide settings – sshd, ntp, ldap, etc • Group settings – Web, email, database • Standardize settings globally • Easy to troubleshoot 134 Chapter 1. Contents: OSU DevOps Bootcamp Documentation, Release 0.0.1 1.26.4 CF Management Tools • CFEngine – Mature, widely used • Puppet – Getting mature, different concept • Chef – Very new Note: Different concepts Feature set different between them Each have their own issues 1.26.5 Puppet Let’s install puppet $ yum install puppet 1.26.6 Declarative Configuration • We �declare’ the desired state of the system • Puppet does the necessary work to make the system match our declaration • We can save these declarations in a repository, just like code 1.26.7 Puppet Resources Puppet knows about “resources” on the system $ puppet describe -l Look at all those things Puppet can manage out of the box. We’re most interested in these: file group package service user - Manages files, including their content, owner ... Manage groups Manage packages Manage running services Manage users 1.26.8 Resources Have Attributes # let’s look at the vagrant user $ sudo puppet resource user vagrant user { ’vagrant’: ensure gid groups home password => => => => => ’present’, ’500’, [’wheel’], ’/home/vagrant’, ’$1$aDsSD/Uu$.tXG5wN.TSit1AP5ZyphB0’, 1.26. Lesson 12: Configuration Management 135 OSU DevOps Bootcamp Documentation, Release 0.0.1 password_max_age password_min_age shell uid => => => => ’99999’, ’0’, ’/bin/bash’, ’500’, } We can declare a value for any of those attributes, and Puppet will make it happen. Note: the password is a password hash, as appears in /etc/shadow - don’t put passwords in puppet manifests! 1.26.9 Puppet Manifests Puppet keeps its declarations in manifest files. We can write a manifest to create a user: $ sudo su $ vim users.pp user {’yournamehere’: ensure => ’present’, home => ’/home/yournamehere’, groups => [’wheel’, ’vagrant’], shell => ’/bin/tcsh’, } 1.26.10 Pull the Strings Lets run our manifest. > puppet apply user.pp Notice: Compiled catalog for devops-bootcamp.osuosl.org in environment production in 0.12 seconds Notice: /Stage[main]/Main/User[yournamehere]/ensure: created Notice: Finished catalog run in 0.13 seconds Note: we are using stand-alone mode, manually running an individual manifest 1.26.11 Declarations Are Idempotent Lets run our manifest again. > puppet apply user.pp Notice: Compiled catalog for devops-bootcamp.osuosl.org in environment production in 0.12 seconds Notice: Finished catalog run in 0.02 seconds The state of the system is already what we declared it should be, so applying the manifest again doesn’t change anything. Note: idempotency is important, the puppet master daemon will run periodically, and it is important that running the same commands over and over does not have cumulative effects 136 Chapter 1. Contents: OSU DevOps Bootcamp Documentation, Release 0.0.1 1.26.12 Packages and Services We can declare that our system should have certain things installed and running. apache.pp: package{’httpd’: ensure => ’present’ } service{’httpd’: ensure => ’running’, enable => ’true’, require => Package[’httpd’], } Note: The �service’ block makes sure that the httpd service is started, and that it is enabled, the �require’ directive tells the service that it must wait until the package �httpd’ is processed. Services are anything you would start with “service x start” and packages anything you would install with “yum install x” 1.26.13 Puppet Config Where does Puppet keep its configuration files? Note: the audience really ought to know where to start looking by this point 1.26.14 /etc/puppet $ ls /etc/puppet auth.conf modules puppet.conf • puppet.conf - systemwide configuration • auth.conf - puppet agent configuration • modules - we’ll talk about that later Note: there isn’t much of anything we need to worry about in any of the config files 1.26.15 The Site Manifest We want to move beyond running individual manifests on �/etc/puppet/manifests/site.pp� is the place to put your site’s configuration. the command line. $ mkdir /etc/puppet/manifests $ vim /etc/puppet/manifests/site.pp 1.26.16 But First, Nodes • Nodes are defined in the site manifest • A node is a single machine, identified by its FQDN (Fully-Qualified Domain Name). 1.26. Lesson 12: Configuration Management 137 OSU DevOps Bootcamp Documentation, Release 0.0.1 • You can define many nodes. • You can add declarations to a node definition. • A special �default’ node will be used if a node’s name can’t be found. We will put our configurations in the default node for now. Note: a node can inherit from another node, but this is discouraged 1.26.17 An Example Site Manifest node default { file {’/etc/issue’: path => ’/etc/issue’, mode => 644, ensure => present, content => "Welcome to the DevOps Bootcamp VM.\n", } package{’httpd’: ensure => ’present’ } service{’httpd’: ensure => ’running’, enable => ’true’, require => Package[’httpd’], } } Note: have we talked about /etc/issue? The file resource lets you declare the filename, ownership, and contents. You can also have it copy files from the module onto the node instead of manually inserting content here. 1.26.18 The Master and the Agent Puppet uses a Master/Agent architecture. • The Master reads the �site.pp� and listens for an Agent to contact it. • Agents run on nodes, they contact the master to get their configuration • Master and Agent can be on the same machine. • When they are on different machines, they need an SSL certificate to authenticate Run the master on your vm: $ puppet master Note: the master will background by default and log to syslog, but you can run it in the foreground with –nodaemonize and get extra logging on stdout with –verbose 138 Chapter 1. Contents: OSU DevOps Bootcamp Documentation, Release 0.0.1 1.26.19 The Agent The agent will look for its master on the host �puppet� by default. Lets add the hostname �puppet� to our local host definition in /etc/hosts, so it will look on the local machine. $ vim /etc/hosts 127.0.0.1 devops-bootcamp.osuosl.org devops-bootcamp localhost localhost.localdomain localhost4 localhost4.localdomain4 puppet ^^^^^^ Now run the agent in test mode: $ puppet agent --test --verbose Note: the agent will also background by default, the –test flag prevents that and shows us what is going on. In a production environment, the master and agent would always be running in the background, usually started as services on boot. 1.26.20 Modules We can keep adding configurations to site.pp, but it’s going to get long and messy. Let’s use modules instead. • Modules are classes • Modules encapsulate a set of related configurations • Modules make it easy to apply configurations to many nodes • Community created modules already exist for almost everything Note: community or puppetlabs modules vary in quality, always read the docs thoroughly 1.26.21 Module Structure /etc/puppet/modules/ modulename/ files/ some_file manifests/ init.pp some_other_manifest.pp Note: that files directory is served to the puppet agent like a fileserver, file resources can declare their source attribute like “puppet:///modules/module_name/some_file” and the file will be copied into place 1.26.22 The Bootcamp Apache Module # $ $ $ $ Let’s create a module for our Apache configuration. cd /etc/puppet/modules mkdir bootcamp_apache mkdir bootcamp_apache/manifests vim bootcamp_apache/manifests/init.pp 1.26. Lesson 12: Configuration Management 139 OSU DevOps Bootcamp Documentation, Release 0.0.1 class bootcamp_apache { package{’httpd’: ensure => ’present’ } package{’mod_wsgi’: ensure => ’present’ } service{’httpd’: ensure => ’running’, enable => ’true’, require => Package[’httpd’], } } Note: it is good practice to namespace the class name of your modules, so instead of just �apache’, we use bootcamp_apache, which won’t collide with any other apache related module. 1.26.23 Site.pp Modularized node default { file {’/etc/issue’: path => ’/etc/issue’, mode => 644, ensure => present, content => "Welcome to the DevOps Bootcamp VM.\n", } include bootcamp_apache } Note: the include statement assumes a module located in modules/ under the pupper config dir. The name is the class name of the the module, which is not necessarily the directory name the module is stored under (but it is much easier to name them the same) 1.26.24 Community Modules We need MySql installed for our SystemView app, as well as a database, user, and permissions. We could do all that with package, service and file resources, but there is a better way, the puppetlabs-mysql module. https://github.com/puppetlabs/puppetlabs-mysql (It’s in Git, how convenient!) $ cd /etc/puppet/modules/ # We’ll clone into a directory named mysql, because that’s the module name $ git clone https://github.com/puppetlabs/puppetlabs-mysql.git mysql We can include this module’s class into our site manifest or our own modules. 1.26.25 The Bootcamp Mysql Module We want to create a database and users, so lets make a module and not clutter up the site.pp 140 Chapter 1. Contents: OSU DevOps Bootcamp Documentation, Release 0.0.1 $ $ $ $ cd /etc/puppet/modules mkdir bootcamp_mysql mkdir bootcamp_mysql/manifests vim bootcamp_mysql/manifests/init.pp class bootcamp_mysql { class { ’::mysql::server’ } } ::mysql::server causes Puppet to install MySql and makes available many methods for managing MySql. Note: Calling the �mysql’ class essentially includes that module, which includes a package declaration insuring mysql is installed. It is easy to explore the module files and see what is in it. 1.26.26 Databases, Users, and Grants class bootcamp_mysql { class { ’::mysql::server’ } mysql_database { ’systemview’: ensure => ’present’, charset => ’utf8’, collate => ’utf8_swedish_ci’, } mysql_user { ’vagrant@localhost’: ensure => ’present’, } mysql_grant { ’vagrant@localhost/systemview.*’: ensure => ’present’, options => [’GRANT’], privileges => [’ALL’], table => ’systemview.*’, user => ’vagrant@localhost’, } } Note: the mysql module has a lot of stuff in it, there isn’t time to get into it all. 1.26.27 Test It Out $ puppet agent --test --verbose 1.26.28 Further Reading • http://docs.puppetlabs.com/learning/introduction.html • https://github.com/puppetlabs/puppetlabs-mysql 1.26. Lesson 12: Configuration Management 141 OSU DevOps Bootcamp Documentation, Release 0.0.1 1.27 Lesson 13: Contributing to Open Source 1.27.1 Getting started in Open Source • Carlos’s CS419 in a nutshell • Relevant books – The cathedral and the bazaar (Eric Raymond) – Producing Open Source Software (Karl Fogel) – Open Advice (Lydia Pintscher) 1.27.2 Joining vs Starting a Project • Scratch an itch. • Research first • Revive dead project vs. rewrite • Get involved with communities even if you plan to start your own – Learn from their examples 1.27.3 Know your licenses Note: I am not a lawyer. http://opensource.org/licenses is pretty cool ” Freely used, modified, and shared” MIT/X11 Short, permissive, says attribution and no liability. Doesn’t discuss copyright. Can convert to Apache 2 Apache Short, permissive, goes in every file, grants patent rights from contributors to users, author keeps copyright. Plays nice with GPL3 (?) BSD Attribution, keep copyright, no liability AGPL Demands source distribution even when software not distributed (for cloud/hosted) GPL Viral, copyleft. Viral = infects entire program if it links to GPL library or uses a single line of GPL’d code LGPL Fixes library linking issue with GPL CC Non-code content • MIT/X11 • Apache • BSD 142 Chapter 1. Contents: OSU DevOps Bootcamp Documentation, Release 0.0.1 • AGPL/GPL/LGPL • Creative Commons • http://choosealicense.com/ 1.27.4 Assessing a new community • Elitism vs welcomeness • Communication style • Documentation and guides • Faster or slower change? 1.27.5 Getting involved • Lurk more • Make accounts – Mailing lists – IRC channels – Wikis – Issue trackers • Your nick is your reputation • It’s okay to make mistakes... But learn from them. 1.27.6 Finding a project Note: First contributions will be metric of how nice they are to newbies 1.27. Lesson 13: Contributing to Open Source 143 OSU DevOps Bootcamp Documentation, Release 0.0.1 144 Chapter 1. Contents: OSU DevOps Bootcamp Documentation, Release 0.0.1 There’s a thing where older project members get grumpy at newbies because they’ve answered the question over and over... read docs/faq then improve them • Openhatch • Easy bugs • GSOC submitters who didn’t get enough interns • Search by language • Search by project type – find something that interests you (web dev? bioinformatics? video games?) • Your immediate payment for contributions will be satisfaction, so pick something satisfying 1.27.7 First steps Note: It will feel like you have only a vague idea what you’re doing. This means you’ve found a project that’s challenging and that you’ll learn from. • Lurk awhile then ask • Write a test • Fix a typo • Deploy and update the installation docs 1.27.8 DevOps Concerns • Configurations often managed in public repos 1.27. Lesson 13: Contributing to Open Source 145 OSU DevOps Bootcamp Documentation, Release 0.0.1 146 Chapter 1. Contents: OSU DevOps Bootcamp Documentation, Release 0.0.1 • Root can’t be handed out to just anyone • Build trust, contribute to project consistently • Practice with the tools they use 1.27.9 Your Homework • Find a project that you’d like to get involved with this summer • Join IRC, mailing lists, etc. • Pull the code and run its tests using what you’ve learned • Find something you can contribute to the project • Discuss how it’s going in #devopsbootcamp on irc.freenode.net 1.27.10 Questions? Any questions about anything from this year? • Conferences: OSBridge, OSCON may have free expo hall passes • In Corvallis? Want to come to the OSL and see what we do, pair program, etc.? • No meeting next week – please leave feedback! 1.28 Lesson 14: Review 1.28.1 Today’s Goals • Working VM • Run Systemview app • Connect to Systemview from browser in host • Understand what’s happening 1.28.2 How to set up a VM 1.28.3 Enable VirtualBox (VT extensions) Install VirtualBox Install Vagrant 1.28.4 Linux # clone git clone https://github.com/DevOpsBootcamp/devopsbootcamp-vagrant.git # start up cd devopsbootcamp-vagrant 1.28. Lesson 14: Review 147 OSU DevOps Bootcamp Documentation, Release 0.0.1 vagrant up # access vm vagrant ssh 1.28.5 IRC • Use the LUG guide • #devopsbootcamp on irc.freenode.net • Also join the Mailing List 1.28.6 GitHub Account SSH keys ssh-keygen -t rsa Your public key is in ~/.ssh/id_rsa.pub by default. GitHub -> Account Settings (icon in upper right) -> SSH keys -> Add SSH key 1.28.7 Joining the Github Org • Ping Ramereth in the IRC channel to add you 1.28.8 Installing the Web App 1.28.9 Prerequisite tools On host machine, in devopsbootcamp-vagrant directory: git pull vagrant up vagrant ssh 1.28.10 The Magic Script Now that you’re in the guest machine: [email protected]:DevOpsBootcamp/catch-up.git cd catch-up ./catch-up.sh 1.28.11 Branches Start one with 148 Chapter 1. Contents: OSU DevOps Bootcamp Documentation, Release 0.0.1 git checkout -b branchname Which are you on git branch Switch with git checkout branchname 1.28.12 Connecting to the App In the guest machine, with virtualenv activated, python systemview.py Point the browser of your host machine at http://127.0.0.1:5050 If changes in the app don’t show up in your browser, use F5 to hard refresh 1.28.13 Where am I? 1.28.14 In virtual machine? • Did you vagrant ssh? 1.28.15 In a repo? • git status 1.28.16 On a branch? # Show current branch $ git branch # create new branch, called branchname $ git checkout -b branchname 1.29 Lesson 16: Email • Email service – How it works – Configuration Postfix – Planning 1.29. Lesson 16: Email 149 OSU DevOps Bootcamp Documentation, Release 0.0.1 1.29.1 Email: System Components • Mail User Agent (MUA) • Mail Transport Agent (MTA) • Delivery Agent (MDA) • Access Agent (MAA) Note: MUA • lets users read & compose mail • Thunderbird, mutt, etc MTA • routes messages to other machines • sendmail, postfix, exim, qmail MDA • places messages in local store • mail.local, procmail MAA access to mail store (i.e IMAP, POP) 1.29.2 Email: System Components Note: The most confusing part about email is understanding the routing. Knowing the different components is important to fully grasping it. 150 Chapter 1. Contents: OSU DevOps Bootcamp Documentation, Release 0.0.1 1.29.3 Transport Agents Accept mail form user agent Postfix More common, easier to configure & use Sendmail Highly configurable, steep learning curve Exim Similar to Postfix Qmail Logging is horrid, but some people like it Note: Postfix is the easiest to learn and understand, but queue management is a “black box” Sendmail & qmail is great for high volume sites, but postfix/exim still perform great. Sendmail has great options for queue management Features to look out for: • SASL (authenticated SMTP) • Queue Management 1.29.4 Delivery Agents & Message Stores • procmail – great filtering • maildrop – newer procmail-like • mail.local • Message Stores – mbox – one large file, locking problems – maildir – one file per message, great for IMAP Note: Consider scaling issues for the mailstore. Generally maildir is the best & most compatible option 1.29.5 Anatomy of a Mail Message • Envelope – Destination email address • Headers – Record of variety of important information – Great for tracking down problems • Body of the message Note: Headers: • Know how to identify and track queue id’s • Originator starts at the bottom 1.29. Lesson 16: Email 151 OSU DevOps Bootcamp Documentation, Release 0.0.1 • Headers can be forged • X- Headers non-RFC headers • Message ID is always unique 1.29.6 MTA Log Files • Track emails via queue ID – Look something like: 03CE18819A • Tracking via message ID • Informational fields – to, from, status, relay, etc • Log files differ between each MTA Note: Being able to read log files is important. 1.29.7 Configuring Postfix • /etc/postfix – main.cf – main config file – master.cf – postfix process config file – /etc/aliases – local email forwarding • Set to relay email to central MTA – relayhost = [smtp.osuosl.org] – myorigin = osuosl.org – /etc/aliases – root: [email protected] Note: relayhost: [smtp.osuosl.org] vs. osuosl.org • [smtp.osuosl.org] goes directly to smtp.osuosl.org • �osuosl.org’ does DNS lookup and uses MX Make sure you run “newaliases” after updating /etc/aliases Reloading postfix is ideal too To test email: echo “this is a test” | mail root@localhost 1.29.8 Sendmail • Config files created via m4 – Makefile 152 Chapter 1. Contents: OSU DevOps Bootcamp Documentation, Release 0.0.1 • Always edit the .mc files not the .cf files • Remember to rebuild .cf files with make • Extremely configurable Note: Config files in /etc/mail usually Primary file to edit should be sendmail.mc 1.29.9 Email: Viruses & Spam • Virus – Clamav – Ensure freshclam is running too • Spam – Spamassassin • All-in-one – Amavis • Check abuse emails Note: Make sure you have enough CPU & RAM for Spam checking Neglecting abuse emails may get you blacklisted For larger infrastructures, have dedicated machines to process spam Important to keep these updated 1.29.10 Email: Infrastructure Implementation • Small sites – Can have MTA/MDA/etc all on the same server • Medium sites – Separate MTA from MDA • Large sites – Split outgoing mail and incoming Note: Consider resources, redundancy, & scalability. MDA is hardest to scale. • Look at Cyrus Murder for large scalability • dovecot is another option 1.29.11 Email: Security • On General servers: – Only listen on localhost – Don’t allow other hosts to relay through it – Relay all outbound mail through central host 1.29. Lesson 16: Email 153 OSU DevOps Bootcamp Documentation, Release 0.0.1 • On Email servers: – Restrict relaying to trusted networks – Implement antivirus & spam protection Note: Always test new configurations to ensure spammers can’t relay mail through your server Having dedicate outbound servers will ensure they always catch spam/viruses/etc 154 Chapter 1. Contents: CHAPTER 2 Indices and tables • genindex • modindex • search 155 OSU DevOps Bootcamp Documentation, Release 0.0.1 156 Chapter 2. Indices and tables Bibliography [Wikipedia] http://en.wikipedia.org/wiki/DevOps [PuppetLabs] http://puppetlabs.com/solutions/configuration-management 157
© Copyright 2024 Paperzz