Reflections: Takeaways - Jenkins User Conference July 26 2015

Jenkins

“Jenkins is a CI platform that has become widely adopted and successful community-driven open source project. It manages and controls development lifecycle processes of all kinds, including build, document, test, package, stage, deployment, static analysis and many more. Today the Jenkins community offers well over 1,000 plugins that allow Jenkins to integrate with almost any popular technology today. The principles behind the Jenkins community – extensibility, inclusiveness, low barriers to participation – have been keys to its success.”¹

JUC 2015

Last month, I had been to the Jenkins User Conference (JUC). Since many teams are using or considering using Jenkins, I am writing this blog for everyone’s benefit. The conference illustrated various features, plugins and best practices of using Jenkins for Continuous Integration and Delivery. The sessions in the conference had different tracks with respect to expertise levels of the users and different job perspectives namely - DevOps, Software Developers and Automation Testers. In addition, Cloudbees invited speakers from companies, HP, FINRA, IBM, and Red Hat, who gave excellent talks and demonstrated their CI/CD lifecycles.

Keynote 1

The conference starts with an enthusiastic keynote by Kohsuke Kawaguchi, the creator of Jenkins. He is a well-respected developer and excellent speaker. He speaks about how CI is becoming increasingly important in Software Development. He describes the evolution of Jenkins into the leading open-source continuous integration server. Today, Jenkins integrates with a 1000 of auxiliary plugins for deployment and automation like WorkFlow, Docker, Chef, Puppet, Git, Vagrant, etc. There is a natural progress of task automation in development, testing, release management and shipping. Automating and integrating these processes are no longer isolated individual concerns but have become an organizational concern and teams are working collectively to address delays and discrepancies in them.

Workflow is a process of defining choreography across all these individual task automations in a flexible way as a simple script using DSL (custom vocabulary on a scripting language) for implementing continuous delivery pipeline. The Workflow plugin suite attempts to make it possible to directly write that script, what people often call a flow, while integrating with Jenkins features like slaves and publishers.

Useful features of this plugin-

· Pipeline definitions are a simple script that can be tested and be loaded into version control.

· Reusable subroutines in DSL.

· Pause and resume execution in flow

· Pipeline sequential stages to throttle concurrency.

Docker is an open source platform for building, shipping, running distributed applications. It manages and runs lightweight Linux containers with isolation and portability.

Kohsuke gives a briefing of the following Jenkins-Docker plugins:

Docker commons plugin: This plugin enables credential management

Docker Hub Notification Trigger: This enables Jenkins to trigger new jobs upon receiving the notification of an image updated in the repository.

Docker Tracebility Plugin: This plugin allows tracking of creation of Docker containers in Jenkins.

Keynote 2

The second Keynote speaker is Gene Kim, a software researcher/ entrepreneur and one of the authors of the book “The Phoenix Project: A Novel about IT, DevOps and Helping Your Business Win”². He focuses mainly on the importance of principles and practices of DevOps in SDLC. He shares anecdotes about how Operations engineers work really long hours to keep production systems up and running.

He explains the following are lessons from his book:

Lesson 1: Business value of DevOps is undervalued.

Many leading technology providers, retailers, manufacturers, government organizations and higher educational organizations are currently using DevOps principles and practices and thereby increasing success rate and profitability.

To support the above fact, Gene shares the following statistics³:

· The companies with DevOps do more frequent deployments with smaller changes and lesser preparation; they have higher lead times than peers.

o 30 times more - frequent deployments

o 8000 times faster - lead times than peers

· High performers are more agile and more reliable.

o 2 times more - change success rate

o 12 times faster - mean time to recover

· High performers win in the Marketplace.

o 2 times more likely to exceed profitability, market share and productivity goals

o 50% higher market capitalization growth over 3 years.

Lesson 2: DevOps is as good for Developer as it is for Ops Engineers

In 2008, for the rollout of Facebook chat, Facebook had been testing the messaging in production for over a year before they released Facebook chat for 98 million users, overnight, without a hitch. Testing in production with frequent fixes was proved to be successful for a large-scale rollout like Facebook chat.

Lesson 3: The need for high-trust management

In 2012 the company, Knight capital group, had a 15-minute deployment failure that caused 440 million trading loss and the company had to be sold immediately over the weekend⁴.

Two possibilities were reported for the failure:

(i) Change control failure

Change control team could’ve had quick detection of the failure and recovery mechanisms.

Management’s reaction:

Add one more level of approval authority.

Phoenix project’s take on this:

The number (levels) of approvers increases => (implies or leads to)

Deployment engineer/ developer not empowered to take dynamic decisions during the deployment => multi million-dollar losses.

Resolution:

Considering the person doing the deployment knows the most about the code change, he must be empowered to take any dynamic decisions during the deployment.

(ii) Testing failure

Proper testing could’ve enabled earlier detection of the failure and prevented it from happening.

Management’s reaction:

Do more testing.

Phoenix project’s take on this:

More testing => longer development cycles => less frequent deployments => bigger chunks of changes in each deployment than before => deployments are more prone to errors.

Resolution:

Do the deployments more frequently, with smaller chunks of changes.

In most scenarios, management would disagree with the above resolutions because of low trust. Thus there is an imperative need to transform low trust organization culture to high trust where people are empowered to deliver software in safe environment.

Lesson 4: DevOps is not just for unicorns (Google, Facebook etc.) it is for horses, too.

Gene gives anecdotes of number of companies (other than the unicorns) that are using DevOps principles and thereby drastically reducing their lead-time from commit through testing through production deployment. Reducing deployment intervals decreases preparation time and technical issues, and increases product efficiency and customer satisfaction.

The attendees were extremely impressed by both the keynotes and many expressed interest in attending “DevOps Enterprise Summit 2015”⁵, a conference where successful companies present case studies of DevOps transformation and best practices of Continuous Delivery.

In the following paragraphs, I have given a gist of sessions that were interesting at JUC.

Session 1: The Secret to Scaling Performance Test Automation with Jenkins and JMeter

Alon Girmonsky, BlazeMeter CEO & Founder, demonstrates an open source test automation framework, Taurus. It is a python based framework for painless integration of Jenkins and JMeter for Automation testing. This framework helps dynamically create JMeter tests from a configuration file and run from Jenkins. It enables simultaneous parallel runs of any number of JMeter tests from a single command line and feeds back the results to Jenkins. It makes JMeter scripts version control friendly and human readable.

Session 2: How to Optimize Automated Testing with Everyone's Favorite Butler

Andrew Phillips, the Vice President of XebiaLabs explains the best practices of performing automation testing with Jenkins that increase confidence in the code and help deliver code faster in production.

Best practices ³

1) Testers are developers

2) Mirror test code to production code

3) Measure quality

4) Link test cases to use cases

5) Slice and dice tests based on teams, use cases, components, project, system-under-test.

6) Tests must not depend on other tests

7) Avoid repetition of boilerplate mock code

8) Radical parallelization – fail faster, avoid nightlies

9) Reuse of components

Session 3: Jenkins Workflow: What’s up

Jesse Glick joined Kohsuke Kawaguchi at CloudBees few years back to create the Workflow functionality. He describes how the Workflow functionality for Jenkins lets the users integrate complex, long-running processes with Jenkins as a management interface, going beyond basic continuous integration (CI).

With Workflow, you can³:

1. Model both simple and complex software pipelines

2. Support workflow process steps that require human interaction

3. Recover from the point of failure, as opposed to starting over

4. Have UI for all this

5. Enable manual intervention- approval

Session 4: “It's Not Called Continuous Integration (CI) for Nothing”

Dan Boutin illustrates that SOASTAs maintains continuous quality and facilitates speed of delivery in its software development lifecycle with Jenkins as follows-

After a code change is made, Jenkins runs unit tests, spins up cloud environment, runs iterative load tests and mobile automation tests. These results are updated daily to the teams and used to manage performance baselines. Real time data is used to iterate development and test plans.

The main takeaways that SOASTA offers are³:

· Understand your requirements

· Identify the cloud advantages for you

· Automate the obvious and the most critical

· Connect to a continuous process

· Align teams with actionable information

Session 5: Scaling Your Jenkins Jobs

Khai Do from HP demonstrates Jenkins Job Builder (JJB) for scaling continuous integration for automating of thousands of jobs.

JJB is an open source, command line tool to create and manage Jenkins jobs that facilitates easy job definitions for multiple jobs.

Benefits of JJB are³:

· Common definitions for multiple jobs

· Easy maintenance

· Human-readable job definitions

· Works with Jenkins –REST api

· Default definitions to reuse parameters

· Macro definitions to reuse plugins

· Template definitions to reuse jobs

· Projects to group related jobs

· Job groups to organize related job templates

· Test jobs before deployment

· Automated job removal

· Cache for idempotent job updates

· Documented

Session 6: Getting Groovy with Jenkins

This was the last and one of the best sessions that I attended. The speaker, Lorelei from IBM, talks about using groovy scripts for enhancing build, test and deploy pipelines in Jenkins via various Groovy plugins.

Two types of Groovy scripts can be run via Jenkins-

1) Groovy script that runs in a forked JVM on the slave where build is run.

2) System Groovy script that runs inside the Jenkins Master JVM, has access to all internal objects of Jenkins and can be used to alter the state of Jenkins.

Groovy scripts can be run at ‘Build step’ or ‘Postbuild step’. Multiple resourceful script examples were given during the presentation which include- some administrative jobs, Jenkins build and post-build jobs like³ -

· Restart Jenkins nodes/Toggle nodes online/offline

· Create and Delete nodes

· Customize emails

· Parse Sub Job Results and Create Tables

Networking & Fun

Apart from the interesting and useful sessions, the conference had some more fun interactive activities.

Networking sessions were organized in the evenings to socialize with the fellow attendees, speakers, sponsors and organizers during the conference.

A smart mobile app, released just for JUC east 2015, provided easy accessibility to interact with the speakers even after the sessions. The app was a platform to share our notes with fellow attendees and get their opinions on the other ongoing sessions.

A sponsor expo was conducted during lunchtime when the sponsors could demonstrate their cool work in CI/CD to the conference attendees.

Many fun contests were organized during the conference. I was thrilled when I won “The Phoenix Project: A Novel about IT, DevOps and Helping Your Business Win” in one of the contests. Gene Kim, the author of the book, is one of the Keynote speakers at JUC.

A leadership board on the mobile app ranked the attendees based on their activity on the app. I was ranked #1 on the leadership board since I shared my notes, responded to others queries and reviewed the speakers.

Conclusion

The future work of Jenkins is to provide more target platforms, more efficiency of master/slave communication and improve test stability. CloudBees follows “Eat own ‘dog food’” approach and uses Jenkins for performing Continuous Delivery. The process consists of a series of containerized services built and tested via Jenkins (and Docker), then deployed to servers via a Puppet master⁶.

The conference was informative and extremely helpful to me. I started using Jenkins for continuous integration in my project. I was able to use a few plugins like Vagrant, Git, WorkFlow, Email-Ext, and Build-pipeline, illustrated during the conference. Best practices of CI/CD presented by at the conference were enlightening and beneficial while working on Jenkins.

To learn more about the other sessions and tracks in the conference, please go through the presentation slides and videos uploaded on the CloudBees website (see References).