DevOps COURSES
COMMUNICATION AND COLLABORATION
Course Name
Course ID
From Zero to Continuous Delivery
3564
Docker Containers Hands-on
3563
Practical DevOps with Windows PowerShell
4117
DevOps Engineer
35640
For Further Information:
Shely: 03-7100769 | shakimi@jbh.co.il
www.johnbryce.com | contact us on-
:‫המבחן של המדינה קבע‬
‫ המרכז הטוב‬,‫ג'ון ברייס הדרכה‬
‫והמקצועי ביותר ללימודי מחשבים‬
!‫זו השנה השביעית ברציפות‬
‫נבחרנו‬
‫במקום‬
!‫הראשון‬
‫תוכנית יומית לכנס‬
‫‪ -08:00-09:00‬ארוחת בוקר והתכנסות‬
‫‪ -09:00-10:30‬הרצאה‬
‫‪ – 10:30-10:45‬הפסקה‬
‫‪ -10:45-12:30‬הרצאה‬
‫‪ -12:30-13:30‬ארוחת צהריים‬
‫‪ -13:30-15:00‬הרצאה‬
‫‪ -15:00-15:15‬הפסקה מתוקה *‬
‫‪ -15:15-16:30‬הרצאה‬
‫בזמן ההפסקה המתוקה צוות גון ברייס יחלקו משובים בכיתות ‪.‬‬
‫אנא הפנו את תשומת ליבם של המשתתפים למשוב והקצו לכך כמה‬
‫דקות כשהמשתתפים חוזרים לכיתה‬
‫לפרטים נוספים והרשמה‪smadart@johnbryce.co.il * 03-7100780 :‬‬
Introduction to the
Course
Introduction to the Course
• In this course we will
cover the basic DevOps
concepts and various
DevOps tools that will
help us to understand
the DevOps
methodology and goals.
1
Who Am I ?
• Ziv Kashtan
• Expert in DevOps architecture, Cloud services, Continuous
Integration (CI), Continuous Testing (CT) and Continuous
Deployment (CD).
• 20 years of technical and management experience in
leading companies (Managed teams world wide) as
Infrastructure and delivery director at Amdocs, TEOCO and
more.
• Since 2012, CEO and founder of AUTOMAT-IT, “The”
DevOps company.
QUESTIONS ?
2
Introduction to
DevOps
The need today
http://blogs.csc.com/2015/02/03/devops-theory-for-beginners/
3
Evolution of Agile
“Agile was instrumental in Development
regaining the trust in the business,
but it unintentionally left IT Operations
behind”
Evolution of Agile
4
CI/CT/CD
Amdocs Premises Stages
CI Dev
Jenkins
Slave
1
Jenkins
Slave
Jenkins
Slave
Jenkins
Slave
Jenkins
Slave
…
FTW
Component
Functional Testing
[Service]
Ready For FTW
2
Component
Functional Testing
[Service]
Component
Functional Testing
[Service]
Component
Functional Testing
[Service]
…
Nightly
Build
Ready For
Global
3
3
Global
Ready For QA
System Integration
5
Jenkins Master
P4 [Perforce]
- Build Modules
- Common
Directory
Common values DevOps
stakeholders agree on (DevOps
Adoption)
C.A.M.S.
Culture
Automation
Culture relates to the people and processes aspects of DevOps. Organizations need to ensure they have 'just
enough' process in place to enable people to effectively communicate and collaborate. Without the right
culture, automation attempts will be fruitless.
Technologies such as release management, configuration management, monitoring and control tools that
enhance the flow of work and enable automation are an important aspect of DevOps.
Measurement
There's an old adage that says 'if you can't measure it, you can't manage it'. Equally it can be said that 'if you
can't measure it, you can't improve it' - A successful DevOps implementation will measure everything –
people, process and technology performance.
Sharing
Sharing is the feedback loop in the CAMS cycle. Creating a culture where people share ideas and problems is
critical, not only because it enables improved communication and collaboration but also because it helps
organisations improve.
DevOps adoption
6
Devops what is it ?
“DevOps is a philosophy of the efficient
development, deployment and operation,
of the highest quality software possible.”
QUESTIONS ?
7
DevOps Tools
DEVOPS CYCLE
8
Tools Overview
• Source control- Git Based (Bitbucket, Git, Github, GitLab), SVN,
Perforce, ClearCase
• Continuous build- Maven, Gradle, NPM
• Continuous integration- Jenkins, Bamboo, Teamcity
• Test automation- Sonar, Junit
• Continuous deployment- Chef, Puppet, Saltstack, Ansible
• Code development- Eclipse, IntelJ, visual Stodio Clover, Crucible
• Governance & Compliance- Nexus, Artifactory, Confluence
DevOps Tools
• Example of DevOps tool
• https://xebialabs.com/periodic-table-of-devopstools/
9
QUESTIONS ?
10
Continuous
integration Tools
Introduction to CI
• Continuous integration (CI) is the practice
of merging all developer working copies to
a shared mainline several times a day.
• There are many CI tools. We will compare
between three- Jenkins, Bamboo and
Teamcity.
11
Introduction to Jenkins
• Jenkins is an open source automation server.
It helps automate the non-human part of the
software development process.
• Jenkins build can be triggers by various
means, for example by commit in a version
control system, by scheduling and after other
build have completed.
• Jenkins is easy to install and configure, and is
very extensible via it’s plug-in architecture.
Introduction to Jenkins
• Jenkins plug-ins have been released to extend its use to
projects written in languages other than Java.
• Plug-ins are available for integrating Jenkins with most
version control systems, databases and build tools.
Jenkins plug-ins are free and can change the way
Jenkins looks or add a new functionality.
• There are also many plug-ins for unit testing and
automated tests. After the tests, Jenkins displays the
reports and can generate trends.
12
Introduction to Jenkins
• Jenkins security depends on two factors- access control
and protection from other external threats.
• Access control can be managed in two waysauthentication (users, passwords and groups) and
authorization (permissions). those two aspects needs to be
individually configured.
• There are security subsystems in Jenkins that protect it
from indirect attacks, like protection from CSRF.
• Jenkins is used by Dell, eBay, EMC, Facebook, LinkedIn,
Netflix, Sony and more.
Introduction to Bamboo
• Bamboo is a CI server from atlassian that is used to build,
test and deploy applications automatically which helps
speed up the release process.
• Bamboo has build-in integration with Git and other
Atalssian software like Jira, Confluence, Bitbucket etc. but
this can be easily achieved in Jenkins with the right plugins.
• Bamboo is used by Citrix, Splunk, Nasa, Sonatype and
more.
13
Introduction to Bamboo
• Bamboo is free for open source projects. Commercial
organizations are charged based on the number of build
agents needed.
• Bamboo supports environment permissions so
developers and QA can deploy to their own environments
on demand.
• Because Bamboo is not an open source software, the
time taken for getting fixes could be longer.
Introduction to Teamcity
• Teamcity is a commercial build
management and CI server from JetBrains.
• Teamcity offers a ‘Freemium’ license for up
to 20 build configurations and 3 build
agents.
• Open source projects can request a free
license.
14
Introduction to Teamcity
• Teamcity supports pre-tested commits, so
you don’t have to wait for a build to finish
to discover mistakes and the base code
stays clean at all times.
• Teamcity can break down a single build
into parts that can be run in sequence or in
parallel.
• Teamcity is used by Airbnb, Siemens, HP
and more.
Basic comparison
Jenkins
Bamboo
Teamcity
Platform
Cross-platform
Servlet container or hosted
Servlet container
Open Source
Yes
No
No
Pricing
Free
Free trial, free for open
source projects but paid for
in large scales
No free trial, free use for
open source projects but
enterprise is paid
Plug-ins
Lots
Yes
Few
Documentation
Yes
Yes
Yes
License
Creative Commons and
MIT
Proprietary
Proprietary
Release
2011
2007
2006
Current Version
2.25
5.12
10.0.1
Written in
Java
Java
Java
15
Jenkins
Bamboo
Teamcity
Windows builders
MSBuild, NAnt
MSBuild, NAnt, Visual Studio
MSBuild, NAnt, Visual Studio,
Duplicates finder for .NET
Java builders
Ant, Maven 2, Kundo
Ant, Maven 1-2-3
Ant, Maven 2-3, Gradle,
IntelliJ IDEA .ipr based and
Inspections and Duplicates
finder
Other builders
Cmake, Gant, Gradle, Grails,
Phing, Rake, Ruby, SCons,
Python, shell script,
command-line
Custom script, command-line
tool, Bash, Xcode, Phing,
Grunt
Rake, FxCop, command-line
Notifications
Android, Email, Google
Calendar, IRC, XMPP, RSS,
Twitter, Slack, Catlight,
CCMenu, CCTray
XMPP, Google Talk, Email,
RSS, Remote API, HipChat
Email, XMPP, RSS, IDE, SysTray
IDEs
Eclipse, IntelliJ IDEA,
NetBeans
IntelliJ IDEA, Eclipse, Visual
Studio
Eclipse, Visual Studio, IntelliJ
IDEA, RubyMine, PyCharm,
PhpStorm, WebStorm
Other integration
Bugzilla, Google Code, Jira,
Bitbucket, Redmine,
FindBugs, Checkstyle, PMD
and Mantis, Trac, HP ALM
FishEye, Jira, Clover,
Bitbucket, GitHub
QUESTIONS ?
12
© All rights reserved to John Bryce Training LTD from Matrix group
16
Jetbrains Youtrack, Jira,
Bugzilla, FishEye, FindBugs,
PMD, dotCover, NCover
Continuous
Integration & Jenkins
Jenkins Introduction
•
•
•
•
•
Branched from Hudson
Java based Continuous Build System
Runs in servlet container Tomcat
Supported by over 400 plugins
SCM, Testing, Notifications, Reporting, Artifact Saving,
Triggers,
• External Integration
• Under development since 2005
17
Jenkins Installation
•
•
•
•
Master Installation
Master Slave
JAR/WAR BASE
Docker BASE
Core Features
• When setting up a project in Jenkins, out of the box you have
the following general options:
• Associating with a version control server
• Triggering builds
• Polling, Periodic, Building based on other projects
• Execution of shell scripts, bash scripts, Ant targets, and
Maven targets Artifact archival
• Publish
• JUnit test results and Javadocs
• Email notifications
• Plugins expand the functionality even further
18
Core Features
• The main page provides a summary of the
projects
• Quick view of What’s building (“No builds in
the queue”)
• Build Executor Status (both “Idle”)
• Status of the projects
Why Jenkins? Flexibility!
• Jenkins is a highly configurable system by
itself
• The additional community developed
plugins provide even more flexibility
• By combining Jenkins with Ant, Gradle, or
other Build Automation tools, the
possibilities are limitless
19
Tomcat-Servlet
• Jenkins is packaged as a WAR, so you can drop it into
whichever servlet container you prefer to use (Tomcat ,
Apache etc)
• Jenkins comes pre-packaged with a servlet if you just want
a lightweight implementation
• Native/Supported packages exist for
• Windows Ubuntu/Debian Redhat/Fedora/CentOS Mac
OSX openSUSE FreeBSD OpenBSD Solaris
• Cloudbees - http://www.cloudbees.com/
Pipeline
• Key Features and Benefits of Pipelines
• Pipeline as Code
• Easily define simple and complex pipelines through the
DSL in a Jenkinsfile.
• Pipeline as code provides a common language to help
teams (e.g. Dev and Ops) work together.
• Easily share pipelines between teams by storing common
"steps" in shared repositories.
20
Jenkins 2.0
•
•
•
•
Built-in support for delivery pipelines.
Improved usability/UX
Fully backwards compatible.
Pipeline as a code
Code Repository - Git
Git -Linus Torvalds (the creator of the Linux
kernel)
• Local projects directory- files.
• Single “.git” directory.
• 10 min exercise https://try.github.io/levels/1/challenges/1
21
Github What is It?
• GitHub is a code hosting platform for
version control and collaboration.
• It lets you and others work together on
projects from anywhere.
GitHub plugin
• Hyperlinks between changes
• Trigger a build when a change is pushed to
GitHub -prferable
• Polling mode - les in use
22
Builds
• Once a project is successfully created in Jenkins, all future
builds are automatic
• Jenkins executes the build in an executer
• By default, Jenkins gives one executor per core on the
build server
• Jenkins also has the concept of slave build servers
– Useful for building on different architectures
– Distribution of load
MSBuild plugin
• This plugin allows you to use MSBuild to build .NET projects.
• Usage
• To use this plugin, specify the location directory of
MSBuild.exe on Jenkin's configuration page.
• Then, on your project configuration page, specify the name of
the build file (.proj or .sln) and any command line arguments
you want to pass in.
• The files are compiled to the directory where Visual Studio
would put them as well.
23
Testing
• Unit testing - Does my small part of code
working properly in my system, my class?
• Integration Testing Does our code work
correctly against code we can’t change?
• Acceptance testing - Does whole system
working properly?
Selenium Plugin
• Selenium automates browsers. That's it!
• This plugin turns your Jenkins cluster into a Selenium2
Grid
• This plugin sets up Selenium Grid in the following way:
– On master, Selenium Grid Hub is started on port 4444,
unless configured otherwise in Jenkins global
configurations. This is where all your tests should
connect to.
24
Continuous deployment (CD)
Reporting & notifications
•
•
•
•
•
Jenkins comes with basic reporting features
Keeping track of build status
Last success and failure
“Weather” – Build trend
These can be greatly enhanced with the use of pre-build
plugins
• Unit test coverage
• Test result trending
• Findbugs, Checkstyle, PMD
25
Configuration Management
• Configuration management (CM) is the detailed recording and
updating of information that describes an enterprise's hardware and
software.
• Popular CM tools
– Chef - Automate your infrastructure • ruby based , agent based , chef solo
– Puppet - The shortest path to driving changes with confidence
• Ruby dsl based , agent based, puppet masterless
– Ansible
• Python, yaml , agent less (ssh)
26
Exercise 2 Jenkins - Quick Guide
Why Jenkins?
Jenkins is a software that allows continuous integration. Jenkins will be installed on
a server where the central build will take place. The following flowchart demonstrates
a very simple workflow of how Jenkins works.
System Requirements
JDK
JDK 1.5 or above
openjdk-7-jre and openjdk-7-jdk recommended
Memory
2 GB RAM (recommended)
Disk Space
No minimum requirement. Note that since all builds will be stored
on the Jenkins machines, it has to be ensured that sufficient disk
space is available for build storage.
Operating
System
Version
Jenkins can be installed on Windows, Ubuntu/Debian, Red
Hat/Fedora/CentOS, Mac OS X, openSUSE, FReeBSD, OpenBSD,
Gentoo.
‫© כל הזכויות שמורות לג'ון ברייס הדרכה בע"מ מקבוצת מטריקס‬
27
Java Container
The WAR file can be run in any container that supports Servlet
2.4/JSP 2.0 or later.(An example is Tomcat 5).
Jenkins - Installation
Download Jenkins
The official website for Jenkins is Jenkins. If you click the given link, you can get the
home page of the Jenkins official website as shown below.
By default, the latest release and the Long-Term support release will be available for
download. The past releases are also available for download. Click the Long-Term
Support Release tab in the download section.
Click the link “Older but stable version” to download the Jenkins war file.
‫© כל הזכויות שמורות לג'ון ברייס הדרכה בע"מ מקבוצת מטריקס‬
28
Starting Jenkins
Open the command prompt. From the command prompt, browse to the directory
where the jenkins.war file is present. Run the following command
D:\>Java –jar Jenkins.war
After the command is run, various tasks will run, one of which is the extraction of the
war file which is done by an embedded webserver called winstone.
D:\>Java –jar Jenkins.war
Running from: D:\jenkins.war
Webroot: $user.home/ .jenkins
Sep 29, 2015 4:10:46 PM winstone.Logger logInternal
INFO: Beginning extraction from war file
Once the processing is complete without major errors, the following line will come in
the output of the command prompt.
INFO: Jenkins is fully up and running
Accessing Jenkins
Once Jenkins is up and running, one can access Jenkins from the link
−http://localhost:8080
This link will bring up the Jenkins dashboard.
Jenkins - Git Setup
For this exercise, you have to ensure that Internet connectivity is present from the
machine on which Jenkins is installed. In your Jenkins Dashboard (Home screen), click
the Manage Jenkins option on the left hand side. If you are using jenkins 2.0 you can
skip that section github plugin is preinstalled
‫© כל הזכויות שמורות לג'ון ברייס הדרכה בע"מ מקבוצת מטריקס‬
29
In the next screen, click the ‘Manage Plugins’ option.
In the next screen, click the Available tab. This tab will give a list of plugins which are
available for downloading. In the ‘Filter’ tab type ‘Git plugin’
The list will then be filtered. Check the Git Plugin option and click on the button ‘Install
without restart’
‫© כל הזכויות שמורות לג'ון ברייס הדרכה בע"מ מקבוצת מטריקס‬
30
The installation will then begin and the screen will be refreshed to show the status of
the download.
Once all installations are complete, restart Jenkins by issue the following command in
the browser. http://localhost:8080/jenkins/restart
After Jenkins is restarted, Git will be available as an option whilst configuring jobs. To
verify, click on New Item in the menu options for Jenkins. Then enter a name for a
job, in the following case, the name entered is ‘Demo’. Select ‘Freestyle project’ as the
item type. Click the Ok button.
‫© כל הזכויות שמורות לג'ון ברייס הדרכה בע"מ מקבוצת מטריקס‬
31
In the next screen, if you browse to the Source code Management section, you will
now see ‘Git’ as an option.
Jenkins – Maven Setup
Step 1: Downloading and Setting Up Maven
The official website for maven is Apache Maven. If you click the given link, you can get
the home page of the maven official website as shown below.
‫© כל הזכויות שמורות לג'ון ברייס הדרכה בע"מ מקבוצת מטריקס‬
32
While browsing to the site, go to the Files section and download the link to the
Binary.zip file.
Once the file is downloaded, extract the files to the relevant application folder. For this
purpose, the maven files will be placed in E:\Apps\apache-maven-3.3.3.
Step 2: Setting up Jenkins and Maven
In the Jenkins dashboard (Home screen), click Manage Jenkins from the left-hand side
menu.
‫© כל הזכויות שמורות לג'ון ברייס הדרכה בע"מ מקבוצת מטריקס‬
33
Then, click on ‘Configure System’ from the right hand side.
‫© כל הזכויות שמורות לג'ון ברייס הדרכה בע"מ מקבוצת מטריקס‬
34
In the Configure system screen, scroll down till you see the Maven section and then
click on the ‘Add Maven’ button.
Uncheck the ‘Install automatically’ option.
Add any name for the setting and the location of the MAVEN_HOME.
Then, click on the ‘Save’ button at the end of the screen.
You can now create a job with the ‘Maven project’ option. In the Jenkins dashboard,
click the New Item option.
‫© כל הזכויות שמורות לג'ון ברייס הדרכה בע"מ מקבוצת מטריקס‬
35
Jenkins - Configuration
You probably would have seen a couple of times in the previous exercises wherein we
had to configure options within Jenkins. The following shows the various configuration
options in Jenkins.
So one can get the various configuration options for Jenkins by clicking the ‘Manage
Jenkins’ option from the left hand menu side.
‫© כל הזכויות שמורות לג'ון ברייס הדרכה בע"מ מקבוצת מטריקס‬
36
You will then be presented with the following screen −
Click on Configure system. Discussed below are some of the Jenkins configuration
settings which can be carried out.
Jenkins Home Directory
 Set "JENKINS_HOME" environment variable to the new home directory before
launching the servlet container.
 Set "JENKINS_HOME" system property to the servlet container.
 Set JNDI environment entry "JENKINS_HOME" to the new directory.
The following example will use the first option of setting the "JENKINS_HOME"
environment variable.
First create a new folder E:\Apps\Jenkins. Copy all the contents from the existing
~/.jenkins to this new directory.
‫© כל הזכויות שמורות לג'ון ברייס הדרכה בע"מ מקבוצת מטריקס‬
37
Set the JENKINS_HOME environment variable to point to the base directory location
where Java is installed on your machine. For example,
OS
Output
Windows
Set Environmental variable JENKINS_HOME to you’re the location you
desire. As an example you can set it to E:\Apps\Jenkins
Linux
export JENKINS_HOME =/usr/local/Jenkins or the location you desire.
In the Jenkins dashboard, click Manage Jenkins from the left hand side menu. Then
click on ‘Configure System’ from the right hand side.
In the Home directory, you will now see the new directory which has been configured.
# of executors
This refers to the total number of concurrent job executions that can take place on the
Jenkins machine. This can be changed based on requirements. Sometimes the
recommendation is to keep this number the same as the number of CPU on the
machines for better performance.
Environment Variables
This is used to add custom environment variables which will apply to all the jobs.
These are key-value pairs and can be accessed and used in Builds wherever required.
Jenkins URL
By default, the Jenkins URL points to localhost. If you have a domain name setup for
your machine, set this to the domain name else overwrite localhost with IP of
machine. This will help in setting up slaves and while sending out links using the email
as you can directly access the Jenkins URL using the environment variable
JENKINS_URL which can be accessed as ${JENKINS_URL}.
Email Notification
‫© כל הזכויות שמורות לג'ון ברייס הדרכה בע"מ מקבוצת מטריקס‬
38
In the email Notification area, you can configure the SMTP settings for sending out
emails. This is required for Jenkins to connect to the SMTP mail server and send out
emails to the recipient list.
Jenkins - Management
To manage Jenkins, click on the ‘Manage Jenkins’ option from the left hand menu side.
So one can get the various configuration options for Jenkins by clicking the ‘Manage
Jenkins’ option from the left hand menu side.
You will then be presented with the following screen −
Some of the management options are as follows −
Configure System
‫© כל הזכויות שמורות לג'ון ברייס הדרכה בע"מ מקבוצת מטריקס‬
39
This is where one can manage paths to the various tools to use in builds, such as the
JDKs, the versions of Ant and Maven, as well as security options, email servers, and
other system-wide configuration details. When plugins are installed. Jenkins will add
the required configuration fields dynamically after the plugins are installed.
Reload Configuration from Disk
Jenkins stores all its system and build job configuration details as XML files which is
stored in the Jenkins home directory. Here also all of the build history is stored. If you
are migrating build jobs from one Jenkins instance to another, or archiving old build
jobs, you will need to add or remove the corresponding build job directories to
Jenkins’s builds directory. You don’t need to take Jenkins offline to do this—you can
simply use the “Reload Configuration from Disk” option to reload the Jenkins system
and build job configurations directly.
Manage Plugin
Here one can install a wide variety of third-party plugins right from different Source
code management tools such as Git, Mercurial or ClearCase, to code quality and code
coverage metrics reporting. Plugins can be installed, updated and removed through
the Manage Plugins screen.
System Information
This screen displays a list of all the current Java system properties and system
environment variables. Here one can check exactly what version of Java Jenkins is
running in, what user it is running under, and so forth.
The following screenshot shows some of the name-value information available in this
section.
‫© כל הזכויות שמורות לג'ון ברייס הדרכה בע"מ מקבוצת מטריקס‬
40
System Log
The System Log screen is a convenient way to view the Jenkins log files in real time.
Again, the main use of this screen is for troubleshooting.
Load Statistics
This pages displays graphical data on how busy the Jenkins instance is in terms of the
number of concurrent builds and the length of the build queue which gives an idea of
how long your builds need to wait before being executed. These statistics can give a
good idea of whether extra capacity or extra build nodes is required from an
infrastructure perspective.
Script Console
This screen lets you run Groovy scripts on the server. It is useful for advanced
troubleshooting since it requires a strong knowledge of the internal Jenkins
architecture.
Manage nodes
Jenkins is capable of handling parallel and distributed builds. In this screen, you can
configure how many builds you want. Jenkins runs simultaneously, and, if you are
using distributed builds, set up build nodes. A build node is another machine that
Jenkins can use to execute its builds.
Prepare for Shutdown
If there is a need to shut down Jenkins, or the server Jenkins is running on, it is best
not to do so when a build is being executed. To shut down Jenkins cleanly, you can
use the Prepare for Shutdown link, which prevents any new builds from being started.
Eventually, when all of the current builds have finished, one will be able to shut down
Jenkins cleanly.
Jenkins - Setup Build Jobs
For this exercise, we will create a job in Jenkins which picks up a simple HelloWorld
application, builds and runs the java program.
‫© כל הזכויות שמורות לג'ון ברייס הדרכה בע"מ מקבוצת מטריקס‬
41
Step 1 − Go to the Jenkins dashboard and Click on New Item
Step 2 − In the next screen, enter the Item name, in this case we have named it
Helloworld. Choose the ‘Freestyle project option’
Step 3 − The following screen will come up in which you can specify the details of the
job.
‫© כל הזכויות שמורות לג'ון ברייס הדרכה בע"מ מקבוצת מטריקס‬
42
Step 4 − We need to specify the location of files which need to be built. In this
example, we will assume that a local git repository(E:\Program) has been setup which
contains a ‘HelloWorld.java’ file. Hence scroll down and click on the Git option and
enter the URL of the local git repository.
Note − If you repository if hosted on Github, you can also enter the url of that
repository here. In addition to this, you would need to click on the Add button for the
credentials to add a user name and password to the github repository so that the code
can be picked up from the remote repository.
Step 5 − Now go to the Build section and click on Add build step → Execute Windows
batch command
‫© כל הזכויות שמורות לג'ון ברייס הדרכה בע"מ מקבוצת מטריקס‬
43
Step 6 − In the command window, enter the following commands and then click on
the Save button.
Javac HelloWorld.java
Java HelloWorld
Step 7 − Once saved, you can click on the Build Now option to see if you have
successfully defined the job.
‫© כל הזכויות שמורות לג'ון ברייס הדרכה בע"מ מקבוצת מטריקס‬
44
Step 8 − Once the build is scheduled, it will run. The following Build history section
shows that a build is in progress.
Step 9 − Once the build is completed, a status of the build will show if the build was
successful or not. In our case, the following build has been executed successfully.
Click on the #1 in the Build history to bring up the details of the build.
‫© כל הזכויות שמורות לג'ון ברייס הדרכה בע"מ מקבוצת מטריקס‬
45
Step 10 − Click on the Console Output link to see the details of the build
‫© כל הזכויות שמורות לג'ון ברייס הדרכה בע"מ מקבוצת מטריקס‬
46
Apart from the steps shown above there are just so many ways to create a build job,
the options available are many, which what makes Jenkins such a fantastic continuous
deployment tool.
Jenkins - Unit Testing
Jenkins provides an out of box functionality for Junit, and provides a host of plugins
for unit testing for other technologies, an example being MSTest for .Net Unit tests. If
you go to the link https://wiki.jenkins-ci.org/display/JENKINS/xUnit+Plugin it will give
the list of Unit Testing plugins available.
Example of a Junit Test in Jenkins
The following example will consider
‫© כל הזכויות שמורות לג'ון ברייס הדרכה בע"מ מקבוצת מטריקס‬
47


A simple HelloWorldTest class based on Junit.
Ant as the build tool within Jenkins to build the class accordingly.
Step 1 − Go to the Jenkins dashboard and Click on the existing HelloWorld project
and choose the Configure option
Step 2 − Browse to the section to Add a Build step and choose the option to Invoke
Ant.
Step 3 − Click on the Advanced button.
‫© כל הזכויות שמורות לג'ון ברייס הדרכה בע"מ מקבוצת מטריקס‬
48
Step 4 − In the build file section, enter the location of the build.xml file.
Step 5 − Next click the option to Add post-build option and choose the option of
“Publish Junit test result report”
‫© כל הזכויות שמורות לג'ון ברייס הדרכה בע"מ מקבוצת מטריקס‬
49
Step 6 − In the Test reports XML’s, enter the location as shown below. Ensure that
Reports is a folder which is created in the HelloWorld project workspace. The “*.xml”
basically tells Jenkins to pick up the result xml files which are produced by the running
of the Junit test cases. These xml files which then be converted into reports which can
be viewed later.
Once done, click the Save option at the end.
Step 7 − Once saved, you can click on the Build Now option.
Once the build is completed, a status of the build will show if the build was successful
or not. In the Build output information, you will now notice an additional section called
Test Result. In our case, we entered a negative Test case so that the result would fail
just as an example.
‫© כל הזכויות שמורות לג'ון ברייס הדרכה בע"מ מקבוצת מטריקס‬
50
One can go to the Console output to see further information. But what’s more
interesting is that if you click on Test Result, you will now see a drill down of the Test
results.
Jenkins - Automated Testing
One of the basic principles of Continuous Integration is that a build should be
verifiable. You have to be able to objectively determine whether a particular build is
ready to proceed to the next stage of the build process, and the most convenient way
to do this is to use automated tests. Without proper automated testing, you find
yourself having to retain many build artifacts and test them by hand, which is hardly
in the spirit of Continuous Integration. The following example shows how to use
Selenium to run automated web tests.
Step 1 − Go to Manage Plugins.
‫© כל הזכויות שמורות לג'ון ברייס הדרכה בע"מ מקבוצת מטריקס‬
51
Step 2 − Find the Hudson Selenium Plugin and choose to install. Restart the Jenkins
instance.
Step 3 − Go to Configure system.
‫© כל הזכויות שמורות לג'ון ברייס הדרכה בע"מ מקבוצת מטריקס‬
52
Step 4 − Configure the selenium server jar and click on the Save button.
Note − The selenium jar file can be downloaded from the locationSeleniumHQ
Click on the download for the Selenium standalone server.
‫© כל הזכויות שמורות לג'ון ברייס הדרכה בע"מ מקבוצת מטריקס‬
53
Step 5 − Go back to your dashboard and click on the Configure option for the
HelloWorld project.
Step 6 − Click on Add build step and choose the optin of “SeleniumHQ htmlSuite Run”
‫© כל הזכויות שמורות לג'ון ברייס הדרכה בע"מ מקבוצת מטריקס‬
54
Step 7 − Add the necessary details for the selenium test. Here the suiteFile is the
TestSuite generated by using the Selenium IDE. Click on Save and execute a build.
Now the post build will launch the selenium driver, and execute the html test.
Jenkins - Notification
Jenkins comes with an out of box facility to add an email notification for a build
project.
Step 1 − Configuring an SMTP server. Goto Manage Jenkins → Configure System. Go
to the E-mail notification section and enter the required SMTP server and user emailsuffix details.
‫© כל הזכויות שמורות לג'ון ברייס הדרכה בע"מ מקבוצת מטריקס‬
55
Step 2 − Configure the recipients in the Jenkins project - When you configure any
Jenkins build project, right at the end is the ability to add recipients who would get
email notifications for unstable or broken builds. Then click on the Save button.
Apart from the default, there are also notification plugin’s available in the market. An
example is the notification plugin from Tikal Knowledge which allows sending Job
Status notifications in JSON and XML formats. This plugin enables end-points to be
configured as shown below.
‫© כל הזכויות שמורות לג'ון ברייס הדרכה בע"מ מקבוצת מטריקס‬
56
Here are the details of each option −





"Format" − This is the notification payload format which can either be JSON or
XML.
"Protocol" − protocol to use for sending notification messages, HTTP, TCP or
UDP.
"Event" − The job events that trigger notifications: Job Started, Job
Completed, Job Finalized or All Events (the default option).
"URL" − URL to send notifications to. It takes the form of "http://host" for
HTTP protocol, and "host:port" for TCP and UDP protocols.
"Timeout" − Timeout in milliseconds for sending notification request, 30
seconds by default.
Jenkins - Reporting
As demonstrated in the earlier section, there are many reporting plugins available with
the simplest one being the reports available for jUnit tests.
In the Post-build action for any job, you can define the reports to be created. After the
builds are complete, the Test Results option will be available for further drill-down.
Jenkins - Code Analysis
‫© כל הזכויות שמורות לג'ון ברייס הדרכה בע"מ מקבוצת מטריקס‬
57
Jenkins has a host of Code Analysis plugin. The various plugins can be found at
https://wiki.jenkins-ci.org/display/JENKINS/Static+Code+Analysis+Plugins
This plugin provides utilities for the static code analysis plugins. Jenkins can parse the
results file from various Code Analysis tools such as CheckStyle, FindBugs, PMD etc.
For each corresponding code analysis tool, a plugin in Jenkins needs to be installed.
Additionally the add-on plugin Static Analysis Collector is available that combines the
individual results of these plugins into a single trend graph and view.
The plugins can provide information such as





The total number of warnings in a job
A showing of the new and fixed warnings of a build
Trend Reports showing the number of warnings per build
Overview of the found warnings per module, package, category, or type
Detailed reports of the found warnings optionally filtered by severity (or new and
fixed)
Jenkins - Distributed Builds
Sometimes many build machines are required if there are instances wherein there are
a larger and heavier projects which get built on a regular basis. And running all of
these builds on a central machine may not be the best option. In such a scenario, one
can configure other Jenkins machines to be slave machines to take the load off the
master Jenkins server.
Sometimes you might also need several different environments to test your builds. In
this case using a slave to represent each of your required environments is almost a
must.
A slave is a computer that is set up to offload build projects from the master and once
setup this distribution of tasks is fairly automatic. The exact delegation behavior
depends on the configuration of each project; some projects may choose to "stick" to
a particular machine for a build, while others may choose to roam freely between
slaves.
‫© כל הזכויות שמורות לג'ון ברייס הדרכה בע"מ מקבוצת מטריקס‬
58
Since each slave runs a separate program called a "slave agent" there is no need to
install the full Jenkins (package or compiled binaries) on a slave. There are various
ways to start slave agents, but in the end the slave agent and Jenkins master needs to
establish a bi-directional communication link (for example a TCP/IP socket.) in order
to operate.
To set up slaves/nodes in Jenkins follow the steps given below.
Step 1 − Go to the Manage Jenkins section and scroll down to the section of Manage
Nodes.
Step 2 − Click on New Node
Step 3 − Give a name for the node, choose the Dumb slave option and click on Ok.
‫© כל הזכויות שמורות לג'ון ברייס הדרכה בע"מ מקבוצת מטריקס‬
59
Step 4 − Enter the details of the node slave machine. In the below example, we are
considering the slave machine to be a windows machine, hence the option of “Let
Jenkins control this Windows slave as a Windows service” was chosen as the launch
method. We also need to add the necessary details of the slave node such as the node
name and the login credentials for the node machine. Click the Save button. The
Labels for which the name is entered as “New_Slave” is what can be used to configure
jobs to use this slave machine.
Once the above steps are completed, the new node machine will initially be in an
offline state, but will come online if all the settings in the previous screen were
entered correctly. One can at any time make the node slave machine as offline if
required.
‫© כל הזכויות שמורות לג'ון ברייס הדרכה בע"מ מקבוצת מטריקס‬
60
Jenkins - Automated Deployment
There are many plugins available which can be used to transfer the build files after a
successful build to the respective application/web server. On example is the “Deploy
to container Plugin”. To use this follow the steps given below.
Step 1 − Go to Manage Jenkins → Manage Plugins. Go to the Available section and
find the plugin “Deploy to container Plugin” and install the plugin. Restart the Jenkins
server.
This plugin takes a war/ear file and deploys that to a running remote application
server at the end of a build.
Tomcat 4.x/5.x/6.x/7.x
JBoss 3.x/4.x
Glassfish 2.x/3.x
Step 2 − Go to your Build project and click the Configure option. Choose the option
“Deploy war/ear to a container”
‫© כל הזכויות שמורות לג'ון ברייס הדרכה בע"מ מקבוצת מטריקס‬
61
Step 3 − In the Deploy war/ear to a container section, enter the required details of
the server on which the files need to be deployed and click on the Save button. These
steps will now ensure that the necessary files get deployed to the necessary container
after a successful build.
Jenkins - Metrics & Trends
There are various plugins which are available in Jenkins to showcase metrics for builds
which are carried out over a period of time. These metrics are useful to understand
your builds and how frequently they fail/pass over time. As an example, let’s look at
the ‘Build History Metrics plugin’.
This plugin calculates the following metrics for all of the builds once installed



Mean Time To Failure (MTTF)
Mean Time To Recovery (MTTR)
Standard Deviation of Build Times
Step 1 − Go to the Jenkins dashboard and click on Manage Jenkins
‫© כל הזכויות שמורות לג'ון ברייס הדרכה בע"מ מקבוצת מטריקס‬
62
Step 2 − Go to the Manage Plugins option.
Step 3 − Go to the Available tab and search for the plugin ‘Build History Metrics
plugin’ and choose to ‘install without restart’.
‫© כל הזכויות שמורות לג'ון ברייס הדרכה בע"מ מקבוצת מטריקס‬
63
Step 4 − The following screen shows up to confirm successful installation of the
plugin. Restart the Jenkins instance.
When you go to your Job page, you will see a table with the calculated metrics.
Metric’s are shown for the last 7 days, last 30 days and all time.
‫© כל הזכויות שמורות לג'ון ברייס הדרכה בע"מ מקבוצת מטריקס‬
64
To see overall trends in Jenkins, there are plugins available to gather information from
within the builds and Jenkins and display them in a graphical format. One example of
such a plugin is the ‘Hudson global-build-stats plugin’. So let’s go through the steps
for this.
Step 1 − Go to the Jenkins dashboard and click on Manage Jenkins
Step 2 − Go to the Manage Plugins option
‫© כל הזכויות שמורות לג'ון ברייס הדרכה בע"מ מקבוצת מטריקס‬
65
Step 3 − Go to the Available tab and search for the plugin ‘Hudson global-build-stats
plugin’ and choose to ‘install without restart’.
Step 4 − The following screen shows up to confirm successful installation of the
plugin. Restart the Jenkins instance.
‫© כל הזכויות שמורות לג'ון ברייס הדרכה בע"מ מקבוצת מטריקס‬
66
To see the Global statistics, please follow the Step 5 through 8.
Step 5 − Go to the Jenkins dashboard and click on Manage Jenkins. In the Manage
Jenkins screen, scroll down and now you will now see an option called ‘Global Build
Stats’. Click on this link.
Step 6 − Click on the button ‘Initialize stats’. What this does is that it gather’s all the
existing records for builds which have already been carried out and charts can be
created based on these results.
‫© כל הזכויות שמורות לג'ון ברייס הדרכה בע"מ מקבוצת מטריקס‬
67
Step 7 − Once the data has been initialized, it’s time to create a new chart. Click on
the ‘Create new chart’ link.
Step 8 − A pop-up will come to enter relevant information for the new chart details.
Enter the following mandatory information





Title – Any title information, for this example is given as ‘Demo’
Chart Width – 800
Chart Height – 600
Chart time scale – Daily
Chart time length – 30 days
The rest of the information can remain as it is. Once the information is entered, click
on Create New chart.
‫© כל הזכויות שמורות לג'ון ברייס הדרכה בע"מ מקבוצת מטריקס‬
68
You will now see the chart which displays the trends of the builds over time.
If you click on any section within the chart, it will give you a drill down of the details of
the job and their builds.
Jenkins - Server Maintenance
‫© כל הזכויות שמורות לג'ון ברייס הדרכה בע"מ מקבוצת מטריקס‬
69
The following are some of the basic activities you will carry out, some of which are
best practices for Jenkins server maintenance
URL Options
The following commands when appended to the Jenkins instance URL will carry out the
relevant actions on the Jenkins instance.
http://localhost:8080/jenkins/exit − shutdown jenkins
http://localhost:8080/jenkins/restart − restart jenkins
http://localhost:8080/jenkins/reload − to reload the configuration
Backup Jenkins Home
The Jenkins Home directory is nothing but the location on your drive where Jenkins
stores all information for the jobs, builds etc. The location of your home directory can
be seen when you click on Manage Jenkins → Configure system.
Set up Jenkins on the partition that has the most free disk-space – Since Jenkins
would be taking source code for the various jobs defined and doing continuous builds,
always ensure that Jenkins is setup on a drive that has enough hard disk space. If you
hard disk runs out of space, then all builds on the Jenkins instance will start failing.
Another best practice is to write cron jobs or maintenance tasks that can carry out
clean-up operations to avoid the disk where Jenkins is setup from becoming full.
Jenkins - Continuous Deployment
Jenkins provides good support for providing continuous deployment and delivery. If
you look at the flow of any software development through deployment, it will be as
shown below.
‫© כל הזכויות שמורות לג'ון ברייס הדרכה בע"מ מקבוצת מטריקס‬
70
The main part of Continuous deployment is to ensure that the entire process which is
shown above is automated. Jenkins achieves all of this via various plugins, one of
them being the “Deploy to container Plugin” which was seen in the earlier lessons.
There are plugins available which can actually give you a graphical representation of
the Continuous deployment process. But first lets create another project in Jenkins, so
that we can see best how this works.
Let’s create a simple project which emulates the QA stage, and does a test of the
Helloworld application.
Step 1 − Go to the Jenkins dashboard and click on New Item. Choose a ‘Freestyle
project’ and enter the project name as ‘QA’. Click on the Ok button to create the
project.
‫© כל הזכויות שמורות לג'ון ברייס הדרכה בע"מ מקבוצת מטריקס‬
71
Step 2 − In this example, we are keeping it simple and just using this project to
execute a test program for the Helloworld application.
So our project QA is now setup. You can do a build to see if it builds properly.
Step 3 − Now go to you Helloworld project and click on the Configure option
‫© כל הזכויות שמורות לג'ון ברייס הדרכה בע"מ מקבוצת מטריקס‬
72
Step 4 − In the project configuration, choose the ‘Add post-build action’ and choose
‘Build other projects’
Step 5 − In the ‘Project to build’ section, enter QA as the project name to build. You
can leave the option as default of ‘Trigger only if build is stable’. Click on the Save
button.
Step 6 − Build the Helloworld project. Now if you see the Console output, you will
also see that after the Helloworld project is successfully built, the build of the QA
project will also happen.
‫© כל הזכויות שמורות לג'ון ברייס הדרכה בע"מ מקבוצת מטריקס‬
73
Step 7 − Let now install the Delivery pipeline plugin. Go to Manage Jenkins → Manage
Plugin’s. In the available tab, search for ‘Delivery Pipeline Plugin’. Click On Install
without Restart. Once done, restart the Jenkins instance.
Step 8 − To see the Delivery pipeline in action, in the Jenkins Dashboard, click on the
+ symbol in the Tab next to the ‘All’ Tab.
Step 9 − Enter any name for the View name and choose the option ‘Delivery Pipeline
View’.
‫© כל הזכויות שמורות לג'ון ברייס הדרכה בע"מ מקבוצת מטריקס‬
74
Step 10 − In the next screen, you can leave the default options. One can change the
following settings −





Ensure the option ‘Show static analysis results’ is checked.
Ensure the option ‘Show total build time’ is checked.
For the Initial job – Enter the Helloworld project as the first job which should
build.
Enter any name for the Pipeline
Click the OK button.
You will now see a great view of the entire delivery pipeline and you will be able to see
the status of each project in the entire pipeline.
‫© כל הזכויות שמורות לג'ון ברייס הדרכה בע"מ מקבוצת מטריקס‬
75
Another famous plugin is the build pipeline plugin. Let’s take a look at this.
Step 1 − Go to Manage Jenkins → Manage Plugin’s. In the available tab, search for
‘Build Pipeline Plugin’. Click On Install without Restart. Once done, restart the Jenkins
instance.
Step 2 − To see the Build pipeline in action, in the Jenkins Dashboard, click on the +
symbol in the Tab next to the ‘All’ Tab.
‫© כל הזכויות שמורות לג'ון ברייס הדרכה בע"מ מקבוצת מטריקס‬
76
Step 3 − Enter any name for the View name and choose the option ‘Build Pipeline
View’.
Step 4 − Accept the default settings, just in the Selected Initial job, ensure to enter
the name of the Helloworld project. Click on the Ok button.
‫© כל הזכויות שמורות לג'ון ברייס הדרכה בע"מ מקבוצת מטריקס‬
77
You will now see a great view of the entire delivery pipeline and you will be able to see
the status of each project in the entire pipeline.
Jenkins - Security
In Jenkins you have the ability to setup users and their relevant permissions on the
Jenkins instance. By default you will not want everyone to be able to define jobs or
other administrative tasks in Jenkins. So Jenkins has the ability to have a security
configuration in place.
To configure Security in Jenkins, follow the steps given below.
Step 1 − Click on Manage Jenkins and choose the ‘Configure Global Security’ option.
‫© כל הזכויות שמורות לג'ון ברייס הדרכה בע"מ מקבוצת מטריקס‬
78
Step 2 − Click on Enable Security option. As an example, let’s assume that we want
Jenkins to maintain it’s own database of users, so in the Security Realm, choose the
option of ‘Jenkins’ own user database’.
By default you would want a central administrator to define users in the system,
hence ensure the ‘Allow users to sign up’ option is unselected. You can leave the rest
as it is for now and click the Save button.
Step 3 − You will be prompted to add your first user. As an example, we are setting
up an admin users for the system.
‫© כל הזכויות שמורות לג'ון ברייס הדרכה בע"מ מקבוצת מטריקס‬
79
Step 4 − It’s now time to setup your users in the system. Now when you go to
Manage Jenkins, and scroll down, you will see a ‘Manage Users’ option. Click this
option.
Step 5 − Just like you defined your admin user, start creating other users for the
system. As an example, we are just creating another user called ‘user’.
‫© כל הזכויות שמורות לג'ון ברייס הדרכה בע"מ מקבוצת מטריקס‬
80
Step 6 − Now it’s time to setup your authorizations, basically who has access to what.
Go to Manage Jenkins → Configure Global Security.
Now in the Authorization section, click on ‘Matrix based security’
Step 7 − If you don’t see the user in the user group list, enter the user name and add
it to the list. Then give the appropriate permissions to the user.
Click on the Save button once you have defined the relevant authorizations.
Your Jenkins security is now setup.
Note − For Windows AD authentication, one has to add the Active Directory plugin to
Jenkins.
Jenkins - Backup Plugin
Jenkins has a backup plugin which can used to backup critical configuration settings
related to Jenkins. Follow the steps given below to have a backup in place.
Step 1 − Click on Manage Jenkins and choose the ‘Manage Plugins’ option.
‫© כל הזכויות שמורות לג'ון ברייס הדרכה בע"מ מקבוצת מטריקס‬
81
Step 2 − In the available tab, search for ‘Backup Plugin’. Click On Install without
Restart. Once done, restart the Jenkins instance
‫© כל הזכויות שמורות לג'ון ברייס הדרכה בע"מ מקבוצת מטריקס‬
82
Step 3 − Now when you go to Manage Jenkins, and scroll down you will see ‘Backup
Manager’ as an option. Click on this option.
‫© כל הזכויות שמורות לג'ון ברייס הדרכה בע"מ מקבוצת מטריקס‬
83
Step 4 − Click on Setup.
Step 5 − Here, the main field to define is the directory for your backup. Ensure it’s on
another drive which is different from the drive where your Jenkins instance is setup.
Click on the Save button.
‫© כל הזכויות שמורות לג'ון ברייס הדרכה בע"מ מקבוצת מטריקס‬
84
Step 6 − Click on the ‘Backup Hudson configuration’ from the Backup manager screen
to initiate the backup.
The next screen will show the status of the backup
‫© כל הזכויות שמורות לג'ון ברייס הדרכה בע"מ מקבוצת מטריקס‬
85
To recover from a backup, go to the Backup Manager screen, click on Restore Hudson
configuration.
The list of backup’s will be shown, click on the appropriate one to click on Launch
Restore to begin the restoration of the backup.
‫© כל הזכויות שמורות לג'ון ברייס הדרכה בע"מ מקבוצת מטריקס‬
86
Jenkins - Remote Testing
Web tests such as selenium tests can be run on remote slave machines via the master
slave and selenium suite plugin installation. The following steps show how to run
remote tests using this configuration.
Step 1 − Ensuring your master slave configuration is in place. Got to your master
Jenkins server. Go to Manage Jenkins → Manage Nodes.
In our node list, the DXBMEM30 label is the slave machine. In this example, both the
master and slave machines are windows machines.
‫© כל הזכויות שמורות לג'ון ברייס הדרכה בע"מ מקבוצת מטריקס‬
87
Step 2 − Click on configure for the DXBMEM30 slave machine.
Step 3 − Ensure the launch method is put as ‘Launch slave agents via Java Web Start’
Step 4 − Now go to your slave machine and from there, open a browser instance to
your Jenkins master instance. Then go to Manage Jenkins → Manage Nodes. Go to
DXBMEM30 and click on
‫© כל הזכויות שמורות לג'ון ברייס הדרכה בע"מ מקבוצת מטריקס‬
88
Step 5 − Click on the DXBMEM30 instance.
Step 6 − Scroll down and you will see the Launch option which is the option to Start
‘Java Web Start’
‫© כל הזכויות שמורות לג'ון ברייס הדרכה בע"מ מקבוצת מטריקס‬
89
Step 7 − You will be presented with a Security Warning. Click on the Acceptance
checkbox and click on run.
You will now see a Jenkins Slave window opened and now connected.
Step 8 − Configuring your tests to run on the slave. Here, you have to ensure that
the job being created is meant specifically to only run the selenium tests.
In the job configuration, ensure the option ‘Restrict where this project can be run’ is
selected and in the Label expression put the name of the slave node.
‫© כל הזכויות שמורות לג'ון ברייס הדרכה בע"מ מקבוצת מטריקס‬
90
Step 9 − Ensure the selenium part of your job is configured. You have to ensure that
the Sample.html file and the selenium-server.jar file is also present on the slave
machine.
Once you have followed all of the above steps, and click on Build, this project will run
the Selenium test on the slave machine as expected.
‫© כל הזכויות שמורות לג'ון ברייס הדרכה בע"מ מקבוצת מטריקס‬
91
Version Control Tools
Introduction to Version control
• Version control systems helps manage
changes to source code over time. It keeps
track of every modification to the code in a
special kind of database. If a mistake is
made, developers can compare earlier
versions to help fix the mistake.
• There are many Version control tools. We will
compare between five- SVN, Git, Perforce,
TFS and Clear Case.
92
Introduction to SVN
• Apache Subversion (SVN) is an open source software
under the Apache license.
• The open source community has used SVN widely, for
example in projects such as the Apache software
foundation, FreeBSD, GCC, Mono, SourceForge and
more.
• SVN was created by CollabNet Inc. and is now a top-level
Apache project being built and used by a global
community of contributors.
Introduction to SVN
• SVN started as an effort to write an open-source version
control system to succeed the widely used CVS, but also to
fix bugs and provide some features missing in CVS.
• SVN is not an Apache extension. It is a set of libraries that
comes with a command-line client that uses them.
• You don’t have to set up Apache to use SVN. If you want to
access a repository, you only need to build a SVN client. If
you want to host a networked repository, then you need to
set up Apache.
93
Introduction to Git
• Git is a free software primarily used for software
development but can be used to keep track of changes in
any files.
• Git is aimed at speed, data integrity and support for
distributed, non-linear workflows.
• Every Git directory on every computer is a full-pledged
repository with complete history and full version tracking
abilities, independent of network access of a central server.
Introduction to Git
• Because Git is a distributed software, it can be used
as a server out of the box.
• Dedicated Git server software helps to add access
control, display the contents of a Git repository via
Web, manage multiple repositories and more helpful
features.
• The main repository management services are
Github, Atalassian’s Bitbucket and GitLab.
94
Introduction to Perforce
• Perforce is a commercial revision control system
developed by Perforce Software.
• The Perforce Helix server manages a central database
and a master repository of file revisions.
• Two GUI clients are available for Perforce users- the
cross-platform P4V, and the windows only P4Win. Both of
them support the majority of end-user operations.
• There is an administration GUI client called P4Admin.
Introduction to Perforce
• On the Perforce server, MD5 hashes of a file
content are stored in the database. The
database tables are stored as binary files.
Checkpoints and journals are written as text files
that can be compressed and offloaded.
• If a database has been corrupted it can be
recovered from the most recent journal and
checkpoint so administrators must plan for
disaster recovery by configuring database
journaling and regular checkpoints.
95
Introduction to TFS
• Team Foundation Server (TFS) is a Microsoft
product that is only available on Windows OS.
• TFS provides other capabilities for the project
development besides managing versions, like
requirements management, testing, release
management and more.
• TFS supports two different types of source
control- its original engine calls Team
Foundation Version Control (TFVC), and Git.
Introduction to TFS
• TFVC supports two different types of workspaces- server
workspaces and local workspaces.
• Server workspaces allows developers to lock files for checkout and provide notification to other developers that the files
are being edited. If the server can’t be connected they need
to “go offline”.
• In local workspaces, files are not read-only and they do not
have to be checked out before working on them. If the files
are on the developer’s machine, the server can be
connected or not. Conflicts are dealt with at check-in time.
96
Introduction to Clear Case
• ClearCase is a family of computer software tools developed
by IBM, that includes revision control and workspace
management.
• An individual database is ClearCase is called VOB
(Versioned Object Base). It is a centralized database that
stores version information about the files and folders.
• Each directory in ClearCase has linear history, meaning
there is no direct relationship between them, so when you
compare two commits you must compare the list of all the
files and directories.
Introduction to Clear Case
• ClearCase commit is file by file and not “set of
modified files”. The transaction is at the file level
and not at the repository level.
• Each file and directory has a unique ID, meaning
they can be renamed at will but the history will not
change since the ID remains. But that also means
that two files with the same size and content will be
stored twice instead of once (unlike in Git that will
give them the same ID- a SHA1 key).
97
SVN
Current
maintainer
Git
Perforce
TFS
CC
Junio
Hamano
Perforce
Software Inc.
Microsoft
IBM Rational
Active
Active
Basic Comparison
Apache
Software
Foundation
Developmen
t Status
Active
Active
Active
Repository
model
Client–server
Distributed
Client–server
GNU GPL
Proprietary
Proprietary
Proprietary
Free for up to
20 users,
open source
projects or
educational
use; else
non-free
Non-free
$4600 per
floating
license
License
Apache
Client–server,
Client–server
Distributed
Cost
Free
Free
Free for up to
5 users in the
Team or for
open source
projects; else
non-free
Concurrency
model
Merge or
lock
Merge
Merge or
lock
Merge or
lock
Merge or
lock
SVN
Git
Perforce
TFS
CC
Language
C
C, shell
scripts, Perl
C++, C,
LabVIEW
C++ and C#
C, Java, Perl
Storage
method
Changeset
and Snapshot
Snapshot
Changeset
Changeset
Changeset
Scope of
change
Tree
Tree
Tree
File and tree
File and tree
Revision IDs
Numbers
SHA-1 hashes
Numbers
Numbers
Numbers
Network
Protocols
svn, ssh,
HTTP, SSL
Git, ssh,
HTTP/HTTPS,
rsync, email,
bundels
custom
SOAP over
HTTP or
HTTPS
CCFS, MVFS, HTTP
Technical Comparison
98
SVN
Git
Perforce
TFS
Atomic commits
Yes
Yes
Yes
Yes
Features
Comparison
File renames
Yes
Partial
Yes
Yes
CC
Partial
Yes
Merge file renames
Partial
Yes
Yes
Yes
Yes
Symbolic links
Yes
Yes
Partial
Yes
Yes
Pre-/Post event hooks
Yes
Yes
Yes
Yes
Yes
Signed revisions
No
Yes
Yes
Yes
Yes
Merge tracking
Yes
Yes
Yes
Yes
Yes
Interactive commits
Yes
Yes
No
Unknow
n
No
Tags
Partial
Yes
Yes
Yes
Yes
Partial checkout
Yes
Yes
Yes
Yes
Yes
Timestamp
preservation
Partial
No
Yes
Unknow
n
Yes
Supports large repos
Yes
Partial
Yes
Yes
Yes
QUESTIONS ?
16
© All rights reserved to John Bryce Training LTD from Matrix group
99
Sonarqube and
Jenkins
Introduction to SonarQube
• SonarQube is an open source platform for
continuous code quality inspection. It supports
languages like Java, JavaScript, C++, C#,
Python and more.
• SonarQube supports fully automated analysis
by integrating with continuous integration (CI)
tools like Jenkins.
100
Introduction to Jenkins
• Jenkins is an open source CI tool and
automation server that is often used in the
software development process.
• Jenkins functionality can be extended with
plug-ins to support building, deploying and
automating different projects on many
platforms.
SonarQube Installation
• Since SonarQube is an open source platform, you can
download the latest SonarQube server and SonarQube
Scanner distributions and start using them very quickly.

The SonarQube General Settings view for an administrator.
101
SonarQube Dashboard
SonarQube Jenkins Plug-in
• The SonarQube Jenkins plugin lets you
centralize the configuration of SonarQube
server and scanner connection details in
Jenkins global configuration.
• After you configured the details, you can use
standard Jenkins build steps to trigger analysis
with SonarQube Scanner.
102
SonarQube Jenkins Plug-in
• To install the SonarQube Jenkins plugin, you need to
access Jenkins plug-ins management screen.
SonarQube Jenkins Plug-in
• After installing the SonarQube Jenkins
plugin, you can define as many SonarQube
servers as you wish.
• You will be able to choose which server to
use for the SonarQube analysis for each
Jenkins job.
103
SonarQube Jenkins Plug-in
•
The SonarQube configuration section under Jenkins system configuration screen.
SonarQube Jenkins Plug-in
• You will also need to add the SonarQube
scanners details in the Jenkins Global Tool
Configuration. You can define as many
SonarQube scanners as you which.
• You will be able to choose which scanner to
use for the SonarQube analysis for each
Jenkins job.
104
SonarQube Jenkins Plug-in
• The SonarQube Scanner configuration section under
Jenkins Global Tool Configuration screen.
SonarQube Jenkins Plug-in
• After adding the SonarQube servers and
scanners, you can add the ‘Execute
SonarQube Scanner’ build step to configure
the SonarQube analysis for you job.
105
SonarQube Jenkins Plug-in
• The ‘Execute SonarQube Scanner’ build step configuration.
SonarQube Jenkins Plug-in
• Once the Jenkins Job is complete, the
SonarQube plugin will detect that a
SonarQube analysis was made during
the build and will display a badge on the
job page with a link to the SonarQube
dashboard to view the report.
106
QUESTIONS ?
15
© All rights reserved to John Bryce Training LTD from Matrix group
107
SonarQube installation on Windows
Prerequisites

Java (JRE or JDK)
Steps
SonarQube server
1. Login to the server
2. Create a directory (like C:\sonarqube)
3. Download and unzip the SonarQube distribution from the SonarQube site
https://www.sonarqube.org/downloads/
4. Execute the following to start the SonarQube server
C:\sonarqube\bin\windows-x86-xx\StartSonar.bat
SonarQube scanner
1. Login to the server
2. Create a directory (like C:\sonar-scanner)
3. Download and unzip the SonarQube scanner from the SonarQube Site
https://docs.sonarqube.org/display/SCAN/Analyzing+with+SonarQube+Scanner
4. Anaylize a project using the SonarQube scanner
cd <Project_Folder>
C:\sonar-scanner\bin\sonar-scanner.bat
5. You can download, unzip and use project samples from the SonarQube site
Create a directory (like C:\sonar-scanning-examples)
cd C:\sonar-scanning-examples\sonarqube-scanner
C:\sonar-scanner\bin\sonar-scanner.bat
6. Access the SonarQube dashboard at http://localhost:9000 to see the results
Use the User "admin" with the Password "admin"
108
CI with Docker
Objectives
• By the end of this session
– You’ll understand Docker technology
– You’ll be able to configure Jenkins with Docker
master and slave
109
[www.docker.com]
What is Docker ?
• Docker is an open-source project
that automates the deployment
of applications inside software containers,
by providing an additional layer of
abstraction and automation of operating
system–level virtualization on Linux
What is Docker ?
• Provides a uniformed wrapper around a
software package:
«Build, Ship and Run Any App, Anywhere»
[www.docker.com]
– Similar to shipping containers: The container is
always the same, regardless of the contents
and thus fits on all trucks, cranes, ships, ...
110
Docker vs. Virtual Machine
Docker’s Technology
• libvirt: Platform Virtualization
• LXC (LinuX Containers): Multiple isolated
Linux systems (containers) on a single host
• Layered File System
[Source: https://docs.docker.com/terms/layer/]
111
Run Platforms
• Various Linux distributions (Ubuntu, Fedora,
RHEL, Centos, openSUSE, ...)
• Cloud (Amazon EC2, Google Compute
Engine, Rackspace)
• Windows 10
• Mac
7
Hello World
Simple Command - Ad-Hoc Container
• Docker run ubuntu echo Hello World
– docker images [-a]
– docker ps –a
8
112
Terminology - Image
• Persisted snapshot that can be run
– images: List all local images
– run: Create a container from an image and execute a
command in it
– tag: Tag an image
– pull: Download image from repository
– rmi: Delete a local image
• This will also remove intermediate images if no
longer used
9
Terminology - Container
• Runnable instance of an image
– ps: List all running containers
– ps –a: List all containers (incl. stopped)
– top: Display processes of a container
– start: Start a stopped container
– stop: Stop a running container
– pause: Pause all processes within a container
– rm: Delete a container
– commit: Create an image from a container
113
10
Image vs. Container
Base Image
ubuntu:latest
base image
New Image
iid1
Container
cid1
run
cmd → new state
commit
run
Container
cid1
Container
Container
cid2
Container
cid3
cid4
11
Dockerfile
• Create images automatically using a build
script: «Dockerfile»
• Can be versioned in a version control
system like Git or SVN, along with all
dependencies
• Docker Hub can automatically build images
based on dockerfiles on Github
114
12
Dockerfile Example
• Dockerfile:
– FROM ubuntu
ENV DOCK_MESSAGE Hello My World
ADD dir /files
CMD ["bash", "someScript"]
• docker build [DockerFileDir]
• docker inspect [imageId]
Mount Volumes
• docker run –ti –v /hostLog:/log ubuntu
• Run second container: Volume can be
shared
– docker run –ti --volumes-from
firstContainerName ubuntu
115
Publish Port
• docker run –t –p 8080:80 ubuntu nc –l 80
– Map container port 80 to host port 8080
– Check on host: nc localhost 8080
• Link with other docker container
– docker run -ti --link containerName:alias ubuntu
– See link info with set
Around Docker
• Docker Images: Docker Hub
• Vagrant: «Docker for VMs»
• Automated Setup
– Puppet, Chef, Ansible, ...
• Docker Ecosystem
– skydock / skydns
– Docker compose
116
Docker Hub
• Public repository of Docker images
– https://hub.docker.com/
– docker search [term]
• Automated: Has been automatically built
from Dockerfile
– Source for build is available on GitHub
Resource Usage
• top / ps / free -m
• Start 100 WebServer containers
– docker run -d -p $hostPort:5000 -e
"PROVIDER=$provider" training/webapp
• docker ps [containerId]
• top / ps / free -m
117
Docker Use Cases
•
•
•
•
•
•
Development Environment
Environments for Integration Tests
Quick evaluation of software
Microservices
Multi-Tenancy
Unified execution environment (dev → test
→ prod (local, VM, cloud, ...)
Docker Compose
• Compose is a tool for defining and running
multi-container Docker applications
118
Jenkins and Docker
Workflow
1. Developer pushes a
commit to GitHub
2. GitHub uses a webhook
to notify Jenkins of the
update
3. Jenkins pulls the GitHub
repository, including the
Dockerfile describing the
image, as well as the
application and test code
4. Jenkins builds a Docker
image on the Jenkins slave
node
5. Jenkins instantiates the
Docker container on the
slave node, and executes
the appropriate tests
6. If the tests are
successful the image is
then pushed up to Docker
Trusted registry
119
Configuring Jenkins Master
After the base Jenkins image is installed, •
and the service is up and running the
GitHub Plugin needs to be installed on the
Jenkins master
This plugin allows for a Jenkins job to be •
initiated when a change is pushed a
designated GitHub repository
Configuring Jenkins Slave
120
Swarm Plugin
as Jenkins Slave in Docker
 This plugin enables slaves to auto-discover nearby Jenkins
master and join it automatically, thereby forming an ad-hoc
cluster
 This plugin consists of two pieces:
 A self-contained CLI client that discovers a nearby Jenkins (via a UDP broadcast)
and joins it
 A plugin that needs to be installed on Jenkins master to accept swarm clients
25
© All rights reserved to John Bryce Training LTD from Matrix group
Swarm Plugin Usage
• Install the swarm plugin from the update center
• Download https://repo.jenkinsci.org/releases/org/jenkins-ci/plugins/swarm-client/
• Run the CLI agent like java -jar path/to/swarmclient-jar-with-dependencies.jar.
121
Summary
• Docker
• Dockerfile
• Configuring Jenkins
– Master
– Slave
• Swarm Plugin
122
Docker compose installation on Linux
Prerequisites
1. 64-bit OS
2. Remove unofficial docker packages
yum -y remove docker
Steps
1. Login to the server
2. Update yum
yum update -y
3. Install yum-utils
yum install -y yum-utils
4. Set up the Docker repository
yum-config-manager \
--add-repo \
https://docs.docker.com/engine/installation/linux/repo_files/centos/docker.repo
5. Install Docker
yum -y install docker-engine
6. Verify the installation by running the hello-world image
docker run hello-world
7. Configure Docker to start on boot
chkconfig docker on
8. Run docker ps -a to show all containers on system
9. Execute the whalesay example
docker run docker/whalesay cowsay boo
10. Run docker images to show all images on system
11. Make a directory and cd into it
mkdir mydockerbuild
cd mydockerbuild
12. Create a new file named Dockerfile
13. Edit the file and add to itFROM docker/whalesay:latest
RUN apt-get -y update && apt-get install -y fortunes
CMD /usr/games/fortune -a | cowsay
14. Save the file
15. Build your image with the docker build commanddocker build -t docker-whale .
16. Run your new docker-whale image
docker images
docker run docker-whale
17. Create a Docker Hub account
https://hub.docker.com/register/?utm_source=getting_started_guide&utm_medium=e
mbedded_MacOSX&utm_campaign=create_docker_hub_account
123
18. Create a repository in the Docker Hub site
19. Run docker images to show all images on system and find the IMAGE ID for
your docker-whale image
20. Use IMAGE ID and the docker tag command to tag your docker-whale imagedocker tag 7d9495d03763 maryatdocker/docker-whale:latest
21. Run docker images to show all images on system and your newly tagged
docker-whale image
22. Use the docker login command to log into the Docker Hub from the commnad
line
23. Use the docker push to push your image to your repository
docker push maryatdocker/docker-whale
24. You will now be able to see your image on your profile on Docker Hub site
25. Run docker images to show all images on system
26. Use docker rmi to remove the image you pushed to the Hub
docker rmi -f 7d9495d03763
docker rmi -f docker-whale
27. Pull the image from your repository using the docker run command
docker run yourusername/docker-whale
Docker will download your image because it is no longer available on your
local system
28. Install Docker compose with the curl commandcurl -L "https://github.com/docker/compose/releases/download/1.9.0/dockercompose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
chmod +x /usr/local/bin/docker-compose
29. Test the installation by running docker-compose --version
30. Make a directory and cd into it
mkdir composetest
cd composetest
31. Create a DockerfileFROM httpd:2.4
EXPOSE 80
CMD ["httpd-foreground"]
32. Create a docker-compose.yml fileversion: '2'
services:
124
tomcat:
image: tomcat
ports:
- "8080:8080"
httpd:
image: httpd
build: .
ports:
- "80:80"
links:
- tomcat
33. Run the command to start your applicationsdocker-compose up --build
34. Access http://localhost:80 to see the result
35. Access http://localhost:8080, you should see the tomcat welcome page
125
Configuration
Management Tools
Introduction to CM
• Configuration management (CM) is the concept and
process of tracking and controlling the changes in the
software projects and managing the different parts of the
system.
• There are many CM tools. We will compare between fourChef, Puppet, Ansible and SaltStack.
126
Introduction to Chef
• Chef is used to streamline the task of configuring and
maintaining servers in various scales.
• The user writes ‘recipes’ (who can be grouped as a
‘cookbook’) that describe how chef will manage, configure
or correct server applications and other resources.
• Chef is used by Airbnb, Mozilla, Expedia, Facebook and
more.
Introduction to Chef
• The recipes and cookbooks the user write and develops
are written in the Chef workstation using the Chef
development kit.
• Later, the user deploys the finished cookbooks to the Chef
server that acts as a central repository for cookbooks and
as a node manager.
• The chef nodes are also called Chef clients, and they
communicate with the chef server to get the latest
configuration instructions.
127
Introduction to Chef
• Chef can be run in client/server mode, or in a standalone
configuration named ‘chef-solo’.
• Chef offers online support, tutorials, community forums
and documentation.
• Chef also has the Chef supermarket, which is a site for
accessing and sharing public cookbooks that can be
customized for any propose. It provides and easily
searchable cookbook repository.
Introduction to Puppet
• Puppet is designed to manage the configuration of
systems declaratively.
• The user describes resources and their state using
Puppet’s custom declarative language or a Ruby DSL in
‘Puppet manifests’.
• Puppet discovers the system information via a utility called
‘Facter’, and compiles the manifests into a system-specific
catalog of resources and their dependencies, which are
applied to the target systems. Any action taken by Puppet
are then reported.
128
Introduction to Puppet
• Puppet is model-driven so limited programming knowledge
is required in order to use it.
• Puppet has documentation on the official site. Puppet also
has the Puppet Forge which is a repository of puppet
modules written by the community of Puppet users.
• Puppet is used by Twitter, Paypal, Disney, Google, Spotify,
Oracle, Intel and more.
Introduction to Puppet
• Puppet usually follows server-client architecture (Puppet
server which is the master and Puppet client which is the
agent) but it can also be used as a stand-alone application.
• Devices can run Puppet agent as a daemon, that can be
triggered periodically or can be run manually whenever
needed.
• Puppet is open source and therefore flexible and can be
customized to suit needs.
129
Introduction to Ansible
• Ansible is an open source automation engine that
automates cloud provisioning, CM and application
deployment.
• Ansible required python and needs to be installed on a
control machine that has it.
• Ansible can deploy to virtualization environments and
public and private cloud environments. It can also deploy
big data, storage and analytics environments.
• Ansible provides monitoring capabilities to help with
supervision of the nodes.
Introduction to Ansible
• In contrast with Chef and Puppet, Ansible has an agentless
architecture.
• Ansible works by connecting through OpenSSH and
pushing ‘Ansible modules’ to the nodes. These modules
are small programs written to be a resource model of the
desired state of the system.
• After executing these modules, Ansible removes them
when finished.
130
Introduction to SaltStack
• SaltStack is a python based open source CM software and
remote execution engine.
• SaltStack is currently the only CM platform to run in either
agent or agentless mode. You can use a ‘salt minion’
agent, or use salt SSH to run salt commands without it.
• SaltStack is used by LinkedIn, Hulu, HP cloud services
and more.
Introduction to SaltStack
• SaltStack uses different module types to manage actions
like remote executions and state changes.
• The six module types are- Execution modules, State
modules, Grains, Renderer modules, Returners and
Runners.
• Execution modules represent the functions available for
direct execution from the remote execution engine.
• State modules execute the code needed to enforce, set up
or change the configuration of the target system.
131
Introduction to SaltStack
• Grains are a system for detecting static information about
a system and storing it in RAM for rapid gathering.
• Renderer modules are used to render the information
passed to the Salt state system.
• Returners manage the arbitrary return locations from the
separate remote execution calls made by Salt .
• Runners are master side convenience applications
executed by the salt-run command.
Basic Comparison
Tool
Release
License
Current Version
Written in
GUI
Enterprise offering
Ansible
2012
GPLv3+
2.2.0.0
Python
Yes
Yes
Chef
2009
Apache 2.0
Client12.14.89
Server- 12.9.1
Ruby
Yes
Yes
Puppet
2005
Apache from
2.7.0, GPL before
4.8.0
Ruby
Yes
Yes
SaltStack
2011
Apache 2.0
2016.3.4
Python
Yes
Yes
132
Platform Comparison
AIX
Linux
OS X
WINDOWS
HP-UX
Solarix
Others
Ansible
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Chef
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Puppet
Yes
Yes
Yes
Yes
Yes
Yes
Yes
SaltStack
Yes
Yes
Yes
Yes
Partial
Yes
Partial
Script
Language
Infrastructure
Centralized point
of control
Script terminology
Task Execution
Order
Requires Agent
software
Ansible
YAML
Controller machine
applies to hosts via
SSH
Any computer can
be a controller
Playbook (Roles)
Sequential
No
Chef
Ruby
Chef Workstation
pushes to Chef Server
that updates Chef
Clients
Yes, Chef Server
Recipes
(Cookbooks)
Sequential
Yes
Puppet
Custom DSL
based of Ruby
Yes, Puppet master
Manifests
(Modules)
Non-Sequential
Yes
SaltStack
YAML
Yes, Salt master
States (Formulas)
Sequential
Both
Puppet master
updates Puppet
agents
Salt master updates
salt minions or Salt
commands via SSH
133
QUESTIONS ?
134