Database monitoring with Zabbix

Database monitoring with Zabbix
Mastering Zabbix
Monitor your large IT environment efficiently with Zabbix
Andrea Dalle Vacche
Stefano Kewan Lee
BIRMINGHAM - MUMBAI
Mastering Zabbix
Copyright © 2013 Packt Publishing
All rights reserved. No part of this book may be reproduced, stored in a retrieval
system, or transmitted in any form or by any means, without the prior written
permission of the publisher, except in the case of brief quotations embedded in
critical articles or reviews.
Every effort has been made in the preparation of this book to ensure the accuracy
of the information presented. However, the information contained in this book is
sold without warranty, either express or implied. Neither the authors, nor Packt
Publishing, and its dealers and distributors will be held liable for any damages
caused or alleged to be caused directly or indirectly by this book.
Packt Publishing has endeavored to provide trademark information about all of the
companies and products mentioned in this book by the appropriate use of capitals.
However, Packt Publishing cannot guarantee the accuracy of this information.
First published: December 2013
Production Reference: 1131213
Published by Packt Publishing Ltd.
Livery Place
35 Livery Street
Birmingham B3 2PB, UK.
ISBN 978-1-78328-349-1
www.packtpub.com
Cover Image by John Michael Harkness ([email protected])
Credits
Authors
Copy Editors
Andrea Dalle Vacche
Sarang Chari
Stefano Kewan Lee
Alfida Paiva
Adithi Shetty
Reviewers
Jan Garaj
Nitish Kumar
K. M. Peterson
Laxmi Subramanian
Project Coordinator
Shiksha Chaturvedi
Gergely Polonkai
Matthew J. Smith
Acquisition Editors
Mary Jasmine Nadar
Nikhil Karkal
Lead Technical Editor
Azharuddin Sheikh
Technical Editors
Sharvari H. Baet
Gauri Dasgupta
Adrian Raposo
Manan Badani
Mrunmayee Patil
Proofreader
Mario Cecere
Indexer
Mariammal Chettiyar
Graphics
Yuvraj Mannari
Abhinash Sahu
Production Coordinator
Kyle Albuquerque
Cover Work
Kyle Albuquerque
About the Authors
Andrea Dalle Vacche is a highly skilled IT Professional with over 12 years of
industry experience. He graduated from Univerista' degli Studi di Ferrara with an
Information Technology certification. This laid the technology foundation, which
Andrea has built on ever since. He has acquired various other industry respected
accreditations, which include Cisco, Oracle, RHCE, ITIL, and of course Zabbix.
Throughout his career he has worked on many large-scale environments, often in roles
which have been very complex on a consultant basis. This has further enhanced his
growing skill set, adding to his practical knowledge base and concreting his appetite
for theoretical technical study. His love for Zabbix came from his time spent in the
Oracle world as a Database Administrator/Developer. His time was spent mainly
reducing "ownership costs" with specialization in monitoring and automation. This is
where he came across Zabbix and the flexibility, both technically and administratively,
it offered. Using this as a launch pad, it inspired Andrea to develop Orabbix, the first
open source software to monitor Oracle completely integrated with Zabbix.
Andrea has published a number of articles on Zabbix-related software such
as DBforBIX. His projects are publicly available on his website http://www.
smartmarmot.com. Currently, Andrea is working for a leading global investment
bank in a very diverse and challenging environment. His involvement is vast
and deals with many aspects of the Unix/Linux platforms as well as paying
due diligence to many different kinds of third-party software, which are
strategically aligned to the bank's technical roadmap.
First, I would like to thank my wife Anna for her support and
encouragement during the writing of this book. I highly appreciate
her help and advice. Many thanks to Fifi for her relaxing company
and fluffy stress relief. I am grateful to my ex-boss Giovanni for
his patience when I used to fill his mailbox with odd Zabbix test
messages. It was nice having been cheered up by my friends and
colleagues: Bav with his precious suggestions and Antonio always
ready to encourage me. Special thanks to the Packt Publishing team:
Abhijit, Nikhil, Sanhita, Mary, and Reshma. Their advice, effort,
and suggestions have been really valuable. The whole team has
been very professional and helpful.
Stefano Kewan Lee is an IT Consultant with 10 years of experience in system
integration, security, and administration. He is a certified Zabbix specialist in Large
Environments, holds a Linux administration certification from the LPI, and a GIAC
GCFW certification from SANS Institute. When he's not busy breaking websites,
he lives in the countryside with two cats and two dogs and practices martial arts.
I would like to thank all my family and friends for their help and
support, my co-author Andrea, and most of all, my partner Roberta
for putting up with me on a daily basis.
About the Reviewers
Jan Garaj is a DevOps engineer, who pulls his hair out, when he sees inefficient
and expensive server/software solutions. It's because he also wrote some dumb
code with terrible exponential O(n2) memory and computational complexity
during software engineering study. However, he learned from a few exceptional
effective programmers that good algorithms don't need a ton of memory and four
core processors. His first touch with Zabbix was in 2009, when he had to choose
a monitoring system for his current employer. He chose Zabbix 1.6. Then, he also
came across Zabbix in another company with high website traffic environment
or on mission critical servers in the automotive sector. He worked also for
Hewlett-Packard, so he knows some proprietary HP monitoring tools. But he
still prefers Zabbix for open source stack/web monitoring. His Zabbix love lasts
to this day, so he is an active Zabbix frontend language maintainer and he may
also be a code contributor one day.
Nitish Kumar is a Wintel Lead at HT Media Ltd. and an independent tech blogger
about various technologies. He has been working on several Microsoft technologies
and open source solutions (including but not limited to Spiceworks, ManageEngine
Products, Zabbix, MS Active Directory, MS Exchange Servers, and so on) for the
past eight years, of which the last couple of years have been spent on bringing
cost-effective solutions to corporates to simplify their complex requirements and to
improve time management for their staff. He is a technology enthusiast and has been
participating at various corporate events and public webinars. Mobile technologies
have been of special interest to him and he has often written about various gadgets
and respective tech. Nitish holds an MS degree in Software from JK Institute
of Applied Physics and Technology and his areas of interest include Microsoft
technologies, open source software, and mobile gadgets.
He occasionally blogs at http://nitishkumar.net and can be reached over e-mail
at [email protected]
Huge thanks to my wife Pooja and colleagues for being there, and to
my kiddo for enduring through the process of writing the book.
Thanks to the Packt Publishing team for their persistence and
patience—it surely was hard to work with a chaotic person like me.
K. M. Peterson has worked across traditional boundaries in Information
Technology, with a passion for listening and teaching, and a focus on helping
those who use technology in an organization work more productively and
efficiently. His experience spans over 20 years in all facets of Information
Technology, across academic, scientific, and commercial environments. He is a
technical manager, with hands-on expertise in infrastructure: computer servers,
network technologies, data management, and the disciplines of creating facilities
that power reliable and scalable applications. He has built data centers and managed
teams of systems administrators, database architects, network engineers, and end
user support specialists. Mr. Peterson has mastered technologies from operating
systems (Windows, Linux, and Mac) to security, Storage Area Networks and
virtualization to network management and routing.
He is an independent consultant. Previously, he was Senior Director of Information
Technology at Basis Technology Corp., where he architected new applications,
enhanced data storage, systems automation, and network services and security.
Prior to his position at Basis, Mr. Peterson worked for John Snow, Inc., a global public
health consulting firm. Earlier, he managed Information Technology operations for
the Whitehead Institute/MIT Center for Genome Research, where he architected the
production systems and managed teams that supported the largest contribution of
data to the Human Genome Project, crucial to the overall success of one of the largest
scientific efforts yet completed.
Gergely Polonkai is a systems engineer for SRV Monitoring, a network monitoring
company, and a freelancer software developer. Educated as an electrician, he got in
closer touch with computers at the age of 11 while messing with BASIC and Pascal
code. In secondary school, he got two Monkey Linux boxes to manage, and his interest
and commitment for Open Source Software remains since. Gergely runs a small blog
of his own (http://gergely.polonkai.eu/blog), which has articles on development
and Linux administration.
Matthew J. Smith has been in Enterprise IT since 1996, involved in a myriad of
technology areas including systems administration, monitoring, and orchestration,
virtualization and cloud, software development, and his favorite area, identity and
access management. Through almost two decades of experience at multiple levels of
the IT organization, leading teams and committees of all sizes through many tactical
and strategic initiatives, Matt has learned the value of bringing open source solutions
to enterprise-scale problems.
Most recently, Matt has taken the opportunity to join Red Hat, Inc., as a Solution
Architect, bringing his passion for and expertise in Enterprise Open Source to those
looking to leverage Red Hat technologies to solve real business problems. Interacting
daily with enterprises in a great variety of global industries, including financial
services, insurance, healthcare, pharmaceutical, automotive services, manufacturing,
and hospitality, Matt experiences the viability and power of Open Source solutions
solving the most complex problems. Matt has come to truly appreciate the values of
Open Source, is a contributing member of multiple Open Source communities, and is
a vocal advocate for Open Source approaches to solving problems.
www.PacktPub.com
Support files, eBooks, discount offers and more
You might want to visit www.PacktPub.com for support files and downloads related
to your book.
Did you know that Packt offers eBook versions of every book published, with PDF
and ePub files available? You can upgrade to the eBook version at www.PacktPub.com
and as a print book customer, you are entitled to a discount on the eBook copy. Get in
touch with us at [email protected] for more details.
At www.PacktPub.com, you can also read a collection of free technical articles,
sign up for a range of free newsletters and receive exclusive discounts and offers
on Packt books and eBooks.
TM
http://PacktLib.PacktPub.com
Do you need instant solutions to your IT questions? PacktLib is Packt's online
digital book library. Here, you can access, read and search across Packt's entire
library of books. Why Subscribe?
• Fully searchable across every book published by Packt
• Copy and paste, print and bookmark content
• On demand and accessible via web browser
Free Access for Packt account holders
If you have an account with Packt at www.PacktPub.com, you can use this to access
PacktLib today and view nine entirely free books. Simply use your login credentials
for immediate access.
Table of Contents
Preface1
Chapter 1: Deploying Zabbix
9
Defining the environment size
Zabbix architectures
Zabbix installation
10
11
14
Prerequisites16
Setting up the server
17
Installing the database
Starting up with Daemon
Some considerations about the database
Sizing of the database
Some considerations about housekeeping
The web interface
21
24
30
32
35
41
Capacity planning with Zabbix
48
Setting up the agent
Installing and creating the package
Configuring the server
The web wizard – frontend configuration
The observer effect
What to monitor
Define a baseline
Load testing
Forecasting the trends
Summary
Chapter 2: Distributed Monitoring
Zabbix proxies
Deploying a Zabbix proxy
Understanding the flow of monitoring data with proxies
Zabbix nodes
Understanding the flow of data with nodes
Deploying a node
18
19
20
42
48
48
50
51
53
54
55
56
57
63
66
67
67
Table of Contents
Proxies versus nodes
Disadvantages of nodes
Choosing between proxies and nodes
Security considerations
No network configuration
Network isolation
Simple tunnels
72
72
73
73
74
76
76
A full-blown VPN
Summary
79
80
Secure Shell
Stunnel
Chapter 3: High Availability and Failover
Understanding high availability
Understanding the levels of IT service
Some consideration about high availability
Automating the switchover/the failover with a resource manager
Replicating the filesystem with DRBD
Implementing high availability on a web server
Configuring HTTPD HA
Understanding Pacemaker and STONITH
Pacemaker – is Quorum really needed?
Pacemaker – the stickiness concepts
Pacemaker – the Apache/HTTPD configuration
Configuring the Zabbix Server for high availability
Database high availability
Clustering of PostgreSQL
Mirrored logical volume with LVM and DRBD
Prerequisite tasks to start with DRBD on LVM
Creating a DRBD device on top of the LVM partition
Enabling resource in DRBD
Defining a primary device in DRBD
Creating a filesystem on a DRBD device
76
78
81
82
83
84
86
86
87
88
90
91
92
92
96
98
100
100
102
103
104
105
105
Pacemaker cluster – integrating DRBD
107
DRBD performance and optimizations
112
Enabling the DRBD configuration
Pacemaker – the LVM configuration
Pacemaker – configuring PostgreSQL
Pacemaker – the network configuration
Pacemaker – the final configuration
Cluster configuration – the final test
107
108
109
109
110
110
Efficient synchronization by DRBD
112
Enabling online verification through DRDB114
DRBD – some networking consideration
115
Summary
118
[ ii ]
Table of Contents
Chapter 4: Collecting Data
Gathering items as raw data
Understanding the data flow for Zabbix items
Understanding Zabbix trapper items
The data flow overview
Database monitoring with Zabbix
Delving into ODBC
Installing database drivers
MySQL ODBC drivers
PostgreSQL ODBC drivers
Oracle ODBC drivers
UnixODBC configuration files
Compiling Zabbix with ODBC
Database monitor items
Some considerations about the ODBC SQL query
119
120
122
125
125
126
127
128
128
130
131
133
134
135
136
Zabbix JMX monitoring
Considering some JMX security aspects
Installing a Zabbix Java gateway
Configuring Zabbix JMX
137
138
139
141
Zabbix SNMP monitoring
SNMP queries
SNMP traps
145
148
151
Web pages monitoring
Authenticating on web pages
Logging out
Aggregated and calculated items
Aggregated items
Calculated items
Summary
158
160
162
164
164
167
168
JMX keys in detail
Issues and considerations about JMX
The snmptrapd process
The Perl trap handler
143
144
152
153
Chapter 5: Visualizing Data
169
Graphs171
Analyzing simple graphs
171
Analyzing custom graphs
174
Reviewing all combinations of graph properties
Visualizing the data through maps
Creating your first Zabbix Map
Some important considerations about macros and URLs
Finally inside the map
Selecting elements
[ iii ]
179
182
186
188
191
194
Table of Contents
Playing with macros inside maps
Visualizing through screens
Creating a screen
Dynamic elements
Visualizing the data through a slide show
Controlling centre slides and the big display challenge
Some consideration about slides on a big display
Automated slide show
IT services
Configuring an IT service
Summary
Chapter 6: Managing Alerts
Understanding trigger expressions
Selecting items and functions
Choosing between seconds or number of measurements
Date and time functions
Trigger severity
Choosing between absolute values and percentages
Understanding operations as correlations
Managing the trigger dependencies
Taking action
Defining an action
The {EVENT.DATE} and {EVENT.TIME} macros
The {INVENTORY.SERIALNO.A} and friends macros
The {NODE.ID} and {NODE.NAME} macros
Defining the action conditions
Choosing the action operations
Steps and escalations
Messages and media
Remote commands
195
197
198
199
201
202
203
204
205
206
209
211
212
213
214
215
216
216
217
221
222
223
224
224
224
225
226
227
228
230
Summary
231
Chapter 7: Managing Templates
233
Creating templates
Adding entities to a template
Using macros
234
234
236
User-defined macros
241
Linking templates to hosts
Nesting templates
Combining templates
Discovering hosts
Low-level discovery
Summary
242
242
244
244
248
252
[ iv ]
Table of Contents
Chapter 8: Handling External Scripts
External checks
The script's placement
Going deep into external checks
Inside the script
General script writing rules
Some consideration about external checks
The UserParameter
The flexible UserParameter
Some considerations about UserParameters
Sending data using zabbix_sender
The new script
Writing a wrapper script for check_ora_sendtrap
Pros and cons of a dedicated script server
Working with Zabbix protocols
The Zabbix get protocol
The Zabbix sender protocol
An interesting undocumented feature
Using clock properties in JSON items
The Zabbix Agent protocol
253
254
255
255
258
260
260
261
261
263
264
265
266
267
268
269
269
271
272
273
Some more possible responses
Communicating with Zabbix
Implementing Zabbix sender in Java
Implementing Zabbix sender in Python
Some considerations about agent development
Summary
Chapter 9: Extending Zabbix
Exploring the Zabbix API
Authenticating through the API
Using the PyZabbix library
Synchronizing templates across different nodes
Mass operations
Redistributing hosts to proxies
Adding or updating users
Exporting data
Extracting tabular data
Creating graphs from data
The Graphviz suite of programs
Creating a trigger dependency graph
Summary
275
276
276
278
280
281
283
284
286
288
291
294
294
295
298
299
302
302
303
306
[v]
Table of Contents
Chapter 10: Integrating Zabbix
An overview of Request Tracker
Setting up RT to better integrate with Zabbix
Creating a custom queue for Zabbix
Customizing tickets – the links section
Customizing tickets – ticket priority
Customizing tickets – the custom fields
Connecting to the Request Tracker API
Setting up Zabbix to integrate with Request Tracker
Creating RT tickets from the Zabbix events
Summary
307
308
309
310
311
312
313
315
318
322
328
Index329
[ vi ]
Preface
Ever since its first public release in 2001, Zabbix has distinguished itself as a very
powerful and effective monitoring solution. As an open source product, it's easy to
obtain and deploy, and its unique approach to metrics and alarms has helped to set
it apart from its competitors, both open and commercial. It's a powerful, compact
package with very low requirements in terms of hardware and supporting software
for a basic yet effective installation. If you add a relative ease of use, it's clear
that it can be a very good contender for small environments with a tight budget.
But it's when it comes to managing a huge number of monitored objects, with a
complex configuration and dependencies, where Zabbix's scalability and inherently
distributed architecture really shines. More than anything, Zabbix can be an ideal
solution in large, complex, distributed environments, where being able to manage
efficiently and extract meaningful information from monitored objects and events is
just as important, if not more important, than the usual considerations about costs,
accessibility, and the ease of use.
The purpose of this book is to help make the most of your Zabbix installation to
leverage all of its power to monitor effectively any large and complex environment.
What this book covers
Chapter 1, Deploying Zabbix, will focus on choosing the optimal hardware and
software configuration for the Zabbix server and database in relation to the current
IT infrastructure, monitoring goals, and possible evolution. This chapter also
includes a section that covers an interesting database-sizing digression, useful
to calculate the final database size using a standard environment as the baseline.
Correct environment sizing and a brief discussion about metrics and measurements
that can also be used for capacity planning will be covered here. The chapter will
contain practical examples and calculations framed in a theoretical approach to give
the reader the skills required to adapt the information to real-world deployments.
Preface
Chapter 2, Distributed Monitoring, will explore the various Zabbix components,
both on the server and agent side. In addition to the deployment and configuration
of agents, proxies and nodes, maintenance, changed management, and security will
all be taken into account. This section will cover all the possible Zabbix architectural
implementations adding the pros and cons considerations.
Chapter 3, High Availability and Failover, will cover the subjects of high-availability
and failover. For each of the three main Zabbix tiers, the reader will learn to choose
among different HA options. The discussion will build on the information provided
in the previous two chapters, in order to end the first part of the book with a few
complete deployment scenarios that will include high-availability server and databases
hierarchically organized in tiered, distributed architectures geared at monitoring
thousands of objects scattered in different geographical locations.
Chapter 4, Collecting Data, will move beyond simple agent items and SNMP queries
to tackle a few complex data sources. The chapter will explore some powerful Zabbix
built-ins, how to use them, and how to choose the best metrics to ensure thorough
monitoring without overloading the system. There will also be special considerations
about aggregated values and their use to monitor complex environments with
clusters or the more complex grid architectures.
Chapter 5, Visualizing Data, will focus on getting the most out of the data visualization
features of Zabbix. This one is a quite useful section especially if you need to explain
or chase some hardware expansion/improvement to the business unit. You will
learn how to leverage live monitoring data to make dynamic maps and how to
organize a collection of graphs for big-screen visualization in control centers and
implement a general qualitative view. This chapter will cover completely the data
center quality view slide show, which is really useful to highlight problems and warn
the first-level support in a proactive approach. The chapter will also explore some
best practices concerning the IT services and SLA reporting features of Zabbix.
Chapter 6, Managing Alerts, will give examples of complex triggers and trigger
conditions, as well as some advice on choosing the right amount of trigger and
alerting actions. The purpose is to help you walk the fine line between being blind
to possible problems and being overwhelmed by false positives. You will also learn
how to use actions to automatically fix simple problems, raising actions without
the need of human intervention to correlate different triggers and events, and how
to tie escalations to your operations management workflow. This section will make
you aware of what can be automated, reducing your administrative workload and
optimizing the administration process in a proactive way.
[2]
Preface
Chapter 7, Managing Templates, will offer some guidelines for effective template
management: building complex templates schemes out of simple components,
understanding and managing the effects of template modification and maintenance
of existing monitored objects, and assigning templates to discovered hosts. This will
conclude the second part of the book, dedicated to the different Zabbix monitoring
and data management options. The third and final part will discuss Zabbix's
interaction with external products and all its powerful extensibility features.
Chapter 8, Handling External Scripts, will help you learn how to write scripts to
monitor objects not covered by the core Zabbix features. The relative advantages
and disadvantages of keeping the scripts on the server side or agent side, how to
launch or schedule them, and a detailed analysis of the Zabbix agent protocol will
also be covered. This section will make you aware of all the possible side effects,
delay, and load caused by script; you will so be able to implement all the needed
external checks, well aware of all that is connected with them and the relative
observer effect. The chapter will include different implementations on working
Bash, Java, and Python so that you can easily write your own scripts to extend
and enhance Zabbix's monitoring possibilities.
Chapter 9, Extending Zabbix, will delve into the Zabbix API and how to use it to
build specialized frontends and complex extensions or to harvest monitoring
data for further elaboration and reporting. It will include some simple example
implementations, written in Python, that will illustrate how to export and further
manipulate data; how to perform massive and complex operations on monitored
objects; and finally, how to automate different management aspects like user
creation and configuration, trigger activation, and the like.
Chapter 10, Integrating Zabbix, will wrap things up discussing how to make other
systems know about Zabbix, and the other way around. This is key to the successful
management of any large or complex environment. You will learn how to use
built-in Zabbix features, API calls, or direct database queries to communicate with
different upstream and downstream systems and applications. To further illustrate
the integration possibilities, there will be a complete and concrete example of
interaction with the Request Tracker trouble-ticket system.
[3]
Preface
Who this book is for
As the book title is Mastering Zabbix, you won't find any detailed, step-by-step tutorials
(well, except the installation that will be covered from scratch, but with some useful
tips) on the basic usage of Zabbix. Although you may find lots of detailed information
about installing the server, or configuring items, triggers, and screens, you are expected
to have at least basic working knowledge of how it all works, so that you can focus on
a more advanced approach for the same subjects. That said, it is possible to profit from
the contents of this book even if you have no previous Zabbix exposure, but in that
case, you are strongly encouraged to refer to the official Zabbix documentation that
you can find at https://www.zabbix.com/documentation/2.0/manual to fill in any
possible gaps in your knowledge.
What you need for this book
Before going deep into the Zabbix setup, it is important to know that the proposed
setup covered here are tested on a large-production environment (more than 1800
hosts monitored, more than 89500 monitored items, and more than 30000 triggers)
and they can be considered valid for most of large and very large environments.
The high-availability solution proposed in this book has been widely tested,
not purely as a disaster recovery exercise but during a real disaster (network
cables accidentally sheared by an excavating machine).
In this book, it is important to understand that most of the choices done has been on
a practical basis and not driven by passion. One of the main choices made is using
PostgreSQL as the official Zabbix RDBMS. We came across PostgreSQL as RDBMS
mostly for the mature and production ready, features offered:
• Hot backup is available by design
• Atomicity, consistency, isolation, and durability: in one word
it is fully ACID compliance
• Many different native standby configurations (hot standby,
synchronous replication, and so on)
• Efficient partitioning
Zabbix's database is a critical component, especially if you need to keep historical
data available and guarantee constant performances day by day while the database
is growing.
[4]
Preface
We have made some assumptions in this book: the packaging system used in our
examples is yum and then the distribution is obviously a Red Hat Enterprise Linux.
Anyway, excluding some details such as package names and packet manager,
the whole book is valid for all the Linux distributions. Furthermore, the proposed
architectures and their implementation are not directly tied to a particular distribution.
We did not use any Red Hat-specific clustering system or a choice that you cannot
reproduce on your favorite Linux distribution.
On reading this book, you will find different open source software, but between all
of them it would be better if you are familiar in particular with the following:
• Apache: http://www.apache.org/
• Pacemaker: http://clusterlabs.org/
• PostgreSQL: http://www.postgresql.org/
• DRBD: http://www.drbd.org
This book also focuses on system administrators that have some programming
skills. We propose different workings for the implemented code snippet. With the
proposed examples all well documented, you should be able to implement your own
plugin or external software fully integrated with Zabbix. The code snippets proposed
are on two different and widely diffused languages: Java and Python. These cover
most of the current programmers' preferences and show, once you know how to
implement the Zabbix protocol, how simple it is to switch between them.
Zabbix is more than monitoring software; it is an open source monitoring solution
that can be explained as you want, and this book will make you aware of all the pros
and cons for the possible solutions.
So now it is time to go deep into the Zabbix's land!
Conventions
In this book, you will find a number of styles of text that distinguish between
different kinds of information. Here are some examples of these styles, and an
explanation of their meaning.
Code words in text, database table names, folder names, filenames, file extensions,
pathnames, dummy URLs, user input, and Twitter handles are shown as follows:
"We can use yum (from root) and install the following package."
[5]
A block of code is set as follows:
# ./configure --enable-server --with-postgresql --with-libcurl
--with-jabber --with-net-snmp --enable-ipv6 --with-openipmi --withssh2 –-with-ldap –-enable-agent
When we wish to draw your attention to a particular part of a code block, the
relevant lines or items are set in bold:
# groupadd zabbixzabbix
# useradd –m –s /bin/bash -g zabbix zabbix
# useradd –m –s /bin/bash -g zabbix zabbixsvr
Any command-line input or output is written as follows:
# yum list postgres*
New terms and important words are shown in bold. Words that you see on the
screen, in menus or dialog boxes for example, appear in the text like this: "The first
screen that you will meet is a welcome page; there is nothing to do there other than
to click on Next".
Warnings or important notes appear in a box like this.
Tips and tricks appear like this.
Reader feedback
Feedback from our readers is always welcome. Let us know what you think about
this book—what you liked or may have disliked. Reader feedback is important for
us to develop titles that you really get the most out of.
To send us general feedback, simply send an e-mail to [email protected],
and mention the book title via the subject of your message.
If there is a topic that you have expertise in and you are interested in either writing
or contributing to a book, see our author guide on www.packtpub.com/authors.
Preface
Customer support
Now that you are the proud owner of a Packt book, we have a number of things to
help you to get the most from your purchase.
Downloading the example code
You can download the example code files for all Packt books you have purchased
from your account at http://www.packtpub.com. If you purchased this book
elsewhere, you can visit http://www.packtpub.com/support and register to have
the files e-mailed directly to you.
Errata
Although we have taken every care to ensure the accuracy of our content, mistakes
do happen. If you find a mistake in one of our books—maybe a mistake in the text or
the code—we would be grateful if you would report this to us. By doing so, you can
save other readers from frustration and help us improve subsequent versions of this
book. If you find any errata, please report them by visiting http://www.packtpub.
com/submit-errata, selecting your book, clicking on the errata submission form link,
and entering the details of your errata. Once your errata are verified, your submission
will be accepted and the errata will be uploaded on our website, or added to any list of
existing errata, under the Errata section of that title. Any existing errata can be viewed
by selecting your title from http://www.packtpub.com/support.
Piracy
Piracy of copyright material on the Internet is an ongoing problem across all media.
At Packt, we take the protection of our copyright and licenses very seriously. If you
come across any illegal copies of our works, in any form, on the Internet, please
provide us with the location address or website name immediately so that we can
pursue a remedy.
Please contact us at [email protected] with a link to the suspected
pirated material.
We appreciate your help in protecting our authors, and our ability to bring you
valuable content.
Questions
You can contact us at [email protected] if you are having a problem with
any aspect of the book, and we will do our best to address it.
[7]
Deploying Zabbix
Most probably, if you are reading this book, you have already used and installed
Zabbix. Most likely on a small/medium environment, but now things have changed
and your environment today is a large one with new challenges coming in regularly.
Nowadays environments are rapidly growing or changing and it is a difficult task to
be ready to support and provide a reliable monitoring solution.
Normally, an initial startup of a system, also a monitoring system, is done by
following some tutorial or how-to and this is a common error. This kind of approach
is valid for smaller environments, when the downtime is not critical, when there are
no disaster recovery sites to handle, or, in few words, when things are easy.
Most likely these setups are not done looking forward to the possible new quantity of
new items, triggers, and events that the server should elaborate. If you have already
installed Zabbix and you need to plan and expand your monitoring solution, or if
instead you need to plan and design the new monitoring infrastructure, this chapter
will help you.
This chapter will also help you perform the difficult task of setting up/upgrading
Zabbix in a large and very large environment. This chapter will cover every aspect
of this task, starting with the definition of a large environment up to using Zabbix
as a capacity planning resource. The chapter will introduce all the possible Zabbix
solutions, doing a practical example with an installation ready to handle a large
environment, going ahead with some possible improvements.
At the end of this chapter, you will understand how Zabbix works, which tables
should be kept under special surveillance and you will know how to improve the
housekeeping on a large environment that, with some years of trends to handle,
is a really heavy task.
Deploying Zabbix
This chapter will cover the following points:
• Knowing when you are in front of a large environment and defining when
an environment can be considered a large environment
• Setting up/upgrading Zabbix on a large and a very large environment
• Installing Zabbix on a three tier setup and having a readymade solution to
handle a large environment
• Database sizing and finally knowing the total amount of space consumed by
our data acquired
• Knowing the database heavy tables and tasks
• Improving housekeeping to reduce the RDBMS load and improving the
efficiency of the whole system
• Learning some fundamental concepts about capacity planning, bearing in
mind that Zabbix is and can be a capacity-planning tool
Defining the environment size
Since this book is focused on a large environment, we need to define or at least
provide some basic fixed points to identify a large environment. There are various
things to consider in this definition; basically, we can identify an environment as
large when:
• There are more different physical locations
• The number of monitored devices is high (hundreds or thousands)
• The number of checks and items retrieved per second (more than 500)
• There is a lot of data to handle (database is larger than 100 GB)
• The availability and performance are both critical
All of the preceding points define a large environment; in this kind of environment,
the installation and maintenance of Zabbix infrastructure play a critical role.
The installation of course is a task well-defined on a timely basis and probably one
of the most critical tasks; it is really important to go live with a strong and reliable
monitoring infrastructure. Also, once we go live with the monitoring in place, it will
not be so easy to move/migrate pieces without any data loss. There are some other
things to consider: we will have a lot of tasks, most of them are daily tasks but in a
large environment they require particular attention.
[ 10 ]
Chapter 1
In a small environment with a small database, a backup will keep you busy for few
minutes, but if the database is quite large, this task will consume a considerable
quantity of time.
The restore and a relative restore plan should be considered and tested periodically
to be aware of the time needed to complete this task in case of a disaster or critical
hardware failure.
Between the maintenance tasks, we need to consider testing and putting on
production the upgrades with a minimal impact and all the daily tasks and
daily checks.
Zabbix architectures
Zabbix can be defined as a distributed monitoring system with a centralized web
interface (on which we can manage almost everything). Among its main features,
we will highlight the following ones:
• It has a centralized web interface
• The server can be run on most Unix-like operating systems
• We can use native agents for most of the Unix, Unix-like, and Microsoft
Windows operation systems
• It has an extremely flexible configuration
• It is easy to integrate with other systems
• It can be monitored via SNMP (v1, v2, and v3), IPMI, JMX, ODBC, and SSH
• It gives us a possibility to create custom items, graphs, and interpolate data
• It is easy to customize
[ 11 ]
Deploying Zabbix
The Zabbix architecture for a large environment is composed of three different
servers/components (that should be configured on HA as well). These three
components are as follows:
• A web server
• A Zabbix server
• An RDBMS server
The whole Zabbix infrastructure in large environments allows us to have two other
actors that play a fundamental role. These actors are the Zabbix agents and the
Zabbix proxies.
An example is represented in the following figure:
[ 12 ]
Chapter 1
On this infrastructure, we have a centralized Zabbix server that is connected to
different proxies, usually one for each server farm or a subnetwork.
The Zabbix server will acquire data from Zabbix proxies and the proxies will
acquire data from all the Zabbix agents connected to it, all the data is stored on a
dedicated RDBMS, and the frontend is exposed with a web interface to the users.
Looking at the technologies used, we see that the web interface is written in PHP
and the server, proxies, and agents are written in C.
The server, proxies, and agents are written in C to give the best
performance and less resource usage possible. All the components
are deeply optimized to achieve the best performance.
We can implement different kinds of architecture using proxies. There are several
type of architectures and in order of complexity we find:
• The single server installation
• One server and many proxies
• Distributed installation
The single server installation is not suggested on a large environment. It is the basic
installation where single servers do the monitoring and it can be considered a good
starting point.
Most probably, in our infrastructure, we might already have a Zabbix installation.
Zabbix is quite flexible and this permits us to upgrade this installation to the next
step: proxy-based monitoring.
Proxy-based monitoring is implemented with one Zabbix server and several proxies,
that is one proxy per branch or datacenter. This configuration is easy to maintain
and offers an advantage to have a centralized monitoring solution. This kind of
configuration is the right balance between large environment monitoring and
complexity. From this point, we can (with a lot of effort) expand our installation to a
complete and distributed monitoring architecture. The installation composed by one
server and many proxies is the one shown in the previous diagram.
The distributed scenarios are the most complex installation that we can implement
with Zabbix. This scenario is composed of one Zabbix server per branch (or
datacenter). This installation is the most complex to set up and maintain and
unfortunately is not centralized but can be combined with proxies.
All the possible Zabbix architecture will be discussed in detail in Chapter 2,
Distributed Monitoring.
[ 13 ]
Deploying Zabbix
Zabbix installation
The installation that will be covered in this chapter is the one composed by a server
for each of the following base components:
• A web frontend
• A Zabbix server
• A Zabbix database
We will start describing this installation because:
• It is a basic installation that is ready to be expanded with proxies and nodes
• Each component is on a dedicated server
• This kind of configuration is the starting point for monitoring
large environments
• It is widely used
• Most probably it will be the starting point of your upgrade and expansion
of monitoring infrastructure.
Actually, this first setup for a large environment, as explained here, can be useful if
you have the task to improve an existing monitoring infrastructure. If your current
monitoring solution is not implemented in this way, the first thing is to plan the
migration on three different dedicated servers.
Once the environment is set up on three tiers, but is still giving poor performance,
you can plan and think which kind of large environment setup will be a perfect fit
for your infrastructure.
When you monitor your large environment, there are some points to consider:
• Use a dedicated server to keep the things easy-to-extend
• Easy-to-extend and implement a high availability setup
• Easy-to-extend an implement a fault tolerant architecture
On this three-layer installation, the CPU usage of server component will not be really
critical, at least for the Zabbix server. The CPU consumption is directly related to the
number of items to store and the refresh rate (number of samples per minute) rather
than the memory.
Indeed the Zabbix server will not consume excessive CPU but is a bit more greedy of
memory. We can consider that a quad core server and 4 GB of RAM can be used for
more than 1000 hosts without any issues.
[ 14 ]
Chapter 1
Basically, there are two ways to install Zabbix:
• Downloading the latest source code and compiling it
• Installing it from packages
There is also another way to have a Zabbix server up and running, that is by
downloading the virtual appliance but we don't consider this case as it is better
to have full control of our installation and be aware of all the steps.
The installation from packages gives us the following benefits:
• It makes the process of upgrading and updating easier
• Dependencies are automatically sorted
The source code compilation also gives us some benefit:
• We can compile only the required features
• We can statically build the agent and deploy it on different Linux "flavors"
• We can have complete control on the update
It is quite usual to have different versions of Linux, Unix, and Microsoft Windows on
a large environment. These kind of scenarios are quite diffused on a heterogeneous
infrastructure; and if we use the agent distribution package of Zabbix on each Linux
server, we will for sure have different versions of the agent and different locations
for the configuration files.
The more standardized we are across the server, the easier it will be to maintain and
upgrade the infrastructure. --enable-static give us a way to standardize the agent
across different Linux versions and releases and this is a strong benefit. The agent
if statically compiled can be easily deployed everywhere and for sure we will have
the same location (and we can use the same configuration file apart from the node
name) for the agent and his configuration file. The deployment will be standardized;
however, the only thing that may vary is the start/stop script and how to register it
on the right init runlevel.
The same kind of concept can be applied to commercial Unix bearing in mind its
compilation by vendors, so the same agent can be deployed on different versions
Unix release by the same vendor.
[ 15 ]
Deploying Zabbix
Prerequisites
Before compiling Zabbix, we need to take a look at the prerequisites. The web
frontend will need at least the following version:
• Apache (1.3.12 or later)
• PHP (5.1.6 or later )
Instead, the Zabbix server will need:
• An RDBMS – The open source alternatives are PostgreSQL and MySQL
• zlib-devel
• mysql-devel – It is used for supporting MySQL (not needed on our setup)
• postgresql-devel – It is used for supporting PostgreSQL
• glibc-devel
• curl-devel – It is used on web monitoring
• libidn-devel – The curl-devel depends on it
• openssl-devel – The curl-devel depends on it
• net-snmp-devel – It is used on SNMP support
• popt-devel – The net-snmp-devel might depend on it
• rpm-devel – The net-snmp-devel might depend on it
• OpenIPMI-devel – It is used for supporting IPMI
• iksemel-devel – it is used for jabber protocol
• Libssh2-devel
• sqlite3 – It is required if SQLite is used as Zabbix backend database
(usually on proxies)
On a Red Hat Enterprise Linux distribution, to install all the dependencies, we can
use yum (from root) and install the following package:
yum install zlib-devel postgresql-devel glibc-devel curl-devel gcc
automake postgresql libidn-devel openssl-devel net-snmp-devel rpmdevel OpenIPMI-devel iksemel-devel libssh2-devel openldap-devel
iksemel-devel is used to send a Jabber message. This is a really useful
feature as it enables Zabbix to send chat messages. Furthermore, Jabber is
managed as a media type on Zabbix and you can also set your working
time, which is a really useful feature to avoid the sending of messages
when you are not in office.
[ 16 ]
Chapter 1
Setting up the server
Zabbix needs a user and an unprivileged account to run. Anyway, if the daemon
is started from root, it will automatically switch to the Zabbix account if this one
is present.
# groupadd zabbix
# useradd –m –s /bin/bash -g zabbix zabbix
# useradd –m –s /bin/bash -g zabbix zabbixsvr
The server should never run as root because it will
expose the server to a security risk.
The preceding lines permit you to enforce the security of your installation. The server
and agent should run with two different accounts otherwise the agent can access
Zabbix server's configuration. Now, using the Zabbix user account we can extract
the tar.gz file:
# tar -zxvf zabbix-2.0.6.tar.gz
To know all the possible configuration options you can check use the --help
parameter as follows
# ./configure -–help
To configure the source for our server, we can use the following options:
# ./configure --enable-server --with-postgresql --with-libcurl
--with-jabber --with-net-snmp --enable-ipv6 --with-openipmi --withssh2 –-with-ldap –-enable-agent
The zabbix_get and zabbix_send command lines are generated
only if --enable-agent is specified during server compilation.
If the configuration is complete without errors, we should see something like this:
Enable server:
Server details:
With database:
WEB Monitoring via:
Native Jabber:
SNMP:
IPMI:
SSH:
ODBC:
yes
PostgreSQL
cURL
yes
net-snmp
openipmi
yes
no
[ 17 ]
Deploying Zabbix
Linker flags:
-rdynamic
-L/usr/lib64
-L/usr/
lib64 -L/usr/lib64 -L/usr/lib -L/usr/lib -L/usr/lib
Libraries:
-lm -lrt -lresolv
-lpq -liksemel
-lcurl -lnetsnmp -lcrypto -lnetsnmp -lcrypto -lssh2 -lOpenIPMI
-lOpenIPMIposix -lldap -llber
Enable proxy:
no
Enable agent:
Agent details:
Linker flags:
Libraries:
yes
Enable Java gateway:
no
-rdynamic
-L/usr/lib
-lm -lrt -lresolv
-lcurl -lldap -llber
LDAP support:
yes
IPv6 support:
yes
***********************************************************
*
Now run 'make install'
*
*
*
*
Thank you for using Zabbix!
*
*
<http://www.zabbix.com>
*
***********************************************************
We will not run make install but only the compilation with # make. To specify a
different location for the Zabbix server, we need to use a --prefix on the configure
options, for example, --prefix=/opt/zabbix. Now, follow the instructions as
explained in the Installing and creating the package section.
Setting up the agent
To configure the sources to create the agent, we need to run the following command:
# ./configure --enable-agent
# make
With the make command followed by the --enable-static option,
you can statically link the libraries and the compiled binary will not
require any external library; this is very useful to distribute the agent
across a different dialect of Linux.
[ 18 ]
Chapter 1
Installing and creating the package
In both the previous sections, the command line ends right before the
installation, indeed we didn't run the following command:
# make install
I advise you to not run the make install command but use the
checkinstall software instead. This software will create the package
and install the Zabbix software.
You can download the software from http://asic-linux.com.mx/~izto/
checkinstall/.
Note that checkinstall is only one of the possible alternatives you have to
create a distributable system package.
We can also use a prebuild checkinstall. The current release is
checkinstall-1.6.2-20.2.i686.rpm (on Red Hat/CentOS);
the package will also need rpm-build:
yum install rpm-build
And we need to create the necessary directories:
mkdir -p ~/rpmbuild/{BUILD,RPMS,SOURCES,SPECS,SRPMS}
The package made things easy; it is easy to distribute and upgrade the software
plus we can create a package for different versions of a package manager: RPM,
deb, and tgz.
checkinstall can produce a package for Debian, Slackware,
and RedHat. This is particularly useful to produce the Zabbix's
agent package (statically linked) and to distribute it around
our server.
Now we need to convert to root or use the sudo checkinstall command followed
by its options:
# checkinstall --nodoc --install=yes -y
[ 19 ]
Deploying Zabbix
If you don't face any issue, you should get the following message:
******************************************************************
Done. The new package has been saved to
/root/rpmbuild/RPMS/i386/zabbix-2.0.6-1.i386.rpm
You can install it in your system anytime using:
rpm -i zabbix-2.0.6-1.i386.rpm
******************************************************************
Now Zabbix is installed. The server binaries will be installed into
<prefix>/sbin, utilities will be on <prefix>/bin, and the man pages
under the <prefix>/share location.
Configuring the server
For the server configuration, we only have one file to check and edit:
/usr/local/etc/zabbix_server.conf
The configuration files are inside the following directory:
/usr/local/etc
We need to change the /usr/local/etc/zabbix_server.conf file and
write the username, relative password, and the database name there.
This location depends on the sysconfdir compile-time installation
variable. Don't forget to take appropriate measures to protect the
access to the configuration file with the following command:
chmod 400/usr/local/etc/zabbix_server.conf
The location of the default external scripts will be:
/usr/local/share/zabbix/externalscripts
It depends on the datadir compile-time installation variable. The alertscripts
directory will be at the following location:
/usr/local/share/zabbix/alertscripts
This can be changed during compilation and it depends
on the datadir installation variable.
[ 20 ]
Chapter 1
Now we need to configure the agent. The configuration file is where we need to
write the IP address of our Zabbix server. Once done, it is important to add two
new services to the right runlevel to be sure that they will start when the server
enters on the right runlevel.
To complete this task we need to install the start/stop scripts on the following:
• /etc/init.d/zabbix-agent
• /etc/init.d/zabbix-proxy
• /etc/init.d/zabbix-server
There are several scripts prebuilt inside the misc folder located at the
following location:
zabbix-2.0.6/misc/init.d
This folder contains different startup scripts for different Linux variants, but this
tree is not actively maintained and tested, and may not be up-to-date with the most
recent versions of Linux distributions so it is better to take care and test it before
going live.
Once the start/stop script is added inside the /etc/init.d folder, we need to add it
to the service list:
# chkconfig --add zabbix-server
# chkconfig --add zabbix-agentd
Now all that is left is to tell the system on which runlevel it should start them, we are
going to use runlevel 3 and 5.
# chkconfig --level 35 zabbix-server on
# chkconfig --level 35 zabbix-agentd on
Currently, we can't start the server; before starting up our server we need to
configure the database.
Installing the database
Once we complete the previous step, we can walk through the database server
installation. All those steps will be done on the dedicated database server. The first
thing to do is install the PostgreSQL server. This can be easily done with the package
offered from the distribution but is recommended to use the latest 9.x stable version.
RedHat is still distributing the 8.x on RHEL6.4. Also, its clones such as CentOS and
ScientificLinux are doing the same. PosgreSQL 9.x has many useful features, at the
moment, the latest stable ready for the production environment is Version 9.2.
[ 21 ]
Deploying Zabbix
To install the PostgreSQL 9.2, there are some easy steps to follow:
1. Locate the .repo files:
°°
Red Hat: It is present at /etc/yum/pluginconf.d/rhnplugin.conf
[main]
°°
CentOS: It is present at /etc/yum.repos.d/CentOS-Base.repo,
[base] and [updates]
2. Append the following line on the section(s) identified in the preceding step:
exclude=postgresql*
3. Browse to http://yum.postgresql.org and find your correct RPM. For
example, to install PostgreSQL 9.2 on the CentOS 6 32 bit, go to http://
yum.postgresql.org/9.2/redhat/rhel-6-i386/pgdg-redhat92-9.2-7.
noarch.rpm.
4. Now to list the entire postgresql package, use the following command:
# yum list postgres*
5. Once you find our package in the list, install it using the following command:
# yum install postgresql92 postgresql92-server postgresql92contrib
6. Once the packages are installed, we need to initialize the database:
# service postgresql-9.2 initdb
Or:
# /etc/init.d/postgresql-9.2 initdb
7. Now we need to change a few things in the configuration file /var/lib/
pgsql/9.2/data/pg_hba.conf. We need to change the listen address, port,
and also need to add a couple of lines for our zabbix_db database
listen_addresses = '*'
port = 5432
# configuration for Zabbix
local
zabbix_db
zabbix
host
zabbix_db
zabbix
<CIDR-address>
md5
md5
The local keyword matches all the connections made in the Unix-domain
sockets. This line is followed by the database name (zabbix_db), the username
(zabbix), and the authentication method (in our case md5).
[ 22 ]
Chapter 1
The host keyword matches all the connections that are coming from TCP/
IP (this includes the SSL and non-SSL connections) followed by the database
name (zabbix_db), the username (zabbix), the network and mask of all the
hosts that are allowed, and the authentication method (in our case md5).
8. The network mask of the allowed hosts in our case should be a network
mask because we need to allow the web interface (hosted on our web server)
and the Zabbix server that is on a different dedicated server for example,
10.6.0.0/24 (a small subnet) or a large network as well. Most likely, the web
interface as well the Zabbix server will be in a different network, so make
sure you express all the networks and relative mask here.
9. Finally, we can start our PosgreSQL server using the following command:
# service postgresql-9.2
start
Or:
# /etc/init.d/postgresql-9.2
start
To create a database, we need to be a Postgres user (or the user that in your
distribution is running PostgreSQL). Create a user for the database (our Zabbix
user) and log in as that user to import the schema with the relative data.
The code to import the schema is as follows:
# su - postgres
-bash-4.1$
Once we become a postgres user, we can create the database (in our example, it is
zabbix_db):
-bash-4.1$ psql
postgres=# CREATE USER zabbix WITH PASSWORD '<YOUR-ZABBIX-PASSWORDHERE>';
CREATE ROLE
postgres=# CREATE DATABASE zabbix_db WITH OWNER zabbix
ENCODING='UTF8';
CREATE DATABASE
postgres=# \q
The database creation scripts are located in the ./database/postgresql folder They
should be installed exactly in this order:
# cat schema.sql | psql -U zabbix zabbix_db
# cat images.sql | psql -U zabbix zabbix_db
# cat data.sql | psql -U zabbix zabbix_db
[ 23 ]
Deploying Zabbix
Starting up with Daemon
Now, finally, it is the time to start our Zabbix server and test the whole setup for our
Zabbix server/database.
# /etc/init.d/zabbix-server start
Starting Zabbix server:
[
OK
]
A quick check on the logfile can give us more information about what is currently
happening in our server. We should be able to get the following lines from the logfile
(the default location is /tmp/zabbix_server.log):
28742:20130609:133955.418
(revision 35158).
28742:20130609:133955.418
28742:20130609:133955.418
28742:20130609:133955.418
28742:20130609:133955.418
28742:20130609:133955.418
28742:20130609:133955.418
28742:20130609:133955.418
28742:20130609:133955.418
28742:20130609:133955.418
28742:20130609:133955.418
Starting Zabbix Server. Zabbix 2.0.6
****** Enabled features ******
SNMP monitoring:
YES
IPMI monitoring:
YES
WEB monitoring:
YES
Jabber notifications:
YES
Ez Texting notifications: YES
ODBC:
NO
SSH2 support:
YES
IPv6 support:
YES
******************************
The preceding lines tell us that our server is up and running. If the logfiles continue
with the following lines, it means that is all fine: the configuration, the connection to
database, and so on:
28745:20130609:133955.469 server #2 started [db watchdog #1]
28744:20130609:133955.471 server #1 started [configuration syncer #1]
28754:20130609:133955.494 server #9 started [trapper #1]
28755:20130609:133955.496 server #10 started [trapper #2]
28756:20130609:133955.498 server #11 started [trapper #3]
28757:20130609:133955.499 server #12 started [trapper #4]
28758:20130609:133955.501 server #13 started [trapper #5]
28763:20130609:133955.519 server #15 started [alerter #1]
28762:20130609:133955.526 server #14 started [icmp pinger #1]
28765:20130609:133955.566 server #17 started [timer #1]
28764:20130609:133955.574 server #16 started [housekeeper #1]
28764:20130609:133955.575 executing housekeeper
[ 24 ]
Chapter 1
Another thing to check is whether our server is running with our user:
# ps -ef |grep zabbix_server
502 28742 1 0 13:39 ?
00:00:00 /usr/local/sbin/zabbix_server
502 28744 28742 0 13:39 ? 00:00:00 /usr/local/sbin/zabbix_server
502 28745 28742 0 13:39 ? 00:00:00 /usr/local/sbin/zabbix_server
...
The preceding lines show that zabbix_server is running with the user 502. We will
go ahead and verify if 502 is the user we previously created:
# getent passwd 502
zabbixsvr:x:502:501::/home/zabbixsvr:/bin/bash
The preceding lines show that is all fine. The most common issue normally is the
following error:
28487:20130609:133341.529 Database is down. Reconnecting in 10
seconds.
We can have different actors that are causing this issue:
• Firewall (local on our servers or infrastructure firewall)
• The Postgres configuration
• Wrong data on zabbix_server.conf
We can try to isolate the problem by running the following command on
the database server:
serverpsql -h 127.0.0.1 -U zabbix zabbix_dbPassword for
user zabbix:psql (9.2.4)Type "help" for help
If we have a connection, we can try the same command from the Zabbix
server; if it fails, it is better to check the firewall configuration. If we get
the fatal identification-authentication failed error, it is better to check the
pg_hba.conf file.
Now it is time to check how to start and stop your Zabbix installation. The scripts
that follow are a bit customized to manage the different users for the server and
the agent.
[ 25 ]
Deploying Zabbix
The following startup script works fine with the standard compilation
without using a --prefix and the zabbixsvr user. If you are running
on a different setup, make sure you customize the executable location
and the user:
exec=/usr/local/sbin/zabbix_server
zabbixsrv=zabbixsvr
For zabbix-server, create the zabbix-server file at /etc/init.d with the
following content:
#!/bin/sh
#
# chkconfig: - 85 15
# description: Zabbix server daemon
# config: /usr/local/etc/zabbix_server.conf
#
### BEGIN INIT INFO
# Provides: zabbix
# Required-Start: $local_fs $network
# Required-Stop: $local_fs $network
# Default-Start:
# Default-Stop: 0 1 2 3 4 5 6
# Short-Description: Start and stop Zabbix server
# Description: Zabbix server
### END INIT INFO
# Source function library.
. /etc/rc.d/init.d/functions
exec=/usr/local/sbin/zabbix_server
prog=${exec##*/}
lockfile=/var/lock/subsys/zabbix
syscf=zabbix-server
The next parameter zabbixsvr is specified inside the start() function and it
determines which user will be used to run our Zabbix server:
zabbixsrv=zabbixsvr
[ -e /etc/sysconfig/$syscf ] && . /etc/sysconfig/$syscf
start()
{
echo -n $"Starting Zabbix server: "
[ 26 ]
Chapter 1
In the preceding code, the user (who will own our Zabbix's server process) is
specified inside the start function.
daemon --user $zabbixsrv $exec
Remember to change the ownership of the server logfile and configuration file of
Zabbix, this to prevent a normal user to access sensitive data that can be acquired
with Zabbix. Logfile is specified as follows:
/usr/local/etc/zabbix_server.conf
On 'LogFile''LogFile' properties
rv=$?
echo
[ $rv -eq 0 ] && touch $lockfile
return $rv
}
stop()
{
echo -n $"Shutting down Zabbix server: "
Here, inside the stop function, we don't need to specify the user as the start/stop
script runs from root so we can simply use killproc $prog as follows:
killproc $prog
rv=$?
echo
[ $rv -eq 0 ] && rm -f $lockfile
return $rv
}
restart()
{
stop
start
}
case "$1" in
start|stop|restart)
$1
;;
force-reload)
restart
;;
status)
status $prog
;;
try-restart|condrestart)
if status $prog >/dev/null ; then
restart
fi
[ 27 ]
Deploying Zabbix
;;
reload)
action $"Service ${0##*/} does not support the reload
action: " /bin/false
exit 3
;;
*)
echo $"Usage: $0 {start|stop|status|restart|try-restart|forcereload}"
exit 2
;;
esac
The following startup script works fine with the standard
compilation without using a --prefix and the zabbix_usr user.
If you are running on a different setup, make sure you customize
the executable location and the user:
exec=/usr/local/sbin/zabbix_agentd
zabbix_usr=zabbix
For zabbix_agent, create the following zabbix-agent file at /etc/init.d/
zabbix-agent:
#!/bin/sh
#
# chkconfig: - 86 14
# description: Zabbix agent daemon
# processname: zabbix_agentd
# config: /usr/local/etc/zabbix_agentd.conf
#
### BEGIN INIT INFO
# Provides: zabbix-agent
# Required-Start: $local_fs $network
# Required-Stop: $local_fs $network
# Should-Start: zabbix zabbix-proxy
# Should-Stop: zabbix zabbix-proxy
# Default-Start:
# Default-Stop: 0 1 2 3 4 5 6
# Short-Description: Start and stop Zabbix agent
# Description: Zabbix agent
### END INIT INFO
# Source function library.
. /etc/rc.d/init.d/functions
exec=/usr/local/sbin/zabbix_agentd
[ 28 ]
Chapter 1
prog=${exec##*/}
syscf=zabbix-agent
lockfile=/var/lock/subsys/zabbix-agent
The following zabbix_usr parameter specifies the account that will be used to run
Zabbix's agent:
zabbix_usr=zabbix
[ -e /etc/sysconfig/$syscf ] && . /etc/sysconfig/$syscf
start()
{
echo -n $"Starting Zabbix agent: "
The next command uses the value of the zabbix_usr variable and permits us to have
two different users, one for server and one for agent, preventing the Zabbix agent
from accessing the zabbix_server.conf file that contains our database password.
daemon --user $zabbix_usr $exec
rv=$?
echo
[ $rv -eq 0 ] && touch $lockfile
return $rv
}
stop()
{
echo -n $"Shutting down Zabbix agent: "
killproc $prog
rv=$?
echo
[ $rv -eq 0 ] && rm -f $lockfile
return $rv
}
restart()
{
stop
start
}
case "$1" in
start|stop|restart)
$1
;;
force-reload)
restart
;;
status)
[ 29 ]
Deploying Zabbix
status $prog
;;
try-restart|condrestart)
if status $prog >/dev/null ; then
restart
fi
;;
reload)
action $"Service ${0##*/} does not support the reload action:
" /bin/false
exit 3
;;
*)
echo $"Usage: $0 {start|stop|status|restart|try-restart|forcereload}"
exit 2
;;
esac
With that setup, we have the agent that is running with zabbix_usr and the server
with Unix accounts of zabbixsvr.
zabbix_usr_ 4653 1 0 15:42 ?
00:00:00 /usr/local/sbin/zabbix_
agentd
zabbix_usr 4655 4653 0 15:42 ?
00:00:00 /usr/local/sbin/zabbix_
agentd
zabbixsvr 4443 1 0 15:32 ?
00:00:00 /usr/local/sbin/zabbix_server
zabbixsvr 4445 4443 0 15:32 ? 00:00:00 /usr/local/sbin/zabbix_server
Some considerations about the database
Zabbix uses an interesting way to keep the database the same size at all times.
The database size indeed depends upon:
• Number of processed values per second
• Housekeeper settings
Zabbix uses two ways to store the collected data:
• History
• Trends
While on history, we will find all the collected data (it doesn't matter what type of
data will be stored on history), trends will collect only numerical data. Its minimum,
maximum, and average calculations are consolidated by hour (to keep the trend a
lightweight process).
[ 30 ]
Chapter 1
All the "strings" items such as character, log, and text do
not correspond to trends since trends store only values.
There is a process called the housekeeper that is responsible to handle the retention
against our database. It is strongly advised to keep the data in history as small as
possible so that we do not overload the database with a huge amount of data and
store the trends as long as we want.
Now since Zabbix will also be used for capacity planning purposes, we need
to consider having a baseline and keeping at least a whole business period.
Normally, the minimum period is one year but it is strongly advised to keep
the trend history on for at least two years. These historical trends will be used
during the business opening and closure to have a baseline and quantify the
overhead of a specified period.
If we indicate 0 as the value for trends, the server will not calculate
or store trends at all. If history is set to "0", Zabbix will be able to
calculate only triggers based on the last value of the item itself as it
does not store historical values at all.
The most common issue that we face when aggregating data is to have values
influenced by some positive spikes or fast drops in our hourly based trends,
this means that huge spikes can produce a mean value per hour that is not right.
Trends in Zabbix are implemented in a smart way. The script creation for the trend
table is as follows:
CREATE TABLE trends(
itemid bigin NOT NULL, clock integer DEFAULT '0'
NOT NULL, num integer DEFAULT '0'
NOT NULL, value_min numeric(16, 4) DEFAULT '0.0000'
NOT NULL, value_avg numeric(16, 4) DEFAULT '0.0000'
NOT NULL, value_max numeric(16, 4) DEFAULT '0.0000'
NOT NULL, PRIMARY KEY(itemid, clock));
CREATE TABLE trends_uint(
Itemid bigint NOT NULL, Clock integer DEFAULT '0'
NOT NULL, Num integer DEFAULT '0'
NOT NULL, value_min numeric(20) DEFAULT '0'
NOT NULL, value_avg numeric(20) DEFAULT '0'
NOT NULL, value_max numeric(20) DEFAULT '0'
NOT NULL, PRIMARY KEY(itemid, clock));
[ 31 ]
Deploying Zabbix
As you can see, there are two tables showing trends inside the Zabbix database:
• Trends
• Trends_uint
The first table, Trends, is used to store the float value. The second table, trends_uint,
is used to store the unsigned integer. Both tables own the concept of keeping for each
hour the:
• Minimum value (value_min)
• Maximum value (value_max)
• Average value (value_avg)
This feature permits us to find out and display the trends graphically using the
influence of spikes and fast drop against the average value and understanding
how and how much this value has been influenced. The other tables used for the
historical purpose are as follows:
• history: It is used to store numeric data (float)
• history_log: It is used to store logs (for example, the text field on
PosgreSQL variable has unlimited length)
• history_str: It is used to store strings (up to 255 characters)
• history_text: It is used to store the text value (here again is a text field,
so it has unlimited length)
• history_uint: It is used to store numeric values (unsigned integers)
Sizing of the database
Calculating the definitive database size is not an easy task because it is hard to
predict how many items and the relative rate per second we will have on our
infrastructure and how many events will be generated. To simplify this, we will
consider the worst scenario where we have an event generated per second.
Summarizing the database size is influenced by:
• Items: The number of items in particular
• Refresh rate: The average refresh rate of our items
• Space to store values: This value depends on RDBMS
[ 32 ]
Chapter 1
The space used to store the data may vary from database to database, but we can
simplify our work by considering some mean values to have a value that quantifies
the maximum space consumed by the database. We can also consider the space used
to store values on history around 50 bytes per value, the space used from a value on
the trend table is normally around 128 bytes and the space used for a single event is
normally around 130 bytes.
The total amount of used space can be calculated with the following formula:
Configuration + History + Trends + Events
Now, let us look into each of the components:
• Configuration: This refers to the Zabbix's configuration for the server,
the web interface, and all the configuration parameters that are stored
into the database; this occupation is normally around 10 MB.
• History: The history component is calculated using the following formula:
History retention days* (items/refresh rate)*24*3600* 50 bytes
(History bytes usage average)
• Trends: The trends component is calculated using the following formula:
days*(items/3600)*24*3600*128 bytes (Trend bytes usage average)
• Events: The event component is calculated using the following formula:
days*events*24*3600*130 bytes (Event bytes usage average)
Now coming back to our practical example, we can consider having 5000 items
refreshed every minute and we would like to have retention of 7 days the used
space will be calculated as follows:
History: retention (in days) * (items/refresh rate)*24*3600* 50
bytes
50 bytes is the mean value for the space consumed by a value stored on history.
Considering 30 days the result is the following:
• History will be calculated as:
30 * 5000/60 * 24*3600 *50 = 10.8GB
• As we said earlier, to simplify, we will consider the worst scenario (one event
per second) and will also consider keeping 5 years of events
[ 33 ]
Deploying Zabbix
• Events will be calculated using the following formula:
retention days*events*24*3600* Event bytes usage (average)
When we calculate an event we have:
5*365*24*3600* 130 = 15.7GB
130 bytes is the mean value for the space
consumed for a value stored on events.
• Trends will be calculated using the following formula:
retention in days*(items/3600)*24*3600*Trend bytes usage (average)
When we calculate trends we have:
5000*24*365* 128 = 5.3GB per year or 26.7GB for 5 years.
128 bytes is the mean value for the space consumed
for a value stored on trends.
The following table shows the retention in days and the space required for
the measure:
Type of measure
Retention in days
Space required
History
30
10.8 GB
Events
1825 (5 years)
15.7 GB
Trends
1825 (5 years)
26.7 GB
Total
N.A.
53.2 GB
The calculated size is not the initial size of our database but we need to keep in mind
that this one will be our target size after 5 years. We are also considering a history
of 30 days, keep in mind that this retention can be reduced in case of issues since the
trends will keep and store our baseline and hourly based trends.
The history and trends retention policy can be changed easily per item. This means
we can create a template with items that have a different history retention by default.
Normally, the history is set to 7 days, but for some kind of measure such as in a web
scenario, or other particular measure, we may need to keep all the values for more
than a week. This permits us to change the value of each item.
[ 34 ]
Chapter 1
In our example we performed a worst-case scenario with retention of 30 days,
but it is good advice to keep the history only for 7 days or maybe less on large
environments. If we perform a basic calculation of an item that is updated every
60 seconds and has history preserved for 7 days, it will generate:
(update interval) * (hours in a day) * (number of days in history) =60*24*7=10080
This means that for each item we will have 10,080 lines in a week and that gives us
an idea of the number of lines that we will produce on our database.
Some considerations about housekeeping
Housekeeping can be quite a heavy process. As the database grows, housekeeping
will require more and more time to complete its work. This issue can be sorted using
the delete_history() function.
There is a way to deeply improve the housekeeping performance and fix this
performance drop. The heaviest tables are: history, history_uint, trends,
and trends_uint.
[ 35 ]
Deploying Zabbix
A solution can be the PosgreSQL table partitioning and partition of the entire tables
on a monthly basis.
The preceding figure displays the standard and nonpartitioned history table on the
database. The following figure shows how a partitioned history table will be stored
in the database.
Partitioning is basically the splitting of a logical large table into smaller physical
pieces. This feature can provide several benefits:
• Performance of queries can be improved dramatically in situations when
there is a heavy access to the table's rows in a single partition
• The partitioning will reduce the index size making it more likely to fit in
memory of the heavy parts that are being used
• Massive deletes can be accomplished by removing partitions, instantly
reducing the space allocated for the database without introducing
fragmentation and heavy load on index rebuilding. The delete partition
command also entirely avoids the vacuum overhead caused by a bulk delete
• When a query updates or requires the access to a large percentage of the
partition, using the sequential scan is often more efficient than the index
usage with a random access or scattered reads against that index
[ 36 ]
Chapter 1
All these benefits are worthwhile only when a table is very large. The strong point of
this kind of architecture is that the RDBMS will directly access the needed partition,
and the delete will simply be a delete of a partition. Partition deletion is a fast
process and requires little resource.
Unfortunately, Zabbix is not able to manage the partitions so we need to disable the
housekeeping and use an external process to accomplish the housekeeping.
To set up this feature, we need to create a schema where we can place all the
partitioned tables:
CREATE SCHEMA partitions AUTHORIZATION zabbix;
Now we need a function that will create the partition. So to connect to Zabbix you
need to run the following code:
CREATE OR REPLACE FUNCTION trg_partition()
RETURNS TRIGGER AS
$BODY$
DECLARE
prefix text:= 'partitions.';
timeformat text;
selector text;
_interval INTERVAL;
tablename text;
startdate text;
enddate text;
create_table_part text;
create_index_part text;
BEGIN
selector = TG_ARGV[0];
IF selector = 'day'
THEN
timeformat:= 'YYYY_MM_DD';
ELSIF selector = 'month'
THEN
timeformat:= 'YYYY_MM';
END IF;
_interval:= '1 ' || selector;
tablename := TG_TABLE_NAME || '_p' || TO_CHAR(TO_TIMESTAMP(NEW.clock),
timeformat);
EXECUTE 'INSERT INTO ' || prefix || quote_ident(tablename) || ' SELECT
($1).*'
USING NEW;
RETURN NULL;
EXCEPTION
[ 37 ]
Deploying Zabbix
WHEN undefined_table THEN
startdate := EXTRACT(epoch FROM date_trunc(selector, TO_TIMESTAMP(NEW.
clock)));
enddate := EXTRACT(epoch FROM date_trunc(selector, TO_TIMESTAMP(NEW.
clock) + _interval));
create_table_part := 'CREATE TABLE IF NOT EXISTS ' || prefix || quote_
ident(tablename) || ' (CHECK ((clock >= ' || quote_literal(startdate)
|| ' AND clock < ' || quote_literal(enddate) || '))) INHERITS (' ||
TG_TABLE_NAME || ')';
create_index_part := 'CREATE INDEX ' || quote_ident(tablename) || '_1
on ' || prefix || quote_ident(tablename) || '(itemid,clock)';
EXECUTE create_table_part;
EXECUTE create_index_part;
--insert it again
EXECUTE 'INSERT INTO ' || prefix || quote_ident(tablename) || ' SELECT
($1).*'
USING NEW;
RETURN NULL;
END;
$BODY$
LANGUAGE plpgsql VOLATILE
COST 100;
ALTER FUNCTION trg_partition()
OWNER TO zabbix;
Now we need a trigger connected to each table that we want to separate.
This trigger will run an INSERT statement and if the partition is not ready
or yet created, the function will create the partition right before the
INSERT statement.
CREATE TRIGGER partition_trg BEFORE INSERT ON history
FOR EACH ROW EXECUTE PROCEDURE trg_partition('day');
CREATE TRIGGER partition_trg BEFORE INSERT ON history_sync
FOR EACH ROW EXECUTE PROCEDURE trg_partition('day');
CREATE TRIGGER partition_trg BEFORE INSERT ON history_uint
FOR EACH ROW EXECUTE PROCEDURE trg_partition('day');
CREATE TRIGGER partition_trg BEFORE INSERT ON history_str_sync
FOR EACH ROW EXECUTE PROCEDURE trg_partition('day');
CREATE TRIGGER partition_trg BEFORE INSERT ON history_log
FOR EACH ROW EXECUTE PROCEDURE trg_partition('day');
CREATE TRIGGER partition_trg BEFORE INSERT ON trends
FOR EACH ROW EXECUTE PROCEDURE trg_partition('month');
CREATE TRIGGER partition_trg BEFORE INSERT ON trends_uint
FOR EACH ROW EXECUTE PROCEDURE trg_partition('month');
[ 38 ]
Chapter 1
At this point, we miss only the housekeeping function that will replace the one built
in Zabbix and disable Zabbix's native one.
The function that will handle housekeeping for us is as follows:
CREATE OR REPLACE FUNCTION delete_partitions(intervaltodelete
INTERVAL, tabletype text)
RETURNS text AS
$BODY$
DECLARE
result RECORD ;
prefix text := 'partitions.';
table_timestamp TIMESTAMP;
delete_before_date DATE;
tablename text;
BEGIN
FOR result IN SELECT * FROM pg_tables WHERE schemaname =
'partitions' LOOP
table_timestamp := TO_TIMESTAMP(substring(result.tablename
FROM '[0-9_]*$'), 'YYYY_MM_DD');
delete_before_date := date_trunc('day', NOW() intervalToDelete);
tablename := result.tablename;
IF tabletype != 'month' AND tabletype != 'day' THEN
RAISE EXCEPTION 'Please specify "month" or "day" instead of %',
tabletype;
END IF;
--Check whether the table name has a day (YYYY_MM_DD) or month
(YYYY_MM) format
IF LENGTH(substring(result.tablename FROM '[0-9_]*$')) = 10
AND tabletype = 'month' THEN
--This is a daily partition YYYY_MM_DD
-- RAISE NOTICE 'Skipping table % when trying
to delete "%" partitions (%)', result.tablename, tabletype,
length(substring(result.tablename from '[0-9_]*$'));
CONTINUE;
ELSIF LENGTH(substring(result.tablename FROM '[0-9_]*$')) = 7
AND tabletype = 'day' THEN
--this is a monthly partition
--RAISE NOTICE 'Skipping table % when trying to delete "%"
partitions (%)', result.tablename, tabletype, length(substring(result.
tablename from '[0-9_]*$'));
CONTINUE;
ELSE
--This is the correct table type. Go ahead and check if it
needs to be deleted
--RAISE NOTICE 'Checking table %', result.tablename;
END IF;
[ 39 ]
Deploying Zabbix
IF table_timestamp <= delete_before_date THEN
RAISE NOTICE 'Deleting table %', quote_ident(tablename);
EXECUTE 'DROP TABLE ' || prefix || quote_ident(tablename) ||
';';
END IF;
END LOOP;
RETURN 'OK';
END;
$BODY$
LANGUAGE plpgsql VOLATILE
COST 100;
ALTER FUNCTION delete_partitions(INTERVAL, text)
OWNER TO zabbix;
Now you have the housekeeping ready to run. To enable housekeeping, we can use
the crontab by adding the following entries:
@daily psql -q -U zabbix -c "SELECT delete_partitions('7 days',
'day')"
@daily psql -q -U zabbix -c "SELECT delete_partitions('24 months',
'month')"
Those two tasks should be scheduled on the database server's crontab. In this
example we are keeping the history of 7 days and trends of 24 months.
Now we can finally disable the Zabbix housekeeping, using the following code:
### Option: DisableHousekeeping
# If set to 1, disables housekeeping.
#
# Mandatory: no
# Range: 0-1
DisableHousekeeping=1
Restart Zabbix, now the built-in housekeeping is disabled and you should see a lot of
improvement in the performance. To keep your database as lightweight as possible,
you can clean up the following tables:
•
•
•
•
•
acknowledges
alerts
auditlog
events
service_alarms
[ 40 ]
Chapter 1
Once you chose your own retention, you need to add a retention policy, for example,
in our case it will be 2 years of retention. With the following crontab entries, you can
delete all the records older than 63072000 (2 years expressed in seconds):
@daily psql -q -U zabbix -c "delete from acknowledges where clock <
(SELECT (EXTRACT( epoch FROM now() ) - 63072000))"
@daily psql -q -U zabbix -c "delete from alerts where clock < (SELECT
(EXTRACT( epoch FROM now() ) - 63072000))"
@daily psql -q -U zabbix -c "delete from auditlog where clock <
(SELECT (EXTRACT( epoch FROM now() ) - 62208000))"
@daily psql -q -U zabbix -c "delete from events where clock < (SELECT
(EXTRACT( epoch FROM now() ) - 62208000))"
@daily psql -q -U zabbix -c "delete from service_alarms where clock <
(SELECT (EXTRACT( epoch FROM now() ) - 62208000))"
To disable housekeeping, we need to drop the triggers created:
DROP
DROP
DROP
DROP
DROP
DROP
DROP
TRIGGER
TRIGGER
TRIGGER
TRIGGER
TRIGGER
TRIGGER
TRIGGER
partition_trg
partition_trg
partition_trg
partition_trg
partition_trg
partition_trg
partition_trg
ON
ON
ON
ON
ON
ON
ON
history;
history_sync;
history_uint;
history_str_sync;
history_log;
trends;
trends_uint;
All those changes need to be tested and changed/modified as they fit your setup.
Also be careful and back up your database.
The web interface
The web interface installation is quite easy there are some basic steps to execute.
The web interface is completely written on PHP so we need a web server that
supports PHP, in our case, we will use Apache with the PHP support enabled.
The entire web interface is contained inside the php folder at frontends/php/
that we need to copy on our htdocs folder:
/var/www/html
Use the following commands to copy the folders:
# mkdir <htdocs>/zabbix
# cd frontends/php
# cp -a . <htdocs>/zabbix
Be careful, you might need proper rights and permissions as all those files
are owned by Apache and they also depend on your httpd configuration.
[ 41 ]
Deploying Zabbix
The web wizard – frontend configuration
Now, from your web browser you need to open the following URL:
http://<server_ip_or_name>/zabbix
The first screen that you will meet is a welcome page; there is nothing to do there
other than to click on Next. When on the first page, you may get a warning on your
browser that informs you that the date/timezone is not set. This is a parameter
inside the php.ini file. All the possible timezones are described on the official
PHP website:
http://www.php.net/manual/en/timezones.php
The parameter to change is the date/timezone inside the php.ini file. If you don't
know the current PHP configuration or where it is located in your php.ini file,
and you need some detailed information about which modules are running or the
current settings then you can write a file, for example, php-info.php inside the
Zabbix directory with the following content:
<?phpphpinfo();phpinfo(INFO_MODULES);
?>
Now point your browser to:
http://your-zabbix-web-frontend/zabbix/php-info.php
You will have your full configuration printed out in a webpage. The next screenshot
is more important, it displays a prerequisites check and as you can see there are three
prerequisites that are not met:
Downloading the example code
You can download the example code files for all Packt books you have
purchased from your account at http://www.packtpub.com . If you
purchased this book elsewhere, you can visit http://www.packtpub.
com/support and register to have the files e-mailed directly to you.
[ 42 ]
Chapter 1
On standard Red-Hat/CentOS, it is easy to find the prerequisites that are not fulfilled:
PHP
PHP
PHP
PHP
PHP
PHP
PHP
PHP
PHP
PHP
PHP
option post_max_size
option max_execution_time
option max_input_time
bcmath
mbstring
gd unknown
gd PNG support
gd JPEG support
gd FreeType support
xmlwriter
xmlreader
8M
30
60
no
no
16M
300
300
2.0
no
no
no
no
no
[ 43 ]
Fail
Fail
Fail
Fail
Fail
Fail
Fail
Fail
Fail
Fail
Fail
Deploying Zabbix
Most of these parameters are contained inside the php.ini file. To fix them, I have
changed the following options inside the /etc/php.ini file:
[Date]
; Defines the default timezone used by the date functions
; http://www.php.net/manual/en/datetime.configuration.php#ini.date.
timezone
date.timezone = Europe/Rome
; Maximum size of POST data that PHP will accept.
; http://www.php.net/manual/en/ini.core.php#ini.post-max-size
post_max_size = 16M
; Maximum execution time of each script, in seconds
; http://www.php.net/manual/en/info.configuration.php#ini.maxexecution-time
max_execution_time = 300
; Maximum amount of time each script may spend parsing request data.
It's a good
; idea to limit this time on productions servers in order to eliminate
unexpectedly
; long running scripts.
; Default Value: -1 (Unlimited)
; Development Value: 60 (60 seconds)
; Production Value: 60 (60 seconds)
; http://www.php.net/manual/en/info.configuration.php#ini.max-inputtime
max_input_time = 300
; Maximum amount of time each script may spend parsing request data.
It's a good
; idea to limit this time on productions servers in order to eliminate
unexpectedly
; long running scripts.
; Default Value: -1 (Unlimited)
; Development Value: 60 (60 seconds)
; Production Value: 60 (60 seconds)
; http://www.php.net/manual/en/info.configuration.php#ini.max-inputtime
max_input_time = 300
[ 44 ]
Chapter 1
To solve the missing library, we need to install the following packages:
• php-xml
• php-bcmath
• php-mbstring
• php-gd
We will use the following command to install these packages:
# yum install php-xml php-bcmath php-mbstring php-gd
The whole list or the prerequisite list is given in the following table:
Prerequisite
Min value
Solution
PHP Version
5.1.6
PHP memory_limit
128M
In php.ini, change memory_
limit=128M.
PHP post_max_size
16M
In php.ini, change post_max_
size=16M.
PHP upload_max_
filesize
2M
In php.ini, change upload_max_
filesize=2M.
PHP max_execution_
time option
300 Seconds
In php.ini, change max_execution_
time=300.
PHP max_input_time
option
300 seconds
In php.ini, change max_input_
time=300.
PHP session.auto_
start
bcmath
Disabled
In php.ini, change the session.auto_
start=0.
Use php-bcmath extension
mbstring
Use php-mbstring extension
sockets
This extension is required for the user
script support
php-net-socket module
gd
PHP GD extension must support PNG
images (--with-png-dir), JPEG
(--with-jpeg-dir) images, and
FreeType 2 (--with-freetype-dir)
[ 45 ]
Deploying Zabbix
Prerequisite
Min value
Solution
libxml
Use php-xml or php5-dom
xmlwriter
Use php-xmlwriter
xmlreader
Use php-xmlreader
ctype
Use php-ctype
session
Use php-session
gettext
Use php-gettext
Every time you change a php.ini file or install a PHP extension the httpd needs
a restart to get the change. Once all the prerequisites are met, we can click on Next
and go ahead. On the next screen, we need to configure the database connection.
We simply need to fill the form with the username, password, IP address,
or hostname and also need to specify the kind of database server we are using.
[ 46 ]
Chapter 1
If the connection is fine (can be checked with a test connection), we can proceed to
the next step.
There is no check for connection available on this page so it is better to verify that it
is possible to reach the Zabbix server from the network. In this form, it is necessary
to fill the Host (or IP address) of our Zabbix server. Since we are installing the
infrastructure on three different servers, we need to specify all the parameters and
check that the Zabbix server port is available on the outside of the server.
Once we fill this form, we can click on Next. After this, the installation wizard
proposes that we to view Pre-Installation summary, which is a complete summary
of all the configuration parameters. If all is fine just click on Next, otherwise we can
go back and change our parameters. When we go ahead, we see the configuration
file has been generated (for example, in this installation in //usr/share/zabbix/
conf/zabbix.conf.php).
It can happen that you may get an error instead of a success notification, and most
probably, it is about the directory permission on our conf directory at /usr/share/
zabbix/conf. Remember to make the directory writable to the httpd user (normally
Apache is writable) at least for the time needed to create this file. Once this step is
completed, the frontend is ready and we can perform our first login.
[ 47 ]
Deploying Zabbix
Capacity planning with Zabbix
Quite often people mix up the difference between capacity planning and performance
tuning. Well the scope of performance tuning is to optimize the system you already
have in place for better performance. Using your current performance acquired as a
baseline, capacity planning determines what your system needs and when it is needed.
Here, we will see how to organize our monitoring infrastructure to achieve this goal
and provide us a useful baseline. Unfortunately, this chapter cannot cover all the
aspects of this argument, we should have one whole book about capacity planning,
but after this section, we will look at our Zabbix with a different vision and will be
aware of what to do with it.
The observer effect
Zabbix is a good monitoring system because it is really lightweight. Unfortunately,
every observed system will spend a bit of its resources to run the agent that acquires
and measures data and metrics against the operating system so it is normal if the agent
introduces a small (normally very small) overhead on the guest system. This is known
as the observer effect. We can only accept this burden on our server and be aware
that this will introduce a slight distortion in data collection, bearing in mind to keep it
lightweight to a feasible extent while monitoring the process and our custom checks.
What to monitor
The Zabbix agent's job is to collect data periodically from the monitored machine
and send metrics to the Zabbix server (that will be our aggregation and elaboration
server). Now, in this scenario there are some important things to consider:
• What we are going to acquire?
• How we are going to acquire these metrics (the way or method used)?
• The frequency with which this measurement is performed
Considering the first point, it is important to think what should be monitored on our
host and the kind of work that our host will do, or in other words what it will serve.
There are some basic metrics of operating systems that are nowadays more or
less standardized, and those are: the CPU workload, percentage of free memory,
memory usage details, usage of swap, CPU time for some process, and all this
family of measure, all of them are built-in on the Zabbix agent.
[ 48 ]
Chapter 1
Having a set of items with built-in measurement means that those are optimized to
produce less workload possible on the monitored host, the whole of Zabbix's agent
code is written in this way.
All the others metrics can be divided by service that our server should provide.
Here templates are really useful! (Also, it is an efficient way
to aggregate our metrics by type.)
Doing a practical example and considering monitoring an RDBMS it will be
fundamental to acquire:
• All the operating system metrics
• Different custom RDBMS metrics
Our different custom RDBMS metrics can be: the number of users connected, or the
usage of caches systems, the number of full table scans, and so on.
All those kind of metrics will be really useful and can be easily interpolated and
compared against the same time period in a graph. Graphs have some strong points:
• Useful to understand (also from the business side )
• Often nice to present and integrate on slides to enforce our speech
Coming back to our practical example, well, currently we are acquiring data from
our RDBMS and our operating system. (we can compare the workload of our
RDBMS and see how this reflects the workload against our OS.) Well and now?
Most probably, our core business is the revenue of a website, merchant site,
or some web application. We assume that we need to keep a website in a three-tier
environment under control because it is a quite common case. Our infrastructure
will be composed of the following actors:
• A web server
• An application server
• RDBMS
[ 49 ]
Deploying Zabbix
In real life, most probably this is the kind of environment Zabbix will be configured
on. We need to be aware that every piece and every component that can influence
our service should be measured and stored inside our Zabbix monitoring system.
Generally we can consider it to be quite normal to see people with a strong system
administration background to be more focused on operating system-related items as
well. We also saw people writing Java code that needs to be concentrated on some
other obscure measure such as the number of threads. The same kind of reasoning
can be done if the capacity planner talks with a database administrator, or the
specific guy of every sector.
This is a quite important point because the Zabbix implementer should have a global
vision and should remember that when buying a new hardware the interface will
most likely be a business unit.
This business unit very often doesn't know anything about the number of threads
that our system can support, but will only understand customer satisfaction,
customer-related issues, or how many concurrent users we can successfully serve.
Having said that, it is really important to be ready to talk in their language, and we
can do that only if we have some efficient items to graph.
Define a baseline
Now if we look at the whole infrastructure from a client's point of view, we can
think that if all our pages are served in a reasonable time the browsing experience
will be pleasant.
Our goal in this case is to make our clients happy and the whole infrastructure
reliable. Now we need to have two kinds of measures:
• The one felt from the user side (response time of our web pages)
• Infrastructure items related with it
We need to quantify the response time related to the user's navigation feeling,
we need to know how much an user can wait in front of a web page to get a
response, keeping in mind that the whole browsing experience needs to be
pleasant. We can measure and categorize our metric with these three levels
of response time:
• 0.2 seconds: It gives the feel of an instantaneous response. The user feels
the browser reaction was caused by him/her and not from a server with a
business logic.
[ 50 ]
Chapter 1
• 1-2 seconds: The users feel the browsing is continuous without any
interruption. The user can move freely rather than waiting for the
pages to load.
• 10 seconds: The liking for our website will drop. The user will want a
better performance and can definitely be distracted by other things.
Now we have our thresholds and we can measure the response of a web page during
normal browsing, in the meantime we can set some trigger level to warn us when the
response time is more than two seconds for a page.
Now we need to relate that with all our other measures: the number of users
connected, the number of sessions in our application server, and the number of
connections to our database and relate all our measures with the response time
and the number of users connected. Now we need to measure how our system
is serving pages to users during normal browsing.
This can be defined as a baseline. It is where we currently are and is a measure of
how our system is performing under a normal load.
Load testing
Now that we know how we are, and we have defined some threshold for our goal
along with the pleasant browsing experience let's move forward.
We need to know which one is our limit, and more importantly how the system should
reply to our requests. Since we can't hire a room full of people that can click on our
website like crazy, we need to use the software to simulate this kind of behavior.
There are some interesting open source software that do exactly this. There are
different alternatives to choose from, one of them is Siege (http://www.joedog.org/
JoeDog/Siege).
Seige permits us to simulate a stored browser history and load it on our server.
We need to keep in mind that users, real users, will never be synchronized with
each other. So it is important to introduce some delay between all the requests.
Remember that if we have a login then we need to use a database of users because
application servers cache their object and we don't want to measure how good the
process is caching in them. The basic rule is to create a real browsing scenario
against our website, so users who login can logout with just a click and without
any random delay.
[ 51 ]
Deploying Zabbix
The stored scenarios should be repeated x times with a growing number of users,
meaning our Zabbix will store our metrics, and at a certain point we will pass our
first threshold (1-2 seconds per web page). We can go ahead until the response time
reaches the value of our second threshold. There is no way to see how much load
our server can take but it is well known that appetite comes with eating, so I will
not be surprised if you go ahead and load your server until it crashes one of the
components of your infrastructure.
Drawing graphs that relate the response time with the number of users' server will
help us see whether our three-tier web architecture is linear or not. Most probably it
will grow in a linear pattern until a certain point. This segment is the one on which
our system is performing fine. We can also see the components inside Zabbix and
from this point we can introduce some kind of delay and draw some considerations.
Now we know exactly what to expect from our system and how the system can serve
our users. We can see which component is the first that suffers the load, and where
we need to plan a tuning.
Capacity planning can be done without digging and going deep into what to
optimize. As said earlier, there are two different tasks: performance tuning and
capacity planning that are related but different. We can simply review our
performance and plan our infrastructure expansion.
A planned hardware expansion is always
cheaper than an unexpected, emergency
hardware improvement.
We can also follow performance tuning, but be aware that there is a relation between
the time spent and the performance obtained, so we need to understand when it is
time to stop our performance tuning.
[ 52 ]
Chapter 1
Forecasting the trends
One of the most important features of Zabbix is the capacity to store historical data.
This feature is of vital importance during the task of predicting trends. Predicting our
trends is not an easy task and is important considering the business that we are
serving, and when looking at historical data we should see if there are repetitive
periods or if there is a sort of formula that can express our trend.
For instance, it is possible that the online web store we are monitoring needs more
and more resources during a particular period of the year, for example, close to
public holidays if we sell travels. Doing a practical example, you can consider the
used space on a specific server disk. Zabbix gives us the export functionality to get
our historical data, so it is quite easy to import them in a spreadsheet. Excel has a
curve fitting option that will help us a lot. It is quite easy to find a trend line using
Excel that will tell us when we are going to reach all our disk space. To add a trend
line into Excel, we need to create at first a "scattered graph" with our data; here it is
also important to graph the disk size. After this we can try to find a mathematical
equation that is closer to our trend. There are different kinds of formula that we can
choose; in this example I used a Linear equation because the graphs are growing
with a linear relation.
The trend line process is also known as a curve
fitting process.
The graph that comes out from this process permits us to know, with a considerable
degree of precision, when we will run out of space.
Now it is clear how important it is to have a considerable quantity of historical data,
bearing in mind the business period and how they influence data.
It is important to keep a track of the trend/regression line used and the
relative formula with R-squared value so that it is possible to calculate it
with precision, and if there aren't any changes in trends, when the space
will be exhausted.
[ 53 ]
Deploying Zabbix
The graph obtained is shown in the following figure, and from this graph it is
simple to see that if the trends don't change, we are going to run out of space on
June 25, 2013.
Summary
In this chapter, we completed a Zabbix setup in a three-tier environment.
This environment is a good starting point to handle all the events generated
from a large and a very large environment.
In next chapter, you will go deep into nodes, proxies, and all the possible infrastructure
evolution and, as you will see, all of them are an improvement on this initial setup.
This does not mean that the extensions described in the next chapter are easy to
implement, but all the infrastructural improvements use this three-tier setup as
a starting point. Basically, in the next chapter, you will learn how to expand and
evolve this setup and also see how the distributed scenarios can be integrated in our
installation. The next chapter will also include an important discussion about security
in a distributed environment, making you aware of possible security risks that may
arise on distributed environments.
[ 54 ]
Distributed Monitoring
Zabbix is a fairly lightweight monitoring application that is able to manage thousands
of items with a single-server installation. However, the presence of thousands of
monitored hosts, a complex network topology, or the necessity to manage different
geographical locations with intermittent, slow, or faulty communications can all
show the limits of a single-server configuration. Likewise, the necessity to move
beyond a monolithic scenario towards a distributed one is not necessarily a matter
of raw performance, and therefore, it's not just a simple matter of deciding between
buying many smaller machines or just one big powerful one. Many DMZs and
network segments with a strict security policy don't allow two-way communication
between any hosts on either side, so it can be impossible for a Zabbix server to
communicate with all the agents on the other side of a firewall. Different branches
in the same company or different companies in the same group may need some sort
of independence in managing their respective networks, while also needing some
coordination and higher-level aggregation of monitored data. Different labs of a
research facility may find themselves without a reliable network connection, so they
may need to retain monitored data for a while and then send it asynchronously for
further processing.
Thanks to its distributed monitoring features, Zabbix can thrive in all these
scenarios and provide adequate solutions whether the problem is about
performance, network segregation, administrative independence, or data
retention in the presence of faulty links.
While the judicious use of the Zabbix agents could be considered from some point of
view as a simple form of distributed monitoring, in this chapter, we will concentrate
on Zabbix's two supported distributed monitoring modes: proxies and nodes. In this
chapter, you will learn the differences between proxies and nodes, their respective
advantages and disadvantages, how to deploy and configure them, and how to mix
and match the two for really complex scenarios.
Distributed Monitoring
There will also be some considerations about security between proxies and nodes,
so that by the end of this chapter, you will have all the information you need to
apply Zabbix's distributed features to your environment.
Zabbix proxies
A Zabbix proxy is another member of the Zabbix suite of programs that sits between a
full blown Zabbix server and a host-oriented Zabbix agent. Just like a server, it's used
to collect data from any number of items on any number of hosts, and it can retain that
data for an arbitrary period of time, relying on a dedicated database to do so. Just like
an agent, it doesn't have a frontend and is managed directly from the central server.
It also limits itself to data collections without triggering evaluations or actions.
All these characteristics make the Zabbix proxy a simple, lightweight tool to deploy
if you need to offload some checks from the central server or if your objective is
to control and streamline the flow of monitored data across networks (possibly
segregated by one or more firewalls) or both.
A basic distributed architecture involving Zabbix proxies would look as follows:
host
host
Zabbix
proxy
host
host
Zabbix
Zabbix Server Firewall
proxy
host
host
Zabbix
proxy
host
[ 56 ]
Chapter 2
By its very nature, a Zabbix proxy should run on a dedicated machine,
possibly different than the main server. A proxy is all about gathering data;
it doesn't feature a frontend and it doesn't perform any complex queries or
calculations; therefore, it's not necessary to assign a powerful machine with a lot
of CPU power or disk throughput. In fact, a small, lean hardware configuration is
often a better choice; proxy machines should be lightweight enough, not only to
mirror the simplicity of the software component but also because they should be
an easy and affordable way to expand and distribute your monitoring architecture
without creating too much impact on deployment and management costs. A possible
exception to the "small, lean, and simple" guideline for proxies can arise if you
end up assigning hundreds of hosts with thousands of monitored items to a single
proxy. In that case, instead of upgrading the hardware to a more powerful machine,
it's often cheaper to just split up the hosts in different groups and assign them to
different smaller proxies. In most cases, this would be the preferred option as you
are not just distributing and evening out the load but you are also considering the
possibility of huge data loss in case a single machine charged with the monitoring
of a large portion of your network goes down for any reason. Consider using small,
lightweight embedded machines as Zabbix proxies. They tend to be cheap, easy to
deploy, reliable, and quite frugal when it comes to power requirements. These are
ideal characteristics for any monitoring solution that aims to leave as little footprint
as possible on the monitored system.
Deploying a Zabbix proxy
A Zabbix proxy is compiled together with the main server if you add --enableproxy to the compilation options. The proxy can use any kind of database backend,
just like the server, but if you don't specify an existing DB, it will automatically create
a local SQLite to store its data. If you do intend to rely on SQLite, just remember to
add --with-sqlite3 to the options as well.
When it comes to proxies, it's usually advisable to keep things light and simple.
A proxy DB will just contain some configuration and measurements data that,
under normal circumstances, is almost immediately synchronized with the main
server. Dedicating a full-blown database to it is usually overkill, so unless you
have some very specific requirements, the SQLite option will provide the best
balance between performance and ease of management.
If you didn't compile the proxy executable the first time you deployed Zabbix,
just run configure again with the options you need for the proxies:
$ ./configure --enable-proxy --enable-static --with-sqlite3 --with-netsnmp –with-libcurl –with-ssh2 --with-openipmi
[ 57 ]
Distributed Monitoring
Compile everything again using the following command:
$ make
Beware that this will compile the main server as well; just remember not to run
make install, or copy the new Zabbix server executable over the old one in the
destination directory.
The only files you need to take and copy over to the proxy machine are the proxy
executable and its configuration file. The $PREFIX variable should resolve to the
same path you used in the configuration command (/usr/local by default):
# cp src/zabbix_proxy/zabbix_proxy $PREFIX/sbin/zabbix_proxy
# cp conf/zabbix_proxy.conf $PREFIX/etc/zabbix_proxy.conf
Next, you need to fill out relevant information in the proxy's configuration file.
The default values should be fine in most cases, but you definitely need to make
sure that the following options reflect your requirements and network status:
ProxyMode=0
This means that the proxy machine is in an active mode. Remember that you
need at least as many Zabbix trappers on the main server as the number of proxies
you deploy. Set the value to 1 if you need or prefer a proxy in the passive mode.
See the Understanding the flow of monitoring data with proxies section for a more
detailed discussion on proxy modes.
Server=n.n.n.n
This should be the IP number of the main Zabbix server or of the Zabbix node that
this proxy should report to.
Hostname=Zabbix proxy
This must be a unique, case-sensitive name that will be used in the main Zabbix
server's configuration to refer to the proxy.
LogFile=/tmp/zabbix_proxy.log
LogFileSize=1
DebugLevel=2
If you are using a small embedded machine, you may not have much disk space to
spare. In that case, you may want to comment all the options regarding the logfile
and let syslog send the proxy's log to another server on the Internet.
# DBHost=
# DBName=
# DBSchema=
# DBUser=
[ 58 ]
Chapter 2
# DBPassword=
# DBSocket=
# DBPort=
Just leave everything commented out and the proxy will automatically create and
use a local SQLite database. Fill out the relevant information if you are using a
dedicated, external DB.
ProxyOfflineBuffer=1
This is the number of hours a proxy will keep monitored measurements if
communications with the Zabbix server go down. You may want to double or triple
it if you know you have a faulty, unreliable link between the proxy and server.
CacheSize=8M
This is the size for the configuration cache. Make it bigger if you have a large number
of hosts and items to monitor.
The only thing that you need to do is to make the proxy known to the server,
and add monitoring objects to it. All these tasks are performed through the
Zabbix frontend.
[ 59 ]
Distributed Monitoring
Note how in the case of an Active proxy, you just need to specify the proxy's name,
as already set in zabbix_proxy.conf. It will be the proxy's job to contact the main
server. On the other hand, a Passive proxy will need an IP address or a hostname
for the main server to connect to. See the Understanding the flow of monitoring data
with proxies section for more details.
[ 60 ]
Chapter 2
You can assign hosts to proxies during the proxy creation process, or in the
proxy's edit screen, or even from the host's configuration screen, as in the
following screenshot:
[ 61 ]
Distributed Monitoring
One of the advantages of proxies is that they don't need much configuration or
maintenance; once they are deployed and you have assigned some hosts to one
of them, the rest of the monitoring activities are fairly transparent. Just remember
to check the number of values per second every proxy has to guarantee as expressed
by the Required performance column in the proxies' list page:
Values per second (vps) is the number of measurements per seconds that a single
Zabbix server or proxy has to collect. It's an average value that depends on the
number of items and the polling frequency for every item. The higher the value,
the more powerful the Zabbix machine must be.
[ 62 ]
Chapter 2
Depending on your hardware configuration, you may need to redistribute the hosts
among proxies or add new ones in case you notice degraded performances coupled
with high vps.
Understanding the flow of monitoring data
with proxies
Zabbix proxies can operate in two different modes, active and passive. An active
proxy, which is the default setup, initiates all connections to the Zabbix server,
both to retrieve configuration information on monitored objects and to send
measurements back to be further processed. You can tweak the frequency of these
two activities by setting the following variables in the proxy configuration file:
ConfigFrequency=3600
DataSenderFrequency=1
Both the preceding values are in seconds. On the server side, in the zabbix_server.
conf file, you also need to set the value of StartTrappers= to be higher than the
number of all active proxies and nodes you have deployed. The trapper processes
will have to manage all incoming information from proxies, nodes, and any item
configured as an active check. The server will fork extra processes as needed, but it's
advisable to prefork as many processes that you already know the server will use.
Back on the proxy side, you can also set a HeartbeatFrequency so that after a
predetermined number of seconds, it will contact the server even if it doesn't have
any data to send. You can then check on the proxy availability with the following
item, where proxy name of course is the unique identifier that you assigned to the
proxy during deployment:
zabbix[proxy, "proxy name", lastaccess]
[ 63 ]
Distributed Monitoring
The item as expressed will give you the number of seconds since the last contact
with the proxy, a value you can then use with the appropriate triggering functions.
A good starting point to finetune the optimal heartbeat frequency is to evaluate how
long you can afford to lose contact with the proxy before being alerted, and consider
that the interval is just over two heartbeats. For example, if you need to know if
a proxy is possibly down in less than five minutes, set the heartbeat frequency to
120 seconds and check whether the last access time was above 300 seconds.
configuration changes requests
port
10051
monitored data sends
heartbeat
Zabbix Server
Zabbix
proxy
active proxy data flow
An active proxy is more efficient in offloading computing duties from the server as
the latter will just sit idle, waiting to be asked about changes in configuration or to
receive new monitoring data. The downside is that proxies will often be deployed
to monitor secure networks, such as DMZs and other segments with strict outgoing
traffic policies. In these scenarios, it would be very difficult to obtain permission for
the proxy to initiate contact with the server. And it's not just a matter of policies;
DMZs are isolated as much as possible from internal networks for extremely good
and valid reasons. On the other hand, it's often easier and more acceptable from a
security point of view to initiate a connection from the internal network to a DMZ.
In these cases, a passive proxy will be the preferred solution.
Connection- and configuration-wise, a passive proxy is almost the mirror image
of the active version. This time, it's the server that needs to connect periodically to
the proxy to send over configuration changes and to request any measurements the
proxy may have taken. On the proxy configuration file, once you've set ProxyMode=1
to signify that this is a passive proxy, you don't need to do anything else. On the
server side, there are three variables you need to check:
• StartProxyPollers=
This represents the number of processes dedicated to manage passive proxies
and should match the number of passive proxies you have deployed.
[ 64 ]
Chapter 2
• ProxyConfigFrequency=
The server will update a passive proxy with configuration changes for the
number of seconds you have set in the preceding variable.
• ProxyDataFrequency=
This is the interval, also in seconds, between two consecutive requests by the
server for the passive proxy's monitoring measurements.
There are no further differences between the two modes of operation for proxies.
You can still use the zabbix[proxy, "proxy name", lastaccess] item to check a
passive proxy's availability, just like for the active one.
configuration changes requests
monitored data sends
port
10051
Zabbix
proxy
Zabbix Server
passive proxy data flow
At the price of a slightly increased workload for the server, when compared to active
proxies, a passive one will enable you to gather monitoring data from otherwise
closed and locked-down networks. At any rate, you can mix and match active and
passive proxies in your environment, depending upon the flow requirements of
specific networks. This way, you will significantly expand your monitoring solution
both in its ability to reach every part of the network and in its ability to handle a
large number of monitored objects, while at the same time keeping the architecture
simple and easy to manage with a strong central core and many simple, lightweight
yet effective satellites.
[ 65 ]
Distributed Monitoring
Zabbix nodes
While proxies are often an adequate solution to many distributed monitoring
problems, sometimes they just lack the features needed in some scenarios.
Consider, for example, the case where you not only have a huge number of items
but also a great number of complex and computing-intensive triggers that can
really impact your server performance. Proxies won't be able to tackle the heart of
the problem in this case. Another case would be that of a group merger of different
companies with an IT infrastructure that's brought together but still managed by
different teams, which still need a bit of operational independence within a shared
context. Or you could have some of your company's branches in distant geographical
locations and it just doesn't make sense to occupy a significant portion of your
bandwidth with every single detail of monitored data. All these scenarios and others
can be managed, thanks to the Zabbix server's ability to take part in a larger tree-like
structure of monitoring nodes.
A Zabbix node is a full-blown server with its own database, frontend, authentication
sources, and monitored objects that has been assigned a node ID and has been given
a place in the hierarchy of Zabbix nodes. As part of this hierarchy, a Zabbix node
will get configuration information from the parent nodes and report back to them
on monitoring measurements and trigger events, while distributing configuration
information to child nodes and retrieve monitoring data from them. A node tree is
strictly hierarchical, and nodes on the same level (that is, sharing the same parent)
won't exchange data in any way.
master
configuration
child
child
child
child
child
child
child
[ 66 ]
monitoring
data
child
child
Chapter 2
Understanding the flow of data with nodes
A parent node will be able to manage almost any aspect of all of its child nodes'
configuration, at any level down the tree. This is not a real-time operation;
every parent keeps a copy of all its child nodes' configuration data (hosts, items,
triggers, templates, screens, and so on) in the database, and periodically calculates
a checksum to compare with each child node. If there are any differences, the child
updates its configuration with the new information sent from the parent.
Conversely, a child can update the information a parent has about it using the
same mechanism. A checksum is sent over to the parent. If there are any differences,
the parent updates its information about the child with the new data sent from the
child itself. This works with modifications to a child node's configuration done
directly from a parent node or with updates from the child to the parent only.
Due to current limitations in the nodes' architecture, there's no easy way to move
or copy configuration information (templates, for example) from one node to
another using the checksum feature, not even from the parent to the child. If you
need to create new templates and distribute them to all nodes, you will have to
resort to some out-of-band strategy. We will see a of possible solution in the
following chapters.
Deploying a node
A Zabbix node is first and foremost a regular Zabbix server installation. You can
refer to Chapter 1, Deploying Zabbix, for indications on server- and database-choice
and deployment.
Once you have installed a Zabbix server, there are a few operations you need to
perform in order to make it into a node that is part of a specific hierarchy. You will
need to make a few changes to the server configuration file, to the database, and to
both the parent's and child's frontends.
First of all, you need to assign a server a node ID number. Zabbix supports up to
1000 nodes, so any number between 0 and 999 will do. You don't need to respect
a sequential order for the nodes hierarchy. A node ID is just a label and it doesn't
matter if Node 1 is a child of Node 57 or vice versa. On the other hand, in order for
every monitored object to be unique across the whole distributed node structure,
this number will be prepended to an object ID in the database of a node. These IDs
will then be synchronized with the parent nodes, as explained in the previous
section, so be sure that every node ID will be unique in your nodes' hierarchy.
[ 67 ]
Distributed Monitoring
A database table for a Zabbix installation that is not part of a node structure will look
like this:
> select itemid, name from items limit 5;
+--------+-----------------------------------------+
| itemid | name
|
+--------+-----------------------------------------+
| 10009 | Number of processes
|
| 10010 | Processor load (1 min average per core) |
| 10013 | Number of running processes
|
| 10014 | Free swap space
|
| 10016 | Number of logged in users
|
+--------+-----------------------------------------+
Note the range of the item IDs. The same ranges can be found for host IDs and any
other monitoring object such as triggers, graphs, maps, and map items.
The main operation to turn a Zabbix installation into a node is to perform an update
of the database, by issuing the following command with the node ID you have
assigned to the server.
$ zabbix_server -n <node id>
For example, assuming that you have assigned node ID 99 to your Zabbix server,
you would issue the following command:
$ zabbix_server -n 99
Dropping foreign keys ...................................................
......... done.
Converting tables .......................................................
..... done.
Creating foreign keys ...................................................
......... done.
Conversion completed successfully.
This will change all the existing and future object IDs by adding the node ID to them:
> select itemid, name from items limit 5;
+------------------+-----------------------------------------+
| itemid
| name
|
+------------------+-----------------------------------------+
| 9909900000010009 | Number of processes
|
| 9909900000010010 | Processor load (1 min average per core) |
| 9909900000010013 | Number of running processes
|
| 9909900000010014 | Free swap space
|
| 9909900000010016 | Number of logged in users
|
+------------------+-----------------------------------------+
[ 68 ]
Chapter 2
As you can see, the same item IDs have now been prepended with the node ID you
have chosen. Before logging on the frontend, you still have to specify the same node
ID in the server configuration file. Once done, just restart the server daemons and
you'll immediately see that the frontend will reflect the recent changes. You'll be able
to select a node from a drop-down menu and all objects will report what node they
are part of. By going to Administration | DM | Nodes, you can change the name
of the local node to reflect the name of the server, so that it's never ambiguous as to
what "local node" you might refer to in your interface:
At this point, you just need to repeat the same procedure for all the nodes in your
hierarchy, and you'll be ready to link them in a tree-like structure. Double-check your
node IDs! You can only execute the zabbix_server -n <node id> command once.
Any other attempt on the same database will corrupt it. So make sure that you are not
re-using any node ID or you'll have to start from scratch for the node you "cloned".
[ 69 ]
Distributed Monitoring
For every parent-child link, you'll need to configure the relationship on both nodes.
Starting from the parent, make sure that you select the right node ID, make sure
the type is "child", and whether you have selected the right parent (Master node).
Keep in mind that parent nodes are identified by their node names and not by their
IDs in the interface, so be sure to keep your nodes names and ID's map handy.
On the child's side, create a node with the same node ID, its name as the parent,
and specify Type as Master, as shown in the following screenshot:
[ 70 ]
Chapter 2
The two nodes are now linked. You will be able to completely administer
the child node from the parent one, and the two nodes will start to synchronize
both configuration and monitoring data as explained in the previous section.
You can control what kind of data a child sends to its master node by setting
the variables:
NodeNoEvents=
NodeNoHistory=
Setting both the values to 1 will instruct the child node not to send the event
information or history data, respectively. This won't have any impact on the
data sent from nodes further down the branch. A node will always pass over
to its master any data received from its child nodes. The configuration update
and data-send frequency are fixed for nodes: two minutes for configuration
and five seconds for history and events.
Once you have historical data or events from child nodes in your master node,
you can use it to create further aggregate items, triggers, or graphs, just like any
other monitoring object.
[ 71 ]
Distributed Monitoring
Proxies versus nodes
At first sight, it could seem that nodes are much more powerful and flexible than
proxies. While that is certainly true to a certain extent, it doesn't mean that they
are automatically the preferred solution to every distributed monitoring scenario.
In fact, there are a number of drawbacks to using nodes.
Disadvantages of nodes
First of all, every node is a full server, with a full-blown database and web frontend.
This means more expensive hardware than that required by a proxy, but most of all,
this means a lot more maintenance in terms of day-to-day system and database
administration, backups, security policy compliance, and so on. It's also not easy to
move nodes around once they are configured, and while they do send data upstream
to the master node, they still need to be managed one-by-one as single entities.
You can't mass-update hosts or items across nodes, for example. More importantly,
if you update a template that is used by more than one node, you'll have to replicate
the update to the other nodes individually. You can try to automate template
synchronization by leveraging the power of the Zabbix API and a lot of custom
code, and in Chapter 9, Extending Zabbix, we'll show you an example of such a setup,
but you'll have to enforce strong template-update policies throughout your node
tree, or you could find yourself with inconsistent data.
Speaking of access policies, these too have to be managed node-by-node, and for
every child node, you'll have to decide not only the group permissions on the node
itself, like a standalone installation, but you'll also have to go up the branch and
figure out group permissions for that node's data on every parent node.
Finally, since configuration-synchronization is performed by calculating checksums,
this operation can add up to a significant amount of CPU time for a master node,
such that you may need to invest in more powerful hardware just to keep up with
synchronization. A master node's database size tends to grow significantly too as it
has to keep information about the child nodes as well.
By contrast, none of these drawbacks exist with proxies. They may be simpler and
less powerful, but they are also far easier to deploy and maintain and easier to move
around in case you need to, and they also consume less computing resources and
less disk space overall.
[ 72 ]
Chapter 2
Choosing between proxies and nodes
To summarize, the power of a node's base architecture comes at a cost:
• It requires more powerful hardware
• It is more complex to configure
• It requires heavy maintenance
• It is not very flexible
On the other hand, a proxy-based architecture is:
• Easy to deploy
• Requires simple hardware
• Easy to configure
• More flexible than a hierarchy of nodes
All in all, a reasonable piece of advice would be to go with proxies whenever
possible and resort to nodes when necessary. Cases such as the merging of two
distinct networks with existing Zabbix installations, distant geographical branches
of the same network or company, and really huge networks where every node has
to manage thousands of nodes are good examples of scenarios where a node-based
solution works best. A typical large environment will probably have a fair amount
of proxies but only a limited number of nodes, such that any maintenance overhead
would be contained as much as possible.
Security considerations
One of the few drawbacks of the whole Zabbix architecture is the lack of built-in
security at the Zabbix protocol level. While it's possible to protect both the web
frontend and the Zabbix API by means of a standard SSL layer to encrypt
communications, and relying on different authorities for identification, there's simply
no standard way to protect communications between agents and server, or between
proxies and server, or among nodes, not even when it comes to message authentication
(the other party is indeed who it says it is), nor when it comes to message integrity
(the data has not been tampered with) and neither when it comes to message
confidentiality (no one else can read or understand the data).
[ 73 ]
Distributed Monitoring
If you've been paying attention to the configuration details of agents, proxies,
and nodes, you may have noticed that all that a Zabbix component needs to know in
order to communicate to another component is its IP address. No authentication is
performed as relying on only the IP address to identify a remote source is inherently
insecure. Moreover, any data sent is clear text, as you can easily verify by running
tcpdump (or any other packet sniffer):
$ zabbix_sender -v -z 10.10.2.9 -s alpha -k sniff.me -o "clear text
data"
$ tcpdump
-s0 -nn -q -A port 10051
00:58:39.263666 IP 10.10.2.11.43654 > 10.10.2.9.10051: tcp 113
[email protected]@.P...........'C..."^......V.......
.Gp|.Gp|{
"request":"sender data",
"data":[
{
"host":"alpha",
"key":"sniff.me",
" value":"clear text data"}]}
Sure, simple monitoring or configuration data may not seem much, but at the very
least, if tampered with, it could lead to false and unreliable monitoring.
While there are no standard counter measures to this problem, there are a few
possible solutions to it that increase in complexity and effectiveness from elementary,
but not really secure to complex and reasonably secure. Keep in mind that this is not
a book on network security, so you won't find any deep, step-by-step instructions on
how to choose and implement your own VPN solution. What you will find is a brief
overview of methods to secure the communication between the Zabbix components,
which will give you a practical understanding of the problem, so you can make an
informed decision on how to secure your own environment.
No network configuration
If, for any reason, you can't absolutely do anything else, you should at the very
least specify a source IP for every Zabbix trapper item, so that it wouldn't be too
easy and straightforward to spoof monitoring data using the zabbix_sender
utility. Use the macro {HOST.CONN} in a template item so that every host will
use its own IP address automatically:
[ 74 ]
Chapter 2
More importantly, make sure that remote commands are not allowed on agents.
That is, EnableRemoteCommands in the zabbix_agentd.conf file must be set to
0. You may lose a convenient feature, but if you can't protect and authenticate the
server-agent communications, the security risk is far too great to even consider
taking it.
[ 75 ]
Distributed Monitoring
Network isolation
Many environments have a management network that is separated and isolated
from your production network via nonrouted network addresses and VLANs.
Network switches, routers, and firewalls typically handle traffic on the production
network, but are reachable and can be managed only through their management
network address. While this makes it a bit less convenient to access them from any
workstation, it also makes sure that any security flaw in your components (consider,
for example, a network appliance that has a faulty SSL implementation that you
can't use or that doesn't support SNMP v3, or has Telnet inadvertently left open) is
contained to a separated and difficult-to-reach network. You may want to put all
of the server-proxy and master-child communications on such an isolated network.
You are just making it harder to intercept monitoring data and you may be leaving
out the server-agent communications, but isolating traffic is still a sensible solution
even if you are going to further encrypt it with one of the solutions outlined in the
following sections.
On the other hand, you certainly don't want to use this setup for a node or proxy
that is situated in a DMZ or another segregated network. It's far more risky to bypass
a firewall through a management network than to have your monitoring data pass
through the said firewall. Of course, this doesn't apply if your management network
is also routed and controlled by the firewall, but it's strongly advised to verify that
this is indeed the case before looking into using it for your monitoring data.
Simple tunnels
So far, we haven't really taken any measures to secure and encrypt the actual data
that Zabbix sends or receives. The simplest and most immediate way to do that is to
create an ad hoc encrypted tunnel through which you can channel your traffic.
Secure Shell
Fortunately, Secure Shell (SSH) has built-in tunneling abilities, so if you have to
encrypt your traffic in a pinch, you already have all the tools you need.
To encrypt the traffic from an active proxy to the server, just log on the proxy's
console and issue a command similar to the following one:
$ ssh -N -f [email protected] -L 10053:localhost:10051
[ 76 ]
Chapter 2
In the preceding command, -N means that you don't want the SSH client to execute
any commands, other than just routing the traffic; the -f option makes the SSH client
go into the background (so you don't have to keep a terminal open, or a start script
executing forever), [email protected] is a valid user (and real hostname or IP
address) on the Zabbix server, and the -L port:remote-server:port sets up
the tunnel. The first port number is what your local applications will connect to,
while the following host:port combination specifies what host and TCP port the
SSH server should connect to as the other end of the tunnel.
Now set your Server and ServerPort options in your zabbix_proxy.conf to
localhost and 10053 respectively.
What will happen is that from now on, the proxy will send data to port 10053 by itself,
where there's an SSH tunnel session waiting to forward all traffic via the SSH protocol
to the Zabbix server. From there, the SSH server will in turn forward it to a local port
10051 and finally to the Zabbix daemon. While all of the Zabbix components don't
natively support data encryption for the Zabbix protocol, you'll still be able to make
them communicate while keeping message integrity and confidentiality; all you will
see on the network with such a setup will be standard, encrypted SSH traffic data on
TCP port 22.
To make a Zabbix server contact a Passive proxy via a tunnel, just set up a
listening SSH server on the proxy (you should already have it in order to remotely
administrate the machine) and issue a similar command as the one given earlier on
the Zabbix server, making sure to specify the IP address and a valid user for the
Zabbix proxy. Change the proxy's IP address and connection-port specifications on
the web frontend, and you are done.
To connect to Zabbix nodes, you need to set up two such tunnels, one from the
master to the child and one from the child to the master.
On the master, run the following command:
$ ssh -N -f [email protected] -L 10053:localhost:10051
On the child, run the following command:
$ ssh -N -f [email protected] -L 10053:localhost:10051
[ 77 ]
Distributed Monitoring
Stunnel
Similar functionalities can be obtained using the stunnel program. The main
advantage of using stunnel over SSH is that with stunnel, you have a convenient
configuration file where you can set up and store all your tunneling configurations,
while with SSH, you'll have to script the preceding commands somehow if you want
the tunnels to be persistent across your machine's reboots.
Once installed, and once you have created the copies of the obtained SSL certificates
that the program needs, you can simply set up all your port-forwarding in the
/etc/stunnel/stunnel.conf file. Considering, for example, a simple scenario
with a Zabbix server that receives data from an active proxy and exchanges data
with another node, after having installed stunnel and SSL certificates on all three
machines, you could have the following setup.
On the Zabbix server's stunnel.conf, add the following lines:
[proxy]
accept = 10055
connect = 10051
[node - send]
accept = localhost:10057
connect = node.server:10057
[node – receive]
accept = 10059
connect = 10051
On the Zabbix proxy's stunnel.conf, add the following lines:
[server]
accept = localhost:10055
connect = zabbix.server:10055
On the other node's stunnel.conf, add the following lines:
[node - send]
accept = localhost:10059
connect = node.server:10059
[node – receive]
accept = 10057
connect = 10051
[ 78 ]
Chapter 2
Just remember to update the host and port information for proxies and servers in
their respective configuration files and web frontend forms.
As you can see, the problem with port-forwarding tunnels is that the more tunnels
you set up, the more different ports you have to specify. If you have a large number
of proxies and nodes, or if you want to encrypt the agent data as well, all the port
forwarding will quickly become cumbersome to set up and keep track of. This is a
good solution if you just want to encrypt your data on an insecure channel among a
handful of hosts, but if you want to make sure that all your monitoring traffic is kept
confidential, you'll need to resort to a more complete VPN implementation.
A full-blown VPN
This is not the place to discuss the relative merits of different VPN implementations,
but if you do use a VPN solution in your network, consider switching all Zabbix
monitoring to your encrypted channel. Of course, unless you want the whole world
to look at your monitoring data, this is practically mandatory when you link two
nodes or a server and a proxy, from distant geographical locations that are connected
only through the Internet. In that case, you hopefully already have a VPN, whether a
simple SSL one or a full-blown IPSEC solution. If you don't have it, protecting your
Zabbix traffic is an excellent reason to set up one.
These workarounds will protect your traffic, and in the best-case scenario, will provide
basic host authentication, but keep in mind that until Zabbix supports some sort of
security protocol on the application level, tunneling and encryption will only be able
to protect the integrity of your monitoring data. Any user who gains access to a Zabbix
component (whether it's a server, proxy, or agent) will be able to send bogus data over
the encrypted channel, and you'll have no way to suspect foul play. So, in addition to
securing all communication channels, you also need to make sure that you have good
security at the host level.
[ 79 ]
Distributed Monitoring
Summary
In this chapter, we saw how to expand a simple, standalone Zabbix installation
into a vast and complex distributed monitoring solution. By now, you should be
able to understand how Zabbix proxies and nodes work, how they pass monitoring
information around, what their respective strong points and possible drawbacks are,
and what is their impact in terms of hardware requirements and maintenance.
You should also know when and how to choose between an active proxy and a
passive one, when to switch to a node-based implementation, and more importantly,
how to mix and match the two features into a tailor-made solution for your own
environment.
Finally, you should have a clear understanding of how to evaluate possible security
concerns regarding monitored data and what possible measures you can take to
mitigate security risks related to a Zabbix installation.
In the next chapter, we will conclude with an overview on how to deploy Zabbix in
a large environment by talking about high availability at the three levels of database,
monitoring server, and web frontend.
[ 80 ]
High Availability and Failover
Now that you have a good knowledge of all the components of a Zabbix infrastructure,
it is time to implement a highly available Zabbix installation. In a large environment,
especially if you need to guarantee that all your servers are up and running, it is of
vital importance to have a reliable Zabbix infrastructure. The monitoring system
and Zabbix infrastructure should survive any possible disaster and guarantee
business continuity.
High availability is one of the solutions that guarantees business continuity and
provides a disaster recovery implementation; this kind of setup cannot be missed
in this book.
This chapter begins with the definition of high availability, and it further describes
how to implement an HA solution.
In particular, this chapter considers the three-tier setup that we described earlier:
• The Zabbix GUI
• The Zabbix server
• Database
For each one of those components, we have described how to set up and configure
it on high availability. All the procedures presented in this chapter have been
implemented and tested on a real environment.
In this chapter, we will cover the following topics:
• Understanding what is high availability, failover, and its service level
• In-depth analysis of all the components (the Zabbix server, the web server,
and the RDBMS server) of our infrastructure and how they will fit in a highly
available installation
• Implement a highly available setup of our monitoring infrastructure
High Availability and Failover
Understanding high availability
High availability is an architectural design approach and associate service
implementation that is used to guarantee the reliability of a service. Availability
is directly associated to the uptime and usability of a service. This means that the
downtime should be reduced to achieve an agreement on that service.
We can distinguish between two kinds of downtime:
• Scheduled or planned downtime
• Unscheduled or unexpected downtime
To distinguish between the scheduled downtime, we can include:
• System patching
• Hardware expansion or hardware replacement
• Software maintenance
• All that is normally a planned maintenance task
Unfortunately, all these downtimes will interrupt your service, but you have to agree
that they can be planned into a maintenance window agreed.
The unexpected downtime normally arises from a failure, and it can be caused by
one of the following reasons:
• Hardware failure
• Software failure
• Physical events
Unscheduled downtimes also include the power outage and over-temperature
shutdown, and all these are not planned; however, they cause an outage.
Hardware and software failure are quite easy to understand, whereas a physical
event is an external event that produces an outage on our infrastructure. A practical
example can be an outage that can be caused by lightning or a flood that leads
to the breakdown of the electrical line with consequences on our infrastructure.
The availability of a service is considered from the service user's point of view;
for example, if we are monitoring a web application, we need to consider this
application from the web user's point of view. This means if all your servers are up
and running, but a firewall is cutting connections and the service is not accessible,
this service cannot be considered available.
[ 82 ]
Chapter 3
Understanding the levels of IT service
Availability is directly tied with service level and is normally defined as a
percentage. It is the percentage of uptime over a defined period. The availability that
you can guarantee is your service level. The following table shows what exactly this
means by considering the maximum admitted downtime for some of the frequently
used availability percentage.
Availability percentage
Max downtime
per year
Max downtime
per month
Max downtime
per week
90% called "one nine"
36.5 days
72 hours
16.8 hours
95%
18.25 days
36 hours
8.4 hours
99% called "two nines"
3.65 days
7.20 hours
1.68 hours
99.5%
1.83 days
3.60 hours
50.4 minutes
99.9% called "three nines"
8.76 hours
43.8 minutes
10.1 minutes
99.95%
4.38 hours
21.56 minutes
5.04 minutes
99.99% called "four nines"
52.56 minutes
4.32 minutes
1.01 minutes
99.999% called "five nines"
5.26 minutes
25.9 seconds
6.05 seconds
99.9999% called "six nines"
31.5 seconds
2.59 seconds
0.605 seconds
99.99999% called "seven nines"
3.15 seconds
0.259 seconds
0.0605 seconds
Uptime is not a synonym of availability. A system can be up and running
but not available; for instance, if you have a network fault, the service will
not be available, but all the systems will be up and running.
The availability must be calculated end-to-end, and all the components required to
run the service must be available. The next sentence may seem a paradox; the more
hardware you add and the more failure points you need to consider, the more it
becomes increasingly difficult to implement an efficient solution. Also, an important
point to consider is how easy the patching of your HA system and its maintenance
will be. A true highly available system implies that human intervention is not needed;
for example, if you need to agree to a five nine service level, the human (your system
administrator) will have only one second of downtime per day, so here the system
must respond to the issue automatically. Instead, if you agree to a two nines service
level agreement (SLA), the downtime per day can be 15 minutes; here, the human
intervention is realistic but unfortunately this SLA is not a common case. Now while
agreeing to an SLA, the mean time for recovery is an important factor to consider.
Mean Time To Recovery (MTTR) is the mean time that a device will take
to recover from a failure.
[ 83 ]
High Availability and Failover
The first thing to do is to keep the architecture as simple as possible and reduce the
number of actors in the play to minimum. Simpler the architecture, lesser the effort
required to maintain, administer, and monitor it. All that the HA architecture needs
is to avoid a single point of failure, and it needs to be as simple as possible. For this
reason, the solutions here presented are easy to understand, tested on production
environments, and quite easy to implement and maintain.
Complexity is the first enemy of high availability.
Unfortunately, a highly available infrastructure is not designed to achieve the
highest performance possible. This is because it is normal that an overhead will
be introduced to keep two servers updated and a highly available infrastructure
is not designed for the maximum throughput. Also, if there are implementations
that consider using the standby server as a read-only server to reduce the load of
a primary node, use an unused server.
A highly available infrastructure is not designed to achieve the
maximum performance or throughput.
Some consideration about high
availability
Every HA architecture has some common problem to solve or common question
to respond:
• How to handle the connection?
• How to manage the failover?
• How the storage is shared or replicated to the other site?
[ 84 ]
Chapter 3
There are some production-stable and widely used solutions for each one of those
questions. Let us study these questions in detail:
• How to handle the connection?
One of the possible responses to this question is one word—VIP (Virtual IP).
Basically, every software component needs to communicate or is
interconnected with different logical layers, and those components are often
deployed on different servers to divide and equalize the workload. Most of
the communications are TCP/IP based, and here the network protocol gives
us a hand.
It is possible to define a VIP that is assigned to the active servers and all
the software required to be configured to use that address. So in case of a
failover, the IP address will follow the service and all the clients will continue
to work. Of course, this solution can't guarantee that there isn't downtime at
all, but it will be time limited and for a small period. From the administration
point, apart from checking the failover, the administrator doesn't need to
reconfigure anything.
• How to manage the failover?
The response to this question is: use a resource manager. You need to think
of a smart way to move a faulty service to the standby node as soon as
possible that is independent of SLA. To achieve the minimum downtime
possible, you need to automate the service failover on the standby node,
and to give the business continuity, the fault needs to be found as soon as
possible when it happens.
• How the storage is shared or replicated to the other site?
This last question can be implemented with different actors, technologies,
and methodologies. You can use a shared disk, a replicated Logical Unit
Number (LUN) between two storages, or use a replicated device with software.
Unfortunately, using a replicated LUN between two storages is quite expensive.
This software should be more close to kernel and should be working on the
lowest layer possible to be transparent from the operating system's perspective,
thereby keeping things easy to manage.
[ 85 ]
High Availability and Failover
Automating the switchover/the failover with
a resource manager
The architecture that you are going to implement needs a component to automate the
switchover or the failover, basically, as said earlier, it requires a resource manager.
One of the resource managers that is widely used and is production mature is
Pacemaker. Pacemaker is an open source high availability resource manager
designed for small and large clusters. Pacemaker is available for download at
the following URL: http://clusterlabs.org/
Pacemakers provide some interesting features that are really useful for your cluster:
• Detecting and recovering server issues at the application level
• Supporting redundant configurations
• Supporting multiple nodes applications
• Supporting startup/shutdown ordering applications
Practically, a Pacemaker replaces you and is automated and fast. Pacemaker does
the work that a Unix administrator normally does in case of node failure. It checks if
the service is no more available and switches all the configured services on the spare
node; plus, it does all this work as quickly as possible. This switchover gives us the
time required to do all the forensic analysis while all the services are still available.
In another context, the service would be simply unavailable.
There are different solutions that provide cluster managements. Red Hat Cluster
Suite is a popular alternative. It is not proposed here as it is not really completely
tied to Red Hat, however, it is definitely developed with this distribution in mind.
Replicating the filesystem with DRBD
Distributed Replicated Block Device (DRBD) has some features that are point of
force for this solution:
• It is a kernel module
• It is completely transparent from the point of view of RDBMS
• It provides real-time synchronization
• It synchronizes writes on both nodes
• It automatically performs resynchronization
• It practically acts like a networked RAID-1
[ 86 ]
Chapter 3
The core functionality of DRBD is implemented on the kernel layer, in particular
DRBD is a driver for a virtual block device, so DRBD works at the bottom of the
system I/O stack.
DRBD can be considered equivalent to a networked RAID-1, below the OS's
filesystem, at the block level.
This means that DRBD synchronization is synced to the filesystem. The worst scenario
and more complex to handle is a filesystem replication for an RDBMS. In this case,
every commit needs to be acknowledged on both nodes before it happens, and all the
committed transaction is written on both nodes; DRBD completely supports this case.
Now what happens when a node is no longer available? It's simple; DRBD will
operate exactly like a degraded RAID-1. This is a strong point because if your
Disaster Recovery site goes down, you don't need to do anything. Once the node
reappears, DRBD will do all the synchronization work for us, that is, rebuilding or
resynchronizing the offline node.
Implementing high availability on a
web server
Now that you know all the software components in play, it's time to go deep into
a web server HA configuration. This proposed design foresees two nodes with
Apache on top that is bonded to a virtual IP address. In this design, the HTTPD
or better Apache is on top of an active/passive cluster that is managed by
Corosync/Pacemaker.
It is quite an easy task to provide a highly available configuration for the Zabbix GUI
because the web application is well defined and does not produce or generate data
or any kind of file on the web server. This allows you to have two nodes deployed
on two different servers, if possible on two distant locations, implementing a highly
available fault-tolerant disaster-recovery setup. On this configuration, since the
web content will be "static", in the sense that it will not change (apart from in case
of system upgrade), you don't need a filesystem replication between the two nodes.
The only other component that is needed is a resource manager that will detect the
failure of the primary node and will coordinate the failover on the secondary node.
The resource manager that will be used is Pacemaker/Corosync.
[ 87 ]
High Availability and Failover
The installation will follow this order:
1. Installation of HTTPD server on both nodes
2. Installation of Pacemaker
3. Deployment of the Zabbix web interface on both nodes
4. Configuration of Apache to bind it on VIP
5. Configuration of Corosync/Pacemaker
6. Configuration of the Zabbix GUI to access RDBMS (on VIP of PostgreSQL)
The next diagram explains the proposed infrastructure:
Configuring HTTPD HA
This software is a sophisticated cluster resource manager that is widely used with a
lot of features. To set up Pacemaker, you need to:
• Install Corosync
• Install Pacemaker
• Configure and start Corosync
[ 88 ]
Chapter 3
It is time to spend a couple of words on this part of the architecture. Corosync is
a software layer that provides the messaging service between servers within the
same cluster.
Corosync allows any number of servers to be a part of the cluster using different
fault tolerant configurations such as Active-Active, Active-Passive, and N+1.
Corosync, in the middle of its tasks, checks that Pacemaker is running and
practically bootstraps all the process that is needed.
To install this package, you can run the following command:
$ yum install pacemaker corosync
Yum will resolve all dependencies for you; once everything is installed, you can
configure corosync. The first thing to do is to copy the sample configuration file
available at the following location:
$ cp /etc/corosync/corosync.conf.example /etc/corosync/corosync.conf
To configure Corosync, you need to choose an unused multicast address and a port:
$ export MULTICAST_PORT=4000
$ export MULTICAST_ADDRESS=226.94.1.1
$ export BIND_NET_ADDRESS=`ip addr | grep "inet " |grep brd |tail -n1 |
awk '{print $4}' | sed s/255/0/`
$ sed -i.bak "s/.*mcastaddr:.*/mcastaddr:\ $MULTICAST_ADDRESS/g" /etc/
corosync/corosync.conf
$ sed -i.bak "s/.*mcastport:.*/mcastport:\ $MULTICAST_PORT/g" /etc/
corosync/corosync.conf
$ sed -i.bak "s/.*bindnetaddr:.*/bindnetaddr:\ $BIND_NET_ADDRESS/g" /etc/
corosync/corosync.conf
Now, you need to tell Corosync to add the Pacemaker service and create the /etc/
corosync/service.d/pcmk file with the following content:
service {
# Load the Pacemaker Cluster Resource Manager
name: pacemaker
ver: 1
}
[ 89 ]
High Availability and Failover
At this point, you need to propagate the files you just configured on node2:
/etc/corosync/corosync.conf
/etc/corosync/service.d/pcmk
After that, you can start Corosync and Pacemaker on both nodes:
$ /etc/init.d/corosync start
$ /etc/init.d/pacemaker start
Check the cluster status using the following command:
$ crm_mon
Examine the configuration using the following command:
$ crm configure show
Understanding Pacemaker and STONITH
Shoot The Other Node In The Head (STONITH) can introduce a weak point in
this configuration; it can cause a split-brain scenario especially if servers are in
two distant locations where there are a wide number of causes that can avoid the
communication between them. The split-brain scenarios happen when each node
believes that the other is broken and it is the first node. Then when the second reboot
occurs, it shoots the first and so on. This is also known as STONITH death match.
There are basically three issues that can cause one node to STONITH the other:
• Nodes are alive but unable to communicate with each other
• A node is dead
• An HA resource failed to stop
The first cause can be avoided ensuring redundant communication paths and by
handling the multicast properly. This involves the whole network infrastructure,
and if you buy a network service from a vendor, you cannot expect safety or trust
and multicasts will not be well managed. The second cause is obvious, and it is
unlikely that the node causes the STONITH death match.
The third cause is not easy to understand. This can be clarified with an example.
Basically, an HA resource is started on a node. If it is started, the resource will be
monitored indefinitely; if the start fails, the resource will be started and stopped and
then restarted in either the current node or the second node. If the resource needs to
be stopped and the stop happens, the resource is restarted on the other node. Now if
the stop fails, the node will be fenced STONITH because it is considered as the safe
thing to do.
[ 90 ]
Chapter 3
If the HA resource can't be stopped and the node is fenced, the worse
action is killing the whole node. This can cause data corruption on your
node especially if there is ongoing transactional activity and this needs
to be avoided. It's less dangerous if the HA service is a resource such as
an HTTP server that provides web pages (without transactional activity
involved); however, this is not safe.
There are different ways to avoid the STONITH death match, but we want the
proposed design to be as easy as possible to implement, maintain, and manage,
and so the proposed architecture can live without the STONITH actor that can
introduce issues if not managed well and configured.
Pacemaker is distributed with STONITH enabled. STONITH is not
really necessary on a two-node cluster setup.
To disable STONITH, use the following command:
$ crm configure property stonith-enabled="false"
Pacemaker – is Quorum really needed?
Quorum refers to the concept of voting; it means each node can vote what can
happen. This is similar to democracy where the majority wins and implements
decisions. For example, if you have a three-node or more cluster and one of the
nodes on the pool fails, the majority can decide to fence the failed node.
With the Quorum configuration, you can also decide the no-Quorum-policy; this
policy can be used for following purposes:
• Ignore: No action is taken if a Quorum is lost
• Stop (default option): It stops all resources on the affected cluster node
• Freeze: It continues running all the existing resources but doesn't start the
stopped ones
• Suicide: It can fence all nodes on the affected partition
All these considerations are valid if you have a three-node or more configuration.
Quorum is enabled by default on most configurations, but this can't apply on the
two-node clusters because there is no majority to elect the winner and get a decision.
The following command needs to be disabled to apply the ignore rule:
$ crm configure property no-quorum-policy="ignore"
[ 91 ]
High Availability and Failover
Pacemaker – the stickiness concepts
It is obviously highly desirable to prevent healthy resources being moved around
the cluster. Moving a resource always requires a period of downtime that can't be
accepted for critical service (such as the RDBMS), especially if the resource is healthy.
To address this, Pacemaker introduces a parameter that expresses how much a service
prefers to stay running where it is actually located. This concept is called stickiness.
Every downtime has its cost, not necessarily represented by an expense that is tied to
the little downtime period needed to switch the resource to the other node.
Pacemaker doesn't calculate this cost associated with moving resources and will do
so to achieve the optimal resource placement.
On a two-node cluster, it is important to specify the stickiness; this will
simplify all the maintenance tasks. Pacemaker can't decide to switch
the resource to a maintenance node without disrupting the service.
Note that the Pacemaker's optimal resource placement does not always agree with
what you would want to choose. To avoid this movement of resources, you can
specify a different stickiness for every resource.
$ crm configure property default-resource-stickiness="100"
Is it possible to use INFINITY instead of a number on the stickiness
properties? This will force the cluster to stay on that node until it's dead,
and once the INFINITY node comes up, all will migrate back to the
primary node.
$ crm configure property default-resourcestickiness="INFINITY"
Pacemaker – the Apache/HTTPD configuration
The Pacemaker resource manager needs access to the Apache server's status to
know the status of HTTPD. To enable the server's status, you need to change the
httpd.conf file as follows:
<Location /server-status>
SetHandler server-status
Order deny,allow
Deny from all
Allow from
127.0.0.1 <YOUR-NETWOR-HERE>/24
</Location>
[ 92 ]
Chapter 3
For security reasons, it makes sense to deny the access to this virtual
location and permit only your network and the localhost (127.0.0.1).
This kind of configuration foresees two web servers that will be called www01 and
www02 to simplify the proposed example. Again, to keep the example as simple as
possible, you can consider the following addresses:
• www01 (eth0 192.168.1.50 eth1 10.0.0.50)
• www02 (eth0 192.168.1.51 eth1 10.0.0.51)
Now, the first step is to configure the virtual address using the following commands:
$ crm configure
crm(live)configure#
primitive vip ocf:heartbeat:IPaddr2 \
> params ip="10.0.0.100"
# not that 10.0.0.100 is the
> nic="eth1" \
> cidr_netmask="24" \
> op start interval="0s" timeout="50s" \
> op monitor interval="5s" timeout="20s" \
> op stop interval="0s" timeout="50s"
crm(live)configure#
show
# make sure
node www01.domain.example.com
node www02.domain.example.com
primitive vip ocf:heartbeat:IPaddr2 \
params ip="10.0.0.100" nic="eth1" cidr_netmask="24" \
op start interval="0s" timeout="50s" \
op monitor interval="5s" timeout="20s" \
op stop interval="0s" timeout="50s"
property $id="cib-bootstrap-options" \
dc-version="1.1.2-f059ec7cedada865805490b67ebf4a0b963bccfe" \
cluster-infrastructure="openais" \
expected-quorum-votes="2" \
[ 93 ]
High Availability and Failover
no-quorum-policy="ignore" \
stonith-enabled="false"
rsc_defaults $id="rsc-options" \
resource-stickiness="INFINITY" \
migration-threshold="1"
crm(live)configure# commit
crm(live)configure# exit
Using commit, you can enable the configuration. Now, to be sure that everything
went well, you can check the configuration using the following command:
$ crm_mon
You should get an output similar to the following one:
============
Last updated: Fri Jul 12 10:59:16 2013
Stack: openais
Current DC: www01.domain.example.com
- partition WITHOUT quorum
Version: 1.1.2-f059ec7cedada865805490b67ebf4a0b963bccfe
2 Nodes configured, unknown expected votes
1 Resources configured.
============
Online: [ www01.domain.example.com
vip
www02.domain.example.com
(ocf::heartbeat:IPaddr2):
]
Started www01.domain.example.com
To be sure that the VIP is up and running you can simply ping it
$ ping 10.0.0.100
PING 10.0.0.100 (10.0.0.100) 56(84) bytes of data.
64 bytes from 10.0.0.100: icmp_seq=1 ttl=64 time=0.012 ms
64 bytes from 10.0.0.100: icmp_seq=2 ttl=64 time=0.011 ms
64 bytes from 10.0.0.100: icmp_seq=3 ttl=64 time=0.008 ms
64 bytes from 10.0.0.100: icmp_seq=4 ttl=64 time=0.021 ms
[ 94 ]
Chapter 3
Now you have the VIP up and running. To configure Apache in the cluster, you need
to go back on the CRM configuration and tell Corosync that you will have a new
service, your HTTPD daemon, and will have to group it with the VIP. This group
will be called webserver.
This configuration will tie the VIP and the HTTPD and both will be up and running
on the same node. We will configure the VIP using the following commands:
$ crm configure
crm(live)configure# primitive httpd ocf:heartbeat:apache \
> params configfile="/etc/httpd/conf/httpd.conf" \
> port="80" \
> op start interval="0s" timeout="50s" \
> op monitor interval="5s" timeout="20s" \
> op stop interval="0s" timeout="50s"
crm(live)configure# group webserver vip httpd
crm(live)configure# commit
crm(live)configure# exit
Now, you can check your configuration using the following commands:
$ crm_mon
============
Last updated: Fri Jul 12 11:03:50 2013
Stack: openais
Current DC: www01.domain.example.com - partition WITHOUT quorum
Version: 1.1.2-f059ec7cedada865805490b67ebf4a0b963bccfe
2 Nodes configured, unknown expected votes
1 Resources configured.
============
Online: [ www01.domain.example.com www02.domain.example.com ]
Resource Group: webserver
vip
example.com
(ocf::heartbeat:IPaddr2):
Started www01.domain.
httpd
example.com
(ocf::heartbeat:apache):
Started www01.domain.
Note that since you are not using Quorum, make sure that the crm_mon
display: partition WITHOUT quorum and unknown expected
votes are normal.
[ 95 ]
High Availability and Failover
Configuring the Zabbix Server for
high availability
A high availability cluster for a Zabbix server is easier to configure compared to
Apache or a database server. Whether it's a standalone server or a node that is a
part of a distributed setup, the procedures are exactly the same.
Once you have installed Corosync and Pacemaker on the two nodes (see the
previous sections for details), you will also install Zabbix on the nodes that will make
the cluster. You will then need to configure Zabbix to listen to the virtual IP that you
have identified for the cluster. To do so, change both SourceIP and ListenIP to the
appropriate value in the zabbix_server.conf configuration file:
SourceIP=10.10.1.9
ListenIP=10.10.1.9
Needless to say, change the IP value to the one that is appropriate
for your environment and you have reserved a virtual IP for the
Zabbix cluster.
[ 96 ]
Chapter 3
You can now proceed to disable STONITH using the following command:
$ crm configure property stonith-enabled="false"
If you have just two nodes, you also need to disable the quorum, otherwise the
cluster wouldn't know how to obtain a majority:
$ crm configure property no-quorum-policy="ignore"
And finally, set the service stickiness high enough so that you don't have a service
going back and forth between the nodes and it stays where it is, unless you manually
move it or the active node goes down:
$ crm configure property default-resource-stickiness="100"
Much like the Apache/HTTPD cluster configuration, you now need to define a
primitive for the virtual IP:
$ crm configure primitive Zbxvip ocf:heartbeat:IPaddr2 \
params ip="10.10.1.9" iflabel="httpvip" \
op monitor interval="5"
For the Zabbix server, define primitive using the following command:
$ crm configure primitive Zabbix lsb::zabbix_server \
op monitor interval="5"
Just like the previous section, all that is now left to do is to group the primitives
together, set up colocation, service StartOrder, and you are done:
$ crm configure group Zbx_server Zbxvip Zabbix meta target-role="Started"
$ crm configure colocation Ip_Zabbix inf: Zbxvip Zabbix
$ crm configure order StartOrder inf: Zbxvip Zabbix
As you can see, the simpler the components, the easier it is to set them up in a cluster
configuration using Pacemaker. While it is still fairly easy and simple, things start
to change when you turn to configure the most critical part of any high availability
setup: database and data storage.
[ 97 ]
High Availability and Failover
Database high availability
Implementing high availability for a database is not an easy task. There are a lot
of ways to implement a high availability configuration using different software
and complexity.
The architecture proposed here is fully redundant; it is one of the possible solutions
that is widely used on large environments. You need two database servers to
implement this solution, and two installations of the same software and operating
system. Obviously, since servers are twins and are tied together, they need to have
the same software, the same release patch, and basically be identical.
Since we are going to have two different servers, it is clear that the data needs
to be replicated between them; this implies that your server needs to be
interconnected with a dedicated network connection that is capable of providing
the needed throughput.
In this design, your server can be placed on the same location or located at two
different data centers that provide a reliable disaster-recovery solution. In this case,
we are going to provide a highly available design.
There is a different way to provide data replication between two servers. They are
as follows:
•
•
•
•
•
•
•
File system replication
Shared disk failover
Hot/warm standby using PITR
Trigger-based master-standby replication
Statement-based replication middleware
Asynchronous multimaster replication
Synchronous master replication
There are positive and negative sides to each one of them. Between all these
options, we can exclude all the solutions that are trigger-based because all of
them introduce an overhead on the master node. Also, adding a user-lever
layer can be imprecise/inexact.
Between these options, there are few solutions that permit having a low or really
low mean time to recovery and are safe from data loss. The solutions that guarantee
that in case of master failure there will no data loss are as follows:
• Shared disk failover
• Filesystem replication
• Statement-based replication
[ 98 ]
Chapter 3
A solution that adopts a shared disk failover cluster implies the usage of a shared
san. This means that if we want to place your server on a separate server farm, on a
different location, this system will be really expensive.
If the solution adopts a warm and hot standby using a point-in-time recovery
(PITR) and your node goes down, you need enough free space to handle and store
all the transaction logfiles generated. This configuration by design needs a secondary
database (identical to the master node) that is a warm standby and waits for the
log transaction. Once arrived, the RDBMS needs to apply the transaction on your
secondary node.
In this case, if secondary node goes down, we need to be warned because the
primary database will produce the archive logfiles that are not shipped, and this can
bring your infrastructure to a halt. On a large environment, the transactional activity
is normally heavy, and if the fault happens to be out of the normal working hours,
this HA configuration needs to be handled.
Another way is the PostgreSQL synchronous replication. If the secondary node goes
down, this configuration would need a reload to prevent hanging of the transaction.
The trigger-based configurations are heavy and dangerous because they imply that
a trigger can go on firing every insert, and replicate the same insert on the secondary
node introducing a feasible overhead. Partitioning with inheritance is not well
supported by this method. Also, this method does not gives us warranty against data
loss when the master fails.
Infrastructures that include a second standby database introduce a second
actor, that is, if it is down or unreachable, it shouldn't cause a master to hang.
Nowadays, with PostgreSQL 9.1, a synchronous replication is a viable solution.
These configurations unfortunately add some constraints: the transmission
must be acknowledged before the commit happens, and the transmission
doesn't guarantee that you will have a reply.
This practically means that if the secondary node goes down, the primary database
will hang until the slave receives the transaction and notifies back to the master
that this one has been acquired. The result is that a primary node can hang for an
indefinite time and this practically doubles the risk of downtime.
An issue on slaves node shouldn't impact the primary node. This practically doubles
the risk of downtime and is not acceptable in the context of high availability.
[ 99 ]
High Availability and Failover
Clustering of PostgreSQL
The cluster presented here is simple and is designed to have as few actors in play
as possible, but with the high availability design in mind.
The architecture shown in the following diagram is efficient. It has a minimum
number of actors in play and is easy to monitor, maintain, and upgrade.
Mirrored logical volume with LVM and DRBD
LVM2 is the Linux implementation of the logical volume manager (LVM) on
the Linux logical device mapper framework. This LVM2, apart from the name,
doesn't have anything in common with the previous one.
The basic concepts of LVM2 are as follows:
• Physical Volume (PV): This is the actual physical partition or storage
system on which the LVM system is built.
[ 100 ]
Chapter 3
• Volume Group (VG): A volume group is the basic administrative unit. It may
include one or more PV. Every VG has a unique name and can be extended at
runtime by adding more additional PVs or enlarging the existing PV.
• Logical Volume (LV): This is available as a regular block device to the Linux
kernel and its components can be created at runtime within the available
volume groups. Logical volumes can be resized when online and also moved
from one PV to another PV if they are on the same Volume Group.
• Snapshot Logical Volume (SLV): This is a temporary point-in-time copy
of LVs. The strong point is that if the size is really big (several hundred
gigabytes), the space required is significantly less than the original volume.
The partition type Linux LVM that owns the signature 0x8E is used
exclusively for LVM partition. This however is not required. LVM
indeed recognizes the physical volume group by a signature written
on the PV initialization.
Since a logical volume once created is simply seen as a block device, you can use
DRBD on it.
[ 101 ]
High Availability and Failover
Prerequisite tasks to start with DRBD on LVM
While setting up DRBD over LVM, there are some basic steps to bear in mind:
• LVM needs to know about your DRBD devices
• LVM caching needs to be disabled
• Remember to update the initramfs with the new kernel device map
LVM by default scans all block device found on /dev looking for PV signatures;
hence, we need to set an appropriate filter on /etc/lvm/lvm.conf:
filter = ["a|sd.*|", "a|drbd.*|", "r|.*|"]
This filter accepts all the SCSI disk plus DRBD disks. Now, we need to rescan all
your volume groups with the following command:
# vgscan
It is important that you remember to disable the LVM caching because DRBD disks
will disappear in case of a failure. This is normal when we face a fault, and if caching
is not disabled, it is possible that you will see the disk is available when in reality it
is not.
This is done by adding the following line on /etc/lvm/lvm.conf:
write_cache_state = 0
Now that the cache has been disabled, it is possible that we still have some portion or
piece of cache on your disks that were previously generated. We need to clean up the
following location:
/etc/lvm/cache/.cache
Now, it's better to regenerate the kernel device map files with the following command:
# update-initramfs –u
Now, it is possible for us to go ahead with the configuration.
[ 102 ]
Chapter 3
Creating a DRBD device on top of the LVM partition
Now that your caching is disabled and the LVM is properly configured, we need
to create your PV. To initialize your SCSI partitions as physical volume, we run the
following commands from the root:
$ pvcreate /dev/sda1
Physical volume "/dev/sda1" successfully created
$pvcreate /dev/sdb1
Physical volume "/dev/sdb1" successfully created
The given output tells us that the volume has been initialized. Now you can create a
low-level VG, vgpgdata:
$ vgcreate vgpgdata /dev/sda1 /dev/sda2
Volume group "vgpgdata" successfully created
Finally, you can create your volume or a better logical volume that will be used as
the DRBD's block device:
$ lvcreate --name rpgdata0 --size 10G local
Logical volume "rpgdata0" created
All these steps need to be repeated in the same order on both your nodes. Now you
need to install DRBD on both nodes using the following command:
$ yum install drbd kmod-drbd
To install the DRBD, it is important to have the EXTRAS
repositories enabled.
Now edit the drbd.conf file located at /etc/drbd.conf and create the rpgdata0
resource as follows:
resource rpgdata0 {
device /dev/drbd0;
disk /dev/local/rpgdata0;
meta-disk internal;
on <host1> { address <address_host1>:<port>; }
on <host2> { address <address_host2>:<port>; }
}
[ 103 ]
High Availability and Failover
Replace host1, host2, address_host1, and address_host2
with the two hostnames and their respective network addresses.
Be sure to have copied the drbd.conf file on both nodes before proceeding
with the next section. Disable automatic start for DRBD because it will be
managed by Pacemaker.
$ chkconfig drbd off
Enabling resource in DRBD
Now that the configuration file has been copied on all your nodes, you can finally
initialize the device and create the required metadata.
This initialization process needs to be executed on both nodes and can be run from
the root using the following command:
$ drbdadm create-md rpgdata0
v08 Magic number not found
Writing meta data...
initialising activity log
NOT initializing bitmap
New drbd meta data block successfully created.
This is the initialization process and needs to be
executed only on a new device.
Now you can enable the rpgdata0 resource:
$ drbdadm up rpgdata0
The process can be observed by looking at the /proc virtual filesystem:
$ tail /proc/drbd
version: 8.4.1 (api:1/proto:86-100)
GIT-hash: 91b4c048c1a0e06837625f65d312b38d47abara80 build by [email protected]
linbit, 2013-02-20 12:58:48
0: cs:Connected ro:Secondary/Secondary ds:Inconsistent/Inconsistent C
r----ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b
oos:524236
[ 104 ]
Chapter 3
The Inconsistent/Inconsistent state here at this point is
normal. You need to specify which node is the master and which
will be the source of this synchronization.
At this point, DRBD has allocated disk and network and is ready to begin
the synchronization.
Defining a primary device in DRBD
The primary promotion is quite easy; you need go to the primary node and run
this command:
$ drbdadm primary rpgdata0
Now the server on which you run this command becomes the master of the
replication server, and you can create the PV on that new device. So, on the
master node, you need to run the following command:
$ pvcreate /dev/drbd0
Physical volume "/dev/drbd0" successfully created
Create your VG, which in this example will be secured_vg_pg:
$ vgcreate secured_vg_pg /dev/drbd0
Volume group "secured_vg_pg" successfully created
Finally, it is possible to create an LV on that PV using the following command:
$ lvcreate -L 6G -n secured_lv_pg secured_vg_pg
In this example, we reserved a space for snapshots; so, if you ever want one,
you have enough space for that. Finally, it is possible to set up the filesystem.
Creating a filesystem on a DRBD device
Now it is important to check whether the DRBD service is disabled from the startup
and shutdown list because this service will be managed directly from Pacemaker.
Once you disable the service, it is possible to create the filesystem on the new device,
but before that it is important to do the following:
• Create a mountpoint
• Create a filesystem
• Mount the filesystem and make it available
[ 105 ]
High Availability and Failover
You can create your own mountpoint, but this step-by-step installation will
use /db/pgdata:
$ mkdir -p -m 0700 /db/pgdata
Now, there are different filesystems supported by most of the distributions;
RedHat 6.0 completely supports XFS. XFS has an important feature that permits
parallel access to filesystem. It supports parallel read/write. XFS allows us to
write the same files from multiple threads concurrently; this obviously is a
big improvement for large database tables and it also reduces the contention
on filesystems.
To install XFS and the relative utils, use the following command:
$ yum install xfsprogs
XFS allows write access to the same file for multiple threads
concurrently; this is interesting especially on DRBD usage where
the contention on filesystems becomes an important factor.
Once installed and available, you can format the logical volume using the
following command:
$ mkfs.xfs /dev/secured_vg_pg/secured_lv_pg
Once created, the filesystem can't be reduced but
only enlarged using the xfs_growfs command.
Now you can mount the filesystem using the following command:
$ mount -t xfs -o noatime,nodiratime,attr2 /dev/secured_vg_pg/secured_lv_
pg /db/pgdata
Do not forget to add this partition on automount (fstab),
otherwise you will lose your partition after a reboot.
All can be changed to your PostgreSQL process owner, usually postgres:
$ chown postgres:postgres /db/pgdata
$ chmod 0700 /db/pgdata
The filesystem creation steps need to
be done only on the primary node.
Now the filesystem is mounted, formatted, and ready for PostgreSQL.
[ 106 ]
Chapter 3
Pacemaker cluster – integrating DRBD
Pacemaker makes DRBD extremely powerful in a really wide variety of scenarios.
There are some notable points that have already been discussed when we presented
Pacemaker/Corosync. These points are as follows:
• Disable STONITH
• Disable Quorum
• Enable stickiness
As discussed earlier in this chapter, it is really important to avoid split-brain
scenarios and STONITH death match. Just as a reminder, to disable STONITH,
you can run the following command:
$ crm configure property stonith-enabled="false"
Since this again is a two-node cluster, it is strongly advised to disable Quorum.
The command that permits us to do this is as follows:
$ crm configure property no-quorum-policy="ignore"
Now, it is preferred to enable stickiness. This argument has been discussed earlier
in the chapter. Anyway, as a quick reminder, we can say that by enabling the
stickiness, we have a guarantee of a preferred node over another. This will help
you to keep your cluster on one side and have a preferred site where everything
should run. The command for this is as follows:
$ crm configure property default-resource-stickiness="100"
Enabling the DRBD configuration
This section explains how to enable the DRBD-backend service in your Pacemaker
cluster. There are some steps to be followed:
• Add DRBD to Pacemaker
• Add and define the master/slave resource
You need to have a master/slave resource that controls which node is primary and
which one is secondary. This can be done with the following command:
$ crm configure primitive drbd_pg ocf:linbit:drbd \
params drbd_resource="rpgdata0" \
op monitor interval="15" \
op start interval="0" timeout="240" \
op stop interval="0" timeout="120"
[ 107 ]
High Availability and Failover
Once done, you need to set up a resource that can promote or demote the DRBD
service on each node. Keep in mind that the service needs to run on both the nodes at
all times with a different state, defining a master/slave resource as follows:
$ crm configure ms ms_drbd_pg drbd_pg \
meta master-max="1" master-node-max="1" clone-max="2" \
clone-node-max="1" notify="true"
Pacemaker – the LVM configuration
Now you need to configure Pacemaker to:
• Manage the LVM
• Manage the filesystem
Because of the design and working of DRBD, the actual active volume will be
invisible on the secondary node. On the secondary node, you can't mount or handle
this volume. Having said that, you need to help DRBD to find devices:
$ crm configure primitive pg_lvm ocf:heartbeat:LVM \
params volgrpname="secured_vg_pg" \
op start interval="0" timeout="30" \
op stop interval="0" timeout="30"
With the preceding configuration, Pacemaker will search for usable volume on
DRBD devices and will be available using the DRBD resource promotion. Since the
adopted filesystem on DRBD device is XFS, you need to define how to mount and
handle this device:
$ crm configure primitive pg_fs ocf:heartbeat:Filesystem \
params device="/dev/secured_vg_pg/secured_lv_pg" directory="/db/pgdata" \
options="noatime,nodiratime" fstype="xfs" \
op start interval="0" timeout="60" \
op stop interval="0" timeout="120"
Since LVM is the last layer on this configuration, you can take advantage of snapshot
capabilities and a good level of isolation
[ 108 ]
Chapter 3
Pacemaker – configuring PostgreSQL
Now you can add the PostgreSQL configuration to the cluster.
PostgreSQL installation is not covered here because it is already
discussed in Chapter 1, Deploying Zabbix.
The following lines add a primitive to the Pacemaker that will set a PostgreSQL
health check every 30 seconds and define a timeout of 60 seconds to retrieve
the response:
$ crm configure primitive pg_lsb lsb:postgresql \
op monitor interval="30" timeout="60" \
op start interval="0" timeout="60" \
op stop interval="0" timeout="60"
This command extends the start and stop timeout because it will
handle quite big databases. It can also happen that a Pacemaker may
need to give the time to complete a checkpoint on shutdown and a
recovery on startup.
Pacemaker uses those parameters in a primary manner to determine if PostgreSQL
is available or not.
Pacemaker – the network configuration
Up until now, you haven't yet configured a predefined IP address for PostgreSQL.
Since it doesn't make sense to have different addresses in case of a switchover or
failover, you need to set up a virtual IP that will follow your service. This prevents
any change of configuration for all your clients. You can use a cluster name or an IP
address. For that, you need to issue the following line:
$ crm configure primitive pg_vip ocf:heartbeat:IPaddr2 \
params ip="192.168.124.3" iflabel="pgvip" \
op monitor interval="5"
NOTE: change the address 192.168.124.3 with your own.
Here it is not specified that the ARP address, IPaddr2, will automatically send five
ARP packets, and this value can be increased if necessary.
[ 109 ]
High Availability and Failover
Pacemaker – the final configuration
Now you have all the required components ready to be tied together in a group that
will contain all your resources. The group is PGServer:
$ crm configure group PGServer pg_lvm pg_fs pg_lsb pg_vip
$ crm configure colocation col_pg_drbd inf: PGServer ms_drbd_pg:Master
The Master server specifies that your PGServer group depends on the master/slave
setup reporting a master status that happens exclusively on an active mode. It is also
true that the PGServer group depends on the DRBD master.
Now, it is important to specify the right order to start and shutdown all the services.
We will use the following command to do so:
$ crm configure order ord_pg inf: ms_drbd_pg:promote PGServer:start
The :promote and :start are fundamentals, they mean that once
ms_drbd_pg is promoted, the PGServer will start. With this precise order
of events, if you omit the :start, Pacemaker can chose the start/stop
order, and it might end in a broken state.
Cluster configuration – the final test
Finally, the cluster is ready! What to do next? It is simple! You can break your own
cluster, play with the configuration, and check that all is fine before we go live with
this new infrastructure.
Faults that need to be tested are as follows:
• The node goes offline
• Manual failover of the cluster
• Primary crash
• Secondary crash
• Forceful synchronization of all the data
Run the following command:
$ crm node standby HA-node2
If all is fine, crm_mon will respond with the following:
Node HA-node2: standby
Online: [ HA-node1 ]
[ 110 ]
Chapter 3
You can easily fix this state by firing the following command:
$ crm node online HA-node2
Until here it is quite easy. Now you can try a failover of the whole cluster using the
following command:
$ crm resource migrate PGServer HA-node2
You can migrate PGServer to the second node. If that becomes
unavailable, Pacemaker will move to the primary node until the
secondary return. This is because the migrate command will
give a higher score to the named node, and this will win on your
specified stickiness.
The server can be migrated back with the following:
$ crm resource unmigrate PGServer
Now, you can switch off the secondary node and Pacemaker will respond with
the following:
Online: [ HA-node1 ]
OFFLINE: [ HA-node2 ]
Master/Slave Set: ms_drbd_pg [drbd_pg]
Masters: [ HA-node1 ]
Stopped: [ drbd_pg:1 ]
After that, you can start up the secondary node again. Now, switch off the secondary
node and Pacemaker will respond with the following:
Online: [ HA-node1 HA-node2 ]
Master/Slave Set: ms_drbd_pg [drbd_pg]
Masters: [ HA-node1 ]
Slaves: [ HA-node2 ]
Now as a final test, you can invalidate all the data on the secondary node with the
following command:
$ drbdadm invalidate-remote all
Or from the secondary node, you can run the following command:
$ drbdadm invalidate all
This will force DRBD to consider all the data on the secondary node out of sync.
Therefore, DRBD will resync all the data on the secondary node before getting it
from the primary node.
[ 111 ]
High Availability and Failover
DRBD performance and optimizations
There are some aspects that can be improved and that should be considered when
you implement a DRBD cluster. There are some optimizations that can be applied.
You need to consider that if your database, or more in general, the second node of
the DRBD cluster is on a different location that is far away from your datacenter,
the network bandwidth can have an efficient synchronization, which plays a
fundamental role. Another thing that needs to be considered on a disaster recovery
site is the bandwidth and its cost. It is also important to calculate and understand
how much data is required and the transfer rate that we can reach or need.
Efficient synchronization by DRBD
Synchronization is a distinct process and can't be considered like device replication.
While replication happens only the first time you start up the device, synchronization,
and resynchronization as well, is decoupled from incoming writes. On the proposed
architecture, synchronization is necessary when:
• The link has been interrupted
• The Server had a fault on the primary node
• The Server had a fault on the secondary node
DRBD doesn't synchronize blocks sequentially and not in the order they were
originally written.
While synchronization is on-going, during the process, on the disk
you will have partly obsolete data and partly updated data.
The service will continue to run on the primary node while background
synchronization is in progress. Since this configuration has a LVM layer on top of
DRBD, it is possible to use snapshots during the synchronization; this is a strong
point of this architecture. While synchronization is ongoing, you are in a delicate
phase because there is a single point of failure; only the primary is working fine,
and if something happens here, you might completely lose all the data and the
secondary node may contain bad data. This critical situation can be mitigated
with the LVM snapshot.
Use of snapshot before beginning the synchronization can give you
hands-on experience of that situation because data on secondary node
are consistent, valid but not recently updated. Enabling snapshot before
beginning synchronization will reduce the Estimate Time to Repair
(ETR) also known as the Recovery Time Objective (RTO).
[ 112 ]
Chapter 3
To automate the snapshot, you can add the following lines on your
DRBD configuration:
resource RESOURCE_NAME {
handlers {
before-resync-target "/usr/lib/drbd/snapshot-resync-target-lvm.
sh";
after-resync-target "/usr/lib/drbd/unsnapshot-resync-target-lvm.
sh";
}
}
The snapshot-resync-target-lvm.sh script is called before we begin the
synchronization and the unsnapshot-resync-target-lvm.sh script will
remove the snapshot once synchronization completes.
If the script fails, the synchronization
will not commence.
To optimize the synchronization DRBD support, a checksum-based synchronization
is required. A checksum-based synchronization is more efficient in the sense that a
brute force overwrites and blocks synchronization; this is not enabled by default.
With these features enabled, DRBD reads blocks before synchronizing them and
calculating a hash of the contents. It compares the hash calculated with the same data
obtained from the same sector on the out-of-sync secondary, and if hash matches,
DRBD omits to rewrite these blocks.
To enable this feature, you need to add the following lines on the DRBD configuration:
resource <resource>
net {
csums-alg <algorithm>;
}
...
}
The <algorithm> tag is any message digest supported by the kernel crypto API,
usually one among sha1, md5, and crc32c.
If this change is done on an existing resource, you need to copy the
changed drbd.conf file on the secondary client and afterwards run:
drbdadm adjust <resource>
[ 113 ]
High Availability and Failover
Enabling online verification through DRDB
Online verification enables a block-by-block data integrity check in a very efficient
way. This is particularly interesting for the efficiency on bandwidth usage plus it
doesn't interrupt or beak redundancy in any way.
Online verification is a CPU-intensive process;
it will impact the CPU load.
DRBD with this functionality will calculate a cryptographic digest of every block on
the first node, and then this hash is sent to the peer node that will do the same check.
If the digest differs, the block will be marked out-of-sync and DRBD will retransmit
only the marked blocks. This feature is not enabled by default and can be enabled by
adding the following lines in drbd.conf:
resource <resource>
net {
verify-alg <algorithm>;
}
...
}
Also, here <algorithm> can be any digest supported by crypto API usually by sha1,
md5, and crc32c. Once configured, it is possible to run the online verification with
the following command:
$ drbdadm verify <resource>
Since the check introduced will ensure that both nodes are
perfectly in sync, it is advised to schedule a weekly or a
monthly check within crontab.
If you have an out-of-sync block, it is possible to resync them simply with the
following command:
$ drbdadm disconnect <resource>
$ drbdadm connect <resource>
[ 114 ]
Chapter 3
DRBD – some networking consideration
When you use a block-based filesystem over a DRBD, it is possible to improve the
transfer rate, enlarging the Maximum Transmission Unit (MTU) to higher values.
Block-based filesystem will have a noticeable improvement. Block-based filesystems
are ext3, ReiserFS, and GFS. The filesystem proposed here on this architecture is
extent-based and is not expected to see high improvement enabling the jumbo frame.
DRBD permits us to set up the synchronization rate. Normally, DRBD will try to
synchronize the data on the secondary node as quickly as possible to reduce the
inconsistent data time. Anyway, you need to prevent degrading of a performance
that is caused by the bandwidth consumed for the synchronization.
Make sure you set up this parameter in relation to the bandwidth
available; for instance, it doesn't make any sense to set up a rate
that is higher than the maximum throughput.
The maximum bandwidth used from the background process of resynchronization is
limited by a parameter: rate expressed in bytes; so, 8192 means 8 MiB. To fix the rate,
you can change the DRBD configuration file by adding in the following code:
resource <resource>
disk {
resync-rate 50M;
...
}
...
}
The rule to calculate the right rate and the resync rate is
MAX_ALLOWED_BANDWIDTH * 0.3. It means we are going
to use 30 percent of the maximum bandwidth available.
The sync rate follows exactly the same rule and can be specified as well on the
drbd.conf file:
resource <resource>
syncer {
rate 50M;
...
}
...
}
[ 115 ]
High Availability and Failover
The syncer rate can be temporarily modified with the
following command:
drbdsetup /dev/drbdnum syncer -r 120M
The resync rate can be temporarily changed with the
following command:
drbdadm disk-options --resync-rate=110M <resource>
Both these rates can be reverted with the following command:
drbdadm adjust resource
DRBD give us other interesting parameters to finetune the system and optimize
performances; of course, those that follow are not solutions to all the throughput
issues. They can vary from system to system but it is useful to know that they exist
and you can get some benefit from them.
In particular, there are two parameters. They are as follows:
• max-buffers
• max-epoch-time
The first properties (max-buffers) represent the maximum number of buffer DRBD.
The second properties (max-epoch-time) represent the maximum number of write
requests permitted between two write barriers. Both can be changed inside the
drbd.conf file:
resource <resource> {
net {
max-buffers 8000;
max-epoch-size 8000;
...
}
...
}
The default value for both is 2048, but they both can be
changed to 8000. This is a reasonable value for most of the
modern raid-SCSI controllers.
[ 116 ]
Chapter 3
There is another network optimization that can be done. Change the send buffer of
the TCP/IP. By default, this value is set to 128 K, but if you are in a high-throughput
network, such as a gigabit network, it make sense to increase this value to 512 K.
resource <resource> {
net {
sndbuf-size 512K;
...
}
...
}
If you set these properties to 0, the DRBD will use the autotuning
feature, adapting the TCP to send buffer to the network.
To close this optimization section, it is important to say that DRBD manages some
other parameters:
• no-disk-barrier
• no-disk-flushes
• no-disk-drain
My personal advice is stay away from them if you don't really know what kind of
hardware you have. Set them to represent a "big iron" on the system raid. These
parameters disable the write barriers, the disk flush, and drain. Usually, all these
features are managed directly from the controller. It doesn't make any sense to
enable DRBD to manage them.
[ 117 ]
High Availability and Failover
Summary
In this chapter, you have learned some fundamental concepts about high availability
and service clustering. You have also learned how to apply them to the Zabbix server
architecture using the open source Pacemaker service manager suite and the filesystem
replication with DRBD. You should also have learned to value keeping things light and
simple by choosing as few nodes as possible while maintaining a robust, redundant
architecture. This completes the first part of the book that was focused on choosing
the optimal Zabbix solution for an environment of any size. By choosing the right
hardware and supporting software (refer to Chapter 1, Deploying Zabbix and Chapter 2,
Distributed Monitoring) and high availability for the most sensitive components,
you should now have a Zabbix installed that is perfectly tailored to your needs
and environment.
In the rest of the book, we will focus on using this setup to actually monitor your
network and servers and make use of the collected data beyond simple alerts.
The next chapter will focus on data collection and using many of Zabbix's built-in
item types to obtain monitoring data from a number of simple or complex or
aggregated sources.
[ 118 ]
Collecting Data
Now that you have a Zabbix installation that is properly sized for your environment,
you will want to actually start monitoring it. While it's quite easy to identify which
hosts and appliances, physical or otherwise, you may want to monitor, it may not be
immediately clear what actual measurements you should take on them. The metrics
you can define on a host are called items, and this chapter will discuss their key
features and characteristics. The first part will be more theoretical, and will focus on
the following:
• Items as metrics, not status checks
• Data flow and directionality for items
• Trapper items as a means to control the data flow
We will then move to a more practical and specific approach and will discuss how to
configure items to gather data from the following data sources:
• Databases and ODBC sources
• Java applications and the JMX console
• SNMP agents
• Web pages
Collecting Data
Gathering items as raw data
One of the most important features that sets Zabbix apart from most other
monitoring solutions is that its main mode of interaction with the monitored
objects is focused on gathering raw data, as opposed to alerts or status updates.
In other words, many monitoring applications have the workflow (or variation)
as shown in the following diagram:
Data measurement
trigger logic
status sent to server
event logging, actions, ...
That is, an agent or any other monitoring probe is asked to not only take a
measurement, but also incorporate some kind of status decision about the
said measurement before sending it to the main server's component for
further processing.
On the other hand, the basic Zabbix workflow is subtly but crucially different,
as shown in the following diagram:
Data measurement
data sent to server
trigger logic
event logging, actions, ...
Here, an agent or monitoring probe is tasked with just the measurement part,
and then it sends the said measurement to the server component for storage,
and eventually for further processing.
The data is not associated to a specific trigger decision (Pass/Fail, OK/Warning/
Error, or any other variation), but is kept on the server as a single data point or
measurement. Where applicable, that is, for numeric types, it's also kept in an
aggregate and trending format as min, max, and average, over different periods
of time. Keeping data separated from the decision logic, but all in a single place,
gives Zabbix two distinct advantages.
[ 120 ]
Chapter 4
The first one is that you can use Zabbix to gather data on things that are not directly
related to the possible alerts and actions that you have to take, but related to the overall
performance and behavior of a system. The classic example is that of a switch with
many ports. You may not be interested in being alerted about anomalous traffic on
every single port (as it may also be difficult to exactly define anomalous traffic on a
single port with no contextual information), but you may be interested in gathering
both port-level and switch-level traffic measurement in order to establish a baseline,
evaluate possible bottlenecks, or plan for an expansion of your network infrastructure.
Similar cases can be made about the CPU and core usage, storage capacity, number of
concurrent users on a given application, and many more. At its simplest, Zabbix could
even be used to gather the usage data, and visualize it in different graphs and plots,
without even touching its powerful trigger and correlation features, and still prove to
be an excellent investment of your time and resources.
Speaking of triggering, the second big advantage of having a full, central database
of raw data as opposed to a single measurement (or at best, just a handful of
measurements of the same item), is that for every trigger and decision logic need that
you may have, you can leverage the whole measurements database to exactly define
the kind of event that you want to monitor and be alerted on. You don't need to rely
on a single measurement; you don't even need to rely on the latest measurement
plus a few of the previous ones of the same item, or limit yourself to items from the
same host. In fact, you can correlate anything with anything else in your item history
database. This is a feature so powerful that we have dedicated an entire chapter to
it, and you can go directly to Chapter 6, Managing Alerts, if that's what you want to
read about. It would suffice to say that all this power is based on the fact that Zabbix
completely separates its data-gathering functions from its trigger logic and action
functions. All of this is based on the fact that measurements are just measurements,
and nothing else.
So, in Zabbix, an item represents a single metric, a single source of data,
and measurements. There are many kinds of native Zabbix items, even without
considering the custom ones that you can define using external scripts. In this
chapter, you will learn about some of the less obvious but very interesting ones.
You will see how to deal with databases, how to integrate something as alien as
SNMP traps to the Zabbix mindset, how to aggregate the existing items together
to represent and monitor clusters, and more. As you lay a solid foundation,
with sensible and strategic item definition and data gathering, you will be able to
confidently rely on it to develop your event management and data visualization
functions, as you will see in the following chapters.
[ 121 ]
Collecting Data
Understanding the data flow for
Zabbix items
A Zabbix item can be understood by its bare essentials—an identifier, data type,
and associated host. These are the elements that are generally more useful for the
rest of Zabbix's components. The identifier (that's usually the name and the
associated item key) and the associated host are used to distinguish a single item
among the thousands that can be defined in a monitoring environment. The data
type is important so that Zabbix knows how to store the data, how to visualize
it (text data won't be graphed, for example), and most importantly, what kind of
functions can be applied to it in order to model triggers and the further processing.
The item's name is a descriptive label that is meant to be easily read,
while the item's key follows a specific syntax and defines exactly the
metric that we want to measure.
Two other very important elements that are common for all the items are the
history (and trends) retention period and item type. We already saw in Chapter 1,
Deploying Zabbix, how history retention directly affects the size of the monitoring
database, how to estimate it, and how to strike a balance between the performance
and data availability. On the other hand, the item type is essential as it tells Zabbix
how the item data is actually going to be made available to the server, in other
words, how is Zabbix going to collect the data: through an agent, an SNMP query,
an external script, and so on.
As you probably already know, there's a fair number of different item types. While it's
fairly easy to understand the difference between an SSH item and an ODBC one,
it's also important to understand how the data is passed around between the server
and its probes, and whether they are a Zabbix agent, a server-side probe, or an external
check of some kind. To this end, we'll first concentrate on the Zabbix agent and the
difference between a passive and active item.
First of all, the active and passive concepts have to be understood from the agent's
point of view, and not the server's. Furthermore, they serve to illustrate the
component that initiates a connection in order to send or receive configuration
information and monitor data.
[ 122 ]
Chapter 4
Zabbix Item (passive)
item definition
monitoring
interval
timer
data request
data response
Zabbix
server
Zabbix
agent
So, a standard Zabbix item is considered passive from the agent's point of view.
This means that it's the server's job to ask the agent, at the time intervals defined
for the item, to get the desired measurement and report it back immediately.
In terms of network operations, a single connection is initiated and brought down
by the server, while the agent is in the listening mode.
On the other end, in the case of a Zabbix active item, it's the agent's job to ask the
server what monitoring data it should gather and at what intervals. It then proceeds
to schedule its own measurements, and connects back to the server to send them
over for further processing. In terms of network operations, the following are the two
separate sessions involved in the process:
• The agent asks the server about items and monitoring intervals
• The agent sends the monitoring data it collected to the server
Unlike standard passive items, you'll need to configure an agent so that it knows
which server it should connect to for the purpose of configuration and data
exchange. This is of course defined in the zabbix_agentd.conf file for every
agent; just set ServerActive as the hostname or IP address of your Zabbix server,
and RefreshActiveChecks to the number of seconds the agent should wait before
checking if there are any new or updated active item definitions.
Zabbix Item (active)
configuration request
item definition
configuration response
item measurements
data receive acknowledge
Zabbix
server
monitoring
interval
timer
Zabbix
agent
[ 123 ]
Collecting Data
Apart from the network connection initiation, the main difference between a passive
and active item is that in the latter, it's impossible to define flexible monitoring
intervals. With a passive item, you can define different monitoring intervals based
on the time of the day and the day of the week. For example, you could check the
availability of an identity management server every minute during office hours and
every ten minutes during the night. On the other hand, if you use an active item,
you are stuck with just one option for monitoring the intervals.
You may also have noticed a more-than-passing resemblance between the Zabbix
active and passive items, and the functionality and features of a Zabbix active and
passive proxy.
In fact, you can choose between the active and passive items in much the same
way and for the same reasons as you chose between an active or passive proxy
in Chapter 2, Distributed Monitoring, to offload some of the server's scheduling job
and to work around restrictions and limitations of your network and routing or
firewall configuration.
There is, of course, one main difference between proxies and agents. It's not the
fact that a proxy can gather monitoring data from many different hosts, while an
agent is theoretically limited to monitoring just the host it's installed on.
The main difference when it comes to data flow is that the mode of operation of a
proxy is applied to all the hosts and items that the proxy manages. In fact, it doesn't
care about the nature of the items a proxy has to monitor. However, when an active
proxy gathers its data (whether with active or passive agent items, external scripts,
SNMP, SSH, and so on), it will always initiate all connections to the server. The same
goes for a passive proxy; it doesn't matter if all the items it has to monitor are all
active agent items; it will always wait for the main server for updates on configuration
and measurement requests.
On the other hand, an active or passive item is just one of the many other items.
A host can be defined by a mix of active and passive items; so you can't assume that
an agent will always initiate all its connections to the server; to do that, all of the
items that rely on the agent have to be defined as active, including the future ones.
[ 124 ]
Chapter 4
Understanding Zabbix trapper items
An extreme version of an active item that still relies on the Zabbix agent protocol
is the Zabbix trapper item. Unique among all other item types, a trapper item does
not have a monitoring interval defined at the server level. In other words, a server
will know if a Zabbix trapper item is defined, its data type, the host it's associated
with, and the retention period for both history and trends. But it will never schedule
a check for the item nor pass the scheduling and monitoring interval information to
any proxy or agent. So it's up to the specific probe to be scheduled in some way,
and then send the information about the gathered data to the server.
Trapper items are, in some respect, the opposite of Zabbix's external checks from
a data flow's point of view. As you probably already know, you define an external
check item type when you want the server to execute an external script to gather
measurements, instead of asking an agent (Zabbix, SNMP, or others). This can exact
an unexpected toll on the server's performance as it has to fork a new process for
every external script it has to execute, and then wait for the response. As the number of
external scripts grows, it can significantly slow down the server operations to the point
of accumulating a great number of overdue checks while it's busy executing scripts.
An extremely simple and primitive, yet effective way to work around this problem
(after reducing the number of external scripts as much as possible, of course) is to
convert all external check items to trapper items, schedule the execution of the same
scripts used in the external checks through the crontab or any other scheduling facility,
and modify the scripts themselves so that they use zabbix_sender to communicate
the measured data to the server. When we talk about the Zabbix protocol in Chapter 8,
Handling External Scripts, you'll see quite a few examples of this setup.
The data flow overview
This is a rundown of item types, classified with the connection type, with a
proposed alternative in case you want, for any reason, to turn it around. As you
can see, a Zabbix Trapper is often the only possible, albeit clunky or clumsy
alternative, if you absolutely need to reverse a connection type. Note that in the
following table, the term Passive means that the connection is initiated by the
server, and Active means that the connection is initiated by whatever probe is
used. While this may seem counter-intuitive, it's in fact coherent with the same
terms as applied to proxies and agents.
[ 125 ]
Collecting Data
Item Type
Direction
Alternative
Zabbix agent
Passive
Zabbix agent (active)
Zabbix agent (active)
Active
Zabbix agent
Simple check
Passive
Zabbix trapper
SNMP agent
Passive
Zabbix trapper
SNMP trap
Active
N/A
Zabbix internal
N/A (data about the server
monitoring itself)
N/A (SNMP traps are
completely different
in nature)
Zabbix trapper
Active
Depends on the nature of the
monitored data
Zabbix aggregate
N/A (uses data already
available in the database)
N/A
External check
Passive
Zabbix trapper
Database monitor
Passive
Zabbix trapper
IPMI agent
Passive
Zabbix trapper
SSH agent
Passive
Zabbix trapper
TELNET agent
Passive
Zabbix trapper
JMX agent
Passive
Zabbix trapper
Calculated
N/A (uses data already in
the database)
N/A
In the next few paragraphs, we'll delve more deeply into some of the more complex
or interesting item types.
Database monitoring with Zabbix
Zabbix offers a way to query any database using SQL queries. The result retrieved
from the database is saved as the item value, and can have, as usual, triggers
associated with it. This functionality is useful in many applications. This gives you
a way to monitor the user currently connected to a database, or the number of users
connected to your web portal, or simply retrieve metrics from the DBMS engine.
[ 126 ]
Chapter 4
Delving into ODBC
ODBC is a layer, a translation layer between database management systems
(DBMS) and the application. Originally, it had been developed by Microsoft as
a standard middleware for C programming language's API to handle the access
to the DBMS. The applications use the ODBC function through the linked ODBC
driver manager. This is basically the driver that passes the queries to the DBMS and
dialogues to the ODBC. The ODBC layer is based on the ODBC device driver model,
which basically encapsulates the logic needed to convert the standard command and
function into specific calls. The ODBC driver has been implemented and developed
from most of the DBMS vendors, to enable their database to interoperate with this
layer. Most of the drivers don't implement all the functions defined in the standard,
because not all of them are supported by all DBMS. The ODBC driver provides the
application with a data source to interoperate with the database. The configuration
file specifies the driver to load all the connections' parameters for each Data Source
Name (DSN), and all the DSNs are enumerated and defined inside this file. DSN
also gives the functionality to present the entire database in a human-readable
format. The DSN file needs to be protected. In the proposed setup, it is advisable to
use a different Unix account for your Zabbix server, which will make things easy.
As there is only one Zabbix server, the only user that needs to access this file is the
Zabbix server user. This file should be owned by this user and be made unreadable
to others. DSNs are contained in the odbc.ini file in the ect folder This file will
contain all the DSNs for all the different databases to which we want to connect.
Take care to protect this file, and prevent other people from accessing this file
because it can contain passwords.
There are two open source versions of ODBC available—UnixODBC and iODBC.
Zabbix can use both of them, but before you can use it, the first thing to do is enable
Zabbix to use ODBC and install the UnixODBC layer. There are two ways to do that:
one is with the package manager, and the other one is to go through the old way of
downloading and compiling it from the source (currently, the latest stable version
is 2.3.1):
$ wget ftp://ftp.unixodbc.org/pub/unixODBC/unixODBC-2.3.1.tar.gz
$ tar zxvf unixODBC-2.3.1.tar.gz
$ cd unixODBC-2.3.1
$ ./configure --prefix=/usr --sysconfdir=/etc
$ make
$ make install
[ 127 ]
Collecting Data
If you are on a 64-bit system, you have to specify the 64-bit version of
libraries with --libdir as follows:
./configure --prefix=/usr --sysconfdir=/etc --libdir=/
usr/lib64
The default locations are: /usr/bin for binary, and /usr/lib or /usr/lib64 for
libraries depending on the version you installed.
Installing database drivers
UnixODBC supports a quite wide and almost complete list of databases. Most of the
following widely diffused databases are supported:
•
•
•
•
•
•
MySQL
PostgreSQL
Oracle
DB2
Sybase
Microsoft SQL Server (via FreeTDS)
The complete list of databases supported by UnixODBC is available at
http://www.unixodbc.org/drivers.html.
MySQL ODBC drivers
Now if you previously installed the UnixODBC via the package manager, you can
follow the same procedure, for example, on RedHat with the following command:
$ yum install mysql-connector-odbc
Otherwise, they are available also as a packet; you only need to download the
package, for example, mysql-connector-odbc-5.1.12-linux-glibc2.5-x8664bit.tar.gz.
Then decompress the package and copy the contents in the /usr/lib/odbc and /
usr/lib64/odbc/ directories as follows:
$ tar xzf mysql-connector-odbc-5.1.12-linux-glibc2.5-x86-64bit.tar.gz
$ mkdir /usr/lib64/odbc/
$ cp /usr/src/ mysql-connector-odbc-5.1.12-linux-glibc2.5-x86-64bit/lib/*
/usr/lib64/odbc/
[ 128 ]
Chapter 4
Now you can check if all the needed libraries are present on your system by using
the ldd command.
This can be done on a 32-bit system with the following command:
$ ldd /usr/lib/odbc/libmyodbc5.so
The same can be done on a 64-bit system by using the following command:
$ ldd /usr/lib64/odbc/libmyodbc5.so
If nothing is marked as Not Found, this means that all the needed libraries are found
and you can go ahead; otherwise, you need to check what is missing and fix it.
All the installed ODBC database drivers are listed in dbcinst.ini; this file,
for MySQL 5, should contain the following:
[mysql]
Description = ODBC for MySQL
Driver
= /usr/lib/odbc/libmyodbc5.so
And the following for a 64-bit system:
[mysql]
Description = ODBC for MySQL
Driver
= /usr/lib64/odbc/libmyodbc5.so
For all the available ODBC options, refer to the official
documentation available at http://dev.mysql.com/doc/
refman/5.0/en/connector-odbc-configurationconnection-parameters.html.
Data sources are defined in the odnc.ini file. You need to create a file with the
following content:
[mysql-test]
# This is the driver name as specified on odbcinst.ini
Driver = MySQL5
Description = Connector ODBC MySQL5
Database = <db-name-here>
USER= <user-name-here>
Password = <database_password-here>
SERVER = <ip-address-here>
PORT = 3306
It is possible to configure ODBC to use a secure connection SSL, but you
need to generate a certificate and configure both the sides (ODBC and
server) to enable that. Refer to the official documentation for this.
[ 129 ]
Collecting Data
PostgreSQL ODBC drivers
In order to access a PostgreSQL database via ODBC, you need to install the
appropriate drivers. They will be used by the Zabbix server, to send the queries to
any PostgreSQL database via the ODBC protocol.
The official ODBC drivers for PostgreSQL are available at http://psqlodbc.
projects.pgfoundry.org/.
Perform the following steps to work with the PostgreSQL database:
1. You can download, compile, and install the psqlODBC driver with the
following commands:
$ tar -zxvf psqlodbc-xx.xx.xxxx.tar.gz
$ cd psqlodbc-xx.xx.xxxx
$ ./configure
$ make
$ make install
2. The configuring script accepts different options; some of the most important
ones are as follows:
--with-libpq=DIR postgresql path
--with-unixodbc=DIR path or direct odbc_config file (default:yes)
--enable-pthreads= thread-safe driver when available (not on all
platforms)
3. Once compiled and installed, you can create a template file with the
following content:
[PostgreSQL]
Description
= PostgreSQL driver for Linux
Driver
= /usr/local/lib/libodbcpsql.so
Setup
= /usr/local/lib/libodbcpsqlS.so
4. Now, odbcinst can be invoked by passing your template to that command.
$ odbcinst -i -d -f template_filepsqlODBC supports
encrypted logins with md5, but not with crypt. Bear in mind that
only the login is encrypted after login. ODBC sends all the queries
in plain text. As of Version 08.01.002, psqlODBC supports SSL
encrypted connections, which will protect your data.
[ 130 ]
Chapter 4
5. As the psqlODBC driver supports threads, you can alter the thread
serialization level for each driver entry. So for instance, the content of
odbcinst.ini will be as follows:
[PostgreSQL]
Description
Driver
Setup
Threading
=
=
=
=
PostgreSQL driver for Linux
/usr/local/lib/libodbcpsql.so
/usr/local/lib/libodbcpsqlS.so
2
6. Now you need to configure the odbc.ini file. You can also use odbcinst
here, providing a template or simply a text editor, as follows.
$ odbcinst -i -s -f template_file
7. You should have inside your odbc.ini file something like the following:
[PostgreSQL]
Description
Driver
Trace
TraceFile
Database
Servername
UserName
Password
Port
Protocol
ReadOnly
RowVersioning
ShowSystemTables
ShowOidColumn
FakeOidIndex
ConnSettings
=
=
=
=
=
=
=
=
=
=
=
=
=
=
=
=
Postgres to test
/usr/local/lib/libodbcpsql.so
Yes
sql.log
<database-name-here>
<server-name-or-ip-here>
<username>
<password>
5432
6.4
No
No
No
No
No
Oracle ODBC drivers
Oracle is another database widely used and provides an ODBC driver as well.
The following is a description on how to install Oracle's ODBC, because at
http://www.unixodbc.org, there isn't much information about it.
1. The first thing to do is get the instant client from the Oracle website. Oracle
provides some of the instant client packets as rpm and tar.gz, as shown in
the following commands:
$ rpm –I oracle-instantclient11.2-basic-11.2.0.1.0-1.i386.
rpm oracle-instantclient11.2-odbc-11.2.0.1.0-1.i386.rpm oracleinstantclient11.2-sqlplus-11.2.0.1.0-1.i386.rpm
[ 131 ]
Collecting Data
2. Then you need to configure some environment variables as follows:
$ export ORACLE_HOME=/usr/lib/oracle/11.2/client
$ export ORACLE_HOME_LISTNER=/usr/lib/oracle/11.2/client/bin
$ export LD_LIBRARY_PATH=$LD_LIBRARY_PATH :/usr/lib/oracle/11.2/
client/lib
$ export SQLPATH=/usr/lib/oracle/11.2/client/lib
$ export TNS_ADMIN=/usr/lib/oracle/11.2/client/bin
3. Now you need to configure the odbcinst.ini file. This file should have
the following content:
[Oracle11g]
Description = Oracle ODBC driver for Oracle 11g
Driver
=
/usr/lib/oracle/11.2/client/lib/libsqora.so.11.1
4. In the odbc.ini file, the relative DSN entry needs to be configured
as follows:
[ORCLTEST]
Driver
=
ServerName =
Database
=
DSN
=
Port
=
Oracle 11g ODBC driver
<enter-ip-address-here>
<enter-sid-here>
ORCLTEST
1521
5. You can test the connection as usual with the following command:
$ isql -v ORCLTEST
+---------------------------------------+
| Connected!
|
|
|
| sql-statement
|
| help [tablename]
|
| quit
|
+---------------------------------------+
6. Now your ODBC connection is fine.
[ 132 ]
Chapter 4
UnixODBC configuration files
Now you are enabled to connect to most of the common databases. To check the
connection, you can test it using isql, as follows:
1. If you didn't specify the username and password inside the odbc.ini file,
it can be passed to the DSN with the following syntax:
$ isql <DSN> <user> <password>
2. Otherwise, if everything is specified, you can simply check the connection
with the following command:
$ isql mysql-test
3. If all goes well, you should see the following output:
+---------------------------------------+
| Connected!
|
|
|
| sql-statement
|
| help [tablename]
|
| quit
|
|
|
+---------------------------------------+
SQL>
If you get the following error from UnixODBC: Data source name not
found and no default driver specified, make sure that the ODBCINI
and ODBCSYSINI environment variables are pointing to the right
odbc.ini file. For example, if your odbc.ini file is in /usr/local/
etc, the environments should be set as follows:
export ODBCINI=/usr/local/etc/odbc.ini
export ODBCSYSINI=/usr/local/etc
4. If a DSN is presenting issues, the following command can be useful:
$ isql -v <DSN>
This enables the verbose mode, which is very useful to debug a connection.
[ 133 ]
Collecting Data
Compiling Zabbix with ODBC
Now if you connect to the target database that is to be monitored, it is time to compile
the Zabbix server with the ODBC support, by performing the following steps.
If your Zabbix is already up and running, don't run the make install
command during a normal installation as it will copy too many files,
and it's possible that some of them will be overwritten. In this case, it is
better to just copy the Zabbix server's executable file.
1. Now you can get the configure command line with all the options used as
specified in Chapter 1, Deploying Zabbix, by adding the --with-unixodbc
parameter as follows:
$ ./configure --enable-server -–with-postgresql --with-net-snmp
--with-libcurl --enable-ipv6 --with-openipmi --enable-agent
--with-unixodbc
2. You should see the following between the output lines:
checking for odbc_config... /usr/local/bin/odbc_config
checking for main in -lodbc... yes
checking whether unixodbc is usable... yes
3. This will confirm that all the needed ODBC binaries are found and
are usable. Once the configuring phase is completed, you can run the
following command:
$ make
4. Once completed, just take a backup of the previous zabbix_server file that
was installed, and copy the new version.
5. On starting zabbix_server, take a look into the logfile, and you should see
the following output:
****** Enabled features ******
SNMP monitoring: YES
IPMI monitoring: YES
WEB monitoring: YES
Jabber notifications: YES
Ez Texting notifications: YES
ODBC: YES
SSH2 support: YES
IPv6 support: YES
******************************
This means that all went well.
[ 134 ]
Chapter 4
Database monitor items
Now it is time to use the Zabbix ODBC functionality. In order to do so, you need to
create an item of the Database monitor type, as shown in the following screenshot:
The item where the retrieved value will be stored is identified by the item key
as follows:
db.odbc.select[<unique short description>]
The <unique short description> is a string that must be unique, and can be
whatever you want. An example is as follows:
db.odbc.select[web_user_connected_on_myapp]
[ 135 ]
Collecting Data
Inside the Additional parameters field, you need to specify the following:
DSN=<database source name>
user=<user name>
password=<password>
sql=<query>
Where the DSN should exist in /etc/odbc.ini, and whether the username and
password are stored in the DSN definition or not, can be specified here. In the last
line, you need to specify the SQL query.
Some considerations about the ODBC SQL query
The following are some restrictions and things to consider on a SQL query:
• The SQL must begin with a select clause
• The SQL can't contain any line breaks
• The query must return only one value
• If the query returns multiple columns, only the first one is read
• If the query returns multiple rows, only the first column of the first row is read
• Macros are not be replaced (for example, {HOSTNAME})
• The SQL command must begin with lowercase, that is, sql=
• The query needs to terminate before the timeout
• The query must exactly return the value type specified, otherwise the item
will be unsupported
As you can see, there are only some limitations that you can accept. In particular,
you can't call a function if that one function returns only one value. You can't execute
a stored procedure; you can only select the data. Also, the query can't contain any
line breaks, so long and complex queries will not be easily readable.
The following are some other points to consider:
• If the database is particularly loaded, it can respond with a delay (the login
can also suffer a delay caused by the workload)
• Every query executes a log in
• If you use proxies, they too need to be compiled with the UnixODBC support
If you consider to have a database that will be under heavy stress, don't have a pool
introduced for an overhead that is not necessary. Also, in this case, it is possible that
just to have a connection, you need to wait for more than five seconds.
[ 136 ]
Chapter 4
The five seconds previously mentioned is not a random value; indeed,
the timeout of a connection is defined when you open a connection.
During the initialization of that, you need to define your expected timeout
before considering the connection impossible.
Zabbix defines this timeout in the following command:
src/libs/zbxdbhigh/odbc.c
On line 130 of that file, we have the definition of the connection timeout for
Zabbix as follows:
SQLSetConnectAttr(pdbh->hdbc, (SQLINTEGER)SQL_LOGIN_TIMEOUT,
(SQLPOINTER)5, (SQLINTEGER)0);
This (SQLPOINTER)5 sets SQL_LOGIN_TIMEOUT to five seconds. If your database
doesn't respond in five seconds, you will get the following error inside the logfile:
[ODBC 3.51 Driver]Can't connect to MySQL server on 'XXX.XXX.XXX.XXX' (4)]
(2003).
In case of SQL_LOGIN_TIMEOUT, you can consider increasing it to
15 seconds and recompile the server and proxy as follows:
SQLSetConnectAttr(pdbh->hdbc,
(SQLINTEGER)SQL_LOGIN_TIMEOUT, (SQLPOINTER)15,
(SQLINTEGER)0);
Zabbix JMX monitoring
Version 2.0 of Zabbix has a native support to monitor applications using JMX.
The actor that monitors the JMX application is a Java daemon called Zabbix
Java gateway. Basically, it works like a gateway. When Zabbix wishes to know
the value of some JMX counter, it simply asks the Java gateway, and the
gateway will do all the work for Zabbix. All the queries are done using the JMX
management API from Oracle.
The Zabbix Java gateway is in early development, providing great functionality
but still experiencing some challenges.
The point of force of this method is that the application only needs to be started
with the JMX remote console enabled, and doesn't need to implement or extend
the class or write a new code to handle the Zabbix request because the entire
request is a standard JMX.
[ 137 ]
Collecting Data
The default way to enable the JMX console is to start the Java application with the
following parameters:
-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.port=<put-your-port-number-here>
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false
With these parameters, you are going to configure the JMX interface on the
application's side. As usual, you need to define a port, the authentication method,
and the encryption.
This basic setup is the simplest and easiest way, but unfortunately, it is not the safest
and most secure configuration.
Considering some JMX security aspects
Now, as you are going to open a door in your application, you are basically exposing
your application to a security attack. The JMX console, on most of the widely
diffused application servers, is not only an entry point to get values from the counter,
but also something that is a lot more sophisticated. Basically, with a JMX console
open in an application server, you can deploy an application. Start it, stop it, and so
on, as you can figure out what a hacker can deploy, and start an application, or cause
an issue on the running one. The JMX console can be called from the application
server looping itself, using the post and get methods. Adding malicious content
in the HEAD section of a web page results in the server that has a JMX console not
secured easily hackable, which is the weakest point of your infrastructure. Once an
application server is compromised, your entire network is potentially exposed, and
you need to prevent all this. This can be done through the following steps:
1. The first thing to do is enable the authentication as follows:
-Dcom.sun.management.jmxremote.authenticate=true
2. Now you need to specify a file that will contain your password as follows:
-Dcom.sun.management.jmxremote.password.file=/etc/java-6penjdk/management/jmxremote.password
There are potential security issues with password authentication
for JMX remote connectors. Once the client obtains the remote
connector from an insecure RMI registry (the default), like for all
the man-in-the-middle attacks, an attacker can start a bogus RMI
registry on the target server, right before the valid original one is
started, and can then steal the client's passwords.
[ 138 ]
Chapter 4
3. Another good thing to do is to profile the users, specifying the
following parameter:
-Dcom.sun.management.jmxremote.access.file=/etc/java-6penjdk/management/jmxremote.access
4. The access file, for instance, should contain something like the following:
monitorRole readonly
controlRole readwrite
5. The password file should be as follows:
monitorRole <monitor-password-here>
controlRole <control-password-here>
6. Now to avoid password stealing, you should enable the SSL as follows:
-Dcom.sun.management.jmxremote.ssl=true
7. This parameter is consequently tied with the following ones:
-Djavax.net.ssl.keyStore=<Keystore-location-here>
-Djavax.net.ssl.keyStorePassword=<Default-keystorepassword>
-Djavax.net.ssl.trustStore=<Trustore-location-here>
-Djavax.net.ssl.trustStorePassword=<Trustore-password-here>
-Dcom.sun.management.jmxremote.ssl.need.client.auth=true
The -D parameter will be written in the startup file of your application
or application server, as after this configuration, your startup file will
contain sensitive data (your keystore and truststore passwords) that
needs to be protected and not readable from other accounts in the same
group, or other users.
Installing a Zabbix Java gateway
To compile the Java gateway, perform the following steps:
1. First, you need to run the following:
$ ./configure --enable-java
2. You should get an output as follows:
Enable Java gateway:
Java gateway details:
Java compiler:
Java archiver:
yes
javac
jar
[ 139 ]
Collecting Data
3. This shows that the Java gateway is going to be enabled and compiled after
the following command is used:
$ make && make install
4. The Zabbix Java gateway will be installed at the following location:
$PREFIX/sbin/zabbix_java
5. Basically, the directory structure will contain the following file—the
java gateway:
bin/zabbix-java-gateway-2.0.5.jar
6. The libraries needed by the gateway are as follows:
lib/logback-classic-0.9.27.jar
lib/logback-console.xml
lib/logback-core-0.9.27.jar
lib/logback.xml
lib/org-json-2010-12-28.jar
lib/slf4j-api-1.6.1.jar
7. The scripts to start and stop the gateway are as follows:
shutdown.sh
startup.sh
8. This is a common script sourced from the start and stop scripts that contain
the following configuration:
settings.sh
9. Now, if you have enabled the SSL communication, you need to enable the
same security level on the Zabbix Java gateway. To do this, you need to add
the following parameter in the startup script:
-Djavax.net.ssl.*
10. Once all this is set, you need to specify the following inside the Zabbix
server configuration:
JavaGateway=<ip-address-here>
JavaGatewayPort=10052
If you would like to use the Java gateway from your proxy, you need
to configure both JavaGateway and JavaGatewayProperties
in the proxy configuration file.
[ 140 ]
Chapter 4
11. Since by default, Zabbix doesn't start any Java poller, you need to specify
that also, as follows:
StartJavaPollers=5
12. Then, restart the Zabbix server or proxy once done.
13. Now you can finally start the Zabbix Java gateway by running the
startup.sh command.
The logs will be available at /tmp/zabbix_java.log with verbosity as "info".
As the Zabbix Java gateway uses the logback library, you can
change the log level or the logfile location by simply changing the
lib/logback.xml file. In particular, the following XML tags need
to be changed:
<fileNamePattern>/tmp/zabbix_java.log.%i</
fileNamePattern><root level="info">
Here you can change all the logrotation parameters as well.
Configuring Zabbix JMX
Now it is time to create a JMX monitored host, with its relatively monitored JMX
items. To do that, inside the host configuration, you need to add a JMX interface and
address as shown in the following screenshot:
Once you have done that for each of the JMX counters you want to acquire, you need
to define an item of the JMX agent type. Inside the definition of the JMX agent,
you need to specify the username and password, and the JMX query string. The JMX
key is composed of the following:
• Object name of MBean
• Attribute name, that is, the attribute name of MBean
[ 141 ]
Collecting Data
The following screenshot shows the Item configuration window:
[ 142 ]
Chapter 4
The field Data type in this configuration window permits to store the unsigned
integer values (such as 0 and 1) as numbers, or as Boolean values (such as true
or false).
JMX keys in detail
MBean is a quite simple string defined in your Java application. The other
component is a bit more complex indeed; the attribute can return primitive
data types or composite data.
The primitive data types are simple types such as integer and strings. For instance,
you can have a query such as the following:
jmx[com.example:Type=Test,weight]
This will return the weight expressed as a numerical floating point value.
If the attribute returns composite data, it is a bit more complicated, but is handled
since dots are supported. For instance, you can have a pen that can have two values
that represent color and the remaining ink, usually dot separated, as shown in the
following code:
jmx[com.example:Type=Test,pen.remainink]
jmx[com.example:Type=Test,pen.color]
Now if you have an attribute name that includes a dot in its name, such as all.pen,
you need to escape the dot, as shown in the following code:
jmx[com.example:Type=Test,all\.pen.color]
If your attribute name also contains a backslash (\), this needs to be escaped twice,
as shown in the following code:
jmx[com.example:Type=Test,c:\\utility]
If the object name or attribute name contains spaces or commas, it needs to be
double-quoted:
jmx[com.example:type=Hello,""c:\\documents and settings""]
[ 143 ]
Collecting Data
Issues and considerations about JMX
Unfortunately, JMX support is not as flexible and customizable as it should be;
at least at the time of writing this book, JMX still had some problems.
For instance, from my personal experience, I know that JBOSS, which is one of
the most widely used application servers, can't be successfully enquired. The JMX
endpoint is currently hardcoded into JMXItemChecker.java as follows:
service:jmx:rmi:///jndi/rmi://" + conn + ":" + port + "/jmxrmi"
Some applications use different endpoints for their JMX management console.
JBOSS is one of them. The endpoint is not configurable as per the host or
frontend, and you can't add a parameter to specify this endpoint on the host's
configuration window.
Anyway, the development is really active and things are getting better and are
improving everyday. At the moment, the status is that the Zabbix Java gateway
needs some improvement. Also, the current implementation of Zabbix Java gateway
suffers because of the workload; if you have more than 100 JMX items per host,
the gateway needs to be restarted periodically. It is possible that you face some
errors inside your logfile of the following kind:
failed: another network error, wait for 15 seconds
Followed by:
connection restored
It is fundamental that the implementer of Zabbix monitoring infrastructure
knows not only all the strong points of the product but also the cons and
limitations. The implementer can then choose if they want to develop something
in-house, use an open source alternative, try to fix the possible issues, or ask the
Zabbix team for a new functionality or fix.
[ 144 ]
Chapter 4
Zabbix SNMP monitoring
Simple Network Monitoring Protocol (SNMP) may not be as simple as the name
suggests; it's a de facto standard for many appliances and applications. It's not
just ubiquitous, it's often the only sensible way one can extract the monitoring
information from a network switch, a disk array enclosure, a UPS battery, and so on.
The basic architecture layout for SNMP monitoring is actually straightforward.
Every monitored host or appliance runs an SNMP agent. This agent can be queried
by any probe (whether it's just a command-line program to do manual queries,
or a monitoring server like Zabbix) and will send back information on any metric
it has made available, or even change some predefined settings on the host itself,
as a response to a set command from the probe. Furthermore, the agent is not just
a passive entity that responds to the get and set commands, but can also send
warnings and alarms as SNMP traps to a predefined host when some specific
conditions arise.
Things get a little more complicated when it comes to metric definitions. Unlike a
regular Zabbix item, or any other monitoring system, an SNMP metric is part of
a huge hierarchy, a tree of metrics that spans hardware vendors and software
implementers across all of the IT landscape. This means that every metric has to be
uniquely identified with some kind of code. This unique metric identifier is called
OID, and identifies both the object and its position in the SNMP hierarchy tree.
OIDs and their values are the actual content that is passed in the SNMP messages.
While this is most efficient from a network traffic point of view, OIDs need to be
translated into something usable and understandable by humans as well. This is
done by using a distributed database called Management Information Base (MIB).
MIBs are essentially text files that describe a specific branch of the OID tree,
with a textual description of its OIDs, their data types, and a human-readable
string identificator.
[ 145 ]
Collecting Data
MIBs let us know, for example, that OID 1.3.6.1.2.1.1.3 refers to the system-uptime
of whatever machine the agent is running on. Its value is expressed as an integer,
in hundredth of a second, and can generally be referred to as sysUpTime.
As you can see, this is quite different from the way the Zabbix agent items work,
both in terms of the connection protocol, and item definition and organization.
Nevertheless, Zabbix provides facilities to translate from SNMP OIDs to Zabbix
items—if you compiled the support for the server in SNMP, it will be able to create
the SNMP queries natively, and with the help of a couple of supporting tools, it will
also be able to process SNMP traps.
[ 146 ]
Chapter 4
This is of course an essential feature if you need to monitor appliances that only
support SNMP, and have no way of installing a native agent on network appliances
in general (switcher, routers, and so forth), disk array enclosures, and so on. But the
following may be some reasons for you to actually choose SNMP as the main
monitoring protocol in your network and completely dispense of Zabbix agents:
• You may not need many complex or custom metrics apart from what is
already provided by an operating system's SNMP OID branch. You most
probably have already set up SNMP monitoring for your network equipment,
and if you just need simple metrics such as uptime, CPU load, free memory,
and so on, from your average host, it might be simpler to rely on SNMP for it
as well, instead of the native Zabbix agent. This way, you will never have to
worry about agent deployment and updates—you just let the Zabbix server
contact the remote SNMP agents and get the information you need.
• The SNMP protocol and port numbers are well-known by virtually all
the products. If you need to send monitoring information across networks,
it might be easier to rely on the SNMP protocol instead of the Zabbix one.
This could be because traffic on the UDP ports 161 and 162 is already
permitted, or because it might be easier to ask a security administrator
to allow access to a well-known protocol instead of a relatively more
obscure one.
• SNMP Version 3 features built-in authentication and security. This means
that, contrary to the Zabbix protocol, as you have already seen in Chapter 2,
Distributed Monitoring, SNMPv3 messages will have integrity, confidentiality,
and authentication. While Zabbix does support all three versions of SNMP,
it's strongly advised that you use Version 3 wherever possible, because it's
the only one with real security features. In contrast, Versions 1 and 2 only
have a simple string sent inside a message as a very thin layer of security.
• While there may be good reasons to use SNMP monitoring as much as
possible in your Zabbix installation, there are still a couple of strong ones to
stick with the Zabbix agent. The Zabbix agent has a few, very useful built-in
metrics that would need custom extensions if implemented through an
SNMP agent. For example, if you want to monitor a logfile, with automatic
log rotation support, and skipping old data, you just need to specify the
logrt[] key for a Zabbix active item. The same goes if you want to monitor
the checksum, or the size of a specific file, the Performance Monitor facility
of the Windows operating system, and so on. In all these cases, the Zabbix
agent is the most immediate and simple choice.
[ 147 ]
Collecting Data
• The Zabbix agent has the ability to discover many kinds of resources that
are available on the host, and report them back to the server, which will in
turn automatically create items and triggers, and destroy them when the said
resources are not available anymore. This means that with the Zabbix agent,
you will be able to let the server create the appropriate items for every host's
CPU, mounted filesystem, number of network interfaces, and so on. While it's
possible to define low-level discovery rules based on SNMP, it's often easier
to rely on the Zabbix agent for this kind of functionality.
So once again, you have to balance the different features of each solution in order to
find the best match for your environment. But generally speaking, you could make the
following broad assessments: if you have simple metrics but need strong security,
go with SNMP v3; if you have complex monitoring or automated discovery needs and
can dispense with strong security (or are willing to work harder to get it, as explained
in Chapter 2, Distributed Monitoring), go with the Zabbix agent and protocol.
That said, there are a couple of aspects worth exploring when it comes to Zabbix
SNMP monitoring. We'll first talk about simple SNMP queries, and then about
SNMP traps.
SNMP queries
An SNMP monitoring item is quite simple to configure. The main point of interest
is that while the server will use the OID you provided to get the measurement,
you'll still need to define a unique name for the item, and most importantly, a unique
item key. Keep in mind that an item key is used in all of Zabbix's expressions that
define triggers, calculated items, actions, and so on. So try to keep it short and simple,
while easily recognizable. As an example, suppose you want to define a metric for
the incoming traffic on network port number 3 of an appliance, the OID would be
1.3.6.1.2.1.2.2.1.10.3, while you could call the key something like port3.ifInOctects,
as shown in the following screenshot:
[ 148 ]
Chapter 4
If you don't already have your SNMP items defined in a template, an easy way to get
them is by using the snmpwalk tool to directly query the host you need to monitor,
and get information about the available OIDs and their data types.
[ 149 ]
Collecting Data
For example, the following command is used to get the whole object tree from the
appliance at 10.10.15.19:
$ snmpwalk -v 3 -l AuthPriv -u user -a MD5 -A auth -x DES -X priv -m ALL
10.10.15.19
You need to substitute the string user with the username for the SNMP
agent, auth with the authentication password for the user, priv with
the privacy password, MD5 with the appropriate authentication protocol,
and DES with the privacy protocol you defined for the agent.
The SNMP agent on the host will respond with a list of all its OIDs. The following is
a fragment of what you could get:
HOST-RESOURCES-MIB::hrSystemUptime.0 = Timeticks: (8609925) 23:54:59.25
HOST-RESOURCES-MIB::hrSystemDate.0 = STRING: 2013-7-28,9:38:51.0,+2:0
HOST-RESOURCES-MIB::hrSystemInitialLoadDevice.0 = INTEGER: 393216
HOST-RESOURCES-MIB::hrSystemInitialLoadParameters.0 = STRING: "root=/dev/
sda8 ro"
HOST-RESOURCES-MIB::hrSystemNumUsers.0 = Gauge32: 2
HOST-RESOURCES-MIB::hrSystemProcesses.0 = Gauge32: 172
HOST-RESOURCES-MIB::hrSystemMaxProcesses.0 = INTEGER: 0
HOST-RESOURCES-MIB::hrMemorySize.0 = INTEGER: 8058172 KBytes
HOST-RESOURCES-MIB::hrStorageDescr.1 = STRING: Physical memory
HOST-RESOURCES-MIB::hrStorageDescr.3 = STRING: Virtual memory
HOST-RESOURCES-MIB::hrStorageDescr.6 = STRING: Memory buffers
HOST-RESOURCES-MIB::hrStorageDescr.7 = STRING: Cached memory
HOST-RESOURCES-MIB::hrStorageDescr.8 = STRING: Shared memory
[ 150 ]
Chapter 4
HOST-RESOURCES-MIB::hrStorageDescr.10 = STRING: Swap space
HOST-RESOURCES-MIB::hrStorageDescr.35 = STRING: /run
HOST-RESOURCES-MIB::hrStorageDescr.37 = STRING: /dev/shm
HOST-RESOURCES-MIB::hrStorageDescr.39 = STRING: /sys/fs/cgroup
HOST-RESOURCES-MIB::hrStorageDescr.53 = STRING: /tmp
HOST-RESOURCES-MIB::hrStorageDescr.56 = STRING: /boot
Let's say that we are interested in the system's memory size. To get the full OID for
it, we will reissue the snmpwalk command using the fn option for the -O switch.
These will tell snmpwalk to display the full OIDs in a numeric format. We will also
limit the query to the OID we need, as taken from the previous output:
$ snmpwalk -v 3 -l AuthPriv -u user -a MD5 -A auth -x DES -X priv -m ALL
-O fn 10.10.15.19 HOST-RESOURCES-MIB::hrMemorySize.0
.1.3.6.1.2.1.25.2.2.0 = INTEGER: 8058172 KBytes
And there we have it. The OID we need to put in our item definition is
1.3.6.1.2.1.25.2.2.0.
SNMP traps
SNMP traps are a bit of an oddball if compared to all the other Zabbix item types.
Unlike other items, SNMP traps do not report a simple measurement, but an
event of some type. In other words, they are the result of some kind of a check or
computation made by the SNMP agent and sent over to the monitoring server as a
status report. An SNMP trap can be issued every time a host is rebooted, an interface
is down, a disk is damaged, or a UPS has lost power and is keeping the servers up
using its battery.
This kind of information contrasts with Zabbix's basic assumption, that is, an item
is a simple metric not directly related to a specific event. On the other hand, there may
be no other way to be aware of certain situations if not through an SNMP trap,
either because there are no related metrics (consider for example, the event "the server
is being shut down"), or because the appliance's only way to convey its status is
through a bunch of SNMP objects and traps.
[ 151 ]
Collecting Data
So traps are of relatively limited use to Zabbix, as you can't do much more than build
a simple trigger out of every trap, and then notify about the event (not much point
in graphing a trap, or building calculated items on it). Nevertheless, they may prove
essential for a complete monitoring solution.
To manage SNMP traps effectively, Zabbix needs a couple of helper tools: the
snmptrapd daemon, to actually handle connections from the SNMP agents, and
some kind of script to correctly format every trap and pass it to the Zabbix server
for further processing.
The snmptrapd process
If you have compiled an SNMP support into the Zabbix server, you should already
have the complete SNMP suite installed, which contains the SNMP daemon, the SNMP
trap daemon, and a bunch of utilities such as snmpwalk and snmptrap.
If it turns out that you don't actually have the SNMP suite installed, the following
command should take care of the matter:
# yum install net-snmp net-snmp-utils
Just as the Zabbix server has a bunch of daemon processes that listen on the TCP port
10051 for incoming connections (from agents, proxies, and nodes), snmptrapd is the
daemon process that listens on the UDP port 162 for incoming traps coming from
remote SNMP agents.
Once installed, snmptrapd reads its configuration options from an snmptrapd.conf
file, which can be usually found in the /etc/snmp/ directory. The bare minimum
configuration for snmptrapd requires only the definition of a community string in
the case of SNMP Versions 1 and 2, which is as follows:
authCommunity log public
Or the definition of a user and a privacy level in the case of SNMP Version 3,
which is as follows:
createUser -e ENGINEID user MD5 auth DES priv
You need to create a separate createUser line for every remote
Version 3 agent that will send traps. You also need to substitute
all the user, auth, priv, MD5, and DES strings to what you have
already configured on the agent, as explained in the previous note.
Most importantly, you need to set the correct ENGINEID for every
agent. You can get it from the agent's configuration itself.
[ 152 ]
Chapter 4
With this minimal configuration, snmptrapd will limit itself to log the trap to syslog.
While it could be possible to extract this information and send it to Zabbix, it's easier
to tell snmptrapd how it should handle the traps. While the daemon has no processing
capabilities of its own, it can execute any command or application by either using
the trapHandle directive, or leveraging its embedded perl functionality. The latter
is more efficient as the daemon won't have to fork a new process and wait for its
execution to finish, so it's the recommended one if you plan to receive a significant
number of traps. Just add the following line to snmptrapd.conf:
perl do "/usr/local/bin/zabbix_trap_receiver.pl";
You can get the zabbix_trap_receiver script from the Zabbix sources.
It's located in misc/snmptrap/zabbix_trap_receiver.pl.
Once restarted, the snmptrapd daemon will execute the perl script of your choice
to process every trap received. As you can probably imagine, your job doesn't end
here—you still need to define how to handle the traps in your script, and find a way
to send the resulting work over to your Zabbix server. We'll discuss both of these
aspects in the following section.
The Perl trap handler
The perl script included in the Zabbix distribution works as a translator from an
SNMP trap format to a Zabbix item measurement. For every trap received, it will
format it according to the rules defined in the script, and will output the result in a
logfile. The Zabbix server will in turn monitor the said logfile, and process every new
line as an SNMP trap item, basically matching the content of the line to any trap item
defined for the relevant host. Let's see how it all works by looking at the perl script
itself and illustrating its logic:
#!/usr/bin/perl
#
#
#
#
#
#
#
#
#
Zabbix
Copyright (C) 2001-2013 Zabbix SIA
This program is free software; you can redistribute it and/or
modify
it under the terms of the GNU General Public License as
published by
the Free Software Foundation; either version 2 of the License,
[ 153 ]
Collecting Data
#
#
#
#
#
#
#
#
#
#
#
#
#
#
or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public
License
along with this program; if not, write to the Free Software
Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA
02110-1301, USA.
#########################################
#### ABOUT ZABBIX SNMP TRAP RECEIVER ####
#########################################
# This is an embedded perl SNMP trapper receiver designed for
# sending data to the server.
# The receiver will pass the received SNMP traps to Zabbix server
# or proxy running on the
# same machine. Please configure the server/proxy accordingly.
#
# Read more about using embedded perl with Net-SNMP:
#
http://net-snmp.sourceforge.net/wiki/index.php/Tut:Extending_
snmpd_using_perl
This first section contains just the licensing information and a brief description of
the script. Nothing worth mentioning, except a simple reminder—check that your
perl executable is correctly referenced in the first line, or change it accordingly.
The following section is more interesting, and if you are happy with the script's
default formatting of SNMP traps, it may also be the only section you will ever
need to customize:
#################################################
#### ZABBIX SNMP TRAP RECEIVER CONFIGURATION ####
#################################################
### Option: SNMPTrapperFile
#
Temporary file used for passing data to the server (or
#
proxy). Must be the same
[ 154 ]
Chapter 4
#
as in the server (or proxy) configuration file.
#
# Mandatory: yes
# Default:
$SNMPTrapperFile = '/tmp/zabbix_traps.tmp';
### Option: DateTimeFormat
#
The date time format in strftime() format. Please make
#
sure to have a corresponding
#
log time format for the SNMP trap items.
#
# Mandatory: yes
# Default:
$DateTimeFormat = '%H:%M:%S %Y/%m/%d';
Just set $SNMPTrapperFile to the path of the file you wish the script to log its trap to,
and set the SNMPTrapperFile option in your zabbix_server.conf file to the same
value. While you are at it, also set StartSNMPTrapper to 1 in zabbix_server.conf
so that the server will start monitoring the said file.
The $DateTimeFormat, on the other hand, should match the format of the actual
SNMP traps you receive from the remote agents. Most of the time, the default value
is correct, but take the time to check it and change it as needed.
The following section contains the actual logic of the script. Notice how the bulk of the
logic is contained in a subroutine called zabbix_receiver. This subroutine will be
called and executed towards the end of the script, but is worth examining in detail.
###################################
#### ZABBIX SNMP TRAP RECEIVER ####
###################################
use Fcntl qw(O_WRONLY O_APPEND O_CREAT);
use POSIX qw(strftime);
sub zabbix_receiver
{
my (%pdu_info) = %{$_[0]};
my (@varbinds) = @{$_[1]};
[ 155 ]
Collecting Data
The snmptrapd daemon will execute the script and pass the trap that it just received.
The script will in turn call its subroutine, which will immediately distribute the trap
information in two lists—the first argument is assigned to the %pdu_info hash and
the second one to the @varbinds array.
# open the output file
unless (sysopen(OUTPUT_FILE, $SNMPTrapperFile,
O_WRONLY|O_APPEND|O_CREAT, 0666))
{
print STDERR "Cannot open [$SNMPTrapperFile]:
$!\n";
return NETSNMPTRAPD_HANDLER_FAIL;
}
Here, the script will open the output file, or fail graciously if it somehow cannot.
The next step consists of extracting the hostname (or IP address) of the agent that sent
the trap. This information is stored in the %pdu_info hash we defined previously:
# get the host name
my $hostname = $pdu_info{'receivedfrom'} || 'unknown';
if ($hostname ne 'unknown') {
$hostname =~ /\[(.*?)\].*/;
$hostname = $1 || 'unknown';
}
Now we are ready to build the actual SNMP trap notification message. The first
part of the output will be used by Zabbix to recognize the presence of a new trap
(by looking for the ZBXTRAP string, and knowing which of the monitored hosts the
trap refers to). Keep in mind that the IP address or hostname set here must match
the SNMP address value in the host configuration as set using the Zabbix frontend.
This value must be set even if it's identical to the main IP/hostname for a given host.
Once the Zabbix server will have identified the correct host, it will discard this part
of the trap notification.
# print trap header
#
timestamp must be placed at the beginning of the
first line (can be omitted)
#
the first line must include the header "ZBXTRAP
[IP/DNS address] "
#
* IP/DNS address is the used to find the
corresponding SNMP trap items
#
* this header will be cut during processing
(will not appear in the item value)
printf OUTPUT_FILE "%s ZBXTRAP %s\n",
strftime($DateTimeFormat, localtime), $hostname;
[ 156 ]
Chapter 4
After the notification header, the script will output the rest of the trap as received by
the SNMP agent.
# print the PDU info
print OUTPUT_FILE "PDU INFO:\n";
foreach my $key(keys(%pdu_info))
{
printf OUTPUT_FILE " %-30s %s\n", $key,
$pdu_info{$key};
}
The printf statement in the previous code will circle over the %pdu_info hash and
output every key-value pair.
# print the variable bindings:
print OUTPUT_FILE "VARBINDS:\n";
foreach my $x (@varbinds)
{
printf OUTPUT_FILE " %-30s type=%-2d value=%s\n",
$x->[0], $x->[2], $x->[1];
}
close (OUTPUT_FILE);
return NETSNMPTRAPD_HANDLER_OK;
}
The second printf statement, printf OUTPUT_FILE " %-30s type=%-2d
value=%s\n", $x->[0], $x->[2], $x->[1];, will output the contents of the @
varbinds array one by one. This array is the one that contains the actual values
reported by the trap. Once done, the logfile is closed, and the execution of the
subroutine ends with an exit message.
NetSNMP::TrapReceiver::register("all", \&zabbix_receiver) or
die "failed to register Zabbix SNMP trap receiver\n";
print STDOUT "Loaded Zabbix SNMP trap receiver\n";
The last few lines of the script set the zabbix_receiver subroutine as the actual
trap handler, and give some feedback about its correct setup. Once the trap handler
will start populating the zabbix_traps.log logfile, you need to define the
corresponding Zabbix items.
[ 157 ]
Collecting Data
As you've already seen, the first part of the log line is used by the Zabbix trap
receiver to match a trap with its corresponding host. The second part is matched to
the said host's SNMP trap items regexp definitions, and its content added to every
matching item's history of values. This means that if you wish to have a startup
trap item for a given host, you'll need to configure an SNMP trap item with an
snmptrap["coldStart"] key, as shown in the following screenshot:
From now on, you'll be able to see the contents of the trap in the item's data history.
Web pages monitoring
In this time and age, web applications are virtually ubiquitous. Some kind of website
or a collection of web pages is typically the final product or the service of a complex
structure that comprises of different databases, application servers, web servers,
proxies, network balancers and appliances, and more. When it comes to monitoring
duties, it makes sense to go just a step further and monitor a resulting site or web
page in addition to all the backend assets that enable the said page.
[ 158 ]
Chapter 4
The advantages as far as warnings and notifications go, are fairly limited, as failure
to reach a web page is certainly a critical event, but it hardly gives any insight on
what may be the actual problem if you haven't set up the correct metrics and triggers
on the backend side. On the other hand, it may be crucial to have a collection of data
about a website's performance, in order to anticipate possible problems, substantiate
SLA reporting, and plan for hardware or software upgrades.
One big advantage of Zabbix's web-monitoring facilities is the scenario concept.
You can define a single web scenario that is composed of many simple steps,
each one building on the previous, and sharing a common set of data. Furthermore,
every such definition includes the automatic creation of meaningful items and
graphs, both at the scenario level (overall performance) and at the single-step level
(local performance). This makes it possible to not only monitor a single web page,
but to simulate entire web sessions, so that every component of a web application
will contribute to the final results. A single scenario can be very complex and
requires a great number of items that would end up being difficult to track and
group together. This is the reason why web monitoring in Zabbix has its own
web configuration tab and interface, separate from regular items, where you can
configure monitoring on a higher level.
Web scenarios support plain HTTP and HTTPS, Basic, NTLM, form-based
authentication, cookies, submission of form fields, and checking of page content
in addition to the HTTP code responses.
For all their power, web scenarios also suffer a few limitations when it comes to
monitoring the modern Web.
First of all, JavaScript is not supported, so you can't simulate a complete AJAX
session just like a human user would experience it. This also means that any kind
of automated page reloads won't be executed in the scenario.
Furthermore, while you can submit forms, you have to know in advance, both,
the name of the fields and their content. If either of them is generated dynamically
from page to page (as many ASP.NET pages do, to keep the session information),
you won't be able to use it in subsequent steps.
These may look like negligible limitations, but they may prove quite important if you
need to monitor any site that relies heavily on client-side elaborations (JavaScript
and friends) or on dynamic tokens and form fields. The reality is that an increasing
number of web applications or frameworks use one or both of these features.
Nevertheless, even with these limitations, Zabbix's web monitoring facilities prove
to be a very useful and powerful tool that you may want to take full advantage of,
especially if you produce a lot of web pages as the final result of an IT pipeline.
[ 159 ]
Collecting Data
Authenticating on web pages
The three methods of authentication supported for web monitoring are Basic,
NTLM and form based. The first two are fairly straightforward and just need to
be defined at the scenario level, as shown in the following screenshot:
A form-based authentication, on the other hand, relies on the ability of the client
(a Zabbix server, in this instance) to keep the session cookies, and is triggered
when the said client submits a form with the authentication data. While defining a
scenario, you'll need to dedicate a step just for the authentication. To know which
form fields you'll need to submit, look at the HTML source of the page containing the
authentication form. In the following example, we'll look at the Zabbix authentication
page. Every form will be slightly different, but the general structure will be largely the
same (here, only the login form in an abbreviated manner is shown).
<form action="index.php" method="post">
<input type="hidden" name="request" class="input hidden" value="" />
<!-- Login Form -->
<div >Username</div>
<input type="text" id="name" name="name" />
[ 160 ]
Chapter 4
<div >Password</div>
<input type="password" id="password" name="password" />
<input type="checkbox" id="autologin" name="autologin"
value="1" checked="checked" />
<input type="submit" class="input" name="enter"
id="enter" value="Sign in" />
</form>
You need to take note of the input tags and their name options, because these are
the form fields you are going to send to the server to authenticate. In this case,
the username field is called name, the password field is called password, and finally,
the submit field is called enter and has the value Sign in.
We are now ready to create a scenario, as shown in the following screenshot:
[ 161 ]
Collecting Data
We will then define the authentication step, as shown in the following screenshot:
From now on, every URL that you'll check, or a form that you'll submit, will be in
the context of an authenticated session, assuming the login process was successful,
of course.
Logging out
One common mistake when it comes to web monitoring is that the authentication
part is taken care of at the start of a scenario, but rarely at the end during logout.
If you don't log out of a website, depending on the system used to keep track of the
logged-in users and active sessions, a number of problems may arise.
Active sessions usually range from a few minutes to a few days. If you are
monitoring the number of logged-in users, and your session's timeouts are on the
longer side of the spectrum, every login scenario would add to the number of active
users reported by the monitoring items. If you don't immediately log out at the end
of the scenario, you may, at the very least, end up with monitoring measurements
that are not really reliable, and they would show a lot of active sessions that are
really just monitoring checks.
[ 162 ]
Chapter 4
Furthermore, every new session uses up a small amount of computing resources,
whether it's disk space or memory. If you create a large number of sessions in a
short time, due to frequent checks, you could end up significantly degrading the
website's performance.
At the very worst, your identity manager and authentication backend may not be
equipped to handle a great number of non-expiring sessions and suddenly stop
working, bringing your whole infrastructure to a grinding halt. We can assure you
that these are not hypothetical situations, but real-life episodes that occurred in the
authors' own experience.
At any rate, you certainly can't go wrong by adding a log-out step to every web
scenario that involves a log in. You'll make sure that your monitoring actions won't
cause any unforeseen problem, and at the very least, you will also test the correct
functioning of your session's tear-down procedures. Log-out steps are also usually
quite easy, as they normally involve just a GET request to the correct URL. In the case
of the Zabbix frontend, you would create the following final step (as shown in the
following screenshot) before ending the scenario:
[ 163 ]
Collecting Data
Aggregated and calculated items
Until now, every item type described in this chapter can be considered as a way of
getting raw measurements as single data points. In fact, the focus of the chapter has
been more on setting up Zabbix to retrieve different kinds of data, than on what is
actually collected. This is because on one hand, a correct setup is crucial for effective
data gathering and monitoring, while on the other hand, the usefulness of a given
metric varies wildly across environments and installations, depending on the specific
needs that you may have.
When it comes to aggregated and calculated items though, things start to become
really interesting. Both types don't rely on probes and measurements at all; instead,
they build on existing item values to provide a whole new level of insight and
elaboration on the data collected in your environment.
This is one of the points where Zabbix's philosophy of decoupling measurements
and triggering logic really pays off, because it would be quite cumbersome otherwise
to come up with similar features, and it would certainly involve a significant amount
of overhead.
The two types have the following features in common:
• Both of them don't make any kind of checks (agent-based, external, SNMP,
JMX, or otherwise) but directly query the Zabbix database to process the
existing information.
• While they have to be tied to a specific host because of how Zabbix data is
organized, this is a loose connection compared to a regular item. In fact,
you could assign an aggregated item to any host regardless of the item's
specifics, although it's usually clearer and easier if you define one or more
simple, dedicated hosts that will contain aggregated and calculated items,
so that they'll be easier to find and reference to.
• Aggregated and calculated items only work with numeric data types—there's
no point in asking for the sum or the average of a bunch of text snippets.
Aggregated items
The simplest of the two types discussed here, aggregated items, can perform
different kinds of calculations on a specific item that is defined for every host in a
group. For every host in a given group, an aggregated item will get the specified
item's data (based on a specified function), and then apply the group function
on all of the values collected. The result will be the value of the aggregated item
measurement at the time that it was calculated.
[ 164 ]
Chapter 4
To build an aggregated item, you first need to identify the host group you are
interested in, and then identify the item, shared by all the group's hosts, which will
form the basis of your calculations. For example, let's say that you are focusing on
your web application servers and you want to know something about the active
sessions of your Tomcat installations. In this case, the group would be something
like Tomcat Servers, and the item key would be jmx["Catalina:type=Manager,
path=/,host=localhost",activeSessions].
Next, you need to decide how you want to retrieve every host's item data. This is
because you are not limited to just the last value, but can perform different kinds of
preliminary calculations. Except for the last function, which indeed just retrieves
the last value from the item's history, all the other functions take a period of time as
a further argument.
Function
avg
Description
sum
The sum of all values in the specified
time period
min
The minimum value recorded in the
specified time period
max
The maximum value recorded in the
specified time period
last
The latest value recorded
count
The number of values recorded in the
specified time period
The average value in the specified time
period
What you now have is a bunch of values that still need to be brought together.
The following table explains the job of the group function:
Function
grpavg
Description
grpsum
The sum of all values collected
grpmin
The minimum value in the collection
grpmax
The maximum value in the collection
The average of all the values collected
[ 165 ]
Collecting Data
Now that you know all the components of an aggregated item, you can build the
key; the appropriate syntax is as follows:
groupfunc["Host group","Item key",itemfunc,timeperiod]
The Host group part can be defined locally to the aggregated item
definition. If you want to bring together data from different hosts
that are not part of the same group, and you don't want to create a
host group just for this, you can substitute the host group name with
a list of the hosts—["HostA, HostB, HostC"].
Continuing with our example, let's say that you are interested in collecting the
average number of active sessions on your Tomcat application server's every hour.
In this case, the item key would look as follows:
grpavg["Tomcat servers",
"jmx["Catalina:type=Manager,path=/,
host=localhost",activeSessions]", avg, 3600]
You could also use 1h or 60m as a time period if you don't
want to stick to the default of using seconds.
Using the same group and a similar item, you would also want to know the peak
number of concurrent sessions across all servers, this time every five minutes,
which can be done as follows:
grpsum["Tomcat servers",
"jmx["Catalina:type=Manager,path=/,host=localhost",maxActive]",
last, 0]
Simple as they are, aggregated items already provide some useful functionality,
which would be harder to match without a collection of measurements as simple
data that is easily accessible through a database.
[ 166 ]
Chapter 4
Calculated items
This item type builds on the concept of item functions expressed in the previous
paragraphs, and takes it to a new level. Unlike aggregated items, with calculated
ones you are not restricted to a specific host group, and more importantly, you are
not restricted to a single item key. With calculated items, you can apply any of
the functions available for the trigger definitions to any item in your database,
and combine different item calculations using arithmetic operations. As with other
item types that deal with specialized pieces of data, the item key of a calculated item
is not used to actually define the data source, but still needs to be unique so that
you can refer to the item in triggers, graphs, and actions. The actual item definition
is contained in the formula field, and as you can imagine, it can be as simple or as
complex as you need.
Keeping with our Tomcat server's example, you could have a calculated item that
gives you a total application throughput for a given server as follows:
last(jmx["Catalina:type=GlobalRequestProcessor,name=http8080",bytesReceived]) +
last(jmx["Catalina:type=GlobalRequestProcessor,name=http8080",bytesSent]) +
last(jmx["Catalina:type=GlobalRequestProcessor,name=http8443",bytesReceived]) +
last(jmx["Catalina:type=GlobalRequestProcessor,name=http-8443",
bytesSent]) +
last(jmx["Catalina:type=GlobalRequestProcessor,name=jk-8009",
bytesReceived]) +
last(jmx["Catalina:type=GlobalRequestProcessor,name=jk8009",bytesSent])
Or you could be interested in the ratio between the active sessions and the maximum
number of allowed sessions, so that later you could define a trigger based on a
percentage value instead of an absolute one as follows:
100*last(jmx["Catalina:type=Manager,path=/,
host=localhost",activeSessions]) /
last(jmx["Catalina:type=Manager,path=/,host=localhost",
maxActiveSessions])
[ 167 ]
Collecting Data
As previously stated, you don't need to stick to a single host either,
in your calculations.
The following is how you could estimate the average number of queries on the
database per single session, on an application server, every three minutes:
avg(DBServer:mysql.status[Questions], 180) /
avg(Tomcatserver:Catalina:type=Manager,path=/,host=localhost",
activeSessions], 180)
The only limitation with calculated items is that there are no easy group functions
like those available to aggregated items. So while calculated items are essentially a
more powerful and flexible version of aggregated items, you still can't dispense with
the latter, as you'll need them for all group-related functions.
Despite this limitation, as you can easily imagine, the sky is the limit when it comes
to calculated items. Together with aggregated items, these are ideal tools to monitor
the host's group performances such as clusters or grids, or to correlate different
metrics on different hosts that contribute to the performance of a single service.
Whether you use them for performance analysis and capacity planning, or as the
basis of complex and intelligent triggers, or both, the judicious use of aggregated and
calculated items will help you to make the most out of your Zabbix installation.
Summary
In this chapter, we delved into various aspects of item definitions in Zabbix. At this
point, you should know the main difference between a Zabbix item and a monitoring
object of other products, and why Zabbix's approach of collecting simple, raw data
instead of monitoring events, is a very powerful one. You should also know the ins
and outs of monitoring the data flow traffic, and how to affect it based on your needs
and environment. You should be comfortable moving beyond the standard Zabbix
agent when it comes to gathering data, and be able to configure your server to collect
data from different sources—database queries, SNMP agents, web pages, JMX
consoles, and so on.
Finally, you should've grasped the vast possibilities implied by the two powerful
item types—aggregated and calculated.
In the next chapter, you'll learn how to present and visualize all the wealth of data
you are collecting using graphs, maps, and screens.
[ 168 ]
Visualizing Data
Zabbix is a flexible monitoring system. Once implemented on an installation, it is
ready to support a heavy workload and will help you acquire a huge amount of
every kind of data. The next step is to graph your data, interpolate, and correlate
the metrics between them. The strong point is that you can relate different types of
metrics on the same axis of time, analyzing patterns of heavy and light utilization,
identifying services and equipment that fail most frequently in your infrastructure,
and capturing relationships between metrics of the connected services.
Zabbix, beyond the standard graphs facility, offers you a way to create your custom
graphs, and to add them on your own template, thus creating an easy method to
propagate your graphs across all the servers. Those custom graphs (and also the
standard and simple) can be collected into screens. Inside Zabbix, a screen can contain
different kind of information—simple graphs, custom graphs, other screens, plain text
information, trigger overview, and so on.
In this chapter, you will learn how to:
• Generate custom graphs
• Create and use maps with shortcuts and nested maps
• Create a dynamic screen
• Create and set up slides for a large monitor display
• Generate an SLA reporting
Visualizing Data
As a practical example, you can think of a big datacenter, where there are different
layers or levels of supports; usually the first level of support needs to have a general
overview of what is happening on your datacenter, the second level can be the first
level of support divided for typology of service, for example, DBA, Application
Servers, and so on. Now your DBA second level of support will need the entire
database related metrics, an application server specialist most probably will need
all the Java metrics, plus some other standard metrics such as CPU Memory usage.
Zabbix's responses to this requirement are maps, screens, and slides.
Once you create all your graphs, and have retrieved all your needed metrics and
messages, you can easily create screens that collect, for instance, all the DBA related
graphs plus some other standard metrics; it will be easy to create a rotation of those
screens. The screen will be collected on slides and each level of support will see its
groups of screens in a slide show, having an immediate qualitative and quantitative
vision of what is going on.
The datacenter support is most probably the most complex slide show to implement,
but in this chapter you will see how easy it is to create them. Once you have all the
pieces (simple graphs, custom graphs, triggers, and so on), you can use them, and also
re-use them on different visualization types. On most of the slides, for instance, all the
vital parameters such as CPU, memory, swap usage, and network I/O needs to be
graphed. Once done, your custom graphs can be re-used in a wide number of dynamic
elements. Zabbix provides another great functionality, that is, the ability to create
dynamics maps. A map is a graphical representation of a network infrastructure.
All those features will be discussed in this chapter.
When you are finally ready to implement your own custom visualization screen,
it is fundamental to bear in mind which one will be the audience, their skills or
background, and their needs. Basically, be aware of what message you will deliver
with your graphs.
Graphs are powerful tools to transmit your message; they are a flexible instrument
that can be used to give more strength to your speech as well as give a qualitative
overview of your service or infrastructure. This chapter is pleasant, and will enable
you to communicate using all the Zabbix graphical elements.
[ 170 ]
Chapter 5
Graphs
Inside Zabbix, you can divide the graphs in two categories—the simple graphs and
the custom graphs. Both of these are analyzed in the next section.
Analyzing simple graphs
Simple graphs in Zabbix are something really immediate since you don't need to put
in a lot of effort to configure this feature. You only need to go to Monitoring | Latest
data, eventually filter by the item name, and click on the graph. Zabbix will show
you the historical graph, as shown in the following screenshot:
Clearly you can graph only numeric items, and all the other kinds of data such as
text can't be shown on a graph. On the Latest data item, you will see instead the
Graph link—a link that will show the history.
No configuration is needed, but you can't
customize this graph.
[ 171 ]
Visualizing Data
On top of the graphs there is the time period selector. If you enlarge this period, you
will see the aggregated data. Until the period is little and you would like to see very
recent data, you will see a single line. If the period is going to enquire the database for
old data, you will see three lines. This fact is tied to history and trends; since the values
are contained in the history table, the graph will show only one line. Once you're
going to retrieve data from the trends, the lines will be three, as shown in the
following screenshot:
In the previous screenshot, we can see three lines that define a yellow area. This area
is designed by the minimum and maximum values, and the green line represents
the mean value. A quite complete discussion about trends/history tables has been
discussed in Chapter 1, Deploying Zabbix. Here it is important to have all those three
values graphed.
Longevity of an item into the history is defined into the item itself in
the field Keep history (in days) as well as the persistence on the
trends table is defined in the field Keep trends (in days).
[ 172 ]
Chapter 5
In the following screenshot, you can see how the mean values may vary with respect
to the minimum and maximum value. In particular, it is interesting to see how the
mean value remains almost the same at 12:00 also. You can see quite an important
drop in the CPU idle time (light green line), that didn't influence the mean value
(green line) too much since most likely it was only a small and quick drop, so it is
basically lost on the mean value, but not on our graph since Zabbix preserves the
minimum and maximum value.
Graphs show the working hours with a white background,
and the non-working hours in gray (using the original
template); working time is not displayed if the graph needs
to show more than three months.
Simple graphs are intended just to graph some on spot metrics, and check a
particular item. Of course, it is important to interpolate the data, for instance on
the CPU you have different metrics and it is important to have all of them.
[ 173 ]
Visualizing Data
Analyzing custom graphs
We have only discussed the graph components here rather than the full interaction
functionality and importance of them to see historical trends, or delving into some
specific time period on some particular date. Zabbix offers the custom graphs
functionality—these graphs need to be created and customized by hand. For instance,
there are some predefined graphs on the standard Template OS Linux. To create a
custom graph, you need to go to Configuration | Hosts (or Templates), and click on
Graphs and then on Create graph.
General graphs should be created into templates so that they can be easily
applied to a group of servers. An example is the graph of CPU utilization
on Template OS Linux. This one is quite general; it is composed of several
metrics aggregated and is nice to have across all your Linux servers.
Graphs on Zabbix are really a strong point of the monitoring infrastructure. Inside this
custom graph, you can choose if you want to show the working time and the legend,
by using different kind of graphs. The details of the CPU Utilization graph is shown in
the following screenshot:
[ 174 ]
Chapter 5
As you can see the following graph is stacked, and shows the legend of the x axis
defined with a fixed y axis scale. In this particular case, it doesn't make any sense to
use a variable for the minimum or maximum values of y axis since the sum of all the
components represent the whole CPU, and each component is a percentage. Since a
stacked graph represents the sum of all the stacked components, this one will always
be 100 percent as shown in the following screenshot:
There are some considerations when it comes to triggers and working hours.
These are only two checks, but they change the flavor of the graph. In the previous
graph, the working hours are displayed on the graph but not triggers, mostly because
there aren't triggers defined for those metrics. The working hours, as mentioned
earlier, are displayed in white. Displaying working hours is really useful in all the
cases where your server has two different life cycles or serves two different tasks.
As a practical example, you can think about a server placed in New York that monitors
and acquires all the market transactions of the U.S. Market. If the working hours—as
on this case—will be coincident with the market opening hours, the server will most
probably acquire data for most of the time. Think about what will happen if the same
trading company works in the Asian market; most probably they will enquire the
server in New York to see what happened while the market was open. Now in this
example, the server will provide a service for two different scenarios, and have the
working hours displayed in a graph, which can be really useful.
[ 175 ]
Visualizing Data
Displaying the working time on graphs is useful.
See if your trigger goes on fire in this period.
Now if you want to display the triggers in your graph, you only need to mark the
Show triggers checkbox and all the triggers defined will be displayed on the graph.
Now it can happen that you don't see any lines about the triggers in your graph,
for instance, looking at the following screenshot:
Now where is your expected trigger line? Well, it is simple. Since the trigger is
defined for a processor load greater than five, to display this line you need to make a
few changes in this graph, in particular the Y axis MIN value and Y axis MAX value
fields. In the default predefined CPU load graph, the minimum value is defined as
zero and the maximum value is calculated. They both need to be changed as follows:
[ 176 ]
Chapter 5
Now refresh your graph. Finally you will see the trigger line, which wasn't visible in
the previous chart because the CPU was almost idle, and the trigger threshold was
too high and not displayed due to the auto scaling on y axis.
As you probably already noticed, Zabbix doesn't display periods shorter
than an hour. The minimum graph's period is about one hour.
Zabbix supports the following kinds of custom graphs:
• Normal
• Stacked
• Pie
• Exploded
[ 177 ]
Visualizing Data
Zabbix also supports different kinds of drawing styles. Graphs that display the
network I/O, for instance, can be made using gradient lines; this will draw an
area with a marked line for the border, so you can see the incoming and outgoing
network traffic on the same scale. An example of this kind is shown in the following
screenshot which is easy to read. Since you don't have the total throughput to have
graphed the total amount from the incoming packet, the outgoing packet is the
better one to be chosen for a stacked graph. In stacked graphs, the two areas are
summarized and stacked, so the graph will display the total consumed bandwidth.
To highlight the difference between a normal graph and a stacked one, the following
screenshot displays the same graph during the same time period, so it will be easier
to compare them:
[ 178 ]
Chapter 5
As you can see, the peaks and the top line are made aggregating the network
input and output of your network card. The preceding graph represents the
whole network traffic handled by your network card.
Reviewing all the combinations of graph properties
Zabbix is a quite flexible system and the graphs are really customizable, to better
explore all the possible combination of attributes, and parameters that can be
customized. All the possible combinations of graphs attributes are reviewed in the
following table:
Parameter
Description
Name
Graph name (Note: needs to be unique)
Width
Graph width in pixels
Height
Graph height in pixels
Graph type
Normal (values displayed as lines, filled region, bold lines,
dots, dashed lines, and gradient lines)
• Stacked values are displayed as stacked areas
• Pie values are displayed as a pie
• Exploded values displayed as pie but exploded
Show legend
If checked, the graph will display the legend.
[ 179 ]
Visualizing Data
Parameter
Description
Show working time
If checked, the non-working hours will be in gray and working
hours in white background.
Show triggers
If checked, a single trigger line is displayed (Note: this is not
available for pie and exploded).
Percentile line (left/
right)
Note: only available on normal graphs. If checked, it displays
a line where the value falls under the percentage (for example,
for 90 it will display a line where 90 percent of the values
fall under).
Y axis MIN/MAX value
Set the minimum and maximum value for the y axis, which
can be any of the following:
• Calculated: The minimum and maximum values will be
calculated on the basis of the displayed area
• Fixed: The fixed value will be used as per maximum
or minimum
• Item: The last value of the selected item will be used as
minimum/maximum
3D View
It displays the graphs in 3D (Note: only available for pie and
exploded pie graphs).
Sort order
This represents the priority for a particular item over the
other and is useful to give priority to the region displayed, for
example, in front of or behind the other items. Here you can
drag-and-drop items in the right order. Note: zero is the first
processed item; Zabbix supports up to 100 items.
Name
Name of the item is displayed here. The metrics name is
composed in the form of <source> : <metric_name>.
This means that if you are inside the host configuration, you
will see <hostname>:<metric_name>. If you are creating
the graphs inside a template, you will see <template_
name>:<metric_name>.
Type
Note: this is available only on pie and exploded pie graphs.
• Simple
• Graph sum
[ 180 ]
Chapter 5
Parameter
Description
Function
What value to display if more than one are present; it can be
one of the following:
• All: Minimum, average, and maximum values
• Min: Only minimum values
• Avg: Only average values
• Max: Only maximum values
Draw style
Available only for normal graph
• Line
• Filled region
• Bold line
• Dot
• Dashed line
Y axis side
Note: available only on stacked and normal graphs,
and defines the y axis side for each elements.
Colour
Other than the standard displayed on the palette, you can set
all the colors that you want in the RGB hex format.
You can easily play with all those functionality and attributes. In Version
2.0 of Zabbix, you have a Preview tab that is really useful when you're
configuring a graph inside a host. If you're defining your graph at a
template level, this tab is useless because it doesn't display the data.
When you are working with templates, it is better to use two windows to
see in real time by refreshing (the F5 key) the changes directly against the
host that inherits the graphs from the template.
All the options previously described are really useful to customize your graphs,
as you have understood that graphs are really customizable and flexible elements.
You can display only three trigger lines, and if the graph is less than
120 pixels, triggers are not displayed; so take care to properly set up
your graphs and check all the changes.
[ 181 ]
Visualizing Data
Visualizing the data through maps
Zabbix provides a powerful element to visualize data and a topological view in
Zabbix, which will help you to create maps. Maps are a graphical representation
of your physical infrastructure, where you can display your server, network
components, and the interconnection between them.
The great thing is that maps on Zabbix are completely dynamic, which means
that you will see your active warning, issues, and triggers represented on the
map, with different icons, or color and labels. This is a powerful representation
of the status of art of your datacentre, or of the service that you're representing.
The elements that you can put in a map are as follows:
• Host
• Host groups
• Triggers
• Images
• Maps
All those elements are dynamically updated by triggers or using macros,
hence providing a complete status of the maps and their elements.
To enable a user to create, configure, or customize maps, the user needs to
be of the Zabbix admin type. This means that there isn't a role dedicated to
map creation. Also, the user needs to have a read/write permission on all
the host that he needs to put into a map. This means that there isn't a way
to restrict the privileges to the map creation only, but you can limit the
administration to a certain number of hosts included in a group.
An example of a map that you can easily produce with Zabbix is shown in the
following screenshot:
[ 182 ]
Chapter 5
In the preceding screenshot, you can see that there is a lot of graphical combination
of icons, round backgrounds, and information. To better understand what this map
represents, it is important to see how Zabbix treats hosts and triggers in a map. In the
following screenshot, you can see all the possible combinations of trigger severity and
status change:
[ 183 ]
Visualizing Data
The preceding screenshot illustrates, from left to right, the following:
• A host that doesn't have a trigger on fire
• A host with a trigger severity Not classified in the alarm
• A host with a trigger severity Information in the alarm
• A host with a trigger severity Warning in the alarm
• A host with a trigger severity Average in the alarm
• A host with a trigger severity High in the alarm
• A host with a trigger severity Disaster in the alarm
The trigger line follows the same classification.
The trigger severity is expressed with the background color and not with
the name that you see under the HOST label. The label in red color is
the name of the trigger. In the preceding screenshot, triggers are callers,
right like their classification, to simplify make the picture more verbose.
Please notice that in case of TRIGGER representation, right after the
TRIGGER label is displayed, the trigger status is displayed, and not the
trigger name that is on fire, as in the case of HOST.
Now if a trigger changed recently, its status will be displayed as shown in the
following screenshot:
Please note that in this case the icon is shown with the arrows because it just changed
the status. The following screenshot shows that there are six problems, and drilling
down which ones those are:
[ 184 ]
Chapter 5
As you can see, there are different triggers with problems. The one that has the most
critical severity is the one that gives the color to the circle around the icon. Once all
the triggers are acknowledged, the icon will show a green circle around it, as shown
in the following screenshot:
The second icon displays a detail of all the problem that your host is facing, and how
many of them are not acknowledged, so you have an immediate status of how many
issues are under control, and how many are new.
The third icon with a square background is a host that has been disabled,
represented in gray; it will be in red once unreachable.
[ 185 ]
Visualizing Data
Creating your first Zabbix Map
Map configuration can be easily reached by navigating to Configuration |
Maps | Create map. The resulting window that you will see is shown in
the following screenshot:
Most of the properties are quite intuitive; the Name field needs to be a unique name,
Width and Height are expressed in pixels.
If you define a large size, and in the second time you want to reduce it,
it is possible that some of your hosts will fall outside the map, and are
no longer visible. Don't be scared; nothing is lost. They are still inside the
map, only not displayed. You can restore it to the original size, and they
will appear again.
[ 186 ]
Chapter 5
Now we will take a look at all the other following parameters:
• Background image: In the background image field you can define your
map's background, choosing between the loaded backgrounds.
Zabbix by default doesn't have any backgrounds
defined. To add your own background, you need to go
to Administration | General, and from the listbox select
Images. Please check to add your image as Background
and not Icon. A good source for royalty free maps is
www.openstreetmap.org.
• Automatic icon mapping: This flag enables a user-defined icon mapping
for a certain host inventory field. This can be defined by navigating to
Administration | General | Icon mapping.
• Icon highlight: This is the flag responsible to generate the round background
around the icon, with the same color of the most critical severity trigger.
• Mark elements on trigger status change: This flag is responsible to highlight
a trigger status change (with the red triangle shown earlier in the screenshot
displaying the status change).
Markers are displayed only for 30 minutes; after which
they will be removed, and the changed trigger status will
become the new normal status.
• Advanced labels: This check enables you to customize the label's type for all
the elements that you can put in a map. So for each one of those items—host,
host group, trigger, map, and image—you can customize the label type.
The possible label types are as follows:
°°
Label: Icon label
°°
IP Address: IP address (note: available only on host)
°°
Element name: Element name, such as hostname
°°
Status only: This is only for the status, so OK/PROBLEM
°°
Nothing: No label at all
°°
Custom label: A free test area (note: macros are allowed)
• Icon label location: This field defines where by default you will see all the
labels. This can be selected among the following values: Bottom, Left, Right,
and Top.
[ 187 ]
Visualizing Data
• Problem display: This listbox permits you to choose between the following:
°°
All: A complete problem count will be displayed
°°
Separated: This displays the unacknowledged problem count
separated as a number of the total problem count
°°
Unacknowledged only: With this selected, only the unacknowledged
problem count will be displayed
• URLs: Here, a URL for each kind of element can be used with a label.
This label is a link, and here you can use macros, for example, {MAP.ID},
{HOSTGROUP.ID}, {HOST.ID}, {TRIGGER.ID}.
Some important considerations about macros
and URLs
The URL section is powerful, but here an example is needed, because the usage is not
so intuitive and simple.
Now if you see a trigger on fire or an alarm that is escalating, most probably the next
action that you will do is to check the latest data of your host or jump to a screen that
will group the graphs, triggers, and data that you need to check to have an idea of
what is happening, and do a first-level analysis. In the practical case of a first level
support, once a server is highlighted and shows triggers with problems, it can be
useful to have a link that will go straight ahead to the latest data of that host and
also the screen. To automate this and reduce the number of clicks, you can simply
copy the link of the desired page, for instance the link to the latest data would be
something as follows:
http://<YOUR-ZABBIX-SERVER>/zabbix/latest.
php?sid=eec82e6bdf51145f&form_refresh=1&groupid=2&hostid=10095
Now looking into the URL to automate the jump to the latest data, you need to
replace the variant part of the URL with the macros wherever available.
The sid value in the URL represents the session ID; it is passed to avoid the
one-click attack also known as session riding. This field can be removed.
The groupid value in the specific latest data example can be omitted, so the
URL can be reduced as follows:
http://<YOUR-ZABBIX-SERVER>/zabbix/latest.php?form_
refresh=1&hostid=10095
[ 188 ]
Chapter 5
Now the link is easy to generate. You can simply replace the hostid value with the
macro {HOST.ID} as follows:
http://<YOUR-ZABBIX-SERVER>/zabbix/latest.php?form_
refresh=1&hostid={HOST.ID}
And configure the URL as shown in the following screenshot:
In the preceding screenshot, you can see that there is a link configured to General
Screen that collects the most important graphs. The following is the URL generated
from the screen page of a particular host:
http://<ZABBIX-SERVER>/zabbix/screens.php?sid=eec82e6bdf51145f&form_
refresh=1&fullscreen=0&elementid=17&groupid=2&hostid=10094&period=360
0&stime=20140807161304
This time again, you can omit the sid value in the preceding URL, since it specifies
a period. If this parameter is absent, you will be taken to a screen that displays the
last hour of data. You can also remove the stime, groupid, and elementid values.
The reduced URL will be as follows:
http://<ZABBIX-SERVER>/zabbix/screens.php?form_refresh=1&fullscreen=0
&hostid=10094&groupid=2
Now, to make it dynamic, you need to replace the values of hostid and groupid
with the macros, as shown in the following URL:
http://<ZABBIX-SERVER>/zabbix/screens.php?form_refresh=1&fullscreen=0
&hostid={HOST.ID}&groupid={HOSTGROUP.ID}
[ 189 ]
Visualizing Data
The result of this customization is shown in the following screenshot:
As you can see, by clicking on the host that has issues, you have two new shortcuts
other than Latest Data and General Screen, with a link dynamically created for
each host.
This kind of behavior allows you to create a master-detail view. In this case, the master
is your map, and the detail can be for instance, the screen or the latest data window.
You can create some custom menus that can run a script, or bring you directly to the
trigger status or the Host screen.
Here you can add some more scripts to run against the
host. To add another script (and see it in the Scripts
section), you need to go to Administration | Scripts.
This will take you to the script's administration panel.
[ 190 ]
Chapter 5
Finally inside the map
Once you have completed this setup, you can begin the nice part of the configuration.
Once inside the map, the menu that you will find is quite simple and user-friendly,
as shown in the following screenshot:
In the map, you can add an element by clicking on the + sign, and remove it by
clicking on the - sign. The element will appear in the upper-left corner of the
map. By clicking on that icon, a configuration panel will appear, as shown in the
following screenshot:
[ 191 ]
Visualizing Data
The element type by default is Icon. In the preceding screenshot, it is marked as
Host, but it can be one of the following:
• Host: This icon will represent the status of all triggers of the selected host
• Map: This icon will represent the status of all the elements of the map
• Trigger: This icon will represent the status of the single trigger selected
• Host group: This icon will represent the status of all the triggers of all hosts
that belong to the selected group
• Image: This icon will just be an image not linked to any source (trigger host
and so on)
The Label section is another strong point of the element. Here you can freely write a
normal text or, use macros.
The next field may vary depending on what you choose as the element type and can
be one of the following:
• Host: This selects the host
• Map: This selects the map
• Trigger: This selects the trigger
• Host group: This selects the host group
• Icon (default): This selects the icon to be used.
With Host group, you can group all your hosts as per
the location, for example, city, nations, or continents.
This will group all the trigger status per location in a nice
representation. You can also add a custom URL.
Host and triggers have already been covered, and are quite intuitive to understand.
Probably it is not immediately understood why we should insert a map inside a map.
An efficient usage of this scenario is that you can produce a nice drill down having
a general map view that gathers together all the sub maps, detailed per location or
nation. This helps you produce a drill down until the final destination, for instance,
try to think about a drill down that comes from nations, down to the city, and deep
into the datacenter, ending on the rack where your server is contained.
[ 192 ]
Chapter 5
The Icon element inside a map is an image that can have a URL associated with
it. Their function is to add some graphic element to your map that contains a URL,
and have the shortcuts directly on your own map.
Right after that, there is the Icons section. Here, if you checked the Automatic icon
selection checkbox, the icon mapping (defined in the map configuration) will be
used to choose the icons to be used.
Defining an icon mapping in the map configuration will save
you a lot of clicks, and also it is a repetitive task. So for instance,
you can define your standard icons for the hosts and they will
then be used here.
If you haven't defined an icon mapping, or if you want to use an item different
from the previous choice, you can specify the icons that will be used on those
cases, which can be one of the following:
• Default
• Problem
• Maintenance
• Disable
The Coordinates section expresses the exact location of the element in pixels, and as
the previous item, you can configure some dedicated URL for this kind of host also.
Imagine that you have produced different kinds of screens (the screens will be
discussed later in this chapter), one that collects all the metrics graphs and triggers
used to monitor an application server, and another one with all the metrics needed
to monitor the status of your database. Well, here if your host is a DBMS, you can
create a URL to jump directly on the RDBMS screen. If it is an application server,
you can create a custom URL that will drive you directly to the application server
screens, and so on.
As you can see, this is an interesting feature and will make your map useful for your
support team.
[ 193 ]
Visualizing Data
Selecting elements
In the map configuration, you can select multiple elements by selecting the first
one, and then keeping the Crtl (or Shift) key pressed, selecting the other elements.
For a multiple selection, you can drag a rectangle area which will then select all
the elements in the drawn rectangle.
Once you have selected more than one element, the element form switches to the
Mass update elements as shown in the following screenshot:
Here you can update the icons, label, and label location for all the selected hosts
in bulk.
To have an efficient update of all the labels, it is strongly advised to use macros.
Now it's time to link your server between them, exactly the way they are physically
connected. To create a link between two hosts, you only need to select the couple of
hosts that need to be linked together, and click on the + symbol in the Link section of
the map.
[ 194 ]
Chapter 5
The links section will appear right below the Mass update elements form, as shown
in the following screenshot:
You can customize your link with labels and also change the representation type and
color. You can choose between a Line, Bold line, Dot, and Dashed line.
An option to keep in mind here is the possibility to connect the link indicator to a
trigger, so basically the link will change its color once a trigger is on fire.
Here you can connect the link to multiple triggers and associate a
different color to each one of them so that you can understand which
trigger is changing your link.
Playing with macros inside maps
Previously, we discussed about the Label section where you can customize the
label in your graphs. Here I think an example can clarify a lot about the power
of this section, and how this can improve and introduce benefits in your maps.
As an example, you can play with macros inside the map. Now you have some
requirement for this, such as, you need to show directly in the map the hostname,
IP address, the status of triggers events (how many acknowledged and how many
unacknowledged), and the network traffic of your network interfaces.
[ 195 ]
Visualizing Data
This seems as a challenging work, and in fact it is; but if you have a bit of knowledge
about macros, this becomes an easy task. The request can be satisfied with the
following code:
{HOSTNAME}
{HOST.CONN}
trigger events ack: {TRIGGER.EVENTS.ACK}
trigger events unack: {TRIGGER.EVENTS.UNACK}
Incoming traffic: {{HOSTNAME}:net.if.in[eth0].last(0)}
Outgoing traffic: {{HOSTNAME}:net.if.out[eth0].last(0)}
The first macro {HOSTNAME} will display the hostname of your selected host.
The second macro {HOST.CONN} will display the IP address. The information
about the trigger's events, whether being acknowledged or unacknowledged is
expressed in the next two lines using the macros {TRIGGERS.EVENTS.ACK} and
{TRIGGER.EVENTS.UNACK}. The last two lines are more interesting because they
are a composition of two nested macros.
In particular, to display the incoming traffic of your first network interface, you can
ask Zabbix to retrieve the last value of the net.if.in[eht0] item. This kind of
expression needs the hostname to be evaluated, so you need to write your hostname,
for example, HOST-A in this example, or use macros.
The last information that Zabbix needs to produce as our requested output is the
hostname. As mentioned earlier, this can be replaced with the {HOSTNAME} macro.
So the complete expression will be as follows:
Incoming traffic: {{HOSTNAME}:net.if.in[eth0].last(0)}
Obviously for the outgoing traffic, the expression is more or less the same, except that
you need to retrieve the net.if.out[eth0] item of the network card. The result is
shown in the following screenshot:
[ 196 ]
Chapter 5
Use {HOSTNAME} or {HOST.NAME} in all the labels and all
the places where it is possible, so it will make things easy in
case of a mass update.
This is a comprehensive and charming output, wherein without any clicks, you have
your needed information directly in your map. In this example, you used the last()
value of your item, but the other function are also supported here such as last(),
min(), max(), and avg().
Macros can be used in the same manner on links; an example is shown in the
following screenshot:
In the preceding screenshot the traffic data on the link is generated using the same
method previously explained. All those macro usages make your maps a lot more
dynamic and appealing.
Visualizing through screens
In the previous section, we discussed about adding custom URLs and introduced
shortcuts to some screen section. Now it's time to go deep into screens. Screens are
easy to generate, and very intuitive to handle. Basically, a screen is a page that can
display multiple Zabbix elements, such as graphs, maps, and text. One of the main
difference between screens and maps is that in maps, you can place a lot of elements,
but you can't for instance, add a graph or the trigger status. They have two different
targets. The screen can group all the elements that are common to a particular kind
of server to have a complete picture of the situation.
[ 197 ]
Visualizing Data
Creating a screen
To create a screen, you need to navigate to Configuration | Screen | Create. A form
will appear, asking for the screen name and the initial size in terms of columns and
rows. After this step, you need to come back inside the screen that you just created.
In this part of the configuration, probably you will notice that there isn't a save
button. This is because screens are saved automatically every time that you complete
an action such as add a graph. The screen appears like a table (basically it is a table),
as shown in the following screenshot:
Now if you need more rows or column, you only need to click on the + sign where
you want to add fields as rows or as column.
On a screen, you can add the following kinds of elements:
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
Data overview: This displays the latest data for a group of hosts
Graph: This displays whether the graphs are single or custom
History of actions: This displays n lines of recent actions
History of events: This displays n lines (you can specify how many) of
latest events
Hosts info: This displays a high-level host related information
Map: This displays a single map
Plain text: This shows the plain text data
Screen: This displays another screen (one screen may contain other screens
inside it)
Server info: This displays the high-level server information
Status of hostgroup triggers: This displays the status of triggers filtered
per hostgroup
Status of host triggers: This displays the status of triggers filtered per host
System status: This displays the system status (it is close to the Dashboard)
Triggers info: This displays the high-level trigger related information
Triggers overview: This displays the status of triggers for a selected
host group
URL: Here you can include content from an external source
[ 198 ]
Chapter 5
All those sources have two common configuration parameters—the column span
and the row span. With the column span you can extend a cell to a certain number
of columns. So also, with the row span, you can extend a cell for a certain number
of rows. For instance, in a table of two columns, if you indicate a column span of
two, the cell will be centered in the table and will be widened with exactly two
fields. This is useful to add a header to your screen.
Once you have inserted and configured your elements on the screen, you can move it
using drag-and-drop, and all your settings will be preserved.
You can freely drag-and-drop your configured element; they will
not lose their settings.
Most of the elements are not dynamic, which means that they will not be
dynamically applied to all your hosts in a group.
Dynamic elements
Zabbix provides some dynamic elements that are really useful, which are as follows:
• Graphs
• Simple graphs
• Plain text
Dynamic items can be identified by having the following option checked:
Now for instance, you can add a map in your screen and some dynamic elements
such as graphs. When you add a dynamic element on top of the screen, you will have
a bar with some listboxes that will help you to change the target of your dynamic
elements, as shown in the following screenshot:
[ 199 ]
Visualizing Data
An example of a screen that mixes dynamic elements and standard elements is
shown in the following screenshot:
Obviously, when you choose and host this, it will affect only the dynamic graphs.
You can switch between your two hosts and change the x axis. This will update
all the dynamic elements on the same time base, making it easy to relate elements
between them.
Pie graphs and exploded pie graphs will display only the average value
for your chosen period. To relate different groups of metrics between
them, it is better to use a custom graph.
[ 200 ]
Chapter 5
Visualizing the data through a slide show
Once you have created all your screens, you can provide a slide show to your
helpdesk that implements a screen rotation.
Creating a slide show is easy; go to Configuration | Slide shows | Create slide
show. A window as shown in the following screenshot will appear:
You can see the slide show configuration in the preceding screenshot.
This configuration screen is really intuitive; Name identifies the name
of your slide show, and in Default delay (in seconds) you need to
specify the default delay that will be applied to all the screens on the
slide show.
In the Slides section—enumerated in the preceding screenshot to have a visualization
order—you can specify a different delay for each one of them. Basically you can
customize how long each screen will be displayed. Once saved, your slide show will be
available to be displayed by going to Monitoring | Screens, and then select the slide
show you just created.
On slide show, you can only create a screen rotation. So to add
a single element such as a graph or a map on your rotation,
you need to create a screen that contains your elements to be
displayed. By using this method, you can basically add all that
can be represented on a screen on the slide show.
[ 201 ]
Visualizing Data
If you want to speed up or slow down your slide show, you can change the refresh
multiplier that will appear by clicking on the Menu icon (on the right-hand side of
the listbox), which will return a window as shown in the following screenshot:
Controlling centre slides and the big
display challenge
Displaying data on a big display is something challenging; first of all you need to
know who will be your target, their skills, and which role do they exactly cover.
After that, it is important to see where the screen is physically and the resolution of
the same.
You will most probably need to create an ad hoc screen for a big display to fit better
on a widescreen. Screens for widescreen need to be developed horizontally. Most of
the screens are usually developed with a web page in mind, so most probably they
need to be scrolled down and up to read all the graphs. Such kinds of screens will
not fit on your widescreen.
You need to bear in mind that the slide show doesn't scroll up and down
automatically. You can add a JavaScript to make it possible, but it is really complex
to implement a screen that will rightly handle this scroll up and down, and all this
effort can be hardly justified. It is better and more productive to produce screens,
so that slide shows fit in the screen dimensions and resolution.
[ 202 ]
Chapter 5
Some consideration about slides on a
big display
Once you have understood which one your target is, their knowledge, and typology
of the work they do, you are already in a stable position. Now you can apply some
best practice which is generally useful when you need to show data on a big display.
Basically they need to be one of the following:
• Easy to read (comprehensive)
• Fit the big screen display
• Non-interactive
• Choose an appropriate delay between the screens
First of all keep things easy. This is a general rule: the easier are the representations,
the lesser the effort required by the helpdesk to read them. An easy and
comprehensive screen will improve the helpdesk's reaction, which is our goal.
You need to provide information in the best way. Never overload your screen with
information; you need to choose the right amount of information to be displayed and
fonts need to be comprehensive. Essentially you need to choose which cut will have
every screen, and the challenge is to choose how to display your monitored services.
If you need to monitor a large amount of services, you need to choose the time to
change the slide; don't spend too much time on the same screen. Keeping a screen for
too much time on the monitor can become annoying, especially when the helpdesk is
waiting to see the next slide. Unfortunately there isn't a rule; you need to spend time
with the first-level support and check with them as to what is the perfect timing.
One thing that simplifies the work is that you don't need to think about
complex screens. A widescreen display doesn't need the drill-down feature
implemented. People will just look at the screen, and the analysis will be
done from their workstation.
Triggers are always welcome as they are easy to read, comprehensive,
and immediate. But take care not to fill a page with them as it will then
be unreadable.
[ 203 ]
Visualizing Data
Automated slide show
Once you're slides are created, and you are ready to run them, it is time to think
about the user. Your widescreen and the relative workstation dedicated to this
task needs to have an account for sure.
In a real-world scenario, we do not want to see the login page on a big display.
To avoid this, it is better to create a dedicated account with some customization.
The requirements are as follows:
• Avoid an automatic disconnection
• Avoid the clicks needed to display the slide show
Both these features will be appreciated from your helpdesk.
To avoid the automatic disconnection, there is a flag on the user configuration form
that is designed for that—the Auto-login flag. Once selected, you need to log in only
the first time.
The Auto-login flag will work only if your browser supports cookies;
please ensure that plugins, antivirus, and so on are not blocked.
Now since you have created a dedicated account, you need to customize the URL
(after login) section; here you need to link the URL to your screen.
To retrieve the appropriate URL, browse to your slide show and copy your link.
For this example, the link would be as follows:
http://<your-zabbix-server>/zabbix/slides.
php?sid=4258s278fa96eb&form_refresh=1&fullscreen=0&elementid=3
Basically, you need to know the elementid value of your slide show. In the
following URL, you can remove the sid parameter. The definitive URL in our
case will be as follows:
http://<your-zabbix-server>/zabbix/slides.php? form_refresh=1&fullscr
een=0&elementid=3
To jump directly to the full screen mode, change the parameter
fullscreen=0 to fullscreen=1. This will further reduce the
human interaction.
[ 204 ]
Chapter 5
Now this account has a first page. After login, the slide show starts in the full screen
mode with really little human interaction.
IT services
The last graphical element that will be discussed in this chapter is a high-level
view of our monitored infrastructure. In a business-level view, there is no provision
for low-level details such as the CPU usage memory consumption and free space.
What the business would like to see is the availability of your services provided,
and the service-level agreements of your IT services.
Zabbix covers this point with IT services. A service is a hierarchical view of your
service. Now imagine that you need to monitor your website (we discussed about
SLAs in Chapter 1, Deploying Zabbix). You need to identify your service components
for example, Web server, Application server, and DB server. For each one of them
you need to identify triggers that tell you whether the service is available or not.
The hierarchical view is the one represented in the following screenshot:
In this hierarchy each node has a status; this status is calculated on trigger basis,
and propagated to the higher level with the selected algorithm. So the lowest level
of IT services is managed via triggers.
Triggers are the core of IT service calculations; so as you can imagine,
they are of particular importance and really critical. You need to find
out which are your effective items to check for this trigger generation.
Triggers with severity as Information and Not classified are not considered,
and don't affect the SLA calculation.
[ 205 ]
Visualizing Data
Configuring an IT service
The way to configure an IT service is by navigating to Configuration | IT services;
you can create your service here. The following screenshot displays the service
previously configured:
By clicking on a service you can add a service, edit the current service, or delete it.
The service configuration is composed of three forms: the first one describes the
service, the second tab is dedicated to the dependencies, and the third is to the time.
In the service tab, you need to define your own service name. In this particular
example, the website SLA is calculated; of course a website is composed of different
components such as the Web server, Application server, and DBMS. In a three-level
environment, they are usually on a dedicated server. Now since all the three
components are vital for our merchant website, we need to calculate the SLA
propagating the problems. This means that if the child of our website has a problem,
the whole website has a problem, and this will reflect in the SLA calculation.
Zabbix provides the following three options on the status calculation algorithm:
• Do not calculate: This option ignores the status calculation completely.
• Problem, if at least one child has a problem: This means that if each
one of our three components has an issue, the service will be considered
unavailable. This is the case when each one of the server doesn't have a
failover node.
• Problem, if all the children has a problem: To propagate the problem, all
the children need to be affected by the problem. This case is typical for a
clustered or load-balanced service, where there are many nodes to provide
a service, and all the nodes need to be down to propagate the issue to the
parent node.
Once you define the algorithm, you need to define the SLA percentage of your
service. This is used to display the SLA issue with different colors on the report.
[ 206 ]
Chapter 5
The next step is the trigger definition that will enable Zabbix to know when your
service has an issue. Since Zabbix provides a hierarchical view, you can have a
service composed of many components so the intermediate level can avoid a trigger
definition that is needed on the lowest level.
The last option is Sort order(0->999). This of course doesn't affect the SLA calculation,
but is only for cosmetic purposes. To visualize on report, for instance, your three levels
are sorted in a logical order as Web server, Application server, and DB server. All that
is previously discussed is shown in the following screenshot:
The following screenshot shows the dependencies; here you don't need to define
each one of them, because they are defined automatically once you design your
hierarchical view. Now it is possible that one of your service is already defined for
some reason in another layer of the service. If this is the case, you only need to mark
the service as soft linked, by ticking the Soft checkbox.
If a service has only soft linked dependencies, it can be deleted. In this
case, you don't need to delete all the children services first; this can be
used to quickly delete the whole service.
[ 207 ]
Visualizing Data
The last tab is used to set the service time. By default, Zabbix considers that a
service needs to be available 24 hours a day, for seven days, and the whole year
(24 x 7 x 365). Fortunately for the system administrators, not all the services
have this requirement. If you are on this case, you can define your Uptime and
Downtime periods, as shown in the following screenshot:
The periods defined here are basically Uptime and Downtime windows.
A problem that occurs during a Downtime window will not affect the
SLA calculation. Here it is possible to also add a one-time downtime,
useful to define an agreed maintenance without an impact on the SLA.
Once you completed the hierarchical definition of your service, the result is available
by navigating to Monitoring | IT services.
[ 208 ]
Chapter 5
Summary
In this chapter, we covered the entire graphical elements that Zabbix provides,
and showed how to use them in an efficient way. This chapter also enabled you
to deliver efficient slide shows to your helpdesk, making you aware of the best
practices in this difficult task. Now you probably understood that this part of
the Zabbix setup will require time to be well implemented. Also, it is easier to
understand for the masses, and the information provided with the graphical
elements has a big impact on your audience. This forces you to be precise and
take a lot of care in this task, but this will be paid back to you providing a lot
of powerful elements to use, to provide more strength to your speech. All those
graphical elements are fundamentals if you need to argue with the business or
the purchase manager to expand the business or buy hardware.
In the next chapter, you will see how to manage complex triggers and trigger
conditions. The next chapter will also make you aware of the right amount of
triggers and alarms you should implement to not be overloaded by alarms,
with the consequence of losing the critical ones.
[ 209 ]
Managing Alerts
Checking conditions and alarms are the most characteristic functions of any
monitoring system, and Zabbix is no exception. What really sets Zabbix apart is
that every alarm condition, or triggers—as they are known in this system—can be
tied, not only to a single measurement but also to an arbitrary complex calculation,
based on all of the data available to the Zabbix server. Furthermore, just as triggers
are independent from items, the actions that the server can take based on trigger
status are independent from the single trigger, as you will see in the
subsequent sections.
In this chapter, you will learn the following things about triggers and actions:
• How to create complex, intelligent triggers
• How to minimize the possibility of false positives
• How to set up Zabbix to take automatic actions based on the trigger status
• How to rely on escalating actions
An efficient, correct, and comprehensive alerting configuration is a key to the success
of a monitoring system. It's based on an extensive data collection, as discussed in
Chapter 4, Collecting Data, and eventually leads to managing messages, recipients,
and delivery media, as we'll see later in the chapter. But all this revolves around the
conditions defined for the checks, and this is the main business of triggers.
Managing Alerts
Understanding trigger expressions
Triggers are quite simple to create and configure—choose a name and a
severity, define a simple expression using the expression form, and you
are done. The expression form, accessible through the Add button, lets you
choose an item, a function to perform on the item's data, and some additional
parameters, and gives an output as shown in the following screenshot:
You can see how there's a complete item key specification, not just the name, to which
a function is applied. The result is then compared to a constant using the greater than
operator. The syntax for referencing the item keys is very similar to that of a calculated
item. In addition to this basic way of referring to item values, triggers also add a
comparison operator that wraps all the calculations up to a Boolean expression.
This is the one great unifier of all triggers; no matter how complex the expression,
it must always return either a True value or a False value. This value is of course
directly related to the state of a trigger, which can only be OK if the expression
evaluates to False, or PROBLEM, if the expression evaluates to True. There are no
intermediate or soft states for triggers.
A trigger can also be in an UNKNOWN state if it's impossible to
evaluate the trigger expression (because one of the items has
no data, for example).
A trigger expression has two main components:
• Functions applied to the item data
• Arithmetical and logical operations performed on the functions' results
[ 212 ]
Chapter 6
From a syntactical point of view, the item and function component has to be enclosed
in curly brackets, as illustrated in the preceding screenshot, while the arithmetical
and logical operators stay outside the brackets.
Selecting items and functions
You can reference as many items as you want in a trigger expression, as long as
you apply a single function to every single item. This means that if you want to
use the same item twice, you'll need to specify it twice completely, as shown in
the following code:
{Alpha:log[/tmp/operations.log,,,10,skip].nodata(600)}=1 |
{Alpha:log[/tmp/operations.log,,,10,skip].str(error)}=1
The previously discussed trigger will evaluate to PROBLEM if there are no new lines in
the operations.log file for more than ten minutes, or if an error string is found in
the lines appended to that same file.
Zabbix doesn't apply short-circuit evaluation of the and and
or (& and | ) operators—every comparison will be evaluated
regardless of the outcome of the preceding ones.
Of course, you don't have to reference items from the same host; you can reference
different items from different hosts, and on different nodes too, if you can access
them, as shown in the following code:
{Node1:Alpha:agent.ping.last(0)}=0 &
{Node2:Beta:agent.ping.last(0)}=0
Here, the trigger will evaluate to PROBLEM if both the hosts Alpha and Beta are
unreachable. It doesn't matter that the two hosts are monitored by two different
nodes. Everything will work as expected as long as the node where the trigger is
defined has access to the two monitored hosts' historical data. In other words,
the trigger has to be configured on a master node that receives data from both
Node1 and Node2.
You can apply all the same functions available for calculated items to your items'
data. The complete list and specification is available on the official Zabbix wiki
(https://www.zabbix.com/documentation/2.0/manual/appendix/triggers/
functions), so it would be redundant to repeat them here, but a few common
aspects among them deserve a closer look.
[ 213 ]
Managing Alerts
Choosing between seconds or number
of measurements
Many trigger functions take a sec or #num argument. This means that you can
either specify a time period in seconds, or a number of measurements, and the
trigger will take all of the item's data in the said period, and apply the function
to it. So, the following code will take the minimum value of Alpha's CPU idle
time in the last ten minutes:
{Alpha:system.cpu.util[,idle].min(600)}
The following code, unlike the previous one, will perform the same operation on
the last ten measurements:
{Alpha:system.cpu.util[,idle].min(#10)}
Will perform the same operation on the last ten measurements.
Instead of a value in seconds, you can also specify
things such as 10m for ten minutes, 2d for two days,
and 6h for six hours.
Which one should you use in your triggers? While it obviously depends on your
specific needs and objectives, each one has its strengths that makes it useful in the
right context. For all kinds of passive checks initiated by the server, you'll often
want to stick to a time period expressed as an absolute value. A #5 parameter will
vary quite dramatically as a time period if you vary the check interval of the relative
item. It's not usually obvious that such a change will also affect related triggers.
Moreover, a time period expressed in seconds may be closer to what you really
mean to check, and thus may be more easy to understand when you visit the
trigger definition at a later date. On the other hand, you'll often want to opt for
the #num version of the parameter for many active checks, where there's no
guarantee that you will have a constant, reliable interval between measurements.
This is especially true for trapper items of any kind, and for logfiles. With these
kinds of items, referencing the number of measurements is often the best option.
[ 214 ]
Chapter 6
Date and time functions
All the functions that return a time value, whether it's the current date, the current
time, the day of the month, or the day of the week, still need a valid item as part of the
expression. These can be useful to create triggers that may change status only during
certain times of day, or during certain specific days, or better yet, to define well known
exceptions to common triggers, when we know that some otherwise unusual behavior
is to be expected. For example, a case where there's a bug in one of your company's
applications that causes a rogue process to quickly fill up a filesystem with huge
logfiles. While the development team is working on it, they ask you to keep an eye on
the said filesystem, and kill the process if it's filling the disk up too quickly. Like with
many things in Zabbix, there's more than one way to approach this problem, but you
decide to keep it simple and find that, after watching the trending data on the host's
disk usage, a good indicator that the process is going rogue is that the filesystem has
grown by more than 3 percent in 10 minutes:
{Alpha:vfs.fs.size[/var,pused].delta(600)}>3
The only problem with this expression is that there's a completely unrelated process
that makes a couple of big file transfers to this same filesystem every night at 2 a.m.
While this is a perfectly normal operation, it could still make the trigger switch to a
PROBLEM state and send an alert. Adding a couple of time functions will take care of
that, as shown in the following code:
{Alpha:vfs.fs.size[/var,pused].delta(600)}>3 &
({Alpha:vfs.fs.size[/var,pused].time(0)}<020000 |
{Alpha:vfs.fs.size[/var,pused].time(0)}>030000 )
Just keep in mind that all the trigger functions return a numerical value, including
the date and time ones, so it's not really practical to express fancy dates like "the first
Tuesday of the month" or "last month" (instead of the last 30 days).
[ 215 ]
Managing Alerts
Trigger severity
Severity is little more than a simple label that you attach to a trigger. The web
frontend will display different severity values with different colors, and you will
be able to create different actions based on them, but they have no further meaning
or function in the system. This means that the severity of a trigger will not change
over time based on how long that trigger has been in a PROBLEM state, nor can you
assign a different severity to different thresholds in the same trigger. If you really
need a warning alert when a disk is over 90 percent full, and a critical alert when
it's 100 percent full, you will need to create two different triggers with two different
thresholds and severity. This may not be the best course of action though, as it could
lead to either warnings that are ignored and not acted upon, or critical warnings that
will fire up when it's already too late and you have already lost service availability,
or even just a redundant configuration with redundant messages and more
possibilities for mistakes, or an increased signal-to-noise ratio.
A better approach would be to clearly assess the actual severity of the possibility for
the disk to fill up, and create just one trigger with a sensible threshold, and possibly
an escalating action if you fear that the warning could get lost among the others.
Choosing between absolute values and percentages
If you look at many native agent items, you'll see that a lot of them can express
measurements either as absolute values or as percentages. It often makes sense
to do the same while creating one's own custom items, as both representations can
be quite useful in and on themselves. When it comes to creating triggers on them
though, the two can differ quite a lot, especially if you have the task to keep track
of available disk space.
Filesystems' sizes and disk usage patterns vary quite a lot between different servers,
installations, application implementations, and user engagement. While a free
space of 5 percent of a hypothetical disk A could be small enough that it would
make sense to trigger a warning and act upon it, the same 5 percent could mean
a lot more space for a large disk array, enough that you don't really need to act
immediately, but can plan a possible expansion without any urgency. This may
lead you to think that percentages are not really useful in these cases, and even that
you can't really put disk-space-related triggers in templates, as it would be better to
evaluate every single case and build triggers that are tailor-made for every particular
disk with its particular usage pattern. While this can certainly be a sensible course
of action for particularly sensible and critical filesystems, it can quickly become too
much work in a large environment where you may need to monitor hundreds of
different filesystems.
[ 216 ]
Chapter 6
This is where the delta function can help you create triggers that are general enough
that you can apply them to a wide variety of filesystems, so that you can still get
a sensible warning about each one of them. You will still need to create more
specialized triggers for those special, critical disks, but you'd have to anyway.
While it's true that the same percentages may mean quite a different thing for disks
with a great difference in size, a similar percentage variation of available space on a
different disk could mean quite the same thing; the disk is filling up at a rate that can
soon become a problem:
{Template_fs:vfs.fs.size[/,pfree].last(0)}<5 &
({Template_fs:vfs.fs.size[/,pfree].delta(1d)} /
{Template_fs:vfs.fs.size[/,pfree].last(0,1d) } > 0.5)
The previously discussed trigger would report a PROBLEM state not just if the available
space is less than 5 percent on a particular disk, but also if the available space has been
reduced by more than half in the last 24 hours (don't miss the time-shift parameter in
the last function). This means that no matter how big the disk is, based on its usage
pattern, it could quickly fill up very soon. Note also how the trigger would need
progressively smaller and smaller percentages for it to assume a PROBLEM state,
so you'd automatically get more frequent and urgent notifications as the disk is
filling up.
For these kinds of checks, percentage values should prove more flexible and easy to
understand than absolute ones, so that's what you probably want to use as a baseline
for templates. On the other hand, absolute values may be your best option if you
want to create a very specific trigger for a very specific filesystem.
Understanding operations as correlations
As you may have already realized, practically every interesting trigger expression is
built as a logical operation between two or more simpler expressions. Naturally, it is
not that this is the only way to create useful triggers. Many simple checks on the status
of an agent.ping item can literally save the day when quickly acted upon, but Zabbix
also makes it possible, and relatively easy, to define powerful checks that would
require a lot of custom coding to implement in other systems. Let's see a few more
examples of relatively complex triggers.
[ 217 ]
Managing Alerts
Going back to the date and time functions, let's say that you have a trigger that
monitors the number of active sessions in an application, and fires up an alert if that
number drops too low during certain hours, because you know that there should
always be a few automated processes creating and using sessions in that window of
time (from 10:30 to 12:30 in this example). During the rest of the day, the number of
sessions is neither predictable nor that significant, so you keep sampling it but don't
want to receive any alert. A first, simple version of your trigger could look like the
following code:
{Appserver:sessions.active[myapp].min(300)}<5 &
{Appserver:sessions.active[myapp].time(0)} > 103000 &
{Appserver:sessions.active[myapp].time(0) } < 123000
The session.active item key could reference a custom
script, a calculated item or anything else. It's used here as
a label to make the example easier to read, and not as an
instance of an actual ready-to-use native item.
The only problem with this trigger is that if the number of sessions drops below five
in that window of time, but it doesn't come up again until after 12:30, the trigger will
stay in the PROBLEM state until the next day. This may be a great nuisance if you have
set up multiple actions and escalations on that trigger, as they would go on for a
whole day no matter what you do to address the actual sessions problems. But even
if you don't have escalating actions, you may have to give accurate reports on these
event durations, and an event that looks like it's going on for almost 24 hours would
be both incorrect in itself and for any SLA reporting. Even if you don't have reporting
concerns, displaying a PROBLEM state when it's not there anymore is a kind of false
positive that will not let your monitoring team focus on the real problems, and over
the time, may reduce their attention on that particular trigger.
A possible solution is to make the trigger return to an OK state outside the target
hours, if it was in a PROBLEM state, as shown in the following code:
({Appserver:sessions.active[myapp].min(300)}<5 &
{Appserver:sessions.active[myapp].time(0)} > 103000 &
{Appserver:sessions.active[myapp].time(0) } < 123000)) |
( {TRIGGER.VALUE}=1 &
{Appserver:sessions.active[myapp].min(300)}<0 &
({Appserver:sessions.active[myapp].time(0)} < 103000 |
{Appserver:sessions.active[myapp].time(0) } > 123000) )
[ 218 ]
Chapter 6
The first three lines are identical to the trigger defined before. This time there is one
more complex condition, as follows:
• The trigger is in a PROBLEM state (see the note about the
TRIGGER.VALUE macro)
• The number of sessions is less than zero (this can never be true)
• We are outside the target hours (the last two lines are the opposite
of those defining the time frame preceding it)
The TRIGGER.VALUE macro represents the current value of the
trigger expressed as a number. A value of 0 means OK, 1 means
PROBLEM, and 2 means UNKNOWN. The macro can be used anywhere
you could use an item.function pair, so you'll typically enclose
it in curly brackets. As you've seen in the preceding example, it can
be quite useful when you need to define different thresholds and
conditions depending on the trigger's status itself.
The condition about the number of sessions being less than zero makes sure that
outside the target hours, if the trigger was in a PROBLEM state, the whole expression
will evaluate to false anyway. False means the trigger is switching to an OK state.
Here, you have not only made a correlation between an item value and a window
of time to generate an event, but you have also made sure that the event will always
spin down gracefully instead of potentially going out of control.
Another interesting way to build a trigger is to combine different items from the
same hosts, or even different items from different hosts. This is often used to spot
incongruities in your systems' state that would otherwise be very difficult to identify.
An obvious case could be that of a server that serves content over the network.
Its overall performance parameters may vary a lot depending on a great number
of factors, and so it would be very difficult to identify sensible trigger thresholds
that wouldn't generate a lot of false positives, or even worse, missed events.
What may be certain though is that if you see a high CPU load while network
traffic is low, then you may have a problem, as shown in the following code:
{Alpha:system.cpu.load[all,avg5].last(0)} > 5 &
{Alpha:net.if.total[eth0].avg(300)} < 1000000
[ 219 ]
Managing Alerts
An even better example would be the necessity to check for hanging or freezed
sessions in an application. The actual way to do it would depend a lot on the specific
implementation of the said application, but for illustrative purposes, let's say that a
frontend component keeps a number of temporary session files in a specific directory,
while the database component populates a table with the session data. Even if you
have created items on two different hosts to keep track of these two sources of data,
each number taken alone will certainly be useful for trending analysis and capacity
planning, but they need to be compared to check if something's wrong in the
application's workflow. Assuming that we have previously defined a local
command on the frontend's Zabbix agent that will return the number of files in a
specific directory, and an odbc item on the database host that will query the DB for the
number of active sessions, we could then build a trigger that compares the two values
and report a PROBLEM state if they don't match:
{Frontend:dir.count[/var/sessions].last(0)} #
{Database:sessions.count.last(0)}
The # term in the expression is the not
equal operator.
Aggregated and calculated items can also be very useful for building effective
triggers. The following one will make sure that the ratio between active workers
and the available servers doesn't drop too low in a grid or cluster:
{ZbxMain:grpsum["grid", "proc.num[listener]", last, 0].last(0)} /
{ZbxMain:grpsum["grid", "agent.ping", last, 0].last(0)} < 0.5
All these examples should help drive home the fact that once you move beyond
checking for simple thresholds with single item values, and start correlating different
data sources together in order to have more sophisticated and meaningful triggers,
there is virtually no end to all the possible variations of trigger expressions that you
can come up with.
By identifying the right metrics, as explained in Chapter 4, Collecting Data,
and combining them in various ways, you can pinpoint very specific aspects
of your systems' behavior; you can check logfiles together with the login events
and network activity to track down possible security breaches, compare a single
server performance with the average server performance in the same group to
identify possible problems in service delivery, and much more.
[ 220 ]
Chapter 6
This is, in fact, one of Zabbix's best kept secrets that really deserves more publicity;
its triggering system is actually a sophisticated correlation engine that draws its
power from a clear and concise method to construct expressions as well as from the
availability of a vast collection of both current and historical data. Spending a bit of
your time studying it in detail, and coming up with interesting and useful triggers
tailor-made for your needs will certainly pay you back tenfold, as you will end up
not only with a perfectly efficient and intelligent monitoring system, but also with a
much deeper understanding of your environment.
Managing the trigger dependencies
It's quite common that the availability of a service or a host doesn't depend only
on the said host in itself, but also on the availability of any other machine that may
provide connectivity to it. For example, if a router goes down isolating an entire
subnet, you would still get alerts about all the hosts in the said network that will
suddenly be seen as unavailable from Zabbix's point of view, even if it's really the
router's fault. A dependency relationship between the router and the hosts behind it
would help alleviate the problem, because it would make the server skip any trigger
check for the hosts in the subnet, should the router become unavailable. While Zabbix
doesn't support the kind of host-to-host dependencies that other systems do, it does
have a trigger-to-trigger dependency feature that can largely perform the same
function. For every trigger definition, you can specify a different trigger upon
which your new trigger is dependent. If the parent trigger is in a PROBLEM state,
the trigger you are defining won't be checked, until the parent returns to an OK state.
This approach is certainly quite flexible and powerful, but it also has a couple of
downsides. The first one is that one single host can have a significant number of
triggers, so if you want to define a host-to-host dependency, you'll need to update
every single trigger, which may prove to be quite a cumbersome task. You can of
course, rely on the mass update feature of the web frontend as a partial workaround.
A second problem is that you won't be able to look at a host definition and see that
there is a dependency relationship with another host. Short of looking at a host's
trigger configuration, there's simply no easy way to display or visualize this kind of
relationship in Zabbix.
[ 221 ]
Managing Alerts
A distinct advantage of having a trigger-level dependency feature is that you can
define dependencies between single services on different hosts. As an example,
you could have a database that serves a bunch of web applications on different web
servers. If the database is unavailable, none of the related websites will work,
so you may want to set up a dependency between the web monitoring triggers and
the availability of the database. On the same servers, you may also have some other
service that relies on a separate license server, or an identity and authentication
server. You could then set up the appropriate dependencies, so that you could end
up having some triggers depend on the availability of one server, and other triggers
depend on the availability of another one, all in the same host. While this kind of
configuration can easily become quite complex and difficult to maintain efficiently,
a select few, well-placed dependencies can help cut down the amount of redundant
alerts in a large environment. This in turn would help you focus immediately on the
real problems where they arise, instead of having to hunt them down in a long list of
trigger alerts.
Taking action
Just as items only provide raw data, and triggers are independent from them as they
can access virtually any item's historical data, triggers in turn only provide a status
change. This change is recorded as an event, just like measurements are recorded
as item data. This means that triggers don't provide any reporting functionality;
they just check their conditions and change the status accordingly. Once again,
what may seem like a limitation and a lack of power turns out to be the exact
opposite, as the Zabbix component in charge of actually sending out alerts,
or trying to automatically resolve some problems, is completely independent
from triggers. This means that just like triggers can access any item's data,
actions can access any trigger's name, severity, or status, so that once again you
can create the perfect mix of very general and very specific actions, without being
stuck in a one-action-per-trigger scheme.
Unlike triggers, actions are also completely independent from hosts and templates
too. Every action is always globally defined, and its conditions checked against every
single Zabbix event. As you'll see in the following paragraphs, this may force you
to create some explicit conditions, instead of implicit conditions, but that's balanced
out by the fact that you won't have to create similar but different actions for similar
events just because they are related to different hosts.
An action is composed of the following three different parts that work together to
provide all the functionality needed:
• Action definition
• Action conditions
[ 222 ]
Chapter 6
• Action operations
The fact that every action has a global scope is reflected in every one of its
components, but it assumes a critical importance when it comes to action conditions,
as it's the place where you decide which action should be executed based on which
events. But let's not get ahead of ourselves and let's see a couple of interesting things
about each component in turn.
Defining an action
This is where you decide a name for the action, and can define a default message that
can be sent as a part of the action itself. In the message, you can reference specific
data about the event, such as the host, item and trigger name, item and trigger
values, and URLs. Here, you can leverage the fact that actions are global by using
macros, so that a single action definition could be used for every single event in
Zabbix, and yet provide useful information in its message.
You can see a few interesting macros already present in the default message when
you create a new action, as shown in the following screenshot:
[ 223 ]
Managing Alerts
Most of them are pretty self-explanatory, but it's interesting to see how you can of
course reference a single trigger, the one that generated the event. On the other hand,
as a trigger can check on multiple items from multiple hosts, you can reference all
the hosts and items involved (up to nine different hosts and/or items) so that you
can get a picture of what's happening by just reading the message.
Other interesting macros can make the message even more useful and expressive.
Just remember that the default message may be sent not only via e-mail, but also via
a chat program or an SMS; you'll probably want to create different default actions
with different messages for different media types, so that you can calibrate the
amount of information provided based on the media available.
You can see the complete list of supported macros in the official documentation wiki
at https://www.zabbix.com/documentation/2.0/manual/appendix/macros/
supported_by_location, so we'll look at just a couple of the most interesting ones.
The {EVENT.DATE} and {EVENT.TIME} macros
These two macros can help differentiate between the time a message is sent and
the time of the event itself. It's particularly useful not only for repeated or escalated
actions, but also for all media where a timestamp is not immediately apparent.
The {INVENTORY.SERIALNO.A} and friends macros
When it comes to hardware failure, information about a machine's location, admin
contact, serial number, and so on, can prove quite useful to track it down quickly,
or to pass it on to external support groups.
The {NODE.ID} and {NODE.NAME} macros
These are quite important in a distributed architecture. If you have thousands of
monitored hosts managed by different nodes, it may not always be immediately
apparent which host is monitored by which node. These macros can really help
avoid wasting time just looking for the right node in order to investigate the event.
[ 224 ]
Chapter 6
Defining the action conditions
This part lets you define conditions based on the event's hosts, trigger, and trigger
values. Just like trigger expressions, you can combine different simple conditions
with a series of AND/OR logical operators, as shown in the following screenshot.
Unlike trigger expressions, there is not much flexibility in how you combine them.
You can either have all AND, all OR, or a combination of the two, where conditions
of different types are combined with AND, while conditions of the same type are
combined with OR:
Observe how one of the conditions is Trigger value = "PROBLEM". Since actions are
evaluated for every event, and a trigger switching from PROBLEM to OK is an event in
itself, if you don't specify this condition, the action will be executed both when the
trigger switches to PROBLEM and when the trigger switches back to OK. Depending
on how you have constructed your default message and what operations you intend
to do with your actions, this may very well be what you intended, and Zabbix will
behave exactly as expected.
[ 225 ]
Managing Alerts
Anyway, if you created a different recovery message in the Action definition form,
and you forget the condition, you'll get two messages when a trigger switches back
to OK—one will be the standard message, and one will be the recovery message.
This can certainly be a nuisance as any recovery message would be effectively
duplicated, but things can get ugly if you are relying on external commands as
part of the action's operations. If you forget to specify the condition Trigger
value = "PROBLEM", the external, remote command would also be executed twice;
once when the trigger switches to PROBLEM (this is what you intended), and once
when it switches back to OK (this is quite probably not what you intended). Just to
be on the safe side, and if you don't have very specific needs for the action you are
configuring, it's probably better if you get into the habit of putting Trigger
value = "PROBLEM" for every new action you create, or at least check that it's
present in the actions you modify.
The most typical application for creating different actions with different conditions
is to send alert and recovery messages to different recipients. This is the part where
you should remember that actions are global.
Let's say that you want all the database problems sent over to the DB Administrators
group and not the default Zabbix Administrators group. If you just create a new
action with the condition that the host group must be DB Instances, and as message
recipients, choose your DB Admins, what will happen is that they will certainly
receive a message for any DB related event, but so will your Zabbix Admins, if the
default action has no conditions configured. The reason is that since actions are
global, they are always executed whenever their conditions evaluate to True. In this
case, both the specific action and the default one would evaluate to True, so both
groups would receive a message. What you could do is add an opposite condition in
the default action so that it would be valid for every event, except for those related
to the DB Instances host group. The problem is that this approach can quickly get
out of control, and you may find yourself with a default action full of the not in group
conditions. Truth is, once you start creating actions specific for message recipients,
you either disable the default action or take advantage of it to populate a message
archive for administration and reporting purposes.
Choosing the action operations
If the first two parts were just preparation, this is where you tell the action what it
should actually do. The following are two main aspects to this:
• Operation steps
• The actual operations available for each step
[ 226 ]
Chapter 6
As with almost everything else in Zabbix, the simplest cases that are very
straightforward are most often self-explanatory; you just have a single step,
and this step consists of sending the default message to a group of defined
recipients. As with almost everything else in Zabbix, this simple scenario
can become increasingly complex and sophisticated, but still manageable,
depending on your specific needs. Let's see a few interesting details about
each part.
Steps and escalations
Even if an action is tied to a single event, it does not mean that it can perform
a single operation. In fact, it can perform an arbitrary number of operations,
called steps, which can even go on for an indefinite amount of time, or until
the conditions for performing the action are not valid anymore.
You can use multiple steps to both send messages as well as perform some kind
of automated operations. Or, you can use the steps to send alert messages to
different groups, or even multiple times to the same group, with the time intervals
that you want, as long as the event is unacknowledged, or even not yet resolved.
The following screenshot shows a combination of different steps:
[ 227 ]
Managing Alerts
As you can see, step 1 starts immediately, and is set to send a message to a user
group, and then delays the subsequent step by just one minute. After one minute,
step 2 starts and is configured to perform a remote command on the host. As step 2
has a default duration (the duration of which is defined in the main Action definition
tab), step 3 will start after about an hour. Steps 3, 4, and 5 are all identical and
have been configured together—they will send a message to a different user group
every 10 minutes. You can't see it in the preceding screenshot, but step 6 will only
be executed if the event is not yet acknowledged, just as step 7 that is still being
configured. The other interesting bit of step 7 is that it's actually set to configure steps
7 to 0. It may seem against intuition, but in this case, step 0 simply means "forever".
You can't really have further steps if you create a step N to 0, because the latter
will repeat itself with the time interval set in the step's Duration(sec) field. Be very
careful in using a step 0 because it will really go on until the trigger's status changes.
Even then, if you didn't add a Trigger status="PROBLEM" condition to your action,
a step 0 can be executed even if the trigger switched back to OK. In fact, it's probably
best never to use step 0 at all, unless you really know what you are doing.
Messages and media
For every message step, you can choose to send the default message that you
configured in the first tab of the Action creation form, or send a custom message
that you can craft in exactly the same way as the default one. You might want
to add more details about the event if you are sending the message via e-mail to
a technical group, or reduce the amount of details, or the wording of the message,
if you are sending it to a manager or supervisor, or if you are limiting the message
to an SMS.
Remember that in the Action operation form, you can only choose recipients as
Zabbix users and groups, while you still have to specify for every user any media
address they are reachable to. This is done in the Administration tab of the Zabbix
frontend by adding media instances for every single user. You also need to keep in
mind that every media channel can be enabled or disabled for a user, or it may be
active only during certain hours of the day, or just for one or more specific trigger
severity, as shown in the following screenshot:
[ 228 ]
Chapter 6
This means that even if you configure an action to send a message, some recipients
may still not receive it based on their own media configuration.
While Email, Jabber, and SMS are the default options for sending messages, you
still need to specify how Zabbix is supposed to send them. Again, this is done in the
Media types section of the Administration tab of the frontend. You can also create
new media types there, which will be made available both in the media section of user
configuration, and as targets for message-sending in the Action operations form.
A new media type can be a different e-mail, jabber, or an SMS server, in case you
have more than one, and you need to use them for different purposes or with
different sender identifications. It can also be a script, and this is where things can
become interesting if potentially misleading.
A custom media script has to reside on the Zabbix server in the directory indicated by
the AlertScriptPath variable of zabbix_server.conf. When called upon, it will be
executed with the following three parameters passed by the server:
• $1: The recipient of the message
• $2: The subject of the message
• $3: The body of the main message
[ 229 ]
Managing Alerts
The recipient will be taken from the appropriate user-media property that you
would have defined for your users while creating the new media type. The subject
and the message body will be the default ones configured for the action, or some
step-specific ones, as explained before. Then, from Zabbix's point of view, the script
should send the message to the recipient by whatever custom methods you intend
to use, whether it's an old UUCP link, or a modern mail server that requires strong
authentication, or a post to an internal microblogging server. The fact is, you can
actually do what you want with the message; you can simply log it to a directory,
send it to a remote file server, morph it to a syslog entry and send it over to a log
server, run a speech synthesis program on it and read it aloud on some speakers,
or record a message on an answering machine; the sky's the limit with custom
media types. This is why you should not confuse a custom media with the execution
of a remote command—while you could potentially obtain roughly the same results
with one or the other. Custom media scripts and remote commands are really two
different things.
Remote commands
These are normally used to try to perform some corrective actions in order to
resolve a problem without human intervention. After you've chosen the target
host that should execute the command, the Zabbix server will connect to it and
ask it to perform it. If you are using the Zabbix agent as a communication channel,
you'll need to set EnableRemoteCommands to 1, or the agent will refuse to execute
any command. Other possibilities are SSH, telnet, or IPMI (if you have compiled
the relative options during server installation).
Remote commands can be used to do almost anything—kill or restart a process,
make space on a filesystem by zipping or deleting old files, reboot a machine,
and so on. They tend to seem powerful and exciting to new implementers, but in
the authors' experience, they tend to be fragile solutions that can break things
almost as often as they fix it. It's harder than it looks to make them run safely,
without accidentally deleting files, or rebooting servers when there's no need
to. The real problem with remote commands is that they tend to hide problems
instead of revealing them, which should really be the job of a monitoring system.
Yes, they can prove useful as a quick patch to ensure the smooth operation of your
services, but use them too liberally and you'll quickly forget that there actually
are recurring problems that need to be addressed, because some fragile command
somewhere is trying to fix things in the background for you. It's usually better to
really try to solve a problem than to just hide it behind some automated temporary
fix, and not just from a philosophical point of view, but because when these patches
fail, they tend to fail spectacularly and with disastrous consequences.
[ 230 ]
Chapter 6
So our advice is to use remote commands very sparingly, and only if you know what
you are doing.
Summary
This chapter focused on what is usually considered the "core business" of a
monitoring system—its triggering and alerting features. By concentrating
separately and alternately on the two parts that contribute to this function:
triggers and actions, it should be clear to you how once again Zabbix's
philosophy of separating all the different functions can give great rewards
to the astute user. You should have learned how to create complex and
sophisticated trigger conditions that will help you have a better understanding
of your environment, and more control over what alerts you should receive.
The various triggering functions and options, as well as some of the finer aspects
of item selection, along with the many aspects of action creation should not be a
secret to you now.
In the next chapter, you will explore the final part of Zabbix's core monitoring
components—the templates and discovery functions.
[ 231 ]
Managing Templates
For all the monitoring powers of Zabbix's items, graphs, maps, and triggers, it would
be incredibly cumbersome to manually create every single one of these objects for
every monitored host. In the case of a large environment, with hundreds or thousands
of monitored objects, it would be practically impossible to manually configure all the
items, graphs, and triggers needed for each host.
By using the templates facility, you'll define different collections of items, triggers,
and, graphs, in order to apply common configurations to any number of hosts,
while still being able to manage any single aspect you may need to tweak for any
single host.
The perfect complement to the template facility is Zabbix's discovery feature.
By using it, you'll define a set of rules to let Zabbix know of new hosts without
having to manually create them. You can also take advantage of the low-level
discovery powers of the Zabbix agent, so that you can automatically assign the
correct items even for those highly variable parts of a system, such as the number
and nature of disks, file systems, and network interfaces.
In this chapter, you will learn to:
• Create and leverage the power of nested templates
• Combine different templates for the same hosts
• Use host discovery and actions to add templates to new hosts
• Configure a low-level discovery to make templates even more general
Let's start from the beginning and see how a template is different from a regular
host, even if they look almost the same.
Managing Templates
Creating templates
A host template is very similar to a regular host. Both are collections of items, triggers,
graphs, screens, and low-level discovery rules. Both need a unique name, just as any
other entity in Zabbix. Both can belong to one or more groups. The crucial difference
is that a host has one or more means to be contacted so that the Zabbix server can
actually take item measurements on it, as illustrated in Chapter 4, Collecting data.
These can be one or more IP addresses, or host names, that represent agent interfaces,
or SNMP, JMX, and IPMI ones. So a host is an object the Zabbix server will ask for
information to, or wait for data from. A template, on the other hand, doesn't have
any access interface, so the Zabbix server will never try to check if a template is alive,
or ask it for the latest item measurements.
The creation of a template is very straightforward and there is not much to say about it.
You navigate to the Configuration | Templates tab and click on the Create template
button. The template creation form that will appear is composed of three different tabs.
We'll look at the Linked templates tab and the Macros tab later in the chapter, as these
are not essential to create a basic template. In fact, the only essential element for a basic
template is its name, but it can be useful to assign it to one or more groups in order
to find it more easily in the other section of the Web interface. If you have configured
some hosts already, you can also assign the template to the hosts you're interested
in directly from the template creation tab. Otherwise, you'll need to go to the Hosts
configuration tab and assign templates there. Once you're done, the template is created
and available in the template list, but it's still an empty object. Your next job is to create
the template's items, trigger, graphs, screens, and discovery rules, if any.
Adding entities to a template
Adding an item or any other entity to a template is virtually identical to the same
operation performed on a regular host. This is especially true for items. As you
already know, item keys are the basic building blocks of the Zabbix monitoring
pipeline and you don't have to specify any kind of address or interface when you
create them, as Zabbix will take this kind of information from the host the item
is assigned to. This means that when you create items for a template, you are
effectively creating items for an ideal host that will be later applied to real ones,
once you have linked the template to the hosts you want to monitor.
[ 234 ]
Chapter 7
Templates, just like hosts, are essentially collections of items, triggers,
and graphs. Since many of the concepts that we will explore apply
equally to items, triggers, and graphs, for the rest of the chapter we'll be
using the term entity to refer to any of the three types of objects. In other
words, you can understand an item, a trigger, or a graph every time you
read entity, and items, triggers, and graphs when you read entities, as a
collective term.
This applies to other types of entities as well, but as they always reference one or
more existing items, you need to make sure that you select the items belonging to
the right template, and not to a regular host:
This may seem obvious but it is far too easy to overlook the Group and Host
selectors at the top of the window when selecting an item and ending with a
specific host's item in a template trigger or graph.
The main difference between template entities and host entities, especially when it
comes to triggers, is that with template entities the use of macros is quite useful to
make trigger and graph names or parameters more expressive and adaptable.
[ 235 ]
Managing Templates
Using macros
As you've already seen in Chapter 6, Managing Alerts, macros are very useful to
make a message general enough that it can be applied to a wide range of events.
It will be the Zabbix server's job to substitute all the macros in a message with
the actual content based on the specific event it's handling. Since an action message
is effectively a template that has to be applied to a particular event, it's easy to
see how the same concept is essential to the effectiveness of host templates.
What changes is the context; while an event has a context that is quite rich, since it
can reference a trigger and one or more different items and hosts, the context of a
simple, regular host is admittedly more limited. This is reflected in the number of
macros available, as they are just a handful:
Macro name
{HOST.CONN}
Macro translates to
Notes
Hostname or IP
address of the host
Will be identical to either {HOST.IP} or
{HOST.DNS} depending on the Connect to
option in the host's configuration form.
{HOST.DNS}
The host's hostname
This must correspond to the host's fully
qualified domain name as defined in the
domain's DNS server.
{HOST.HOST}
The host's name as
defined in Zabbix
This is the main host identifier. It must be
unique for the specific Zabbix server. If using
an agent, the same name must be present in
the agent's configuration on the host.
{HOST.IP}
The host's IP address
A host can have more than one IP address.
You can reference them using {HOST.IP1},
{HOST.IP2}, and so on up to {HOST.IP9}.
{HOST.NAME}
The host's visible name
as defined in Zabbix
This will be the name visible in lists, maps,
screens, and so on.
To better clarify the differences between the various {HOST.*} macros, let's see an
example host configuration:
[ 236 ]
Chapter 7
In this case, {HOST.HOST} would resolve to ZBX Main, {HOST.NAME} to Main
Zabbix Server, {HOST.IP} to 127.0.0.1, and {HOST.DNS} to zabbix.example.
com. Finally, since the Connect to option is set to IP, {HOST.CONN} would resolve
to 127.0.0.1 as well.
The most obvious use of these macros is to make trigger and graph names more
dynamic and adaptable to the actual hosts they will be used into. Since a graph's
name is displayed as a header when viewing the graph, it's vital to distinguish
different graphs of the same type belonging to different hosts, especially when they
are displayed together in a screen, as explained in Chapter 5, Visualizing Data.
[ 237 ]
Managing Templates
A less obvious use of these macros is inside an item's key definition. We touched
briefly on external scripts in Chapter 4, Collecting Data, and you'll meet them again
in the next chapter, so we won't get into too much detail about them here. It's suffice
to say that from an item creation point of view, all you need to know about an
external script is its name and any parameters you may need to pass in order to
execute it correctly.
Since external scripts, as is their nature, don't share any information with the rest
of Zabbix other than the arguments they are passed and their return value, it's often
essential to include the host's IP address or hostname as one of the arguments.
This ensures that the script will connect to the right host and collect the right data.
A single, well-configured script can perform the same operation on many different
hosts thanks to the template systems and macros like {HOST.CONN}, {HOST.IP},
and so on.
Take, for example, a script that checks if a particular application is alive using a
custom protocol. You could have an external script, let's call it app_check.sh,
which takes a host name or IP address as an argument, connects to it using the
application's protocol and returns 1 if it's alive and well, and 0 if the check fails.
Your template item's key would look like the following screenshot:
[ 238 ]
Chapter 7
In these cases, using a macro as the argument to the item's key is the only way to
make an external check for a part of a template and is useful for any regular host.
Another example would be that of a bunch of Zabbix hosts that don't represent
regular operating system machines, physical or virtual, but single applications or
single database instances. In a scenario like this, all the application hosts would
share the same connections and interfaces, those of the actual server hosting the
applications, and they would be linked to a template holding only items relevant to
application-level (or database-level) measurements. To keep things simple, let's say
that you have an application server (Alpha) hosting three different applications:
• a document archival facility (doku)
• a customer survey form manager (polls)
• a custom, internal microblogging site (ublog)
For each of these applications you are interested, by and large, in taking the
same measurements
• number of active sessions
• amount of memory consumed
• number of threads
• network I/O
• number of connections to the database
Again, for simplicity's sake, let's say that you have a bunch of external scripts that,
given an IP address and an application name, can measure exactly the preceding
metrics. External script keys tend to be easy to read and self-explanatory, but all of
this can be equally applied to JMX console values, Windows performance counters,
database queries, and any other kind of items.
One way to monitor this set up is to create only one host, Alpha, and, in addition
to all the regular OS and hardware monitoring items, a number of items dedicated
to application measurements, repeated for each one of them. This can certainly
work, but if you have to add a new application, you'll need to create all the items,
triggers, and graphs related to it, even if they differ from the rest by just the
application's name.
[ 239 ]
Managing Templates
Seeing as that is the only difference in an otherwise identical collection of entities,
a more flexible solution would be to split the monitoring of every application to a
different host and apply a common template.
A host, from Zabbix's point of view, is just a collection of entities
with one or more connection interfaces. It doesn't have to be an
actual, physical (nor virtual!) machine with a regular operating
system. Any abstract but coherent collection of metrics and a
means to retrieve them can be configured as a host in Zabbix.
Typical examples are applications, database instances, and so on.
Instead of creating many similar items, triggers, and so forth for the host Alpha,
you would create a custom application template and fill it with items that would
look like the following screenshot:
[ 240 ]
Chapter 7
You could then create one host for each application, with Alpha's IP address as
the connection interface, and with the application's name as host name. Linking the
template you just created to the hosts would give you the same basic results as before,
but with much more flexibility; adding an application to be monitored now is a
simple matter of creating a host and linking it to the correct template. If you move an
application from one server to another, you just need to update its IP address. If you
put all these application hosts in a separate group, you can even grant access to their
monitoring data to a specific group of users, without necessarily giving them access
to the application server's monitoring data. And, it goes without saying, adding,
deleting, or modifying an entity in the template applies immediately to all the
monitored applications.
User-defined macros
A special class of macros is user-defined, template- and host-level macros. You can
configure them in the Macros tab of every host or template creation and administration
form. They are quite simple, as they only provide a translation facility from a custom
label to some predefined, fixed value:
When used in a template, they prove quite useful in defining common thresholds for
triggers, so if you need to modify a bunch of time-out triggers, you can just modify the
{$NODATA} macro instead of changing every single trigger that uses it. User-defined
macros can be used everywhere that built-in macros can be used.
The usefulness is even greater when used in connection with nested templates,
as we'll see in a short while.
[ 241 ]
Managing Templates
Linking templates to hosts
To link a template to a host, you can either select the hosts you want to link from
the template's configuration form, as we've seen in the Creating templates section,
or you can choose the template you need for a host in that host's configuration
form by going into the Template tab.
Once linked, a host will inherit all of the template's entities. Previously existing entities
with the same name will be overwritten, but entities not included in the template will
remain as they are and not touched in any way by the linking operation.
All entities will maintain their original template's name when displayed in
the configuration section of the Web interface, even when viewed from a host
configuration page. However, this doesn't mean that modifying them from a
template's configuration tab is the same as modifying them from a linked host's
configuration tab.
If you modify an entity (item, trigger, graph, and so on) from a template's
configuration tab, the modifications will apply immediately to all linked
hosts. On the other hand, if you modify a template entity from a particular
host's configuration tab, the changes will only apply to that host, and not on
a template level. While this can be useful to address any special circumstances
for an otherwise regular host, it can also generate some confusion if you make
many local changes that can become hard to keep track of. Moreover, not every
aspect of a template entity can be modified at the host level. You can change
the frequency of an item, for example, but not its key.
Unlinking a template from a host doesn't eliminate its entities, unless you unlink
and clear it. Be careful with this operation as all the items' history and trends would
become unavailable. If you have collected any actual data, it's probably better to
just unlink a template from a host, and then disable any unused items and triggers,
while retaining all of their historical data.
Nesting templates
Just as you can link a template to a host, you can also link a template to another
template. The operation is identical to linking a template to a host; you navigate
to the Linked templates tab in a template's configuration form, and choose the
templates you want to link.
While this may seem an awkward operation, it can prove quite useful in two cases.
[ 242 ]
Chapter 7
The first application of nested templates is to make user macros even more general.
Since a template inherits all of its linked templates entities and properties, any custom
macro will also be inherited and thus made available to the actual monitored hosts.
To make a concrete example, let's say you have a Template Macros template containing
a {$PFREE} user macro, with a value of 5. You could use this macro to represent the
amount of free disk space, in percentages, to check against, or free available memory,
or any other such threshold. This template could be linked to both Template OS
Linux and Template OS Windows templates, and the {$PFREE} macro used in these
templates' triggers. From now on, if you ever need to modify the default value of the
free disk space percentage to check against, you'll just need to change the original
Template Macros template and the updated value will propagate through the linked
templates down to the monitoring hosts.
A second, somewhat more limited but still useful, way to use nested templates is
to extend the inheritance beyond macros, to all the other entities. This may become
an advantage when you have a common set of features on a given technological
layer, but different uses on other layers. Let's take for instance the case where you
have a large number of virtually identical physical servers, that host just a couple of
versions of operating systems (Linux and Windows, for simplicity's sake), but that in
turn perform many different specialized functions, database, file server, web server,
and so on.
You could certainly create a few monolithic templates with all the items you need
for any given server, from hardware checks to OS checks to application specific ones.
Or, you could create a sort of hierarchy of templates. A common, hardware level
template that enables IPMI checks will be inherited by a couple of OS specific
templates. These, in turn, will be inherited by some application specific templates
that will have names, such as Linux Apache Template or Win Exchange Template.
These templates will have all the items, triggers, and graphs specific to the
applications they are meant to monitor, in addition to all the OS specific checks
they have inherited from the OS level templates and the hardware specific ones
they have inherited from the HW level templates. This means that, when creating
a host, you will still just need to link it to a single template, but you'll also have
a lot of flexibility in creating new templates and nesting them, or modifying
existing ones in only one place and watching the changes propagate along the
template-linking chain. This also means having maximum generality while
still maintaining the ability to make host specific customizations if you need to.
[ 243 ]
Managing Templates
Combining templates
Another way to make templates modular is to create specific templates for any
given technological layer and product, but not link them in a hierarchy at the
template level.
You can instead link them—as many as you want—directly to the host you need
to monitor, as long as they don't have any conflicting or overlapping item names
or keys. As in the preceding scenario, Host A could have linked an IPMI checks
template, an OS Linux one, and an Apache server one, while Host B could also
have an IPMI checks template and an OS Linux one, but then also have a
PostgreSQL database template.
The end result is practically the same as the nested templates solution described
previously, so which one should you choose? This is largely a matter of preference,
but a possible criterion could be that if you have a relatively low number of low-level
templates and good consistency in you hardware, OS, and technological configuration,
the nested solution might be easier to manage. You'll only have to connect the
templates once and then use them on a large number of hosts. This approach also
works well with the host discovery facility as it keeps things simple when linking
templates to hosts. If, on the other hand, you have a great number of low-level
templates, and a great variability in your technological configuration and landscape,
you may just as well pick and choose the templates you need when you create your
hosts. Any preconfiguration, in fact, would only prove too rigid to be really useful.
This approach works well if you want to always ponder how you are creating and
configuring your hosts, and also need a great deal of local customization that would
make any aggressive inheritance feature a moot point.
Discovering hosts
A third way to link templates to hosts is to let the server do it automatically by
combining Zabbix's host-discovery facility with discovery actions.
Zabbix's discovery facilities consist of a set of rules that periodically scan
the network, looking for new hosts, or disappearing ones, according to
predetermined conditions.
The three methods in Zabbix that can be used to check for new or disappeared hosts,
given an IP range, are:
• Availability of a Zabbix agent
• Availability of a SNMP agent
[ 244 ]
Chapter 7
• Response to simple external checks (FTP, SSH, and so on)
• These checks can also be combined, as illustrated in the following example:
As you can see, when enabled, this rule will check every hour, in the IP range
192.168.1.1-255, for any server that:
• Responds to a ICMP ping probe
• Has a correctly configured Zabbix agent that will return a value when asked
for the system.uname item
• An SMTP listening port, which is usually associated to a mail server.
As usual with all things Zabbix, a discovery rule will not do anything by itself,
except generate a discovery event. It will then be the job of Zabbix's actions facility
to detect the said event and decide if and how to act on it. Discovery event actions
are virtually identical to trigger event actions. As you've already seen trigger event
actions in Chapter 6, Managing Alerts, following are the only differences when it
comes to discovery events.
[ 245 ]
Managing Templates
First, action conditions are a bit different, as can be expected, as shown in this
following screenshot:
Instead of host names and trigger specifications, you can set conditions based
on things, such as Discovery status, Service type, and Uptime/Downtime.
The Received value condition is of particular interest, as it allows things like
differentiating between operating systems, application versions, and any other
information you could get from a Zabbix or an SNMP agent query.
This kind of information will be critical when it comes to configuring the
action's operations:
[ 246 ]
Chapter 7
Sending a message and executing a remote command are the exact equivalent of
the same operations available to trigger event actions. On the other hand, if adding
(or removing) a host is quite a self-explanatory action, when it comes to adding to a
host group, or linking to a template, it becomes clear that a good set of actions with
specific received value conditions and template linking operations can give a high
level of automation to your Zabbix installation.
[ 247 ]
Managing Templates
This high-level of automation is probably more useful in rapidly changing
environments that still display a good level of predictability depending on the
kind of hosts you can find, such as fast growing grids or clusters. In these kinds
of environments you can have new hosts appearing on a daily basis, and maybe
old hosts disappearing at almost the same rate, but the kind of host is more or
less always the same. This is the ideal premise for a small set of well-configured
discovery rules and actions, so you don't have to constantly and manually add or
remove the same types of hosts. On the other hand, if your environment is quite
stable, or you have very high host-type variability, you may want to look more
closely at what and how many hosts you are monitoring as any error can be much
more critical in such environments.
On the other hand, limiting discovery actions to sending messages about discovered
hosts can prove quite useful in such chaotic environments, or where you don't directly
control your systems' inventory and deployment. In such a case, getting simple
alerts about new hosts, or disappearing ones, can help the monitoring team keep
Zabbix updated despite any communication failure between IT departments,
accidental or otherwise.
Low-level discovery
An even more useful and important feature of Zabbix templates is their ability
to support special kind of items, called low-level discovery rules. Once applied
to actual hosts, these rules will query the host for whatever kind of resources they
are configured to look for, filesystems, network interfaces, SNMP OIDs, and more.
For every resource found, the server will then dynamically create items, triggers,
and graphs according to special entity prototypes connected to the discovery rules.
The great advantage of low-level discovery rules is that they take care of the more
variable parts of a monitored host, such as the type and number of network interfaces,
in a dynamic and general way. This means that, instead of manually creating specific
items and triggers of every host's network interfaces or filesystems, or creating huge
templates with any possible kind of item for a particular operating system, and
keeping most of these items disabled, you can have a reasonable number of general
templates that will adapt themselves to the specifics of any given host, by creating
on the fly, any entity needed based on discovered resources and previously
configured prototypes.
Out of the box, Zabbix supports three discovery rules:
• Network interfaces
• Filesystems types
• SNMP OIDs
[ 248 ]
Chapter 7
As discovery rules are effectively special kinds of items, you can create your own
rules, provided you understand their peculiarity compared to regular items.
If we don't consider the fact that you need to create and manage low-level discovery
rules in the Discovery rules section of template configuration, and not in the usual
Items section, the main difference between the two kinds of items is that while a
regular item usually returns a single value, as explained in Chapter 4, Collecting Data,
a discovery item always returns a list, expressed in JSON, of macro value pairs.
This list represents all the resources found by the discovery items, together with
means to reference them.
The following table shows Zabbix's supported discovery items and their return
values, together with a generalization that should give you an idea on how to create
your own rules:
Discovery item key
Item type
Return values
vfs.fs.discovery
Zabbix
agent
{"data": [
{"{#FSNAME}":<path>",
"{#FSTYPE}":"<fstype>"},
{"{#FSNAME}":<path>",
"{#FSTYPE}":"<fstype>"},
{"{#FSNAME}":<path>",
"{#FSTYPE}":"<fstype>"},
…
] }
net.if.discovery
Zabbix
agent
{"data":[
{"{#IFNAME}":"<name>"},
{"{#IFNAME}":"<name>"},
{"{#IFNAME}":"<name>"},
…
]}
snmp.discovery
SNMP
(v1, v2 or
v3) agent
{"data":[
{"{#SNMPINDEX}":"<idx>",
"{#SNMPVALUE}":"<value>},
{"{#SNMPINDEX}":"<idx>",
"{#SNMPVALUE}":"<value>},
{"{#SNMPINDEX}":"<idx>",
"{#SNMPVALUE}":"<value>},
…
]}
[ 249 ]
Managing Templates
Discovery item key
custom.discovery
Item type
any
Return values
{"data":[
{"{#CUSTOM1}":"<value>","{#CUSTOM2}":
"<value>"},
{"{#CUSTOM1}":"<value>","{#CUSTOM2}":
"<value>"},
{"{#CUSTOM1}":"<value>","{#CUSTOM2}":
"<value>"},
…
]}
As with all SNMP items, an item key is not really important, as long as
it is unique. It's the SNMP OID value that you ask an agent that makes
the difference, you can create different SNMP discovery rules that look
for different kind of resources, by changing the item key and looking
for different OID values. The custom discovery example is even more
abstract, as it will depend on the actual item type.
As you can see, a discovery item always returns a list of values, but the
actual contents of the list change depending on what resources you are
looking for. In the case of a filesystem, the returned list will contain values
like {#FSNAME}:/usr, {#FSTYPE}:btrfs, and so on for every discovered
filesystem. On the other hand, a network discovery rule will return a list
of the names of the discovered network interfaces.
When configuring a template's discovery rules, you don't need to care about the
actual values returned in such lists, nor the lists' length. The only thing you have to
know is the name of the macros that you can reference in your prototypes. These are
the second half of the low-level discovery mechanisms. You create them as regular
template entities, making sure you use the discovery item macros where needed,
as exemplified in the following screenshot:
[ 250 ]
Chapter 7
When you apply the template to a host, it will create items, triggers, and graphs
based on the resources discovered by the discovery items and configured according
to the discovery prototypes.
Custom discovery rules, from this point of view, work exactly in the same way as
custom items, whether you decide to use agent-side scripts (thereby using a custom
zabbix.agent item key), external scripts, database queries, or anything else. The only
things you have to make sure of is that your custom items' return values must respect
the JSON syntax as shown in the preceding table, and that you reference your custom
macros in the entities' prototypes that you will create.
[ 251 ]
Managing Templates
Summary
This chapter concludes the central part of the book, dedicated to developing a deeper
understanding of the Zabbix monitoring system's core functions. The effective
configuration and use of templates builds on all the knowledge gained from using
and analyzing items, graphs and screens, triggers, and actions. To this knowledge,
this chapter has added a few template-specific aspects that should help to tie all
of the previous chapters together. At this point of the book, you should be able to
perform all the tasks associated with implementing and managing a monitoring
system, from choosing what to monitor and how to configure different item types,
to putting together information-rich visualization items. You should also be able
to select the triggers and actions that are most significant in order to maximize the
expressiveness of your alerts while avoiding false positives. Finally, you should not
have any problems bringing it all together through the use of macros, nested, and
combined templates, in order to apply a consistent and meaningful configuration to
a wide range of hosts, and to further automate these operations through host-level
discovery actions and the low-level discovery facilities of Zabbix's templates.
The final part of the book will be about further customization options for Zabbix,
how to extend its functionality, and how to really integrate it with the rest of your
IT management systems, in order to bring out its full power.
The next chapter will focus on writing extension scripts for Zabbix and its
monitoring protocol.
[ 252 ]
Handling External Scripts
So far, you have learned how most part of a server's components work and how to
leverage Zabbix to acquire data from various external sources. Considering that,
set up your monitoring system in a large heterogeneous and complex infrastructure.
Most probably, you will find a different custom device, server appliance,
and proprietary hardware. Usually, all those devices have an interface to be
enquired; but unfortunately, it often happens that most of the metrics are not
exposed via Simple Network Management Protocol (SNMP) or any other
standard query method.
Let us consider a practical example. Nowadays, all the UPSs own a temperature
sensor and if you are in a complex infrastructure, it is possible that those UPSs are
custom made, out of standard, and most probably, this sensor can be enquired
only with a tool provided from the UPS vendor. Now the temperature of a UPS
is a critical parameter, especially if the UPS is a big, custom made UPS. It is really
important to monitor these metrics.
Imagine that your cooling system is not working properly; receiving an alarm just
when the temperature reaches over the warning level is fundamental. On the other
hand, predicting the failure will save a lot of money. Also, even if the physical damage
is not really expensive, the downtime can cost a lot of money and have a terrible
impact on your business. A good example is the case of a trading company. In this
scenario, everything should be in perfect working order. In this environment, there is
terrible competition to achieve better performance against competitors—buying a
stock option some milliseconds before the others is a big advantage. Here it is easy to
understand that if servers are not performing well, it is already an issue; if they are
down, it is a complete disaster for the company. This example explains how critical
it is to predict a failure.
Handling External Scripts
Moreover, it is important to understand how critical it is to retrieve all functioning
parameters of your infrastructure. This is where Zabbix comes to the rescue,
providing interesting methods to retrieve data that interacts with the operating
system, eventually enabling you to use a command-line tool. Zabbix's responses to
this kind of requirement are as follows:
• External checks (server side)
• UserParameter (agent side)
• Zabbix_sender binary (can be used on both agent and server side)
• A simple, efficient, and easy-to-implement communication protocol
This chapter will entirely explore those alternatives to interact with the operating
system and receive data from external sources. In this chapter, you will learn that
there isn't a general, optimal, valid solution for all the problems, but each of them has
its pros and cons. This chapter will make you aware of all the things that need to be
considered when you implement a custom check. The analysis proposed will enable
you to choose the best solution for your problems.
This chapter will cover the following points:
• Writing a script and making it available as external scripts
• Advantages and disadvantages of scripts on the server side or agent side
• Exploring some alternative methods to send data to your Zabbix server
• Detailed documentation about the Zabbix protocol
• Some commented educational implementation of the Zabbix sender protocol
External checks
Zabbix provides some features to cover all the items that cannot be retrieved with the
standard agent. In real life, it is possible that you are not able to install the standard
agent on the device that you would like to monitor. A practical example is the UPS or
all the servers that for some reason cannot be compromised when installing external
software, or all the appliances that cannot have custom software installed.
Now for all those reasons, you cannot have an agent on your device but you need to
monitor some of the vital parameters of this devices, the only feasible solution is to
use an external check.
[ 254 ]
Chapter 8
The script's placement
The script's location on Zabbix is defined into the zabbix_server.conf
configuration file. Since Zabbix Version 2.0, the default location has changed
to /usr/local/share/zabbix/externalscript.
The default location depends on compile time from the variable
datadir. Actually, the default location is ${datadir}/
zabbix/externalscripts. This rule is valid for both proxy
and server components.
Previously, it was defined as /etc/zabbix/externalscripts; anyway, you can
change it by simply specifying a different location on zabbix_server.conf,
using the ExternalScript parameter.
### Option: ExternalScripts
# Mandatory: no
# Default:
# ExternalScripts=${datadir}/zabbix/externalscripts
ExternalScripts=/usr/local/zabbix/externalscripts
Zabbix Version 2.0 also accepts an empty string as a valid value. In the
previous releases, an empty value was treated as an unsupported item
causing issues.
There are some important enhancements to external script functionality introduced
in Zabbix 2.0
• Key syntax now supports multiple comma-separated parameters
• Supports multilines values
Going deep into external checks
Now it is time for a practical example. This is an easy example to understand,
how External Scripts works. In the following example, we will count the number
of open files for a specified user. The first thing to do is create the script and place
it in the ExternaleScripts location. The script will be called lsof.sh and will
contain the following code:
#!/bin/bash
if grep -q $1 /etc/passwd
then lsof -u $1 | tail -n +2 |wc -l
else
echo 0
fi
[ 255 ]
Handling External Scripts
This software requires the username as an input parameter, check if the username
exists on the system and then return the number of open files for that account.
Now you only need to create a new item of type External check. In the Key field,
enter lsof.sh["postgres"] as shown on the following screenshot:
Now, on navigating to Monitoring | Latest Data it is possible to see the data
retrieved by our script.
Now that you know how ExternalScript works, it is time to see how we can
implement something more complex thanks to this feature.
In the next example, we will monitor some remote Oracle instances. There are
some prerequisites to have this setup fully working: an Oracle client installed
with a working sqlplus and tnsping and an account configured on your Oracle
database target.
The last version of this software is available for download at http://www.
smartmarmot.com/product/check_ora.
Anyway, it is interesting to see how it evolved from Version 1.0. Version 1.0 is
available for download directly on the Zabbix forum available at https://www.
zabbix.com/forum/showthread.php?t=13666.
[ 256 ]
Chapter 8
This script is a good example of an external check. Basically, to have everything
properly configured you need to do the following:
1. Create a user account on all your monitored databases.
2. Configure your Oracle client.
3. Decompress the package on your external script location.
4. Configure your database account on the following location:
<EXTERNAL_SCRIP_LOCATION>/check_ora/credentials
5. Create a host with the same name as your database instance
The last point is of particular importance and it is a particular mode to use Zabbix.
This method can be reused everytime that you need to aggregate some metrics that
are not tied to a real host but to a service. To do a practical example, if you have a
DBMS that can failover against another server, you can simply create a Zabbix fake
host, called with the same name as that of your database. Now if the services do
failover, you don't have an interruption on your collected data because the failover
process is transparent from the server that is providing the service. This method
is applied here because the Oracle client will handle a failover automatically,
once properly configured.
Now, you can go ahead and create a host with the same name as that of your SID,
for example, you have an Oracle instance to monitor that is defined as ORCL on
your tnsnames.ora; thus, the Zabbix host will be ORCL.
You can create hosts tied to the name of the service; this
enables you to abstract the service from the host that is
providing the service.
The detailed configuration of an Oracle client is out of the scope of this book.
Once you complete the configuration, you can test the script by simply running
the following command:
check_ora.sh[-i <instance> -q <query>]
In the preceding command line, <instance> represents your instance name and
<query> is the query file that you would like to run. There is a large amount of
query files prebuilt in the check_ora directory, you can check all of them against
your database.
[ 257 ]
Handling External Scripts
The usage of Oracle SID or an Oracle instance name like the hostname
on Zabbix is really useful here. It can be expanded by the {HOSNAME}
macros, so you can simply create a key such as check_ora.sh [-i
{HOSTNAME} –q query] on your template and it will be expanded
across all your databases.
Now in the Zabbix's host, you need to create the item to retrieve the external check
and the key will be as follows:
check_ora.sh[-i {HOSTNAME} –q <queryhere>]
For example:
key="check_ora.sh[-i {HOSTNAME} -q lio_block_changes]"
The template is available on the forum at the same location. Note that {HOSTNAME} is
expanded with the hostname, which in our case is exactly the Oracle instance name.
You can have a generalized template using the {HOSTNAME} macro and their items
are propagated across all your databases' hosts.
Now, the lifecycle of this item will be the following:
1. Zabbix calls the script.
2. The script will perform the following:
°°
Login to the database
°°
Execute the query and retrieve the value
°°
Return the value on the standard output, Zabbix will receive the
value and if it is valid, it will be stored
Inside the script
The core function of check_ora.sh is the execquery(). The function is
the following:
execquery () {
start_time=$(date +%s)
#
echo "Time duration: $((finish_time - start_time)) secs."
echo "BEGIN check_ora.sh $1 $2 `date`" >> /tmp/checkora.log
cd $SCRIPTDIR;
sqlplus -S $1 <<EOF | sed -e 's/^\ *//g'
set echo off;
set tab off;
set pagesize 0;
set feedback off;
[ 258 ]
Chapter 8
set trimout on;
set heading off;
ALTER SESSION SET NLS_NUMERIC_CHARACTERS = '.,';
@$2
EOF
finish_time=$(date +%s)
echo "END check_ora.sh $1 $2 `date`" >> /tmp/checkora.log
echo "check_ora.sh $1 $2 Time duration: $((finish_time - start_time))
secs." >> /tmp/checkora.log
}
This function will begin producing some log information on /tmp/checkora.log
start_time=$(date +%s)
#
echo "Time duration: $((finish_time - start_time)) secs."
echo "BEGIN check_ora.sh $1 $2 `date`" >> /tmp/checkora.log
Those are useful to understand which external check is on-going and against which
database. Plus in the logfile, you will find the elapsed time for the whole operation:
finish_time=$(date +%s)
echo "END check_ora.sh $1 $2 `date`" >> /tmp/checkora.log
echo "check_ora.sh $1 $2 Time duration: $((finish_time - start_time))
secs." >> /tmp/checkora.log
}
Since this file is shared (between the check_ora.sh process), and the Zabbix calls
are not serialized, it is important to report the script calling line twice so that you can
identify exactly which stating line correspond to which finish line. Here, to avoid any
doubt, the elapsed is calculated and reported on the finish message.
After the script, call sqlplus
sqlplus -S $1 <<EOF | sed
-e 's/^\ *//g'
Here, sed cleans up the white space at the beginning of the output. This is because
the returned data is a number that cannot begin with blank spaces; if that happens,
the item will become unsupported!
The following code snippet makes an Oracle client less verbose:
set
set
set
set
set
set
echo off;
tab off;
pagesize 0;
feedback off;
trimout on;
heading off;
The preceding lines are important to avoid noise in the output.
[ 259 ]
Handling External Scripts
The following code snippet explains the separator that should be used:
ALTER SESSION SET NLS_NUMERIC_CHARACTERS = '.,';
This section is important because you can have databases installed for different
reasons with different character sets. Also, the client can use a different separator for
decimals. You need to avoid all the uncontrolled charset conversions, and this is a
general rule.
Finally, the script executes the query file in the following way:
@$2
EOF
The output is returned in a standard output and is collected from Zabbix.
General script writing rules
This script covers all the critical points that you need to pay attention to:
• Do not introduce unwanted characters into the output
• Be aware of the type, so if a number is expected, remove all the unneeded
characters (such as heading spaces)
• Avoid local conversions of numbers, the case of the dot and comma is a
good example
• Have a log, keeping in mind that external script are not serialized, so you can
have your log messages mixed in your logfile
• Those scripts of course run with the Zabbix server user, so maybe you need
to take care of file permissions
Some consideration about external checks
In this section, you have seen how external checks can be executed and a complex task,
such as database monitoring, is handled with them. If you have few external checks
to implement, this can be a feasible solution to retrieve some metrics. This kind of
approach with external checks, unfortunately, is not the solution to all the problems.
On the other hand, you need to consider that they are really resource intensive and
were once widely applied. Since external checks are on the server side, it is better
not to overload the Zabbix server. The Zabbix server is the core component of you
monitoring infrastructure and you can't steal resource from this server.
[ 260 ]
Chapter 8
The UserParameter
The simple thing to do is to avoid extensive resource usage by your script by placing
the script on the agent side. Zabbix provides this alternative method, and the script
should instead be on server side and load the Zabbix server; it can be offloaded to
the agent side with UserParameters.
UserParameter are defined on the agent configuration file. Once they are
configured, they are treated like all the other Zabbix agent items by simply using the
key specified in the parameter option. To define a user parameter, you need to add
on the agent configuration file something like the following:
UserParameter=<key>,<shell command>
Where key must be unique and the shell command represents the command to
execute. The command can be specified here inline and doesn't need to be a script
like in the following example:
UserParameter=process.number, ps -e |wc -l
In this example, the process.number key will retrieve the total amount of the
process number on your server.
With the same kind of approach, you can check the number of users currently
connected with the following code:
UserParameter=process.number, who |wc -l
The flexible UserParameter
It is easy to understand that using this method you are going to define a huge
amount of entry inside the agent configuration file. This is not the right approach
because it is better keep the configuration file simple.
Zabbix provides an interesting UserParamenter feature to avoid the proliferation
of those items on the agent side, the flexible user parameter. This feature is enabled
with an entry of this kind:
UserParameter=key[*],<shell command>
Here, the key still needs to be unique and the [*] term defines that this key accepts
the parameters. The content between the square bracket is parsed and substituted
with $1...$9; please note that the $0 refers to the command itself.
[ 261 ]
Handling External Scripts
An example of UserParameter can be the following:
UserParameter=oraping[*],tnsping $1 | tail -n1
This command will execute the tnsping to your SID, passing it as $1. You can apply
the same method in the the process for counting specified users as follows:
UserParameter=process.number[*], ps -e |grep ^$1 | wc -l
Then, if we want to move to the agent side, for the first script that returns the number
of open files for a defined user, the configuration will be the following:
UserParameter=lsof.sh[*],/usr/local/bin/ /lsof.sh $1
Once added, you only need to restart the agent. On the server side, you need to
switch the item Type to Zabbix agent and save it.
With the same method, you can configure the check_ora.sh script to check the
database with the following code:
UserParameter=check_ora.sk[*],check_ora.sh –i $1 –q $2
On the Zabbix server side, you need to create the item of Zabbix agent type or Zabbix
agent (active) type and on the key you need to specify:
check_ora.sk[<databasename> <query_to_execute>]
[ 262 ]
Chapter 8
You can test UserParameter using the command line, as previously
described, or using the zabbix_get utility. With zabbix_get,
you don't need to wait to see data between the latest data and it is
easier to debug what is happening on the agent side.
There are methods to test if your UserParameter is working fine and the agent is
able to recognize it. The first one is with zabbix_get, for example, in the case of
losf.sh from the Zabbix server, we can use the following:
# zabbix_get -s 127.0.0.1 -p 10050 -k lsof.sh["postgres"]
2116
The response is the result of the operation. Alternatively, we can log on to the
monitored host and run the following command:
#
/usr/sbin/zabbix_agentd -t lsof.sh["postgres"]
lsof.sh[postgres][/usr/local/bin/lsof.sh postgres] [t|2201]
Again, this will display once the output and the script that is called.
Some considerations about UserParameters
With UserParameter, you moved the script from the server side to the agent side.
The workload introduced by script is now on the agent side and you avoided
resource stealing on the server side. Another point to consider is that this approach
divides the workload between multiple servers. Obviously, each agent will monitor
the database present on its hosts.
The UserParameter parameters are really flexible. To enable them on the agent side,
you need to change the configuration file and restart the agent. Also, here you need
to be sure that the returned value is properly set or it will be discarded.
Now, between the cons you need to consider the observer effect (discussed in
Chapter 1, Deploying Zabbix) introduced with this kind of monitoring. You need
to keep things as lightweight as possible, especially because the agent runs on
the same server that provides the service.
[ 263 ]
Handling External Scripts
UserParameter usage implies that you need to distribute the scripts and the relative
updates across all your servers. In this example, where you want to monitor Oracle,
you need to consider how many different versions of operating systems and software
you need to handle. It is possible that in some time you will need to handle a myriad
of different flavors of your scripts and software. This myriad of script, version, and
so on will force you to have a centralized deployment, that is, all the versions of the
scripts are stored in a centralized repository and so on. In addition, you need to take
care of the workload added by your scripts and, if they don't handle all the possible
exceptions well, this can be really complex to manage.
UserParamenter is really good, flexible, and sometimes indispensable to solve some
monitoring requirements, but are not designed for a massive monitoring against the same
host. For all these reasons, it is time to explore another way to massively monitor the
items that Zabbix doesn't support natively.
The following are some very important points about external scripts and
Userparameters:
• All input is passed as parameters to the script and should properly be
sanitized within the script to prevent command injection.
• All values are returned via STDOUT and should be in the format of the
expected return type. Returning nothing will cause the Zabbix server to flag
this item as unsupported.
• Make sure that all scripts terminate in a short amount of time.
• Make sure that scripts do not share or lock any resources, or have any
other side effects, to prevent race conditions or incorrect interactions from
multiple executions.
Sending data using zabbix_sender
Until here, you have seen how to implement external checks on both the server
and agent side, moving the workload from the monitoring host to the monitored
host. You can understand how both methods in the case of heavy and extensive
monitoring are not the best approach, since we are thinking of placing Zabbix in a
large environment. Most probably, it is better to have a server dedicated to all our
checks and use those two functionalities for all the checks that are not widely run.
Zabbix provides utility designed to send data to the server. This utility is
zabbix_sender, and with it, you can send item data to your server, using the
items of a Zabbix trapper type.
[ 264 ]
Chapter 8
To test the zabbix_sender utility, you simply add a Zabbix trapper item to an
existing host and run the command:
zabbix_sender -z <zabbixserver> -s <yourhostname> -k <item_key> -o
<value>
You will get a response like the following:
Info from server: "Processed 1 Failed 0 Total 1 Seconds spent 0.0433214"
sent: 1; skipped: 0; total: 1
You see, the zabbix_sender utility is really easy to use. That said, now we can
dedicate a server to all our resource intensive scripts.
The new script
Now, we can change the script that has been previously used as an external check
and UserParameter in a new version that sends traps to your Zabbix server.
The core part of the software will be as follows:
CONNECTION=$( grep $HOST\; $CONNFILE | cut -d\; -f2) || exit 3;
RESULT=$( execquery $CONNECTION $QUERY.sql);
if [ -z "$RESULT" ]; then
send $HOST $KEY "none"
exit 0;
fi
send $HOST $QUERY "$RESULT"
exit 0;
This code executes the following steps:
1. Retrieve the connection string from a file.
CONNECTION=$( grep $HOST\; $CONNFILE | cut -d\; -f2) || exit 3;
2. Execute the query specified into the $QUER.sql file.
RESULT=$( execquery $CONNECTION $QUERY.sql);
3. Check the result of the query and if it is not empty, send the value to Zabbix;
otherwise, the value is replaced with "none".
if [ -z "$RESULT" ]; then
send $HOST $KEY "none"
exit 0;
fi
send $HOST $KEY "$RESULT"
[ 265 ]
Handling External Scripts
In this code, there are two main functions in play: one is the execquery() function
that basically is not changed and the other is the send() function. The send()
function plays a key role in delivering data to the Zabbix server.
send () {
MYHOST="$1"
MYKEY="$2"
MYMSG="$3"
zabbix_sender -z $ZBX_SERVER -p $ZBX_PORT -s $MYHOST -k $MYKEY -o
"$MYMSG";
}
This function sends the values passed using a command line like the one already
used to test the zabbix_sender utility. The value sent on the server side will have
the corresponding Zabbix trapper item of kind trapper and Zabbix will receive and
store your data.
Now to automate the whole check process, you need a wrapper that polls between all
your configured Oracle instances, retrieve the data, and send it to Zabbix. The wrapper
acquires the database list and the relative credential to log in from a configuration file
and you need to call your check_ora_sendtrap.sh script recursively.
Writing a wrapper script for
check_ora_sendtrap
Since this script will run from crontab, as the first thing, it will properly set up the
environment source a configuration file:
source /etc/zabbix/externalscripts/check_ora/globalcfg
Then, go down to the script directory. Please note that the directory structure has not
been changed for compatibility purposes:
cd /etc/zabbix/externalscripts
Then, it begins to execute all the queries against all the databases:
for host in $HOSTS; do
for query in $QUERIES; do
./check_ora_sendtrap.sh -r -i $host -q ${query%.sql} &
sleep 5
done;
./check_ora_sendtrap.sh -r -i $host -t &
sleep 5
./check_ora_sendtrap.sh -r -i $host -s &
done;
[ 266 ]
Chapter 8
Note that this script for each database executes all the queries and retrieves the
tnsping time and the connection time. There are two environment variables that are
used to cycle between hosts and queries; they are populated with two functions:
HOSTS=$(gethosts)
QUERIES=$(getqueries)
The gethost functions retrieve the database name from the configuration file:
/etc/zabbix/externalscripts/check_ora/credentials
gethosts () {
cd /etc/zabbix/externalscripts/check_ora
cat credentials | grep -v '^#' | cut -d';' -f 1
}
The getquery function goes down into the directory tree, retrieving all the query
files present.
getqueries () {
cd /etc/zabbix/externalscripts/check_ora
ls *.sql
}
Now you only need to schedule the wrapper script on crontab
*/5 * * * * /etc/zabbix/externalscripts/check_ora_cron.sh
And your Zabbix server will store and graph data.
All the software discussed here is available on SourceForge https://
sourceforge.net/projects/checkora released on GPLv3 and
on http://www.smartmarmot.com/.
Pros and cons of a dedicated script server
With this approach, we have a dedicated server that retrieves data. This means you
do not overload the server that provides your service or the Zabbix server, and this is
really a good point.
Unfortunately, this kind of approach lacks flexibility, and in this specific case, all the
items are refreshed every five minutes. On the other hand, with the external check or
UserParameter, the refresh rate can vary and be customized per item.
[ 267 ]
Handling External Scripts
In this particular case, where a database server is involved, there is an observer
effect introduced by our script. The query can be as lightweight as you want, but to
retrieve an item, sqlplus will ask Oracle for a connection. This connection will be
used only for a few seconds, the time needed to retrieve the item, after which the
connection is closed. All this workflow basically lacks connection pooling. Using
connection pooling, you can sensibly reduce the observer effect on your database.
Reducing the overhead with connection pooling is a general concept and
it is not tied with a vendor-specific database. Databases in general will
suffer if they are hammered with frequent requests of a new connection
and a close connection.
Pooling the connection is always a good thing to do in general. To better understand
the benefit of this methodology, you can consider a complex network with a path
that crosses different firewalls and rules before arriving to a destination; this is
the clear advantage of having a persistent connection. To have a pool of persistent
connections kept valid with keep-alive packed, reduce the latency to retrieve the
item from your database, and in general, the network workload. Creating a new
connection involves the approval process of all the firewalls crossed. Also, you need
to consider that if you are using Oracle, first is a connection request made against
the listener that will require a call back once accepted and so on. Unfortunately,
the connection pooling can't be implemented with the shell components. There are
different implementations of connection pooling, but before we go deep into the
programming side, it is time to see how the Zabbix protocol works.
Working with Zabbix protocols
The Zabbix protocol is quite simple; this is a strong point because it is simple to
implement your own custom agent or software that sends data to Zabbix.
Zabbix supports the different versions of protocols. We can divide the protocols into
three families:
• Zabbix get
• Zabbix sender
• Zabbix agent
[ 268 ]
Chapter 8
The Zabbix get protocol
The Zabbix get protocol is really simple and easy to implement. Practically, you only
need to send data to your Zabbix server at the port 10050.
This protocol is so simple that you can implement it with a shell script as well!
This is a textual protocol and is used to retrieve data from the agent
directly. [[email protected]]# telnet 127.0.0.1 10050
Trying 127.0.0.1...
Connected to 127.0.0.1.
Escape character is '^]'.
agent.version
ZBXD2.0.6Connection closed by foreign host.
This example shows you how to retrieve the agent version simply with a telnet.
Please note that the data is returned with a header that is ZBXD, followed by the data
that represent the actual response 2.0.6.
This simple protocol is useful to retrieve data directly from the agent installed into
our server and use it in a shell script.
This protocol is useful to identify the agent version without logging on to the server,
or check all the instances of UserParameter defined against an agent.
The Zabbix sender protocol
The Zabbix sender is a JSON-based protocol. The message composition is the
following:
<HEADER><DATA_LENGTH><DATA>
The <HEADER> section is of five bytes and it is in the form ZBXD\x01. Actually, only the
first four bytes are the header, the next byte is used to specify the protocol version.
Currently, only Version 1 is supported (0 x 01 HEX).
The <DATA_LENGTH> section is eight bytes in length and in hex format. So, for instance,
1 is formatted as 01/00/00/00/00/00/00/00, an eight bytes (or 64 bit) number in a
hex format.
[ 269 ]
Handling External Scripts
It is followed by <DATA>. This section is expressed in the JSON format.
Zabbix from Version 2.0.3 can receive only 128 MB of data to prevent
the server from running out of memory. This limit has been added to
protect the server from crashes caused by a large amount of data input.
To send the value, the JSON message needs to be in the following form:
<HEADER><DATALEN>{
"request":"sender data",
"data":[
{
"host":"Host name 1",
"key":"item_key",
"value":"XXX",
"clock":unix_time_format
},
....
{
"host":"Host name 2",
"key":"item_key",
"value":"YYY"
}
],
"clock":unix_time_format
}
In the previous example, as you can see, multiple items are queued on the same
message if they come from different hosts or referred to as different items keys.
The "clock" term is optional in this protocol and can be omitted
on the JSON object as well as at the end of the data section.
Once all the items are received, the server will send back the response. The response
has the following structure:
<HEADER><DATALEN>{
"response":"success",
"info":"Processed 6 Failed 1 Total 7 Seconds spent 0.000283"
}
[ 270 ]
Chapter 8
This example reports a response message; the following are some considerations:
• The response has a status that can be [success|failure] and is refers to the
whole transmission of your items list to the Zabbix server.
• It is possible, like in this example, that some of the items failed. You simply
receive a notification and you can't do much more than notify and write this
status in a logfile.
It is important to keep track of the time spent to send your item
list because if this value becomes high, or has a sensible variation,
it means that our Zabbix server suffers on receiving items.
Unfortunately, this protocol does not give you feedback of which items failed and
the reason for the failure. At the time of this writing, there were two requested
features that are still pending:
• To have a more readable output, visit https://support.zabbix.com/
browse/ZBXNEXT-935
• To identify the failed items, visit https://support.zabbix.com/browse/
ZBXNEXT-246
Now you know how the Zabbix sender protocol works on Version 1.8 and later.
An interesting undocumented feature
There is an interesting sender's feature that is not widely known and not well
documented. While going deep into the protocol analysis, the first thing to do
is read the official documentation and the second is to check how Zabbix will
implement it; because is it possible that not all the minor changes are updated
in the documentation.
Then, looking into the zabbix_sender code, you can find the section where the
protocol is implemented:
zbx_json_addobject(&sentdval_args.json,
zbx_json_addstring(&sentdval_args.json,
ZBX_JSON_TYPE_STRING);
zbx_json_addstring(&sentdval_args.json,
JSON_TYPE_STRING);
zbx_json_addstring(&sentdval_args.json,
value, ZBX_JSON_TYPE_STRING);
[ 271 ]
NULL);
ZBX_PROTO_TAG_HOST, hostname,
ZBX_PROTO_TAG_KEY, key, ZBX_
ZBX_PROTO_TAG_VALUE, key_
Handling External Scripts
The preceding code snippet implements the Zabbix JSON protocol and in particular
this section:
"host":"Host name 1",
"key":"item_key",
"value":"XXX",
Until here, the protocol has been well documented. Right after these lines are
interesting sections that add one more property to our JSON item.
if (1 == WITH_TIMESTAMPS)
zbx_json_adduint64(&sentdval_args.json, ZBX_PROTO_TAG_CLOCK,
atoi(clock));
Here a timestamp is provided within the item and is added as a property of the
JSON object, after which the item is closed as follows:
zbx_json_close(&sentdval_args.json);
The clock is defined as unsigned int64.
This is a really important property because if you write your own zabbis_sender,
you can specify the timestamp of when the item has been retrieved.
The important thing is that testing this section, Zabbix stores in its database the item,
at the specified clock time.
Using clock properties in JSON items
Now this property can be used to optimize your sender. Zabbix supports 128 MB of
data for a single connection. Of course, it is better to be far from that limit; because if
we reach that limit, it is a sign that our implementation is not well done.
Clock can be used in two scenarios:
• If buffer items need to be sent and if they are sent inside a single
connection in burst
• If the server is not available, you can cache and send the item later.
The first usage of this feature is clearly an optimization to keep the whole
communication as lightweight as possible, reducing the number of connections
against our Zabbix server can prevent issues.
[ 272 ]
Chapter 8
The second way to enable this is implement a robust sender, which can handle a
Zabbix server downtime and preserve your item in a cache, ready to be sent once
the server is backed up and running. Please be aware to not flood the server if it is
not reachable for a long period. Manage the communication by sending a reasonable
number of items and not a long trail of items.
The Zabbix Agent protocol
This protocol is a bit more complex because it involves more phases and the dialogue
is more articulated. When an active agent starts, the first thing it does is connect to
the server and ask for a task to perform, in particular, which item should be retrieved
and the relative timing.
Also, in the following, the form of the protocol is the same as used previously:
<HEADER><DATA_LENGTH><DATA>
Where <HEADER>, <DATA_LENGTH>, and <DATA> are as explained in the
previous section.
The dialogue begins when the agent sends the following request to the server:
<HEADER><DATALEN>{
"request":"active checks",
"host":"<hostname>"
}
With this kind of request, the agent is going to ask in the active checklist for a
specified hostname. The server response will, for instance, be as follows:
<HEADER><DATALEN>{
"response":"success",
"data":[{
"key":"log[\/var\/log\/localmessages,@errors]",
"delay":1,
"lastlogsize":12189,
"mtime":0
},
{
"key":"agent.version",
"delay":"900"
}]
"regexp":[
{
"name":"errors",
[ 273 ]
Handling External Scripts
"expression":"error",
"expression_type":0,
"exp_delimiter":",",
"case_sensitive":1
}]
}
Zabbix server must respond with success, followed by the list of items and the
relative delay.
In the case of log and logrt items, the server should respond with
lastlogsize. The agent needs to know this parameter to continue
the work, also, mtime is needed for all the logrt items.
"regexp", which in this example is the response back to the agent, will exist only
if you have defined some global or regular expression. Note that if a user macro
is used, the parameter key is resolved and the original key is sent as key_orig.
The original key is the user macro name.
Once the response is received, the agent will close the TCP connection and will
parse it. Now the agent will start to collect the items at their specified period.
Once collected, the items will be sent back to the server:
<HEADER><DATALEN>{
"request":"agent data",
"data":[
{
"host":"HOSTNAME",
"key":"log[\/var\/log\/localmessages]",
"value":"Sep 16 18:26:44 linux-h5fr dhcpcd[3732]: eth0: adding default
route via 192.168.1.1 metric 0",
"lastlogsize":4315,
"clock":1360314499,
"ns":699351525
},
{
"host":"<hostname>",
"key":"agent.version",
"value":"2.0.1",
"clock":1252926015
}
],
"clock":1252926016
}
[ 274 ]
Chapter 8
While implementing this protocol, pay attention while sending
back lastlogsize for all the log-type items and mtime for
the logrt items.
The server will respond with
{
"response":"success",
"info":"Processed 2 Failed 0 Total 2 Seconds spent 0.000110"
}
Also, here there is a possibility that some items have not been accepted, but there
isn't currently a way to know which ones they are.
Some more possible responses
To complete the protocol description, you need to know that there are some
particular cases to handle:
• When a host is not monitored
• When a host does not exist
• The host is actively monitored but there aren't active items
In the first case, when a host is not monitored, the agent will receive the following
response from the server:
<HEADER><DATALEN>{
"response":"failed",
"info":"host [Host name] not monitored"
}
In the second case, when the host does not exist, the agent will receive the
following response:
<HEADER><DATALEN>{
"response":"failed",
"info":"host [Host name] not found"
}
[ 275 ]
Handling External Scripts
And in the last case, when the host is monitored, but it does not have active items,
the agent will receive an empty dataset:
<HEADER><DATALEN>{
"response":"success",
"data":[]
}
Communicating with Zabbix
Now you know how the Zabbix protocol works and so it is time to see some code
that implements this protocol. To keep things easy, we have described an example
of the zabbix_sender protocol, the simplest way to send data to Zabbix.
Zabbix uses JSON to describe the object contained into data. There are a lot of
efficient JSON libraries that can be used, but to make things more easy here,
those libraries will not be used.
Implementing Zabbix sender in Java
Here you will see a really simple implementation of the zabbix_sender protocol
that as you know, is the easy way to send a trap to Zabbix.
The piece of code that follows has been kept as simple as possible and the scope
is to provide an example from which you can start to develop your own Zabbix
monitoring component.
private String buildJSonString(String host, String item,Long
timestamp, String value){
return "{"
+ "\"request\":\"sender data\",\n"
+ "\"data\":[\n"
+
"{\n"
+
"\"host\":\"" + host + "\",\n"
+
"\"key\":\"" + item + "\",\n"
+
"\"value\":\"" + value.replace("\\", "\\\\") + "\",\n"
+
"\"clock\":" + timestamp.toString()
+
"}]}\n" ;
}
This piece of code simply returns the JSON message to send it as a body. You only
need to provide the host, item or better the item key, the value, and timestamp to
include it into the message and it will return a JSON-formatted string object.
[ 276 ]
Chapter 8
Now, once you have retrieved all your item values, you simply need to generate
the JSON message, open a connection, and send the message. To open a connection
against your Zabbix server, we can use the following lines of code:
String data = buildJSonString( host,item,value);
zabbix = new Socket(zabbixServer, zabbixPort);
zabbix.setSoTimeout(TIMEOUT);
out = new OutputStreamWriter(zabbix.getOutputStream());
int length = data.length;
In this code, as you see the program open a socket, define the timeout and retrieve
the message length, now it is ready to send the message. Please remember that the
message is composed with <HEADER><DATALEN><MESSAGE>, a simple way to send the
header and the data length is the following:
out.write(new byte[] {
'Z', 'B', 'X', 'D',
'\1',
(byte)(length & 0xFF),
(byte)((length >> 8) & 0x00FF),
(byte)((length >> 16) & 0x0000FF),
(byte)((length >> 24) & 0x000000FF),
'\0','\0','\0','\0'});
This portion of code writes the message on the socket that actually contains the
host, item, and value:
out.write(data);
Remember to flush the data and close the socket once the delivery is completed
as follows:
out.flush();
out.close();
Now, we need to see what the Zabbix server will say about our item:
in = zabbix.getInputStream();
final int read = in.read(response);
String respStatus = (String) getValue(response);
if (read !=2 || respStatus.equals(ZBX_SUCCESS)) {
in.close();
}
If the response is a success, you can close the InputStream.
[ 277 ]
Handling External Scripts
This example is fully working but it is only for educational purpose. There are different
things to improve before considering it ready for production. Anyway, this is a good
starting point. This example can be extended by handling multiple JSON objects on
the data section, increasing the number of objects passed per connection. You need to
limit the connection numbers and avoid flooding your Zabbix server with connections
just to send an item. Items can be buffered and sent together, for instance, if you have a
group of items with the same schedule all of them can be sent together.
When you retrieve your items, it is important keep track of the timestamps. For doing
so, you can add the timestamp to your item, and know when it has actually retrieved
this metric.
In the previous example, the timestamp is not sent since it is optional, but it is a
good practice to include it, especially if you're buffering some item; when you send
it, the items will have the right timestamp.
Implementing Zabbix sender in Python
Nowadays, a lot of applications are written in Python and it is a programing
language widely diffused and known. For this reason, it is an example of a
fundamental treat that can be the starting point for your custom zabbix_sender
in python. This piece of code can be extended and integrated directly into your
software. Having a functionality integrated in the application can be really
interesting because the application itself can send its health status to your Zabbix
server. Now, it is time to take a look at the piece of code and how it works.
First, you need to define the structure and import simplejson used here, to add in
JSON format the host, key, item value, and clock.
import simplejson as smplj
items_data = []
Now, retrieve the timestamp from the items; if it is null, we will get the
current timestamp:
clock = zbxit.clock or time.time()
[ 278 ]
Chapter 8
Now you can begin to create the JSON object to include it in the Zabbix message:
items_data.append(('\t\t{\n'
'\t\t\t"host":%s,\n'
'\t\t\t"key":%s,\n'
'\t\t\t"value":%s,\n'
'\t\t\t"clock":%s}') % (smplj.dump(zbxit.
host), smplj.dump(zbxit.key), smplj.dump(zbxit.value), clock))
Now that your item has been transformed into a JSON object, it is time for
the header:
json_items = ('{\n'
'\t"request":"sender data",\n'
'\t"data":[\n%s]\n'
'}') % (',\n'.join(items_data))
The next step is to retrieve the length of our message to add it on the header:
data_len = struct.pack('<Q', len(json_items))
As previously discussed, here the message is put on the form
<HEADER><DATALEN>+<JSON ITEM> as follows:
packet = 'ZBXD\1' + data_len + json_items
Then, the socket is going to be open and the packet sent:
zabbix = socket.socket()
zabbix.connect((zabbix_host, zabbix_port))
zabbix.sendall(packet)
Once the packet has been sent, it is time to retrieve the Zabbix server response:
resp_hdr = _recv_all(zabbix, 13)
Next, check if it is valid:
if not resp_hdr.startswith('ZBXD\1') or len(resp_hdr) != 13:
return False
resp_body_size = struct.unpack('<Q', resp_hdr[5:])[0]
resp_body = zabbix.recv(resp_body_size)
zabbix.close()
resp = smplj.loads(resp_body)
if resp.get('response') != 'success':
return False
return True
This piece of code is a good starting point to develop the Zabbix sender in python.
[ 279 ]
Handling External Scripts
Some considerations about agent
development
Now you probably don't see the time to begin the development of your software that
sends a trap to Zabbix. But before beginning to write the code, it is fundamental to
keep in mind the requirements and the problem.
Until here, you have two examples and you can easily start to send a trap to the
Zabbix server, even if they are not completely engineered components.
First it is important to understand if it is only needed to send the data to Zabbix
at a specified time schedule that is not directed from the Zabbix server. Those two
pieces of code implement the Zabbix sender protocol, but the frequency with which
the items are retrieved and sent can't be defined from the Zabbix server. Here, it
is important to keep in mind who will drive your software, is it the Zabbix server
or your software? To enable Zabbix to drive the sampling frequency, you need to
implement the Zabbix agent protocol. The agent protocol is a bit more articulated
and a bit more complex to implement. Anyway, the two examples proposed have all
the components needed to properly handle the Agent protocol.
There is another point to consider. Usually, developers have their own preference for
a programming language. Here, it is important to use the right instrument to solve
the problem. A practical example would be to monitor your Oracle database. So your
software will need to interact with the commercial software; the easy and logical
choice is to use Java. Now, all the Python fans will stick up their nose! Here, more
than the preference, it is important to keep in mind what is better supported from the
monitored entity.
Oracle and databases in general produce standard industry engineered drivers
for Java to interact with them. Most of the database vendors provide and, more
importantly, update, fix, and develop their JDBC drivers continuously. It is better to
delegate a bit of work to vendors. Also, they know their products better and you can
get assistance on that.
Java has a lot of well-engineered components that will make your life easy in the
difficult task of monitoring a database. For instance, the JDBC framework together
with the database driver will provide an efficient connection pooling that can be
configured to the following:
• Handle a minimum and maximum number of connections
• Validate the connection before using it for your software
• Send a keep-alive packet (useful to avoid firewall issues).
[ 280 ]
Chapter 8
• Handle a reap time, removing all the idle connections (reducing the total
amount of connection on the monitored server)
Those are only some of the points covered by JDBC all these points will help you to
keep the monitoring lightweight and efficient.
An example of software made to monitor databases in general is
DBforBIX available at http://sourceforge.net/projects/
dbforbix/ or http://www.smartmarmot.com/product/
dbforbix/
Summary
In this chapter, we introduced you to the all the possible ways that will help you
interact with the server, enabling Zabbix to acquire items and metrics that are
otherwise unsupported. In this chapter, we have seen the steps required to move
the Oracle monitoring script from the server side to the client side, and then to its
final destination, the dedicated server. Here, you have learned how a simple script
grows until it becomes a complex external software. In each step, you have seen
the progress and an analysis of the pros and cons of each location the script passed.
This does not mean that you need a dedicated server for all your checks, but if your
monitoring of the script is widely and extensively used, then it is a good practice.
For each location passed, you have seen the positive side and the negative side
of that particular placement. Now, you have a global vision of what can be done,
and which one is the right place or point to act. Now the Zabbix protocol doesn't
have more secrets and you can extend Zabbix ideally without any limits.
In the next chapter, you will learn how to interact with Zabbix using the API.
The next chapter will explain how to take advantages of the Zabbix API for massive
deployments of hosts, users, and massive and repetitive operations in general.
[ 281 ]
Extending Zabbix
Understanding the Zabbix monitoring protocol allows us to write scripts, agents,
and custom probes. In other words, it allows us to freely extend Zabbix's monitoring
capabilities by expanding its means to collect data.
When it comes to actually controlling and administrating its monitoring objects,
though, the only point of access we have mentioned until now is the web frontend.
Whether you need to add a user, change the sampling frequency of an item, or look
at historical data, you always need to use the frontend as a user interface.
This is certainly a convenient solution for day-to-day activities, as all you need to
have access to is a simple browser. The frontend itself is also quite powerful and
flexible, as you can conveniently perform mass operations on many objects of the
same type, and control different nodes and proxies from the same spot.
On the other hand, not every large and complex operation can be performed
conveniently through a web application, and sometimes you don't need to just
visualize some data, but you need to export it and feed it to other programs in order
to further analyze it. This is where the Zabbix API comes in. As you will learn in this
chapter, Zabbix's JSON RPC API provides all the functions available to the frontend,
including user management, monitoring configurations, and access to historical data.
In the following pages, you will learn to:
• Write code to connect to the API and make queries through it
• Create custom operations to manage your installation
• Write complex and conditional mass operations
• Export monitoring data in a number of different formats
Let's start with a look at the general API architecture and the way to set up your code
in order to interact with it.
Extending Zabbix
Exploring the Zabbix API
Zabbix provides an entry point to interact, manipulate, configure, or create objects in
Zabbix. This API is available through its PHP frontend at the following URL:
http://<your-zabbix-server>/zabbix/api_jsonrpc.php
The communication protocol is JSON based, and the medium used is obviously
http/https.
Zabbix's JSON RPC API provides a nice interface and exposes a lot of functionality.
Once authenticated, it will allow you to perform any kind of operation on the Zabbix
objects. Now, if you need to configure Zabbix in a large or in a very large network,
this Zabbix API can be really useful. As a practical example, you can consider that
you may need to add a large amount of devices that most probably are already
defined in a separate document. The API provides the entry point to add all of them
in Zabbix by simply using a dedicated script.
Zabbix API was introduced with the Zabbix Version 1.8, and it went through some
changes up until the current 2.0 Version. This version can be considered more
stable and mature, but it is still officially in draft state, so things may change a little
in the future versions. This does not mean that it's not suitable for a production
environment; on the contrary, the bigger the installation, the more crucial can there
be sensible use of the API to script complex and time-consuming operations.
The following code is a simplified JSON request to the Zabbix API:
{
"jsonrpc": "2.0",
"method": "method.name",
"params": {
"param_1_name": "param_1_value",
"param_2_name": "param_2_value"
},
"id": 1,
"auth": "159121ba47d19a9b4b55124eab31f2b81"
}
The following points explain what the preceding code lines represent:
"jsonrpc": "2.0": This is a standard JSON PRC parameter used to identify the
protocol version; this will not change across your requests
[ 284 ]
Chapter 9
"method": "method.name": This parameter defines the operation that should be
performed, for instance, it can be host.create or item.update
• "params": This specifies the parameter needed by the method in JSON.
Here if you want to create an item, the most common parameters, will be
"name" and "key_"
• "id": 1: This field is useful to tie a JSON request to its response.
Every response will have the same "id" provided in the request. This "id"
is useful when you are going to send multiple requests at once if those
requests don't need to be serialized or be sequential.
• "auth": "159121ba47d19a9b4b55124eab31f2b81": This is the
authentication token used to identify an authenticated user; for more
details please see the next section.
For a detailed description of all the possible parameters and
methods, please refer to the official Zabbix documentation
available at the following website:
https://www.zabbix.com/documentation/2.0/manual/
appendix/api/api
Now it is important to remember that the whole communication usually is on HTTP.
This is something to consider if we interact with Zabbix from our workstation or
from a different network location. To interact with the Zabbix API, the first thing you
need is authentication by the server, and here it is clear how important it is to have
the whole communication encrypted and to use a secured channel. There are two
different exposures for you to consider:
• Use http instead of https; otherwise the whole authentication will be in clear
format, and readable
• Be aware of the sensitivity of the data being transmitted.
Now it is time to do the first step here with the API. The first thing you can do is
the authentication.
[ 285 ]
Extending Zabbix
Authenticating through the API
Here we discuss an example using a Unix shell because this will show us how simple
communication is; later we will analyze an example with Python since it is widely
used for rapid application development.
To test the authentication from our shell we can use curl. This is a common tool
used to transfer data from/to a service using different protocols, and here we use
https to make a secure channel. Now, for this test, you can log on to your Zabbix
server and write the following command:
$ curl --insecure --include --netrc --requesX POST --header "Content-Type:
application/json" [email protected] https://127.0.0.1/zabbix/api_jsonrpc.php
Please note that –-insecure specifies to curl to not verify the server certificate.
This option produces a less-secure connection, but since we are the localhost, it
is acceptable and will avoid a lot of certificate issues. Doing a practical example
without the --insecure curl will respond with the following error:
curl: (60) Peer certificate cannot be authenticated with known CA
certificates
More details here: http://curl.haxx.se/docs/sslcerts.html
Between the options, we set the Content-Type header as JSON and enable curl to
receive data from the standard input with [email protected] Once this command is run you
can paste the following JSON envelope:
{
"jsonrpc": "2.0",
"method": "user.login",
"params": {
"user": "Admin",
"password": "my secret password"
},
"auth": null,
"id": 0
}
Make sure you replace the "password" properties with your own password,
and then you can close the standard input using Ctrl + D.
[ 286 ]
Chapter 9
curl will take care to manage the whole https connection and will return the server's
full HTTP response. In this case, we are interested in the authentication token:
HTTP/1.1 200 OK
The remaining output is the following:
Content-Type: application/json
{"jsonrpc":"2.0","result":"403bbcdc3c01d4d6e66f68f5f3057c3a","id":0}
This response contains the token that we need to use for all the following queries on
the Zabbix Server.
The token will expire according to the auto-logout option set
for the user who is authenticating.
Now to see how all this works, we can use curl again:
# curl --insecure --include --netrc –request POST --header "Content-Type:
application/json" [email protected] https://127.0.0.1/zabbix/api_jsonrpc.php.
In this example, we are going to ask our server which is the last history value for the
item "Processor load (15 min average per core)" then in this particular case, on this
server, the JSON envelope will be the following lines of code:
{ "jsonrpc": "2.0",
"method": "history.get",
"params": {
"output": "extend",
"history": 0,
"hostids": "10096",
"itemid": "23966",
"sortfield": "clock",
"sortorder": "DESC",
"limit": 1
},
"auth": "403bbcdc3c01d4d6e66f68f5f3057c3a",
"id": 1
}
Remember that the request must contain the authentication token previously
obtained using the "user.authenticate" method.
[ 287 ]
Extending Zabbix
The server response in this case is the following:
HTTP/1.1 200 OK
…response header removed…
{"jsonrpc":"2.0",
"result":[
{"hosts":
[{"hostid":"10096"}],
"itemid":"23840",
"clock":"1381263380",
"value":"0.1506",
"ns":"451462502"}
],"id":1}
In this example, you have seen a way to use the authentication token to query the
historical data for a particular host/item. Of course, shell scripting is not the best
method to interact with Zabbix API because it requires a lot of coding to manage
the "auth" token, and it is better to use something more user friendly.
Using the PyZabbix library
Now that we have a clear understanding of the API's architecture and its JSON-RPC
protocol, we can move beyond the manual construction of the JSON objects and rely
on a dedicated library. This will allow us to focus on the actual features of the API
and not on the specifics of the implementation.
There are quite a few Zabbix API libraries available for different languages, but the
one we'll use for the rest of the chapter is PyZabbix, written by Luke Cyca (https://
github.com/lukecyca/pyzabbix/wiki). It's a small, compact Python module that
stays quite close to the API while still being easy to use. Moreover, Python's interactive
console makes it quite convenient to try some features and build a prototype before
moving seamlessly to a complete script or application.
You can install PyZabbix very easily through Pip, the python package installer:
$ pip install pyzabbix
[ 288 ]
Chapter 9
Once the module has been installed, you'll be able to import it and use it in your
scripts to manage a Zabbix installation.
The first thing to do is to create an object for the API server and get an
authentication token.
The following code fragments are shown as part of an interactive
session, but they can also be part of any Python code:
>>> from pyzabbix import ZabbixAPI
>>> zh = ZabbixAPI("https://127.0.0.1/zabbix/")
>>> zh.login("Admin", "zabbix")
Needless to say, you have to use your actual Zabbix frontend URL and user
credentials for this code to work in your environment. If all goes well, this is
actually all there is to it. From now on, you can use the object handler to access
any API method in the following way:
>>> zh.host.get(output="refer")
The "refer" options will give you only the primary key and foreign key for any
returned object:
[{'hostid': '9909900000010084'}, {'hostid': '9909900000010085'},
{'hostid': '9909900000010086'}, {'hostid': '9909900000010087'},
{'hostid': '9909900000010088'}]
Another advantage of using a Python library is that JSON datatypes map very
cleanly onto Python ones, so much so that most of the time you won't even need
to perform any additional type conversion. Here is a table that shows the specific
types supported by the Zabbix API and some examples of how they look both in
JSON and within PyZabbix function calls:
Type
bool
JSON
pyzabbix
{"jsonrpc" : "2.0"
"method": "host.get",
"params" : {
"editable" : "true" }
"auth" : <....>
"id" : 1
}}
[ 289 ]
zh.host.
get(editable="true")
Extending Zabbix
Type
JSON
pyzabbix
flag
{"jsonrpc" : "2.0"
"method": "host.get",
"params" : {
"countOutput" : "1" }
"auth" : <....>
"id" : 1
}}
zh.host.
get(countOutput=1)
integer
{"jsonrpc" : "2.0"
"method": "host.get",
"params" : {
"limit" : 10}
"auth" : <....>
"id" : 1
}}
zh.host.get(limit=10)
string
{"jsonrpc" : "2.0"
"method": "host.get",
"params" : {
"sortfield": "name" }
"auth" : <....>
"id" : 1
}}
zh.host.
get(sortfield="name")
timestamp
{
zh.event.get(time_
from="1349797228",
time_till=
"1350661228")
"jsonrpc": "2.0",
"method": "event.get",
"params": {
"time_from":
"1349797228",
"time_till":
"1350661228",
},
"auth": <...>,
"id": 1
}
array
{"jsonrpc" : "2.0"
"method": "host.get",
"params" : {
"hostids" : [1001, 1002,
1003] }
"auth" : <....>
"id" : 1
}}
[ 290 ]
zh.host.
get(hostids=[1001,
1002, 1003])
Chapter 9
Type
JSON
pyzabbix
object
{"jsonrpc" : "2.0"
"method": "host.get",
"params" : {
"filter": { "name": ["Alpha",
"Beta"] }
"auth" : <....>
"id" : 1
} }
zh.host.
get(filter={"name":
["Alpha", "Beta"]})
query
{"jsonrpc" : "2.0"
"method": "host.get",
"params" : {
"output": "extend" }
"auth" : <....>
"id" : 1
}}
zh.host.
get(output="extend")
The library creates the method requests on the fly, so it's fairly future-proof, which
means any new or updated methods in the API will be automatically supported.
We can now move on to explore a few concrete examples of API usage. In order
to keep the code more readable and to focus on the API, and not on general
programming issues, all the examples will have a very simplistic and direct approach
to data handling, without much data validation or error management. While you
can certainly use the following fragments in interactive sessions or as part of more
complex applications (or even to build a suite of dedicated command-line tools),
you are strongly encouraged to make them more robust with the appropriate error
handling and data validation controls.
Synchronizing templates across
different nodes
One of the problems with a distributed node hierarchy that we talked about in
Chapter 2, Distributed Monitoring, is that there is no built-in way to keep all templates
synchronized across all nodes. You may recall that every node is an independent
entity when it comes to configuration and object monitoring. Yes, it's possible to
update a slave node's configuration from a master node, but it's not possible to
transfer an existing configuration from a master node to a slave node. To do this,
you need to manually replicate every change if you want to keep things consistent
across nodes.
[ 291 ]
Extending Zabbix
A possible solution is to use the export feature on your main node and then import
and update your templates on all other nodes. In order to do that, you need to
establish a workflow similar to this one:
• Maintain an updated list of nodes, along with API URLs and user credentials
• Update or create new templates only on the root node
• From the root node, export all templates
• Import exported templates in all nodes, updating existing ones and creating
new ones
The two main limitations of this setup are that local template modifications on the
different nodes will be overwritten, and as a consequence, only the users who have
access to the root node will be able to define and distribute templates across the node
hierarchy. Unless you have very specific needs, however, this should not really be
a problem, and the distinct advantage of this setup is that it becomes very easy to
automate the export-import process using the API. Let's see a simple script that
does just that:
#!/usr/bin/python
from pyzabbix import ZabbixAPI
rootmasterUser = "Admin" masterPwd = "zabbix"
masterUrl = "https://127.0.0.1/zabbix"
nodefile = "nodes.csv"
with open(nodefile, 'r') as f:
nodes = f.readlines()
The first few lines are used to set up a few variables, namely, the root node's
connection information and the location of the URLs of other nodes. Be sure to put
your actual URL and credentials instead of those shown here. In the node.csv file,
you will put a line per Zabbix node, with a semicolon-separated list of node name,
frontend URL, user, and password. Just remember to keep it updated and secured!
The file is then read into a "nodes" array that will be used later to connect to the
single nodes.
importRules = {"templates": {"createMissing": 'true',
"updateExisting": 'true'},
"templateLinkage": {"createMissing": 'true'},
"templateScreens": {"createMissing": 'true',
"updateExisting": 'true'}
}
[ 292 ]
Chapter 9
dict will be used to define the entities the import method should actually import.
If, for any reason, you just want to propagate new templates but don't want to lose
any local modifications, just omit the "updateExisting" options.
masterZh = ZabbixAPI(masterUrl)
masterZh.login(user=masterUser, password=masterPwd)
As previously seen, these two lines connect to the main Zabbix server's API.
templates = masterZh.template.get(output=['templateid'])
tlist = []
for t in templates:
tlist.append(t['templateid'])
exports = masterZh.configuration.export(format="json",
options={"templates": tlist})
Here we first get a clean list of template IDs and then use it to create an export of all
the templates available.
for node in nodes:
(name, url, usr, pwd) = node.rstrip('\n').split(";")
nzh = ZabbixAPI(rootUrl)
nzh.login(user=rootUser, password=rootPwd)
result = nzh.confimport(format="json", rules=importRules,
source=exports)
nzh.user.logout()
This is the main part of the script. For every node, we get its connection information,
and then we connect to its API and execute a configuration.import method with
the preceding options and the exported templates from the root node. Run this script
every time you update a template or create a new one on the root node.
Needless to say, this is just the simplest possible version of such a script.
Besides making it more robust with some error checking, you can add logging
information or even periodically save every export in a file and only run the
import part if two consecutive exports differ.
[ 293 ]
Extending Zabbix
Mass operations
Another common use of the API facility is to automate some operations that you can
perform from the web frontend, but they may be cumbersome or prone to errors.
Things such as adding many users or updating the host IP addresses after merging
two different networks fall under this category. The following fragments will assume
that you already have a Zabbix API handle just as shown in the previous paragraphs.
In other words, from now on, it will be assumed that your code will start with
something like the following (remember that the Zabbix URL and user credentials
here are just examples! Use your own URL and credentials):
#!/usr/bin/python
from pyzabbix import ZabbixAPI
user='Admin'
pwd='password'
url = 'https://127.0.0.1/zabbix/'
zh = ZabbixAPI(url)
zh.login(user=user, password=pwd)
Redistributing hosts to proxies
We have seen in Chapter 2, Distributed Monitoring, that you can add hosts to a proxy
through the proxy configuration form or by updating every single host's monitored
by property. Both of these methods can be too slow and cumbersome if you have a
great number of hosts, and you need to update more than just a handful of them.
If you just need to move an entire group of hosts from one proxy to another, you could
also use the mass update functionality of the frontend, but if you need to distribute
hosts to different proxies or work on just some hosts from many different groups,
this approach won't scale well.
Here is one way to redistribute all the hosts monitored by a proxy to all the other
proxies in your Zabbix installation. A possible reason to do this is that you may be
doing some proxy maintenance, and you need to bring it down for a while, but you
don't want to suspend monitoring for a whole bunch of hosts, so you redistribute
them to other proxies.
First, let's get the proxy ID, knowing the proxy name:
proxy_name = "ZBX Proxy 1"
proxy_id = zh.proxy.get(filter={"host": proxy_name}, output="refer")
[0]['proxyid']
[ 294 ]
Chapter 9
Once you have the proxy's ID, get the list of monitored hosts:
hlist = zh.proxy.get(selectHosts=['hostid'], proxyids=proxy_id,
output="refer")[0]['hosts']
hosts = [x['hostid'] for x in hlist]
Next, for simplicity's sake, let's just get the list of all other proxies excluding the one
you are removing hosts from:
proxies =
proxyid]
[x['proxyid'] for x in zh.proxy.get() if x['proxyid'] !=
Now we need to split the host list in as many roughly equal-sized chunks as the
number of proxies available:
nparts = int(round(len(hosts)/len(proxies))
hostchunks = [list(hosts[i:i+nparts]) for i in
range(0,len(hosts),nparts)]
The preceding code will divide your host list in as many sublists as the number of
proxies you have. All that is left to do is to actually assign the hosts to the proxies:
for c in len(hostchunks):
zh.proxy.update(proxyid=proxies[c], hosts=hostchunks[c])
And that's it. The proxy.update method will automatically reassign hosts, so you
don't even have to remove them first from the original one. You can of course make
things more robust by selecting only proxies in the same network as the one you are
doing maintenance on, or by saving the host list so you can reassign it to the original
proxy once it's available.
Adding or updating users
Even if you rely on some external authentication method for your Zabbix users,
such as an LDAP server or Active Directory, no new user account will have any
media information set nor will it belong to any group. This means that you'll still
need to manually configure every user unless you create new users or update
existing ones through some kind of code. For simplicity's sake, let's assume that you
already have a list of usernames and email addresses, and the groups they should
belong to, all gathered in a semicolon-separated users.csv file that looks like this:
adallevacche,[email protected],Admins
slee,[email protected],Admins
jdoe,[email protected],DB admins; App Servers
mbrown,[email protected],Net admins
[ 295 ]
Extending Zabbix
The script you are going to write will assume that the first field of every line will
contain the username (called alias in the API). The second field will contain an
e-mail address while the last field will be a comma-separated list of user groups
that the user should belong to. Updating your users' information is quite simple.
First, you need to read into your script the contents of the users.csv file:
with open('users.csv', 'r') as f:
users = f.readlines()
Assuming that your Zabbix API connection object is called zh, you can now define a
couple of helper functions and variables. The mediatypeid will be needed to update
your users' media information. Assuming that you have only one e-mail media type
defined in your Zabbix installation, you can get its ID by calling the following:
mediatypeid = zh.mediatype.get(output="refer",
filter={"description": ['Email']})[0]['mediatypeid']
Unless you want to extend your .csv file with information about severity and
the time period for each one of your users' media, you can also define a common
template for all e-mail contacts:
def mkusermedia(mediatype='', email='', mediaid=''):
return { 2mediaid2"mediaid": mediaid
"mediatypeid": mediatype,
"sendto": email,
"active": 0,
"severity": 63,
"period": "1-7,00:00-24:00"
}
Please note how 0 means enabled, while 1 means disabled for the active property.
Also, while the period property is fairly self-explanatory, the severity property
could look quite puzzling at first. It's actually a binary bitmap value and can be
more easily understood if you take into consideration the trigger severity values
and put them in order. Each severity level occupies a position of a six bits value
Severity
Disaster
High
Average
Warning
Information
Not
classified
Enabled?
1
1
1
1
1
1
Decimal value
111111 = 63
[ 296 ]
Chapter 9
Since 63 equals 111111 in binary form, this means that the user will receive
notifications for every severity level. If you want to receive notifications only for
the disaster severity, you will have a 100000 bitmap and so a severity value of 32:
Severity
Disaster
High
Average Warning
Information Not
classified
Enabled?
1
0
0
0
0
0
Decimal value 100000 = 32
Similarly, to get notifications for disaster and higher levels, you'll need a 110000
bitmap and a severity value of 48, and so on.
Severity
Disaster
High Average Warning
Information
Not
classified
Enabled?
1
1
0
0
0
0
Decimal value 110000 = 48
The following helper function will get a list of group names and return a list of user
group IDs that actually exist, ignoring non-existing group names:
def getgroupids(grouplist):
return zh.usergroup.get(output=['usergrpid'], filter={"name":
grouplist.split(",")})
We can now proceed to actually work the user list to either update existing users or
create new ones:
for u in users:
(alias, email, groups) = u.split(",")
user = zh.user.get(output=['userid'], filter={"alias": [alias]})
if not user:
zh.user.create(alias=alias,
passwd="12345",
usrgrps=getgroupids(groups),
user_medias=[mkus
ermedia(mediatype=mediatypeid, email=email)])
The if statement checks if the user exists. If not, the user.create method will take
care of creating it, adding it to the appropriate groups, and creating the media contact
as well. You need to define a password even if your users will authenticate from an
external source. Depending on your password management policies, your users
should be strongly encouraged to change it as soon as possible, or better yet, you could
directly generate a random password instead of using a fixed string.
[ 297 ]
Extending Zabbix
The second part of the if construct will get the userid and update the
user's information.
else:
userid=user[0]['userid']
zh.user.update(userid=userid,
usrgrps=getgroupids(groups))
usermedia = zh.usermedia.get(filter={"userid" : userid},
output=['mediaid'])
zh.user.updatemedia(users = [userid],
medias=[mkusermedia(
mediaid=usermedia[0]['mediaid'],
mediatype=mediatypeid,
email=email)])
Please note the way you need to call three different methods here for user groups
and media, instead of just one. The first one will update group information; the
second one will check for an already-defined e-mail address, and the third will
update the said address or create a new one if it doesn't exist.
You can run the preceding code periodically to keep your user accounts updated.
Obvious improvements would be adding each user's name and surname or getting
user data directly from an LDAP server or any other source instead of from
a .csv file.
Exporting data
Besides directly manipulating and monitoring internal objects, another compelling
use of the Zabbix API is to extract data for further analysis outside of the Zabbix
frontend. Maps, screens, graphs, triggers, and history tables can be excellent
reporting tools, but they are all meant to be used inside the frontend. Sometimes you
may need the raw data in order to perform custom calculations on it—especially
when it comes to capacity planning—or you may need to produce a document with a
few custom graphs and other data. If you find yourself with such needs on a regular
basis, it makes sense to write some code and extract your data through the API. An
interesting feature of the get methods, which are the fundamental building blocks of
any data extraction code, is that they come with quite a few filters and options out of
the box. If you are willing to spend some time studying them, you'll find that you'll
be able to keep your code small and clean as you won't usually have to get lots of
data to filter through, but you'll be able to build queries that can be quite focused
and precise.
Let's see a few short examples in the following paragraphs.
[ 298 ]
Chapter 9
Extracting tabular data
Zabbix provides a way to group similar items in a host in order to navigate them
more easily when looking at the latest monitoring data. These item containers,
called Applications, come in very handy when the number of items in a host is quite
consistent. If you group all CPU monitoring items together under the label, say CPU,
all filesystem-related items under Filesystems, and so on, you could find the data you
are looking for more easily. Applications are just labels tied to a specific template
or host and are used just used to categorize items. This makes them simple and
lightweight. But it also means that they are not really used elsewhere in the
Zabbix system.
Still, it's sometimes useful to look at the trigger status or event history, not just by
host but by application too. A report of all network-related problems regardless of
the host, host group, or specific trigger, could be very useful for some groups in your
IT department. The same goes for a report of filesystem events, database problems,
and so on.
Let's see how to build a script that will export in a .csv file all events related to a
specific application. The setup is basically the same as the previous examples:
#!/usr/bin/python
from pyzabbix import ZabbixAPI
import sys
import csv
from datetime import datetime
appname = sys.argv[1]
timeformat="%d/%m/%y %H:%M"
zh = ZabbixAPI("http://locahost/zabbix")
zh.login(user="Admin", password="zabbix")
As you can see, the application name is taken from the command line while the
API's URL and credentials are just examples. When you use your own you can also
consider using an external configuration file for more flexibility. Since events are
recorded using a Unix timestamp, you'll need to convert it to a readable string later
on. The timeformat variable will let you define your preferred format. Speaking of
formats, the .csv module will let you define the output format of your report with
mode flexibility than a series of manual prints.
Now you can proceed to extract all applications that share the name you passed on
the command line:
applications = zh.application.get(output="shorten", filter={"name":
[appname]})
[ 299 ]
Extending Zabbix
Once we have the list of applications, you can get the list of items that belong to the
said application:
items = zh.item.get(output="count", applicationids=[x['applicationid']
for x in applications])
And from there, you still need to extract all the triggers that contain the given items
before moving to the actual events:
triggers = zh.trigger.get(output="refer", itemids=[x['itemid'] for x
in items])
Now you can finally get the list of events that are related to the application you are
interested in:
events = zh.event.get(triggerids=[j['triggerid'] for j in triggers])
Here only the event IDs are extracted. You didn't ask for a specific time period, so
it's possible that a great number of events will be extracted. For every event, we'll
also need to extract all related hosts, triggers, and items. To do that, let's first define a
couple of helper functions to get the host, item, and trigger names:
def gethostname(hostid=''):
return zh.host.get(hostids=hostid, output=['host'])[0]['host']
def getitemname(itemid=''):
return zh.item.get(itemids=itemid, output=['name'])[0]['name']
def gettriggername(triggerid=''):
return zh.trigger.get(triggerids=triggerid,
expandDescription="1", output=['description'])[0]['description']
Finally, you can define an empty eventstable table and fill it with event
information based on what you have extracted so far:
eventstable = []
triggervalues = ['OK', 'problem', 'unknown']
for e in events:
eventid = e['eventid']
event = zh.event.get(eventids=eventid,
selectHosts="refer",
selectItems="refer",
selectTriggers="refer",
output="extend")
host = gethostname(event[0]['hosts'][0]['hostid'])
item = getitemname(event[0]['items'][0]['itemid'])
[ 300 ]
Chapter 9
trigger = gettriggername(event[0]['triggers'][0]['triggerid'])
clock = datetime.fromtimestamp(int(event[0]['clock'])).
strftime(timeformat)
value = triggervalues[int(event[0]['value'])]
eventstable.append({"Host": host,
"Item": item,
"Trigger": trigger,
"Status": value,
"Time" : clock
})
Now that you have all the events' details, you can create the output .csv file:
filename = "events_" + appname + "_" + datetime.now().
strftime("%Y%m%d%H%M")
fieldnames = ['Host', 'Item', 'Trigger', 'Status', 'Time']
outfile = open(filename, 'w')
csvwriter = csv.DictWriter(outfile, delimiter=';',
fieldnames=fieldnames)
csvwriter.writerow(dict((h,h) for h in fieldnames))
for row in eventstable:
csvwriter.writerow(row)
outfile.close()
The report's filename will be automatically generated based on the application you
want to focus on and the time of execution. Since every event in the eventstable
array is a dict, a fieldnames array is needed to tell the csv.DictWriter module in
what order the fields should be written. Next, a column header is written before finally
looping over the eventstable array and writing out the information we want.
There are a number of ways you can expand on this script in order to get even
more useful data. Here are a few suggestions, but the list is limited only by
your imagination:
• Ask for an optional time period to limit the number of events extracted
• Order events by host and trigger
• Perform some calculations to add event duration based on trigger
state change
• Add acknowledge data if present
[ 301 ]
Extending Zabbix
Creating graphs from data
At this point in the book, you should be familiar with Zabbix's powerful data
visualization possibilities. On the frontend, you can create and visualize many kinds
of graphs, maps, and charts that can help you analyze and understand item history
data, trigger status change over time, IT services availability, and so on. Just as any
other of Zabbix's capabilities, all of the visualization functions are also exposed
through the API. You can certainly write programs to create, modify, or visualize
screens, graphs, and maps, but unless you are building a custom frontend, it's quite
unlikely that you'll ever need to do so.
On the other hand, it may be interesting to extract and visualize data that is otherwise
too dispersed and hard to analyze through the frontend. A good example of such data
is trigger dependency. You may recall from Chapter 5, Visualizing Data, that a trigger
can depend on one or more other triggers such that it won't change to a PROBLEM
state if the trigger it depends on is already in a PROBLEM state.
As useful as this feature is, there's no easy way to see at a glance the triggers that
depend on other triggers, and if those triggers in turn depend on other triggers, and so
on. The good news is that with the help of the Graphviz package and a couple of lines
of Python code, you can create a handy visualization of all trigger dependencies.
The Graphviz suite of programs
Graphviz (http://www.graphviz.org) is a suite of graph visualization software
utilities that enables you to create arbitrary complex graphs from especially
formatted text data. The suite provides many features for data visualization and can
become quite complex to use, but it is quite simple to create a basic, functional setup
that you can later build on.
If you do not have it installed in your system, Graphviz is just a yum install
command away if you are on a Red Hat Enterprise Linux system:
# yum install 'graphviz*'
The program you will use to create your graphs is called dot. Dot takes a graph text
description and generates the corresponding image in a number of formats. A graph
description looks like this:
digraph G {
main → node1 → node2;
main → node3;
main → end;
node2 → node4;
node2 → node5;
[ 302 ]
Chapter 9
node3 → node4;
node4 → end;
}
If you put the preceding graph in a graph.gv file and run the following command:
$ dot -Tpng graph.gv -o graph.png
You will obtain a PNG image file that will look something like the following:
As you can see, it should be fairly simple to create a trigger-dependency graph once
we have extracted the right information through the API. Let's see how we can do it.
Creating a trigger dependency graph
The following is a Python script that will extract data about trigger dependencies
and output a dot language graph description that you can later feed into the dot
program itself:
#!/usr/bin/python
from pyzabbix import ZabbixAPI
zh = ZabbixAPI("https://127.0.0.1/zabbix")
zh.login(user="Admin", password="zabbix")
def gettriggername(triggerid=''):
return zh.trigger.get(triggerids=triggerid,
output=['description'])[0]['description']
[ 303 ]
Extending Zabbix
In the first part, there are no surprises, a Zabbix API session is initiated, and a simple
helper function, identical to the one shown before, is defined.
tr = zh.trigger.get(selectDependencies="refer", expandData="1",
output="refer")
dependencies = [(t['dependencies'], t['host'], t['triggerid']) for t
in tr if t['dependencies'] != [] ]
The next two lines extract all triggers and their dependencies and then create a list,
filtering out triggers that don't have any dependencies.
outfile = open('trigdeps.gv', 'w')
outfile.write('digraph TrigDeps {\n')
outfile.write('graph[rankdir=LR]\n')
outfile.write('node[fontsize=10]\n')
Here the first few lines of the graph are written out to the output file, setting up the
graph direction, from left to right, and the font size for the nodes' labels.
for (deps, triggerhost, triggerid) in dependencies:
triggername = gettriggername(triggerid)
for d in deps:
depname = gettriggername(d['triggerid'])
dephost = d['host']
edge = '"{}:\\n{}" -> "{}:\\n{}";'.format(dephost, depname,
triggerhost, triggername)
outfile.write(edge + '\n')
This is the core of the script. The double for loop is necessary because a single
trigger may have more than one dependency, and you want to map out all of
them. For every dependency → trigger relationship found, an edge is defined in
the graph file.
outfile.write('}\n')
outfile.close()
Once the script reaches the end of the list, there is nothing more to do except to close
the graph description and close the output file.
If you execute the script:
$ chmod +x triggerdeps.py
$ ./triggerdeps.py
[ 304 ]
Chapter 9
You will get a trigdeps.gv file that will look something like this:
digraph TrigDeps {
graph[rankdir=LR]
node[fontsize=10]
"Template IPMI Intel SR1630:\nPower" -> "Template IPMI Intel SR1630:\
nBaseboard Temp Critical [{ITEM.VALUE}]";
"Template IPMI Intel SR1630:\nBaseboard Temp Critical [{ITEM.VALUE}]"
-> "Template IPMI Intel SR1630:\nBaseboard Temp Non-Critical [{ITEM.
VALUE}]";
"Template IPMI Intel SR1630:\nPower" -> "Template IPMI Intel SR1630:\
nBaseboard Temp Non-Critical [{ITEM.VALUE}]";
"Template IPMI Intel SR1630:\nPower" -> "Template IPMI Intel SR1630:\
nBB +1.05V PCH Critical [{ITEM.VALUE}]";
"Template IPMI Intel SR1630:\nBB +1.05V PCH Critical [{ITEM.VALUE}]"
-> "Template IPMI Intel SR1630:\nBB +1.05V PCH Non-Critical [{ITEM.
VALUE}]";
"Template IPMI Intel SR1630:\nPower" -> "Template IPMI Intel SR1630:\
nBB +1.05V PCH Non-Critical [{ITEM.VALUE}]";
"Template IPMI Intel SR1630:\nPower" -> "Template IPMI Intel SR1630:\
nBB +1.1V P1 Vccp Critical [{ITEM.VALUE}]";
Just run it through the dot program in order to obtain your dependencies graphs:
$ dot -Tpng trigdeps.gv -o trigdeps.png
The resulting image will probably be quite big; the following is a close up of a
sample resulting image:
[ 305 ]
Extending Zabbix
Once again, there are many ways to improve the script, from improving the layout
and the node shapes, to integrating the graph generating part into Python with its
graphviz bindings. Moreover, you could feed the image back to a Zabbix map using
the API, or you could invert the process and define trigger dependencies based on an
external definition.
Summary
In this chapter, we have barely scratched the surface of what is possible once
you begin playing with the powerful Zabbix API. If you have worked through
the examples, you should be comfortable with the JSON-RPC protocol that is the
foundation of the API. You should know how to explore the various methods
and have some ideas on how to use them to make your Zabbix management
tasks easier, or to further expand the system's data manipulation possibilities.
With the discussion of the API, we conclude our exploration of Zabbix's features.
The next and final chapter will build upon the knowledge we have gained until now
and use it to make Zabbix a more integral part of your IT infrastructure by opening
communication channels with other management systems.
[ 306 ]
Integrating Zabbix
A monitoring system is, by definition, all about connecting and communicating
with other systems. On one hand, it needs to connect to its monitored objects in
order to take measurements and evaluate service status. On the other hand, it needs
to be able to communicate the collected information outside of itself so that system
administrators can act on the data and alarms it raised. In the previous chapters of
the book, we focused mainly on the first part of the equation, namely collecting data,
and always assumed that the second part, exposing data and warnings, would involve
sending a series of messages to human operators. While this is certainly the most
common setup, the one that will be at the core of every Zabbix deployment, it's also
true that it can prove to be quite limited in a large, complex IT environment.
Every managing system has a specific, detailed view of its environment that is directly
dependent on the function it must perform. Identity management systems know all
about users, passwords, and permissions, while inventory systems keep track of
hardware and software configurations and deployment. Trouble ticketing systems
keep track of current issues with users, while monitoring systems keep track of the
availability status and performance metrics of anything they can connect to. As many
of these systems actually share some common data among them, whether it is user
information, connection configuration or anything else, it is vital that as much of this
data as possible is allowed to pass from one system to the next, without constant
manual intervention on the part of the administrators.
It will be impossible for Zabbix, or any monitoring system, to come with an
out-of-the-box integration with any other arbitrary system in the world. Its open
source nature, clear protocol, and powerful APIs make it relatively easy to
integrate your monitoring system with any other IT management tools you have
deployed in your environment. This can be the subject of a book in itself, but we
will try to get you started on the path of Zabbix integration by looking at one
such integration possibility.
Integrating Zabbix
In this chapter, you will see a concrete example of integration between Zabbix and
Request Tracker, the open source trouble ticket management system from Best
Practical (http://bestpractical.com/rt/). By the end of the chapter, you will be
able to do the following:
• Rely an alert to a trouble ticketing system
• Keep track of which tickets relate to which events
• Update events' acknowledgments based on the status of a ticket
There won't be any new concepts or Zabbix functionality explained here, as we'll
explore some of the real-world applications made possible by the features we have
already learned about in the rest of the book.
An overview of Request Tracker
Quoting from the Best Practical website:
"RT is a battle-tested issue tracking system which thousands of organizations
use for bug tracking, help desk ticketing, customer service, workflow processes,
change management, network operations, youth counseling and even more.
Organizations around the world have been running smoothly thanks to RT
for over 10 years."
In other words, as powerful yet simple open source package it's a good candidate
to showcase a Zabbix integration. This is not to say that it is the only issue tracking
system that you can use with Zabbix; once the principles behind the following
sample implementation are clear, you will be able to integrate any product with
your monitoring system.
Request Tracker (RT) is a web application written in Perl that relies on a web server
to expose its frontend and on a relational database to keep all its data on. The main
means of interaction with the system is through the web interface, but it also features
a powerful e-mail-parsing utility that can categorize an e-mail message, turn it into a
full-fledged ticket, and keep track of the subsequent mail exchange between the user
and the support staff. Closer to our interests, it also features a simple yet effective
REST API, that we'll rely on in order to create and keep track of existing tickets from
Zabbix. On the other hand, a powerful scripting engine that can execute custom
chunks of code called scrips, not only allows RT to automate its internal workings
and create custom workflows but also allows it to communicate with external
systems using any available protocol.
[ 308 ]
Chapter 10
The following diagram shows the basic application architecture. All the data is kept
in a database, while the main application logic can interact with the outside world
either through the web server or via e-mail and custom scrips.
database
main application logic
scrips
email
web server
frontend
client
email
client
browser
operator
browser
REST API
external
system
Basic Request Tracker architecture
This is not the place to cover an in-depth installation and configuration of RT,
so we will assume you already have a working RT server with at least a few users
and groups already set up. If you need to install RT from scratch, the procedure
is quite simple and well-documented; just follow the instructions detailed here
http://www.bestpractical.com/docs/rt/4.2/README.html. Refer to the
Request Tracker website link provided earlier for further information.
Setting up RT to better integrate
with Zabbix
The two basic elements of RT are tickets and queues. The function of a ticket is
to keep track of the evolution of an issue. The basic lifecycle of a ticket can be
summarized in the following points:
• A ticket is created with the first description of the problem
• An operator takes the ownership of the ticket and starts working on it
• The evolution of the problem is recorded in the ticket's history
• After the problem's resolution, the ticket is closed and archived
[ 309 ]
Integrating Zabbix
All of the ticket's metadata, from its creation date to the amount of time it took to
close it, from the user who created it to the operator that worked on it, and so on,
is recorded and grouped with all the other tickets' metadata in order to build
statistics and calculate service-level agreements.
A queue, on the other hand, is a specific collection of tickets and a way to file new
tickets under different categories. You can define different queues based on different
organization departments, or different products you are providing support for,
or any other criteria that can make it easier to organize tickets.
Let's see what can we do to configure RT queues and tickets so that we can transfer
all the information we need to and from Zabbix, while keeping any existing
functionality as a generic issue tracker as is.
Creating a custom queue for Zabbix
A nice feature of queues is that you can customize every aspect of a ticket that
belongs to a specific queue, from the fields that need to be filled out to the details
of the workflow. The first step is, therefore, to create a specific queue for all
tickets created from a Zabbix event action. This will allow us to define specific
characteristics for the corresponding Zabbix tickets.
Creating a queue is quite simple. Just go to Admin | Queues | Create and fill in the
form. For our purpose, you don't need to specify more than a name for the queue
and an optional description, as shown in the following screenshot:
[ 310 ]
Chapter 10
After the queue is created, you will be able to configure it further by going to Admin
| Queues | Select and choosing the Zabbix queue. At the very least, you should
grant the user and staff rights to a user group or to some specific users so that your
IT staff can work on the tickets created by Zabbix. You will also want to create some
custom fields, as we will see in a couple of paragraphs.
First, let's move on to look at what parts of a ticket are most interesting from an
integrator's point of view.
Customizing tickets – the links section
Keeping in mind our goal to integrate Zabbix actions and events with RT, the Links
section of a ticket is of particular interest to us. As the name suggests, you can define
links to other tickets as dependencies or even define links to other systems as further
referrals. You can insert useful links during ticket creation or while editing it.
As you probably already imagined, we'll be relying on the Refers to: link field to
link back to the Zabbix event that created the ticket. As we will see in the following
pages, the event's acknowledged field will in turn show a link to the corresponding
ticket so that you can move easily from one platform to the other in order to keep
track of what's happening.
[ 311 ]
Integrating Zabbix
Customizing tickets – ticket priority
Another interesting field in the Basics section of a ticket is the ticket's priority. It's an
integer value that can go from 0 to 100, and it's quite useful to sort tickets depending
on their severity level. There is no official mapping of RT priority and some more
descriptive severity levels such as those used by the Zabbix triggers. This means that
if you want to preserve information about trigger severity in the ticket, you have
two choices:
• Ignore the ticket's priority and create a custom field that shows the trigger
severity as a label
• Map the trigger severity values to the ticket's priority values as a convention,
and refer to the mapping while creating tickets
The only advantage of the first option is that the single ticket will be easily
readable, and you will immediately know about the severity of the corresponding
trigger. On the other hand, the second option will allow you to sort your tickets
by priority and act first on the more important or pressing issues with a more
streamlined workflow.
While creating a ticket from Zabbix, our suggestion is, therefore, to set ticket
priorities based on the following mapping:
Trigger severity label
Trigger severity value
Ticket priority value
Not classified
0
0
Information
1
20
Warning
2
40
Average
3
60
High
4
80
Disaster
5
100
There is nothing to configure either on the Zabbix or RT side. This mapping will
use the full range of priority values so that your Zabbix tickets will be correctly
sorted not only in their specific queue, but also anywhere in RT.
[ 312 ]
Chapter 10
Customizing tickets – the custom fields
As we have seen in Chapter 6, Managing Alerts, a Zabbix action can access a
great number of macros and thus expose a lot of information about the event
that generated it. While it makes perfect sense to just format this information in
a readable manner while sending e-mails, with the availability of custom fields
for RT tickets, it makes less sense to limit all of the event details just to the
ticket description.
In fact, one great advantage of custom fields is that they are searchable and filterable
just like any other native ticket field. This means that if you put the host related to a
ticket event in a custom field, you'll be able to search all tickets belonging to the said
host for reporting purposes, or assign a host's specific tickets to a particular user,
and so on. So, let's go ahead and create a couple of custom fields for the tickets in the
Zabbix queue that will contain information, which we'll find useful later on. Go to
Admin | Custom Fields | Create and create a Hosts custom field as shown in the
following screenshot:
Be sure to select Enter multiple values as the field type. This will allow us to
specify more than a single host for those complex triggers that reference items
from different hosts.
[ 313 ]
Integrating Zabbix
Speaking of triggers and items, you can follow the same procedure to create other
custom fields for the trigger name and item names or keys. After you are done,
you will need to assign these fields to the tickets in the Zabbix queue. Select the
Zabbix queue by going to Admin | Queues | Select and for the Tickets form, go to
Custom fields | Tickets. Select the fields that you wish to assign to your tickets:
After you are done, you will see the following fields in every ticket of the
Zabbix queue:
[ 314 ]
Chapter 10
Depending on your needs, you can create as many custom fields as you want for
trigger and event acknowledged history, host's IP interfaces, custom macros, and so
on. You will be able to search for any of them, and for the three shown earlier, you can
do so by selecting the Zabbix queue in the search page of the RT frontend. At the
bottom of the search form, you can see the newly created fields, just as expected:
Connecting to the Request Tracker API
RT exposes a REST-type API that relies directly on the HTTP protocol to handle
requests and responses. This means that the API is easily tested and explored by
using tools such as wget and netcat. Let's do that to get a feel of how it works
before introducing the Python library that we'll use for the rest of the chapter.
The base URL for the RT API is located at ../REST/1.0/ after the base URL of the
Request Tracker itself. This means that if your base URL is http://your.domain.
com/rt, the API will be accessible at http://your.domain.com/rt/REST/1.0/.
[ 315 ]
Integrating Zabbix
If you try to connect to it, you should get a message asking for credentials (some
response headers are removed to improve readability):
$ ncat example.com 80
GET /rt/REST/1.0/ HTTP/1.1
Host: example.com
HTTP/1.1 200 OK
[…]
Content-Type: text/plain; charset=utf-8
22
RT/4.2.0 401 Credentials required
The API doesn't have a special authentication mechanism separated from the rest
of the application, so the best way to authenticate is to get a session cookie from the
main login form and use it for each API request. To get the cookie, let's use wget:
$ wget --keep-session-cookies --save-cookies cookies.txt --post-data
'user=root&pass=password' http://example.com/rt/
The command that will save the session cookie in the cookies.txt file is as follows:
$ cat cookies.txt
# HTTP cookie file.
# Generated by Wget on 2013-11-02 10:16:58.
# Edit at your own risk.
localhost FALSE /rt FALSE 0 RT_SID_example.com.80
2bb04e679236e58b406b1e554a47af43
Now that we have a valid session cookie, we can issue requests through the API.
Here is the GET request for the general queue:
$ ncat localhost 80
GET /rt/REST/1.0/queue/1 HTTP/1.1
Host: localhost
Cookie: RT_SID_example.com.80=2bb04e679236e58b406b1e554a47af43
HTTP/1.1 200 OK
[...]
Content-Type: text/plain; charset=utf-8
RT/4.2.0 200 Ok
[ 316 ]
Chapter 10
id: queue/1
Name: General
Description: The default queue
CorrespondAddress:
CommentAddress:
InitialPriority: 0
FinalPriority: 0
DefaultDueIn: 0
As you can see, the API is quite easy to interact with without any special encoding or
decoding. For our purposes, however, it is even easier to use a library that will spare
us the burden of parsing each HTTP request. Rtkit is a Python 2.x library that makes
it even easier to connect to the API from within a Python program, allowing us to
send requests and get responses using native Python data structures. The installation
is very simple using pip:
$ pip install python-rtkit
Once installed, the library will be available by importing various Rtkit modules.
Let's see the same preceding interaction (authenticating and requesting the general
queue) from within a Python 2.x session:
$ python2
Python 2.7.5 (default, Sep 6 2013, 09:55:21)
[GCC 4.8.1 20130725 (prerelease)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from rtkit.resource import RTResource
>>> from rtkit.authenticators import CookieAuthenticator
>>> from rtkit.errors import RTResourceError
>>>
>>> res = RTResource('http://localhost/rt/REST/1.0/', 'root', 'password',
CookieAuthenticator)
>>>
>>> response = res.get(path='queue/1')
>>> type(response)
<class 'rtkit.resource.RTResponse'>
>>> type(response.parsed)
<type 'list'>
>>> response.parsed
[[('id', 'queue/1'), ('Name', 'General'), ('Description', 'The
default queue'), ('CorrespondAddress', ''), ('CommentAddress', ''),
('InitialPriority', '0'), ('FinalPriority', '0'), ('DefaultDueIn', '0')]]
[ 317 ]
Integrating Zabbix
As you can see, a response is parsed into a list of tuples with all the attributes of
an RT object.
Now that we have a custom queue and custom fields for Zabbix tickets, we are
able to interact with the API through the Python code, and the setup process on
the RT's side is complete. We are ready to actually integrate the Zabbix actions
and the RT tickets.
Setting up Zabbix to integrate with
Request Tracker
Our goal is to define a Zabbix action step that when executed will:
• Create a ticket with all the relevant event information
• Link the ticket back to the Zabbix event that generated it
• Acknowledge the event with a link to the ticket just created
While the first point can be covered with a simple e-mail action to RT, we need
some custom code to take care of the other two. The best way to do this is to define
a new media type in Zabbix as a custom alert script. The script will:
• Take the action message
• Parse it to extract relevant information
• Create a ticket with all custom fields and link the referrals filled out
• Get back the ticket ID
• Write a link to the created ticket in the event's acknowledgment field
Before actually writing the script, let's create the media type and link it to a user
(you can assign the media type to any user you want; the custom rt_tickets
user will be used here).
[ 318 ]
Chapter 10
While linking the media type to the user, use the RT base URL in the Send to field,
so you won't need to define it statically in the script:
[ 319 ]
Integrating Zabbix
Once saved, you'll see all relevant information at a glance in the Media tab. Just after
the URL address, you'll find the notification periods for the media, and after that
a six-letter code that shows the active severity levels. If you disabled any of them,
the corresponding letter would be gray.
Now let's create an action step that will send a message to our rt_tickets user
through the custom media type. Needless to say, the rt_tickets user won't receive
any actual message as the alert script will actually create a RT ticket, but all of this is
completely transparent from the point of view of a Zabbix action. You can put any
information you want in the message body, but, at the very least, you should specify
the trigger name in the subject and the event ID, severity, hosts, and items in the
body so that the script will parse them and fill them in the appropriate ticket fields:
[ 320 ]
Chapter 10
We are now ready to actually write the script and use it to create tickets from Zabbix.
[ 321 ]
Integrating Zabbix
Creating RT tickets from the Zabbix
events
Zabbix will search for custom alert scripts in the directory specified by
AlertScriptsPath in the zabbix_server.conf file. In the case of a default
install, this would be /usr/local/share/zabbix/alertscripts/.
This is where we will put our script called rt_mkticket.py. The Zabbix action we
configured earlier will call this script with the following three arguments, in order:
• recipient
• subject
• message
As we have seen, the content of the subject and the message are defined in the action
operation and depend on the specifics of the event triggering action. The recipient
is defined in the media type configuration of the user receiving the message, and it
is usually an e-mail address. In our case, it will be the base URL of our Request
Tracker installation.
So let's start the script by importing the relevant libraries and parsing the arguments:
#!/usr/bin/python2
from pyzabbix import ZabbixAPI
from rtkit.resource import RTResource
from rtkit.authenticators import CookieAuthenticator
from rtkit.errors import RTResourceError
import sys
import re
rt_url = sys.argv[1]
rt_api = rt_url + 'REST/1.0/'
trigger_name = sys.argv[2]
message= sys.argv[3]
[ 322 ]
Chapter 10
Now we need to extract at least the event URL, the trigger severity, a list of host
names, and a list of item names from the message. To do this, we will use the
powerful regular expression functions of Python:
event_id = re.findall(r'^Event: (.+)$', message, re.MULTILINE)[0]
severity = re.findall(r'^Trigger severity: (.+)$', message,
re.MULTILINE)[0]
hosts = re.findall(r'^Host: (.+)$', message, re.MULTILINE)
items = re.findall(r'^Item: (.+)$', message, re.MULTILINE)
lines = re.findall(r'^(?!(Host:|Event:|Item:|Trigger severity:))
(.*)$', message, re.MULTILINE)
desc = '\n'.join([y for (x, y) in lines])
While the event ID has to be unique, a trigger can reference more than one item,
and thus more than one host. The preceding code will match any line beginning with
Host: to build a list of hosts. In the preceding action message, we just put one Host:
{HOST.NAME} line for readability purposes, but your actual template can contain
more than one (just remember to use {HOST.NAME1}, {HOST.NAME2}, {HOST.NAME3},
and so on, or you'll end up with the same host value repeatedly). Of course, the same
goes for item names. The rest of the message is then extracted with the opposite of
the regexps used before and joined back in a single multiline string.
Now, the macro we used for trigger severity is {TRIGGER.SEVERITY}. This means that
it will be substituted by a string description and not a numerical value. So, let's define
a simple dictionary with severity labels and RT ticket priority values mapped,
as explained earlier in the chapter:
priorities = {
'Not classified': 0,
'Information': 20,
'Warning': 40,
'Average': 60,
'High': 80,
'Disaster': 100 }
[ 323 ]
Integrating Zabbix
We also need to know in advance the name of the queue we are creating the ticket in,
or better yet, its ID number:
queue_id = 3
Now that we have everything we need, we can proceed and build the request to
create a new ticket and then send it over to Request Tracker.
ticket_content = {
'content': {
'Queue': queue_id,
'Subject': trigger_name,
'Text': desc,
'Priority': priorities[severity],
'CF.{Hosts}': ','.join(hosts),
'CF.{Items}': ','.join(items),
'CF.{Trigger}': trigger_name
}
}
links = {
'content': {
'RefersTo': event_url
}
}
First, we create two dictionaries, one for the main ticket content, and the second for
the links section, which must be edited separately.
[ 324 ]
Chapter 10
Then we get to the main part of the script: first we log in to the RT API (be sure to
use your actual username and password credentials!), then we create a new ticket,
and then we get the ticket ID and input the link to the Zabbix event page:
rt = RTResource(rt_api, 'root', 'password', CookieAuthenticator)
ticket = rt.post(path='ticket/new', payload=ticket_content,)
(label,ticket_id) = ticket.parsed[0][0]
refers = rt.post(path=ticket_id + '/links', payload=links,)
We are almost done. All that is left to do is to acknowledge the Zabbix event with a
link back to the ticket we just created:
event_id = re.findall(r'eventid=(\d+)', event_url)[0]
ticket_url = rt_url + 'Ticket/Display.html?id=' + ticket_id.split('/')
[1]
print(ticket_url)
zh = ZabbixAPI('http://localhost/zabbix')
zh.login(user='Admin', password='zabbix')
ack_message = 'Ticket created.\n' + ticket_url
zh.event.acknowledge(eventids=event_id, message=ack_message)
This preceding code is fairly straightforward. After extracting the eventid value
and creating the URL to the ticket, we connect to the Zabbix API and edit the
acknowledge field of the event, effectively closing the circle.
Now that the script is complete, remember to give ownership to the zabbix user
and set the executable bit on it:
$ chown zabbix rt_mkticket.py
$ chmod +x rt_mkticket.py
[ 325 ]
Integrating Zabbix
The next time the action condition you defined in your system returns true, and the
action operation is carried out, the script will be executed with the parameters we've
seen before. A ticket will be created with a link back to the event, and the event itself
will be acknowledged with a link to the ticket.
Here is an example event. The link in the acknowledgement field corresponds to the
URL of the ticket:
And here is the corresponding ticket. The Refers to: field contains a clickable link
to the event shown earlier, while the Custom Fields section reports host, item,
and trigger information, just as expected:
[ 326 ]
Chapter 10
The script, in much the same way as those explained in Chapter 9, Extending Zabbix,
is little more than a proof of concept, with as much focus on readability and ease of
explanation as on pure functionality. Be sure to add as many condition checks and
error-reporting functions as possible if you want to use it in a production environment.
[ 327 ]
Integrating Zabbix
Summary
We have finally reached the end of our journey to mastering the Zabbix monitoring
system. In the course of the book, you have learned how to plan and implement the
general monitoring architecture; how to create flexible and effective items, triggers,
and actions; and how to best visualize your data. You have also learned how to
implement custom agents by understanding the Zabbix protocol and how to write
the code to manipulate every aspect of Zabbix through its API. In this chapter,
we have barely scratched the surface of what's possible once you start taking
advantage of what you now know about Zabbix to integrate it better with your IT
infrastructure. Many more integration possibilities exist, from getting and updating
users and groups through an identity management system to getting inventory
information through an asset management system, or vice versa feeding inventory
information to a CMDB database, and much more. Our hope is that by following
the steps necessary to integrate Zabbix with a trouble ticket management system,
you have learned how to prepare two systems such that they can share and exchange
data and how to use each system's API in a coordinated manner in order to get the
systems to talk to each other.
Our hope is that with the skills you just learned, you will be able to bring out the
full potential of the Zabbix monitoring system and make it a central asset of your
IT infrastructure. In doing so, your time and effort will be repaid tenfold.
[ 328 ]
Index
Symbols
{EVENT.DATE} macros 224
{EVENT.TIME} macros 224
{HOST.CONN} macros 236
{HOST.DNS} macros 236
{HOST.HOST} macros 236
{HOST.IP} macros 236
{HOST.NAME} macros 236
{INVENTORY.SERIALNO.A} macros 224
{NODE.ID} macros 224
{NODE.NAME} macros 224
A
action
about 222
action conditions 222
action definition 222
action operations 223
defining 223, 224
action conditions
defining 225, 226
action definition
{EVENT.DATE} macros 224
{EVENT.TIME} macros 224
friends macros 224
{INVENTORY.SERIALNO.A} macros 224
{NODE.ID} macros 224
{NODE.NAME} macros 224
action operations
aspects 226
media step 228, 229
message step 228, 229
remote commands 230
selecting 226
steps 227, 228
active property 296
active Zabbix proxy 63-65
aggregated items 164-166
architecture, RT 309
architecture, Zabbix
for large environment 12, 13
RDBMS server 12
types 13
web server 12
Zabbix server 12
architecture, Zabbix proxy 56, 57
Auto-login flag 204
avg() function 165, 197
C
calculated items 164, 167
CentOS 22
check_ora_sendtrap
wrapper script, coding for 266, 267
clock property
using 272
configuration component 33
configuration, Corosync 89
configuration files, UnixODBC 133, 134
configuration, HTTPD 88-95
configuration, IT services 206, 208
configuration, JMX 141-143
configuration, Pacemaker
testing 110, 111
configuration, PostgreSQL 109
configuration, web interface 42-47
configuration, Zabbix Server
performing 20, 21
for high availability 96, 97
configure command 134
Corosync
configuring 89
count function 165
curve fitting process 53
custom graphs
about 171
analyzing 174-178
exploded graphs 177
normal graphs 177
pie graphs 177
stacked graphs 177
custom media script 229
D
data
exporting 298
graphs, creating from 302
sending to Zabbix server, new script
used 265, 266
sending, zabbix_sender utility used 264
visualizing, through maps 182-185
visualizing, through slide show 201, 202
database
considerations 30, 32
high availability, implementing for 98, 99
installing 21-23
sizing 32-35
database drivers
installing 128
database management systems (DBMS) 127
database monitoring
ODBC 127
with Zabbix 126
database monitor items 135, 136
data flow
for Zabbix item 122-124
monitoring, with Zabbix node 67
monitoring, with Zabbix proxy 63-65
overview 125
Data Source Name (DSN) 127
date function 215
DBforBIX
URL 281
dedicated script server
pros and cons 267
delete_history() function 35
delete partition command 36
discovery rules, Zabbix
filesystems types 248
network interfaces 248
SNMP OIDs 248
Distributed Replicated Block Device. See DRBD
downtime
scheduled downtime 82
unscheduled downtime 82
DRBD
about 86
configuration, enabling 107
creating, on LVM partition 103, 104
features 86, 87
filesystem, creating on 105, 106
filesystem, replicating with 86, 87
integrating, with Pacemaker 107
networking consideration 115-117
online verification, enabling 114
optimizations 112
performance 112
primary device, defining in 105
resource, enabling in 104, 105
setting, over LVM 102
synchronization 112, 113
dynamic elements, screens
about 199, 200
graphs 199
plain text 199
simple graphs 199
E
elements, maps
about 191-193
selecting 194, 195
entities
adding, to templates 234, 235
Estimate Time to Repair (ETR) 112
[ 330 ]
event component 33
execquery() function 266
external checks
about 254
consideration 260
example 255-258
external script, location 255
external script, working 258, 259
script writing rules 260
external script
location 255
working 258, 259
F
failover
automating, with resource manager 86
filesystem
creating, on DRBD 105, 106
replicating, with DRBD 86, 87
types 249
friends macros 224
function
selecting 213
G
get command 145
gethost function 267
get method 298
getquery function 267
graph.gv file 303
graphs
3D View parameter 180
about 170
Colour parameter 181
creating, from data 302
custom graphs 171-178
Draw style parameter 181
Function parameter 181
Graph type parameter 179
Height parameter 179
Name parameter 179
Percentile line (left/right) parameter 180
properties 179-181
Show legend parameter 179
Show triggers parameter 180
Show working time parameter 180
simple graphs 171-173
Sort order parameter 180
Type parameter 180
Width parameter 179
Y axis MIN/MAX value parameter 180
Y axis side parameter 181
Graphviz
about 302, 303
URL 302
grpavg function 165
grpmax function 165
grpmin function 165
grpsum function 165
H
high availability
about 82
considerations 84, 85
failover, automating with
resource manager 86
implementing, for database 98, 99
implementing, on web server 87
Zabbix Server, configuring 96, 97
history component 33
host entities
vs, template entities 235
hosts
discovering 244-248
redistributing, to proxy 294, 295
templates, linking to 242
housekeeping
about 35
considerations 35-41
HTTPD
configuring 88-95
I
INSERT statement 38
installation, database 21-23
installation, database drivers 128
installation, package 19
installation, PostgreSQL 9.2 22, 23
[ 331 ]
installation, PyZabbix 288
installation, web interface 41
installation, Zabbix
performing 14, 15
prerequisites 16
installation, Zabbix Java gateway 139-141
iODBC 127
items
selecting 213
IT service level
understanding 83
IT services
about 205
configuring 206, 208
J
Java
Zabbix sender protocol,
implementing in 276, 277
JMX
configuring 141-143
considerations 144
issues 144
monitoring 137
security aspects 138, 139
Zabbix Java gateway, installing 139-141
JMX keys 143
L
large environment 10, 11
last() function 165, 197
ldd command 129
logical operation 217-220
Logical Volume (LV) 101
logical volume manager. See LVM
low-level discovery rules 248-251
LVM
about 100, 101
configuring 108
DRBD, setting over 102
LVM2
about 100
basic concepts 100
LVM partition
DRBD, creating on 103, 104
M
macros
about 188-190
URL, documentation 224
user defined macros 241
using 236-241
working with 195-197
Management Information Base (MIB) 145
maps
about 170
data, visualizing through 182-185
elements 191-193
elements, selecting 194, 195
macros 188-190
macros, working with 195-197
URLs 188-190
Zabbix Map, creating 186-188
max() function 165, 197
Maximum Transmission Unit (MTU) 115
MBean 143
Mean Time To Recovery (MTTR) 83
min() function 165, 197
monitored by property 294
MySQL ODBC drivers 128, 129
N
nesting templates 242, 243
network configuration 109
network interfaces 249
new script
used, for sending data to
Zabbix server 265, 266
node.csv file 292
nodes
templates, synchronizing across 291-293
O
observer effect 48
ODBC
about 127
database drivers, installing 128
database monitor items 135, 136
iODBC 127
MySQL drivers 128, 129
[ 332 ]
Oracle drivers 131, 132
PostgreSQL drivers 130, 131
SQL query, considerations 136
UnixODBC 127
UnixODBC configuration files 133
Zabbix, compiling 134
OID 145
online verification
enabling 114
Oracle ODBC drivers 131, 132
P
Pacemaker
about 90
configuration, testing 110, 111
configuring 110
DRBD configuration, enabling 107
DRBD, integrating with 107
features 86
HTTPD, configuring 92-95
LVM configuration 108
network configuration 109
PostgreSQL, configuring 109
Quorum 91
stickiness concepts 92
URL, for downloading 86
package
creating 19
installing 19
partitioning
about 36
benefits 36
passive Zabbix proxy 63-65
period property 296
perl trap handler 153-158
Physical Volume (PV) 100
planned downtime. See scheduled downtime
point-in-time recovery (PITR) 99
PostgreSQL
clustering 100
configuring 109
PostgreSQL 9.2
installing 22, 23
URL 22
PostgreSQL ODBC drivers
about 130, 131
URL 130
primary device
defining, in DRBD 105
proxy
hosts, redistributing to 294, 295
proxy.update method 295
Python
Zabbix sender protocol,
implementing in 278, 279
PyZabbix
installing 288
URL 288
using 288-291
Q
queues, RT
about 310
creating 310, 311
Quorum 91
R
Recovery Time Objective (RTO) 112
Red Hat 22
remote commands 230
Request Tracker. See RT
resource
enabling, in DRBD 104, 105
resource manager
failover, automating with 86
RT
about 308, 309
architecture 309
queues, creating 310, 311
setting up, to integrate with Zabbix 309
RT API
connecting to 315-318
S
scheduled downtime
vs, unscheduled downtime 82
screens
creating 198, 199
[ 333 ]
dynamic elements 199, 200
elements, adding 198
visualizing through 197
scrips 308
script writing rules 260
Secure Shell. See SSH
security, Zabbix
network isolation 76
tunnels 76
VPN 79
without network configuration 74, 75
send() function 266
service level agreement (SLA) 83
session riding 188
set command 145
severity 216
severity property 296
shell command 261
Shoot The Other Node In The Head. See STONITH
Siege
URL 51
simple graphs
about 171
analyzing 171-173
Simple Network Monitoring Protocol. See SNMP
slide show
customization 204, 205
on big display 202
on big display, considerations 203
visualizing through 201, 202
Snapshot Logical Volume (SLV) 101
SNMP 145, 253
SNMP monitoring
about 145-148
SNMP queries 148-151
SNMP traps 151, 152
SNMP OIDs 249
SNMP queries 148-151
snmptrapd process 152, 153
SNMP traps
about 151, 152
perl trap handler 153-158
snmptrapd process 152, 153
snmpwalk command 151
SourceForge
URL 267
SQL query
considerations 136
SSH 76, 77
start() function 26
steps, action operations 227, 228
stickiness 92
STONITH 90
stop() function 27
stunnel 78, 79
sudo checkinstall command 19
sum function 165
switchover. See failover
synchronization, DRBD 112, 113
T
tabular data
extracting 299-301
template entities
vs, host entities 235
Template OS Linux 174
templates
about 234, 235
combining 244
creating 234
entities, adding to 234, 235
linking, to hosts 242
low-level discovery rules 248-251
nesting 242, 243
synchronizing, across nodes 291-293
tickets, RT
about 310
creating, from Zabbix 322-327
custom fields, customizing 313-315
links section, customizing 311
ticket priority, customizing 312
time function 215
trends component 33
trigger dependencies
managing 221
trigger dependency graph
creating 303-306
[ 334 ]
trigger expression
about 212
components 212
function, selecting 213
items, selecting 213
logical operation 217-220
severity 216
trigger function
absolute value, selecting 216
#num argument, selecting 214
percentage, selecting 216
sec argument, selecting 214
tunnels
about 76
SSH 76, 77
stunnel 78
U
unexpected downtime. See unscheduled
downtime
UnixODBC
about 127
configuration files 133
URL 128
unscheduled downtime
vs, scheduled downtime 82
URLs 188-190
user defined macros 241
user parameter
about 261
considerations 263
flexibility 261, 262
users.csv file 295
users, Zabbix API
adding 295-298
updating 295-298
V
Values per seconds (vps) 62
verbose mode 133
Volume Group (VG) 101
VPN 79
W
web interface
configuration 42-47
installing 41
web pages
authenticating 160-162
logging out 162, 163
monitoring 158, 159
web server
high availability, implementing on 87
wrapper script
coding, for check_ora_sendtrap 266, 267
Y
yum install command 302
Z
Zabbix
about 11, 55
action 222
architecture, for large environment 12, 13
architecture, types 13
baseline, defining 50
capacity planning 48
compiling, with ODBC 134
database monitoring 48, 49, 126
data sending, Zabbix sender
protocol used 276
data, visualizing through maps 182-185
data, visualizing through
slide show 201, 202
external checks 254
features 11
graphs 171
historical data, storing 53
installing 14, 15
installing, prerequisites 16
IT services 205
load testing 51, 52
macros, using 236-241
observer effect 48
performance tuning 48
RT, setting up 309
[ 335 ]
screens, visualizing through 197
security concerns 73, 74
setting up, to integrate with RT 318-321
templates, creating 234
trigger dependencies, managing 221
trigger expression 212
URL, for documentation 285
user parameter 261
Zabbix agent protocol 268
about 273-275
developing, considerations 280
Zabbix agents
about 13
setting up 18
Zabbix API
about 284, 285
array datatype 290
authenticating 286-288
bool datatype 289
data, exporting 298
datatypes 289, 291
flag datatype 290
graphs, creating from data 302
hosts, redistributing to proxy 294, 295
integer datatype 290
object datatype 291
operations, performing 294
PyZabbix library, using 288-291
query datatype 291
string datatype 290
tabular data, extracting 299, 301
timestamp datatype 290
trigger dependency graph, creating 303-306
URL 284
users, adding 295-297
users, updating 295-298
Zabbix architecture
database, considerations 30-32
database, installing 21-23
database, sizing 32-35
housekeeping, considerations 35-41
web interface, installing 41
Zabbix installation 14, 15
Zabbix server, setting up 17, 18
Zabbix server, starting up 24-30
Zabbix forum
URL 256
zabbix_get command 17
Zabbix get protocol 268 269
Zabbix item
aggregated items 164-166
calculated items 164, 167
data flow 122-124
gathering, as raw data 120, 121
types 125, 126
Zabbix trapper item 125
Zabbix Java gateway
about 137
installing 139-141
Zabbix Map
creating 186-188
Zabbix node
about 66
data flow, monitoring 67
deploying 67-71
disadvantages 72
vs, Zabbix proxy 72, 73
Zabbix protocol
about 268, 269
possible responses 275
Zabbix agent 268-275
Zabbix get 268, 269
Zabbix sender 268-271
Zabbix proxy
about 13, 56, 57
active proxy 63-65
architecture 56, 57
data flow, monitoring with 63-65
deploying 57-62
passive proxy 63-65
vs, Zabbix node 72, 73
zabbix_send command 17
Zabbix sender protocol
about 268-271
clock property, using 272
implementing, in Java 276, 277
implementing, in Python 278, 279
undocumented feature 271, 272
used, for sending data to Zabbix 276
[ 336 ]
zabbix_sender utility
used, for sending data 264
Zabbix server
about 13
configuring 20, 21
configuring, for high availability 96, 97
data sending, new script used 265, 266
setting up 17, 18
starting up 24-30
Zabbix trapper item 125
Zabbix Version 1.8 284
Zabbix Version 2.0
about 255
enhancements 255
[ 337 ]
Thank you for buying
Mastering Zabbix
About Packt Publishing
Packt, pronounced 'packed', published its first book "Mastering phpMyAdmin for Effective
MySQL Management" in April 2004 and subsequently continued to specialize in publishing
highly focused books on specific technologies and solutions.
Our books and publications share the experiences of your fellow IT professionals in adapting
and customizing today's systems, applications, and frameworks. Our solution based books
give you the knowledge and power to customize the software and technologies you're using
to get the job done. Packt books are more specific and less general than the IT books you have
seen in the past. Our unique business model allows us to bring you more focused information,
giving you more of what you need to know, and less of what you don't.
Packt is a modern, yet unique publishing company, which focuses on producing quality,
cutting-edge books for communities of developers, administrators, and newbies alike. For
more information, please visit our website: www.packtpub.com.
About Packt Open Source
In 2010, Packt launched two new brands, Packt Open Source and Packt Enterprise, in order to
continue its focus on specialization. This book is part of the Packt Open Source brand, home
to books published on software built around Open Source licences, and offering information
to anybody from advanced developers to budding web designers. The Open Source brand
also runs Packt's Open Source Royalty Scheme, by which Packt gives a royalty to each Open
Source project about whose software a book is sold.
Writing for Packt
We welcome all inquiries from people who are interested in authoring. Book proposals
should be sent to [email protected] If your book idea is still at an early stage and you
would like to discuss it first before writing a formal book proposal, contact us; one of our
commissioning editors will get in touch with you.
We're not just looking for published authors; if you have strong technical skills but no writing
experience, our experienced editors can help you develop a writing career, or simply get some
additional reward for your expertise.
Zabbix 1.8 Network Monitoring
ISBN: 978-1-847197-68-9
Paperback: 428 pages
Monitor your network's hardware, servers, and web
performance effectively and efficiently
1.
Start with the very basics of Zabbix, an
enterprise-class open source network
monitoring solution, and move up to more
advanced tasks later
2.
Efficiently manage your hosts, users, and
permissions
3.
Get alerts and react to changes in monitored
parameters by sending out e-mails, SMSs, or
even execute commands on remote machines
Zenoss Core 3.x Network and
System Monitoring
ISBN: 978-1-849511-58-2
Paperback: 312 pages
A step-by-step guide to configuring, using and
adapting this free Open Source network monitoring
system
1.
Designed to quickly acquaint you with the core
feature so you can customize Zenoss Core to
your needs
2.
Discover, manage, and monitor IT resources
3.
Build custom event-processing and alerting
rules
4.
Write custom device reports to extract, display,
and analyze monitoring data
Please check www.PacktPub.com for information on our titles
Oracle VM Manager 2.1.2
ISBN: 978-1-847197-12-2
Paperback: 244 pages
Manage a flexible and elastic data center with Oracle
VM Manager
1.
Learn quickly to install Oracle VM Manager
and Oracle VM Servers
2.
Learn to manage your Virtual Data Center
using Oracle VM Manager
3.
Import VMs from the Web, template,
repositories, and other VM formats such as
VMware
4.
Learn powerful Xen Hypervisor utilities such
as xm, xentop, and virsh
Tcl 8.5 Network Programming
ISBN: 978-1-849510-96-7
Paperback: 588 pages
Build network-aware applications using Tcl, a
powerful dynamic programming language
1.
Develop network-aware applications with Tcl
2.
Implement the most important network
protocols in Tcl
3.
Packed with hands-on-examples, case studies,
and clear explanations for better understanding
Please check www.PacktPub.com for information on our titles
Was this manual useful for you? yes no
Thank you for your participation!

* Your assessment is very important for improving the work of artificial intelligence, which forms the content of this project

Download PDF

advertisement