IRMA Documentation
IRMA Documentation
Release 1.5.3
Quarkslab
Sep 02, 2017
Contents
1
Introduction
1.1 Purpose . . . . . . . . .
1.2 File Analysis Process . .
1.3 Infrastructure Overview
1.4 Hardware requirements
1.5 Supported Analyzers . .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
3
3
3
4
5
6
Automated Install
2.1 Requirements . . . . . .
2.2 Ansible scripts . . . . .
2.3 Predefined Environments
2.4 Using Debian repos . .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
9
9
9
9
12
3
Manual Installation
3.1 Brain . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3.2 Frontend . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3.3 Probe . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
13
13
22
29
4
Database migration
4.1 Requirements .
4.2 Content . . . .
4.3 Usage . . . . .
4.4 Tips and tricks
2
5
6
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
39
39
39
40
42
References
5.1 Disclaimer . . . . . . . . .
5.2 License . . . . . . . . . . .
5.3 Apache License, version 2.0
5.4 Authors . . . . . . . . . . .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
45
45
45
45
45
Frequently Asked Questions
6.1 Playing with tags . . . . . . . . . . .
6.2 SSL settings . . . . . . . . . . . . .
6.3 How to debug . . . . . . . . . . . . .
6.4 How to migrate . . . . . . . . . . . .
6.5 API documentation . . . . . . . . . .
6.6 Connect to a vagrant box through ssh
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
47
47
49
51
54
55
57
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
i
6.7
6.8
Enable SSL using OpenSSL in ansible scripts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Speed up your Vagrant VMs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
57
58
7
Resources
59
8
Screenshots
8.1 Command Line Interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
8.2 Web Interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
61
61
62
ii
IRMA Documentation, Release 1.5.3
IRMA is an asynchronous & customizable analysis system for suspicious files.
This guide will explain how to set up IRMA, use it and customize it at will.
Contents
1
IRMA Documentation, Release 1.5.3
2
Contents
CHAPTER
1
Introduction
This is an introductory chapter to IRMA. It recalls IRMA’s overall architecture, hardware required to run it and the
recommended order for installing the IRMA’s components.
Purpose
IRMA intends to be an open-source platform designed to help identifying and analyzing malicious files.
However, today’s defense is not only about learning about a file, but it is also getting a fine overview of the incident
you dealt with: where / when a malicious file has been seen, who submitted a hash, where a hash has been noticed,
which anti-virus detects it, ...
An important value with IRMA comes from you keep control over where goes and who gets your data. Once you
install IRMA on your network, your data stays on your network.
Each submitted files is analyzed in various ways. For now, we focus our efforts on multiple anti-virus engines, but we
are working on other “probes” (feel free to submit your own).
File Analysis Process
1. An analysis begins when a user uploads files to the Frontend.
2. Frontend checks for existing files and results in SQL. If needed, it stores the new files and calls asynchronously
scan jobs on Brain.
3. Brain worker sends as much subtasks to Probe(s) as needed.
4. Probe workers process their jobs and send back results to Brain.
5. Brain sends results to Frontend.
3
IRMA Documentation, Release 1.5.3
Infrastructure Overview
A drawing is better than a lot of explanations (sometimes ;)
4
Chapter 1. Introduction
IRMA Documentation, Release 1.5.3
Hardware requirements
IRMA platform is divided in three major components: the Frontend, the Brain and one or multiple Probes.
These three components can be installed on a unique host or on multiple hosts, according to the kind of probes you
are using.
The Frontend and the Brain must be installed on a GNU/Linux system1 . We recommend to use a Debian Stable
distribution which is supported and known to work.
According to the kind of probes and their dependencies, each analyzers can be installed on a separate hosts or share
1 Theorically, it should be possible, with some efforts, to make IRMA work on Microsoft Windows systems as most of the components used for
the platform are known to work or to have equivalents on these systems.
1.4. Hardware requirements
5
IRMA Documentation, Release 1.5.3
the same host as far as they do not interfere with each other2 . So forth, only Debian Stable and Microsoft Windows 8
and 10 hosts have been tested.
We can not give you any specific numbers. On one hand we managed to run the whole IRMA platform on a single
machine by hosting it with multiple systems inside virtual machines: this setup gives fairly high throughput as long
as it has reasonable IO (ideally, SSDs), and a good amount of memory (our setup was an i7 cpu with 16 GB ram
on regular drives (at least 200 GB required), on the other hand, a lighter version of the system with the three parts
together3 was successfully installed on a single virtual machine (1 GB of Ram and 4 virtual processors).
For a large company, in theory, given a single high-memory machine, with 16+ cores, and SSDs, you could run IRMA
platform and bear the workload load with reasonable response time.
Supported Analyzers
We have mainly focused our efforts on multiple anti-virus engines but we are working on other kind of “probes”. We
enumerate the analyzers that are bundled with IRMA probe application. Feel free to submit your own probes.
Antiviruses
So far, we have instrumented the following antiviruses from their CLI:
Probe Name
ASquaredCmd
Avira
AvastCoreSecurity
AVGAntiVirusFree
BitdefenderForUnices
ClamAV
ComodoCAVL
DrWeb
EsetNod32
EScan
FProt
FSecure
GData
Kaspersky
McAfeeVSCL
Sophos
Symantec
VirusBlokAda
Zoner
Anti-Virus Name
Emsisoft Command Line
Avira
Avast
AVG
Bitdefender
ClamAV
Comodo Antivirus for Linux
Dr.Web
Eset Nod32 Business Edition
eScan
F-Prot
F-Secure
G Data Antivirus
Kaspersky Internet Security
McAfee VirusScan Command Line
Sophos
Symantec Endpoint Protection
VirusBlokAda
Zoner Antivirus
Platform
Microsoft Windows CLI
Microsoft Windows CLI
GNU/Linux CLI
GNU/Linux CLI
GNU/Linux CLI
GNU/Linux CLI
GNU/Linux CLI
GNU/Linux CLI
GNU/Linux CLI
GNU/Linux CLI
GNU/Linux CLI
GNU/Linux CLI
Microsoft Windows CLI
Microsoft Windows CLI
GNU/Linux - Microsoft Windows CLI
GNU/Linux - Microsoft Windows CLI
Microsoft Windows CLI
GNU/Linux CLI
GNU/Linux CLI
External analysis platforms
So far, we query the following external analysis platforms:
Probe Name
VirusTotal
Analysis Platform
VirusTotal
Description
Report is searched using the sha256 of the file which is not sent
2 For instance, we managed to host several GNU/Linux anti-viruses on an unique probe by preventing it to launch daemons at startup. This is
difficult for Microsoft systems on which it is not recommended to install multiple anti-viruses on a single host.
3 with a limited set of probes
6
Chapter 1. Introduction
IRMA Documentation, Release 1.5.3
File database
So far, we query the following file databases:
Probe
Name
NSRL
Database
Description
National Software Reference
Library
collection of digital signatures of known, traceable software
applications
Metadata
So far, we implemented the following analyzers:
Probe Name
StaticAnalyzer
PEiD
Yara
Description
PE File analyzer adapted from Cuckoo Sandbox
PE File packer analyzer
Checks if a file match yara rules
1.5. Supported Analyzers
7
IRMA Documentation, Release 1.5.3
8
Chapter 1. Introduction
CHAPTER
2
Automated Install
IRMA platform can be easily installed with a set of ansible roles and playbooks. It will help you to build, install or
maintain different setups.
Requirements
• Ansible 2.2.1.0;
Ansible scripts
Get IRMA ansible scripts on github:
$ git clone https://github.com/quarkslab/irma
$ cd irma/ansible
Predefined Environments
There are 2 different IRMA setups available. Dev/Testing will be installed in one or multiple virtual machines while
production could be used to install IRMA on physical machines or virtual machines already setup:
Development environment
This environment has been designed to help you to modify IRMA’s components and redeploy and test them. In this
setup, everything is installed in a single virtual machine with sources rsync-ed between the host and the guest.
9
IRMA Documentation, Release 1.5.3
Requirements
• Vagrant 1.8 or higher has to be installed
• As the installation work only for Virtualbox, you will need to install it
• Rsync to synchronize directories from host to VMs
• Read the Ansible introduction
Run Vagrant and create your VMs
To initialize and provision the Virtualbox VM.
$ cd <IRMA_SRC_DIR>/ansible
$ VM_ENV=your_environment_name vagrant up
The template will be downloaded automatically and configured using environments/dev.yml file.
Note: Optionally, if you want to use your own environment, create it in environments directory and run:
$ VM_ENV=your_environment_name vagrant up
Configure your .ini files
Note: You can bypass this step, as this provisioning is sync with default username and password used in (frontend|brain|probe) config files.
As your config/*.ini file are transferred from host to VMs, you will need to modify them individually in the
frontend, probe and brain directories to match the user and password defined in playbooks/group_vars/
*.
Modify your host and open IRMA frontend
Then, for proper use, update your /etc/hosts file and add:
172.16.1.30
www.frontend.irma
Then, with your web browser, IRMA allinone is available at www.frontend.irma.
Sync files between host and guest
Once rsync is installed inside your virtual machine and your environment is correctly set. You could easily sync your
code with:
$ vagrant rsync # or vagrant rsync-auto to automatically initiates an rsync
# transfer when changes are detected
Then, reload the modified application.
10
Chapter 2. Automated Install
IRMA Documentation, Release 1.5.3
Production environment
This environment is used to install IRMA on production-ready Debian servers.
Requirements
• One or multiple 64-bit Debian 7 servers.
Preparing servers
Create an account that is going to be used to provision IRMA on the server via Ansible, or use one which has already
been created. To speed up provisioning, you can:
• Authorize your SSH key for password-less authentication (optional):
# On your local machine
$ ssh-copy-id user@hostname # -i if you want to select your identity file
• If you do not want to have to type your password for sudo command, consider adding your user to sudoers,
using visudo command (optional):
user ALL=(ALL) NOPASSWD: ALL
Configure for your installation
Modify settings in playbooks/group_vars/all especially the default_ssh_keys: section. You will need
to add your public keys for SSH password-less connection to the default irma server user.
Configuration file used by the brain, the frontend and the probes applications are generated with default values that are
specified in playbooks/group_vars/brain, playbooks/group_vars/frontend and playbooks/
group_vars/probe respectively. Make sure to adapt xxx_deployment_configs variables accordingly to
your installation. It is recommended to change all the default passwords defined in group_vars/* configuration
files (password variables for most of them).
Finally, you will need to customize the hosts/example file and adapt it to describe your own server infrastructure. There is three sections, one for each server role (frontend, brain, probe). Please refer to Ansible Inventory
documentation for the expected syntax.
Run the Ansible Playbook
To run the main playbook with the hosts/example file you have defined, use the following command. Ansible
will ask you the sudo password (-K option).
$ cd <IRMA_SRC_DIR>/ansible
$ ansible-playbook -i ./hosts/example playbooks/playbook.yml -u <your_sudo_username> ˓→K
To run one or more specific actions and avoid running all the playbook, you can use tags. For example, if you want to
re-provision Nginx, run the same command, but append --tags=nginx. You can combine multiple tags separated
with commas.
2.3. Predefined Environments
11
IRMA Documentation, Release 1.5.3
Deploy a new version of IRMA
Assuming that you have already provisioned and deployed a version of IRMA, which you want to upgrade, you will
need to run the deployment script:
$ ansible-playbook -i ./hosts/example ./playbooks/deployment.yml -u irma
Make sure to replace irma with the default user if you have changed it in the group_vars/all file.
Access your IRMA installation
Once the provisioning and the deployment have finished, you should be able to perform scans and get their results via
the frontend hostname you specified.
Using Debian repos
If you planned to use only Debian official repository, you’ll need to change in playbooks/group_vars/all:
default_use_debian_repo: yes # no is the default value
12
Chapter 2. Automated Install
CHAPTER
3
Manual Installation
Warning: Manual installation needs an update since 1.5.3
as all repositories layout have changed. Try Automated install or ask for help on IRC.
In some specific cases where automatic install is not possible, you should go with manual installation.
Each major component of the IRMA platform comes with their own python-based application. As the Brain is the
nerve center of the whole platform, it is recommended to install it first before installing other components. One can
then install either the Frontend or the Probes he wants.
Please refer to specific documentation for the manual installation process for each component.
Brain
The Brain is a python-based application that only dispatches analysis requests from different frontends1 to the available Probes. Analyses are scheduled by the Brain on Probes through Celery, an open source task framework for
Python.
Architecture
Let us recall first the inner architecture of the Brain. It uses multiple technologies with a specific purpose each:
• a Celery worker that handles scan requests from Frontends and results returned by the Probes.
• a RabbitMQ server used by Celery as a backend and as a broker for task queues and job queues used to schedule
tasks for Probes (for scan jobs) and the Frontend (for scan results).
• an SFTP server where files to be scanned are uploaded by Frontends and downloaded by Probes,
1
This feature is not ready yet, we are currently working on its implementation.
13
IRMA Documentation, Release 1.5.3
Installing and Configuring RabbitMQ
The following explains how to install and to setup RabbitMQ for your setup.
Install RabbitMQ
RabbitMQ server can be installed with this command on Debian:
$ sudo apt-get install rabbitmq-server
Configuring RabbitMQ
RabbitMQ serves all components of IRMA platform. Each component has their own virtual host (i.e., message queue
where to get data), a username and a password.
To configure RabbitMQ for IRMA platform, you have to create users, vhosts and add permissions for each user to
corresponding vhost. To easily create virtual hosts and users, we provide scripts in extras/ directory:
$ sh ./extras/scripts/rabbitmq/rmq_adduser.sh
Usage: sudo rmq_adduser.sh <user> <password> <vhost>
For instance, to create 3 users with the following parameters, one can do:
Username
brain
probe
frontend
Password
brain-rmqpassword
probe-rmqpassword
frontendrmqpassword
Virtual Command
Host
mqbrain sudo ./extras/scripts/rabbitmq/rmq_adduser.sh brain
brain-rmq-password mqbrain
mqprobe sudo ./extras/scripts/rabbitmq/rmq_adduser.sh probe
probe-rmq-password mqprobe
mqfron- sudo ./extras/scripts/rabbitmq/rmq_adduser.sh
tend
frontend frontend-rmq-password mqfrontend
The script simply execute the following three commands:
$ sudo rabbitmqctl add_user <username> <password>
$ sudo rabbitmqctl add_vhost <vhostname>
$ sudo rabbitmqctl set_permissions -p <vhostname> <username> ".*" ".*" ".*"
Warning: Important
Make sure to note down the username, the password and the virtual host you just defined. You will be asked to
retype them on each application configuration file (brain, frontend and probe)
Note: Disclaimer
Please ensure that only trusted sources can communicate with your RabbitMQ server, by setting up firewall rules for
instance, as your RabbitMQ may be exposed to Internet.
14
Chapter 3. Manual Installation
IRMA Documentation, Release 1.5.3
Verifying RabbitMQ configuration
We can verify that the RabbitMQ server has taken into account our modifications with some commands:
Checking for vhosts
$ sudo rabbitmqctl list_vhosts
Listing vhosts ...
mqbrain
/
mqfrontend
mqprobe
mqadmin
...done.
If the defined virtual host are not listed by the above command, please execute once more the script.
Checking for users
$ sudo rabbitmqctl list_users
Listing users ...
probe
[]
brain
[]
frontend
[]
...done.
If the defined users are not listed by the above command, please execute once more the script.
Changing password
If you do not remember the password you just typed, you can change it with rabbitmqctl command:
$ sudo rabbitmqctl change_password brain brain-rmq-password
Changing password for user "brain" ...
...done.
Restarting the service
You may want to restart the service. Thus, the following command can be done:
$ sudo invoke-rc.d rabbitmq-server restart
Installing and configuring sftp
A SFTP server is used to transfer files Frontends and meant to be analyzed by Probes. We describe in the following
how to set it up.
3.1. Brain
15
IRMA Documentation, Release 1.5.3
Installing sshd
sshd should already been installed but if not install it with:
$ sudo apt-get install openssh-sftp-server
Creating FTP specific users and groups
$
$
$
$
sudo groupadd sftpusers
sudo useradd -g sftpusers -M -s /etc <frontend username>
sudo useradd -g sftpusers -Match -s /etc <probe username>
sudo passwd frontend
<enter frontend password>
$ sudo passwd probe
<enter probe password>
Configure sshd
Edit /etc/sshd_config
Subsystem sftp internal-sftp
Match User <frontend username>
AllowTcpForwarding no
ChrootDirectory /sftp/<frontend username>
ForceCommand internal-sftp -u 2
PermitTunnel no
X11Forwarding no
Match User <probe username>
AllowTcpForwarding no
ChrootDirectory /sftp
ForceCommand internal-sftp -u 2
PermitTunnel no
X11Forwarding no
Create SFTP directories
The frontends need an account with /sftp/<frontend-name>/uploads as home directory and a single account
is shared between probes ( uploads directory is needed as in chrooted environment home is not writeable). The later
needs to access to all frontends, thus the associated home directory is simply /sftp/. For instance, a frontend named
frontend, execute the following to create the directories:
$ sudo
mkdir:
mkdir:
mkdir:
$ sudo
$ sudo
$ sudo
$ sudo
$ sudo
16
mkdir -pv /sftp/frontend/uploads
created directory `/sftp'
created directory `/sftp/frontend'
created directory `/sftp/frontend/uploads'
chown -R root:sftpusers /sftp
chmod 0750 /sftp
chmod 0750 /sftp/frontend
chown -R frontend:sftpusers /sftp/frontend/uploads
chmod -R 0775 /sftp/frontend/uploads
Chapter 3. Manual Installation
IRMA Documentation, Release 1.5.3
Restart the service
You may want to restart the service:
$ sudo invoke-rc.d sshd restart
Installation
The Brain must be installed on a GNU/Linux distribution. With some efforts, it should be possible to run it on a
Microsoft Windows system, but this has not been tested yet.
This section describes how to get the source code of the application for the Brain and to install it.
Pre-requisites
We assume that you have a command line interface on your system with the following tools installed:
• python 2.7.x (see Python for Windows for prebuild MSI installer)
• python-pip (see Install pip for the recommended way to install pip package manager)
Furthermore, we assume that you have created an user irma that will be used to run the python application.
Note: On GNU/Linux, one can create the user (and the group) irma with:
$ sudo adduser --system --no-create-home --group irma
Installation on GNU/Linux
On GNU/Linux system, we will assume that the code for the Brain will be installed in /opt/irma/irma-brain/
current directory, which is the configured default installation directory.
The source code is hosted on github.com. One can fetch an up-to-date version with the following commands. Let us
note that there is a common submodule named common that is a soft-link (lib), do not forget the dereference
option of cp (-L):
# If src repository not already cloned
$ git clone https://github.com/quarkslab/irma /opt/irma/src
$ cp -rL /opt/irma/src/brain /opt/irma/irma-brain/current
We need few dependencies for future steps:
$ sudo apt-get install python-virtualenv python-dev
then, we need to install python dependencies manually in a virtualenv :
$ virtualenv --system-site-packages /opt/irma/irma-brain/current/venv
$ /opt/irma/irma-brain/current/venv/bin/pip install -r /opt/irma/irma-brain/current/
˓→requirements.txt
[...]
If everything has gone well, you should have installed the python application on your system. The next step is to install
the other components it relies on and finally to configure it for your platform.
3.1. Brain
17
IRMA Documentation, Release 1.5.3
Configuration
The configuration file is located at config/brain.ini in the installation directory. Update it with your specific
info.
Note: Detailed meaning of each field in config/brain.ini:
Section
log
broker_brain
broker_probe
broker_frontend
sqldb
ftp
ftp_brain
interprocess_
lock
ssl_config
18
Key
syslog
prefix
debug
sql_debug
host
port
vhost
username
password
queue
host
port
vhost
username
password
queue
host
port
vhost
username
password
queue
dbms
dialect
username
password
host
dbname
Type
integer
string
boolean
boolean
string
integer
string
string
string
string
string
integer
string
string
string
string
string
integer
string
string
string
string
string
string
string
string
string
string
tables_prefix
protocol
host
port
auth
key_path
username
password
path
string
string
string
integer
string
string
string
string
string
activate_ssl
ca_certs
keyfile
certfile
boolean
string
string
string
Default
Description
0
enable rsyslog (experimental)
irma-brain:
prefix to append to rsyslog entries
False | enable Debug log
False | enable SQL debug log
hostname for the RabbitMQ server
5672
port for the RabbitMQ server
virtual host configured for brain
username used for brain on the RabbitMQ server
password used for brain on the RabbitMQ server
queue to poll new tasks on the RabbitMQ server
hostname for the RabbitMQ server
5672
port for the RabbitMQ server
virtual host configured for probes
username used for probes on the RabbitMQ server
password used for probes on the RabbitMQ server
queue to poll new tasks on the RabbitMQ server
hostname for the RabbitMQ server
5672
port for the RabbitMQ server
virtual host configured for frontend
username used for frontend on the RabbitMQ server
password used for frontend on the RabbitMQ server
queue to poll new tasks on the RabbitMQ server
sqlite
dbapi engine
sqlalchemy dialect
database username
database password
database host
/var/irma/
database name
db/brain.db
database tables prefix
“sftp”
choose File Transfer Protocol (“sftp” or “ftps”)
hostname for the FTP server
21
port for the FTP server
“password”| SFTP authentication method (“password” or “key”)
sftp private key absolute path
username used by probe on the FTP server
password used by the probe on the FTP server
/var/run/
Concurrency file lock
lock/irmabrain.lock
False
Enable RabbitMQ ssl
RabbitMQ SSL certs
RabbitMQ SSL keyfile
RabbitMQ SSL certfile
Chapter 3. Manual Installation
IRMA Documentation, Release 1.5.3
Generate a SQLite database for scan tracking
You could easily generate the user database by running the following command. The path of the database is taken
from the configuration file and the folder where the database is going to be stored must be created beforehand.
Note: The default path for the database is /var/irma/db/ make sure it exists before creating user database.
$ cd /opt/irma/irma-brain/current/
$ ./venv/bin/python -m scripts.create_user
usage: create_user <username> <rmqvhost> <ftpuser>
with <username> a string
<rmqvhost> the rmqvhost used for the frontend
<ftpuser> the ftpuser used by the frontend
example: create_user test1 mqfrontend frontend
To create an entry in the database for the frontend named frontend and which uses the mqfrontend virtual host
on the RabbitMQ server, simply run the following commands:
$ ./venv/bin/python -m scripts.create_user frontend mqfrontend frontend
Note: There is a limitation due to SQLite. The folder where the database is stored, plus the database file must be
writable by the user running the worker:
$ sudo chown irma:irma /var/irma/db/brain.db
$ sudo chmod a+w /opt/irma/irma-brain
Installation Checks
Celery Workers
Before going further, you should check that the python applications manages to communicate with the RabbitMQ
server through Celery. To ensure that, from the installation directory, execute both Celery workers:
On GNU/Linux:
$ cd /opt/irma/irma-brain/current
$ ./venv/bin/celery worker --app=brain.scan_tasks
----------------- **** ------- * *** * --- * - **** --- ** ---------- ** ---------- ** ---------- ** ---------- *** --- * ---- ******* ----
3.1. Brain
celery@brain v3.1.23 (Cipater)
Linux-3.16.0-4-amd64-x86_64-with-debian-8.2
[config]
.> app:
.> transport:
.> results:
.> concurrency:
scantasks:0x7fbd7ee4c350
amqp://brain:**@127.0.0.1:5672/mqbrain
amqp://
2 (prefork)
19
IRMA Documentation, Release 1.5.3
--- ***** ----- [queues]
-------------- .> brain
exchange=celery(direct) key=brain
[2016-07-15 15:00:36,155: WARNING/MainProcess] celery@brain ready.
This worker is responsible for splitting the whole scan job in multiples job per probe per file.
$ cd /opt/irma/irma-brain/current
$ ./venv/bin/celery worker --app=brain.results_tasks
----------------- **** ------- * *** * --- * - **** --- ** ---------- ** ---------- ** ---------- ** ---------- *** --- * ---- ******* ------ ***** ------------------
celery@brain v3.1.23 (Cipater)
Linux-3.16.0-4-amd64-x86_64-with-debian-8.2
[config]
.> app:
.> transport:
.> results:
.> concurrency:
[queues]
.> results
resultstasks:0x7fa68f9aa590
amqp://probe:**@127.0.0.1:5672/mqprobe
disabled://
2 (prefork)
exchange=celery(direct) key=results
[2016-07-15 14:59:01,799: WARNING/MainProcess] celery@brain ready.
And this worker is responsible for collecting and tracking results.
If your Celery worker does not output something similar to the above output, you should check twice the parameters
in the application configuration file you are using.
SFTP accounts
Try to login as frontend and upload a sample file in home dir (should raise an error as it is non writeable) then in
uploads dir.
$ sftp frontend@localhost
frontend@localhost's password:
Connected to localhost.
sftp> put test
Uploading test to /test
remote open("/test"): Permission denied
sftp> ls
uploads
sftp> cd uploads/
sftp> put test
Uploading test to /uploads/test
test
˓→100%
10
0.0KB/s
00:00
Running Brain applications at startup
We have ensured that the freshly installed Brain is ready to be integrated to your IRMA platform. Now, we can go a
step further and make it launch automatically all daemons when the system starts up so you will not need to relaunch
them manually every time.
20
Chapter 3. Manual Installation
IRMA Documentation, Release 1.5.3
We are using supervisor to manage our celery daemons.
Installing Supervisor
Install it with apt:
$ sudo apt-get install python-virtualenv python-dev
$ sudo pip install supervisor
We will create two new applications called scan_app and result_app.
Configure Scan APP
Create a file called scan_app located at /etc/supervisor/conf.d with the following content:
[program:scan_app]
numprocs = 1
stopwaitsecs = 600
killasgroup = True
stderr_logfile = /var/log/supervisor/scan_app.log
command = /opt/irma/irma-brain/current/venv/bin/celery worker -A brain.scan_tasks -˓→hostname=scan_app.%%h --loglevel=INFO --without-gossip --without-mingle --without˓→heartbeat --soft-time-limit=60 --time-limit=300 -Ofair
user = irma
autostart = True
directory = /opt/irma/irma-brain/current
stdout_logfile = /var/log/supervisor/scan_app.log
Configure Result APP
Create a file called result_app located at /etc/supervisor/conf.d with the following content:
[program:result_app]
numprocs = 1
stopwaitsecs = 600
killasgroup = True
stderr_logfile = /var/log/supervisor/result_app.log
command = /opt/irma/irma-brain/current/venv/bin/celery worker -A brain.results_tasks ˓→-concurrency=1 --hostname=result_app.%%h --loglevel=INFO --without-gossip --without˓→mingle --without-heartbeat --soft-time-limit=60 --time-limit=300
user = irma
autostart = True
directory = /opt/irma/irma-brain/current
stdout_logfile = /var/log/supervisor/result_app.log
Ensure supervisor will read our files by checking /etc/supervisor/supervisord.conf last lines should be:
[...]
[include]
files = /etc/supervisor/conf.d/*
Restart supervisor:
3.1. Brain
21
IRMA Documentation, Release 1.5.3
$ sudo service supervisor restart
Restart applications if needed (should be done automatically):
$ sudo supervisorctl restart all
scan_app: stopped
result_app: stopped
scan_app: started
result_app: started
Frontend
The Frontend handles scan submission to the Brain, stores the results of the scanned files. These results can be
displayed through a web graphical user interface or via the command line interface.
Architecture
Let us recall first the inner architecture of the Frontend. It uses multiple technologies with each a specific purpose:
• A client through which an user submits a file and get the analysis results. There are two clients bundled in the
repository: a web user interface and a command-line client.
• A python-based restful API, served by a nginx web server and a uWSGI application server. It gets the results of
a file scan by querying a database.
• A worker that will handle scan submission to the Brain and store the results of analyzes scheduled by the Brain.
The worker relies on Celery, a python-based distributed task queue.
• An database server (PostgreSQL) is used to store results of analyzes made on each file submitted either by the
web graphical interface or the CLI client.
Installation
The Frontend must be installed on a GNU/Linux system. With some efforts, it should be possible to run it on a
Microsoft Windows system, but this has not been tested yet.
This section describes how to get the source code of the application and to install it.
Pre-requisites
We assume that you have a command line interface on your system with the following tools installed:
• python 2.7.x (see Python for Windows for prebuild MSI installer)
• python-pip (see Install pip for the recommended way to install pip package manager)
Furthermore, we assume that you have created an user irma that will be used to run the python application.
Note: On GNU/Linux, one can create the user (and the group) irma with:
$ sudo adduser --system --no-create-home --group irma
22
Chapter 3. Manual Installation
IRMA Documentation, Release 1.5.3
Installation on GNU/Linux
On GNU/Linux system, we will assume that the code for the Frontend will be installed in /opt/irma/
irma-frontend/current directory, which is the configured default installation directory.
The source code is hosted on github.com. One can fetch an up-to-date version with the following commands. Let us
note that there is a common submodule named common that is a soft-link (lib), do not forget the dereference
option of cp (-L):
$ git clone https://github.com/quarkslab/irma /opt/irma/src
$ cp -rL /opt/irma/src/frontend /opt/irma/irma-frontend/current
We need few dependencies for future steps:
$ sudo apt-get install python-virtualenv python-dev
then, we need to install python dependencies manually in a virtualenv :
$ virtualenv --system-site-packages /opt/irma/irma-frontend/current/venv
$ /opt/irma/irma-frontend/current/venv/bin/pip install -r /opt/irma/irma-frontend/
˓→current/requirements.txt
[...]
Building the web client
The first step that should be performed when installing the frontend from the code source is to build the web client
which is composed of static HTML files mixed with javascript web framework (in particular, AngularJS).
One must install first some tools that are necessary to build the whole web client:
$ curl -sL https://deb.nodesource.com/setup | sudo bash [...]
$ sudo apt-get install -y nodejs
[...]
$ curl -sL https://www.npmjs.org/install.sh | sudo bash [...]
Once the tools installed, you can build the static files for the web user interface with the following commands, executed
from the installation directory:
$
$
$
$
cd irma-frontend/current/web
sudo npm install
sudo node_modules/.bin/bower install --allow-root
sudo node_modules/.bin/gulp dist
A new directory or an updated directory web/dist should appear now with HTML and javascript files that have
been minified and “obfuscated” by gulp. For more details on the web interface, please refer to the dedicated chapter.
If everything has gone well, you should have installed the python application on your system. The next step is to
configure it for your platform and to install the other components it relies on.
Configuration
The configuration file is located at config/frontend.ini in the installation directory.
Note: Detailed meaning of each field in config/frontend.ini:
3.2. Frontend
23
IRMA Documentation, Release 1.5.3
Section
log
sqldb
samples_storage
celery_brain
celery_frontend
broker_brain
broker_frontend
ftp
ftp_brain
cron_frontend
interprocess_ lock
ssl_config
Key
syslog
prefix
debug
sql_debug
username
password
host
dbname
tables_prefix
path
timeout
timeout
host
port
vhost
username
password
queue
host
port
vhost
username
password
queue
protocol
host
port
auth
key_path
username
password
clean_db_file _max_age
clean_db_cron _hour
clean_db_cron _minute
clean_db_scan _day_of_week
path
activate_ssl
ca_certs
keyfile
certfile
Type
integer
string
boolean
boolean
string
string
string
string
string
string
integer
integer
string
integer
string
string
string
string
string
integer
string
string
string
string
string
string
integer
string
string
string
string
integer
integer
integer
integer
string
boolean
string
string
string
Default
0
irma-frontend:
False
False
60 (sec)
30 (sec)
5672
5672
“sftp”
22
“password”
0
0
0
*
/var/run/lock/ir ma-frontend.lock
False
Description
enable rsyslog (experimental
prefix to append to rsyslog en
enable Debug log
enable SQL debug log
database username
database password
database host
database name
database tables prefix
Samples storage path
time before considering that t
time before considering that t
hostname for the RabbitMQ s
port for the RabbitMQ server
virtual host configured for br
username used for brain on th
password used for brain on th
queue to poll new tasks on th
hostname for the RabbitMQ s
port for the RabbitMQ server
virtual host configured for thi
username used for this fronte
password used for this fronte
queue to poll new tasks on th
choose File Transfer Protoco
hostname for the FTP server
port for the FTP server
SFTP authentication method
sftp private key absolute path
username used by this fronten
password used by this fronten
remove file after X days (0 m
cron hour settings
cron minute settings
cron day of week settings
Concurrency file lock
Enable RabbitMQ ssl
RabbitMQ SSL certs
RabbitMQ SSL keyfile
RabbitMQ SSL certfile
Note: The default path for samples is /var/irma/samples/ make sure it exists with correct rights for irma user before
launching your first scan.
Installing and configuring uWSGI
The restful API is served by an uWSGI application server. This section will explain how to install an uWSGI server
and configure it for the Frontend.
24
Chapter 3. Manual Installation
IRMA Documentation, Release 1.5.3
Installation
On Debian, installing uWSGI to serve a python application is pretty straightforward:
$ sudo apt-get install uwsgi uwsgi-plugin-python
[...]
Please refer to the documentation of your preferred distribution to install an uWSGI server on it.
Installing SQL server
Note: Since version 1.5.0, IRMA required a postgreSQL server as we are using JSONB column.
The Frontend relies on a PostgreSQL database to keep track of all scans info. On Debian, one can install a PostgreSQL
server in few commands:
Install it with the following dependencies:
$ sudo apt-get install postgresql-9.4 python-psycopg2
[...]
You need to manually create the configured database and db user, the tables are automatically created on uwsgi startup.
Running Frontend applications at startup
We have ensured that the freshly installed Frontend is ready to be integrated to your IRMA platform. Now, we can go
a step further and make it launch automatically all daemons when the system starts up so you will not need to relaunch
them manually every time.
We are using supervisor to manage our python daemons (uswgi for IRMA API and celery for IRMA frontend workers).
Installing Supervisor
Install it with apt:
$ sudo apt-get install python-virtualenv python-dev
$ sudo pip install supervisor
We will create two new applications called frontend_api and frontend_app. Frontend_api is the python uwsgi server
and frontend_app the frontend celery daemon.
Configure Frontend API
Create a file called frontend_api located at /etc/supervisor/conf.d with the following content:
[program:frontend_api]
numprocs = 1
redirect_stderr = True
stopwaitsecs = 600
killasgroup = True
stderr_logfile = /var/log/supervisor/frontend_api.log
3.2. Frontend
25
IRMA Documentation, Release 1.5.3
stopsignal = INT
command = uwsgi -s 127.0.0.1:8081 --disable-logging --master --workers 4 --need-app -˓→plugins python --chdir /opt/irma/irma-frontend/current --home /opt/irma/irma˓→frontend/current/venv --python-path /opt/irma/irma-frontend/current/venv --mount /
˓→api=frontend/api/base.py --lazy
user = irma
autostart = True
stdout_logfile = /var/log/supervisor/frontend_api.log
Configure Frontend APP
Create a file called frontend_app located at /etc/supervisor/conf.d with the following content:
[program:frontend_app]
numprocs = 1
stopwaitsecs = 600
killasgroup = True
stderr_logfile = /var/log/supervisor/frontend_app.log
command = /opt/irma/irma-frontend/current/venv/bin/celery worker -A frontend.
˓→tasks:frontend_app --hostname=frontend_app.%%h --loglevel=INFO --without-gossip -˓→without-mingle --without-heartbeat --soft-time-limit=60 --time-limit=300 --beat -˓→schedule=/var/irma/frontend_beat_schedule
user = irma
autostart = True
directory = /opt/irma/irma-frontend/current
stdout_logfile = /var/log/supervisor/frontend_app.log
Ensure supervisor will read our files by checking /etc/supervisor/supervisord.conf last lines should be:
[...]
[include]
files = /etc/supervisor/conf.d/*
Restart supervisor:
$ sudo service supervisor restart
Restart applications if needed (should be done automatically):
$ sudo supervisorctl restart all
frontend_api: stopped
frontend_app: stopped
frontend_app: started
frontend_api: started
Installing and configuring nginx
In the Frontend, we use a nginx web server to serve the uWSGI application and the static web site that query the API
in order to get results of scanned files and to present them to the user.
26
Chapter 3. Manual Installation
IRMA Documentation, Release 1.5.3
Installation
On Debian, installing nginx is done with few commands:
$ sudo apt-get install nginx
[...]
Please refer to the documentation of your preferred distribution to install an uWSGI server on it.
Configuration
We provide several template scripts in the extras/ repository located at the root of the installation directory. Templates for nginx are located in extras/nginx/. Copy the file to sites-available in nginx configuration
folder:
$ sudo cp extras/nginx/frontend /etc/nginx/sites-available/
The template is configured for a default installation of the frontend in /opt/irma/irma-frontend/current.
You may need to edit it for you setup. In particular, please ensure that the root directive points to the folder web/
dist (for a production ready version of the web site) or web/app (for a development version of the web site) in
your installation directory.
Activate the web site
By default, all websites in apps-available are not activated. One can enable the website described in frontend
configuration file by creating a soft-link into the sites-enabled folder:
$ sudo ln -s /etc/nginx/sites-available/frontend /etc/nginx/sites-enabled/frontend
Note: The case of HTTPs
A template to set up a HTTPs server with nginx is also provided in the extras/nginx folder. Here is the way to
setup it:
$ sudo cp extras/nginx/frontend-https /etc/nginx/sites-available/
$ sudo ln -s /etc/nginx/sites-available/frontend-https /etc/nginx/sites-enabled/
˓→frontend-https
$ sudo mkdir /etc/nginx/certificates/
$ cd /etc/nginx/certificates/
Generate the required Diffie Hellman Ephemeral Parameters:
$ sudo openssl dhparam -out dhparam.pem 4096
Finally, generate a self signed certificate:
$ sudo openssl req -x509 -nodes -days 365 -newkey rsa:4096 -subj "/CN=$(hostname -˓→fqdn)/" -keyout frontend.key -out frontend.crt
Relaunch the service
The final step is to relaunch the service:
3.2. Frontend
27
IRMA Documentation, Release 1.5.3
$ sudo invoke-rc.d nginx restart
Installation Checks
Celery Workers
Before going further, you should check that the python application manages to communicate with the RabbitMQ server
and the Redis server through Celery. To ensure that, from the installation directory, execute the Celery worker:
On GNU/Linux:
$ cd /opt/irma/irma-frontend/current
$ ./venv/bin/celery worker --app=frontend.tasks:frontend_app
----------------- **** ------- * *** * --- * - **** --- ** ---------- ** ---------- ** ---------- ** ---------- *** --- * ---- ******* ------ ***** ------------------
celery@frontend v3.1.23 (Cipater)
Linux-3.2.0-4-amd64-x86_64-with-debian-7.6
[config]
.> app:
.> transport:
.> results:
.> concurrency:
frontend.tasks:0x1e18750
amqp://probe:**@brain:5672/mqfrontend
disabled
2 (prefork)
[queues]
.> frontend
exchange=celery(direct) key=frontend
[2014-08-20 15:28:58,745: WARNING/MainProcess] celery@frontend ready.
If your Celery worker does not output something similar to the above output, you should check twice the parameters
in the application configuration file you are using. Let us note that the Celery worker gives us useful information on
the analyzer that are enabled.
SFTP accounts
Defaut File Transport Protocol since v1.4.0 is now SFTP, you can check whether the configured account is valid.
$ sftp <user>@<hostname of the brain>
FTP-TLS accounts
Additionnally, if you have configured IRMA to use FTP-TLS, you can check whether the configured account is valid.
On Debian, this can be done with the ftp-ssl package:
$ sudo apt-get install ftp-ssl
[...]
$ ftp-ssl <hostname of the brain>
Connected to brain.
220---------- Welcome to Pure-FTPd [privsep] [TLS] ---------220-You are user number 1 of 50 allowed.
220-Local time is now 18:55. Server port: 21.
220-This is a private system - No anonymous login
220-IPv6 connections are also welcome on this server.
28
Chapter 3. Manual Installation
IRMA Documentation, Release 1.5.3
220 You will be disconnected after 15 minutes of inactivity.
Name (brain:root): frontend-ftp
500 This security scheme is not implemented
234 AUTH TLS OK.
[SSL Cipher DHE-RSA-AES256-GCM-SHA384]
200 PBSZ=0
200 Data protection level set to "private"
331 User probe OK. Password required
Password: frontend-ftp-password
230 OK. Current directory is /
Remote system type is UNIX.
Using binary mode to transfer files.
ftp>
Restful API
One can verify that the restful API is up and running by querying a specific route on the web server or by checking the
system logs:
$ curl http://localhost/api/v1.1/probes
{"total": 9, "data": ["ClamAV", "ComodoCAVL", "EsetNod32", "FProt", "Kaspersky",
˓→"McAfeeVSCL", "NSRL", "StaticAnalyzer", "VirusTotal"]}
$ sudo cat /var/log/supervisor/frontend_api.log
[...]
added /opt/irma/irma-frontend/current/venv/ to pythonpath.
*** uWSGI is running in multiple interpreter mode ***
spawned uWSGI master process (pid: 3943)
spawned uWSGI worker 1 (pid: 3944, cores: 1)
spawned uWSGI worker 2 (pid: 3945, cores: 1)
spawned uWSGI worker 3 (pid: 3946, cores: 1)
spawned uWSGI worker 4 (pid: 3947, cores: 1)
mounting frontend/api/base.py on /api
mounting frontend/api/base.py on /api
mounting frontend/api/base.py on /api
mounting frontend/api/base.py on /api
WSGI app 0 (mountpoint='/api') ready in 0 seconds on interpreter
˓→(default app)
WSGI app 0 (mountpoint='/api') ready in 0 seconds on interpreter
˓→(default app)
WSGI app 0 (mountpoint='/api') ready in 0 seconds on interpreter
˓→(default app)
WSGI app 0 (mountpoint='/api') ready in 0 seconds on interpreter
˓→(default app)
0x99a3e0 pid: 3945
0x99a3e0 pid: 3946
0x99a3e0 pid: 3944
0x99a3e0 pid: 3947
Probe
The Probes are python-based application that host a single or multiple analyzers. Each analyzer listens on a specific
work queue and waits for an analysis to be scheduled by the Brain through Celery, an open source task framework for
Python.
3.3. Probe
29
IRMA Documentation, Release 1.5.3
Architecture
Probes are mainly Celery workers that handle scan requests from Brain
Installation
Probes can be installed either on GNU/Linux and on Microsoft Windows systems. The installation procedure may
differs between the two systems.
Downloading the source code from github.com
The source code is hosted on github.com. One can fetch an up-to-date version with the following commands. Let us
note that there is a common submodule named common that is a soft-link (lib), do not forget the dereference
option of cp1 (-L):
# If src repository not already cloned
$ git clone https://github.com/quarkslab/irma /opt/irma/src
$ cp -rL /opt/irma/src/probe /opt/irma/irma-probe/current
We assume that you have a command line interface on your system with the following tools installed:
• python 2.7.x (see Python for Windows for prebuild MSI installer)
• python-pip (see Install pip for the recommended way to install pip package manager)
Furthermore, we assume that you have created an user irma that will be used to run the python application.
Note: On GNU/Linux, one can create the user (and the group) irma with:
$ sudo adduser --system --no-create-home --group irma
Installation on Microsoft Windows
On windows system, we will assume that the code for the Probe will be installed at the root of the C:\ drive, namely
in C:\irma\irma-probe\current.
$ C:\Python27\Scripts\pip.exe install virtualenv
$ C:\Python27\Scripts\virtualenv C:\irma\irma-probe\current\venv
$ cd C:\irma\irma-probe\current\
$ .\venv\Scripts\pip.exe install -r requirements.txt
[...]
Installation on GNU/Linux
On GNU/Linux system, we will assume that the code for the Probe will be installed in /opt/irma/irma-probe/
current directory.
1
30
On Microsoft Windows, a Linux-like lightweight command line interface can be installed by installing MSYS or Git for windows.
Chapter 3. Manual Installation
IRMA Documentation, Release 1.5.3
The source code is hosted on github.com. One can fetch an up-to-date version with the following commands. Let us
note that there is a common submodule named common that is a soft-link (lib), do not forget the dereference
option of cp (-L):
# If src repository not already cloned
$ git clone https://github.com/quarkslab/irma /opt/irma/src
$ cp -rL /opt/irma/src/probe /opt/irma/irma-probe/current
We need few dependencies for future steps:
$ sudo apt-get install python-virtualenv python-dev
then, we need to install python dependencies manually in a virtualenv :
$ virtualenv --system-site-packages /opt/irma/irma-probe/current/venv
$ /opt/irma/irma-probe/current/venv/bin/pip install -r /opt/irma/irma-probe/current/
˓→requirements.txt
[...]
If everything has gone well, the python application is now installed on your system. The next step is to configure it for
your platform and to enable the analyzers you need.
Configuration
The configuration file is config/probe.ini located in the installation directory.
Note: We recall in the following the meaning of each field in config/probe.ini:
Section
log
probe
broker probe
ftp_brain
Key
syslog
prefix
name
host
port
vhost
username
password
queue
host
port
auth
key_path
username
password
Type
integer
string
string
string
integer
string
string
string
string
string
integer
string
string
string
string
Default
0
irma-probe:
Description
enable rsyslog (experimental)
prefix to append to rsyslog entries
name to give to the probe
hostname for the RabbitMQ server
5672
port for the RabbitMQ server
virtual host configured for probes
username used for probes on the RabbitMQ server
password used for probes on the RabbitMQ server
queue to poll new tasks on the RabbitMQ server
hostname for the FTP server
21
port for the FTP server
“password”| SFTP authentication method (“password” or “key”)
sftp private key absolute path
username used by probe on the FTP server
password used by the probe on the FTP server
Enabling Analyzers
The python application auto-discovers available analyzers. As long as the requirements are fulfilled for an analyzer,
it is automatically discovered and enabled. By default, no analyzers are available. The following enumerates the
requirements to enable each analyzer bundled with the application.
3.3. Probe
31
IRMA Documentation, Release 1.5.3
ClamAV - GNU/Linux
To enable ClamAV on Debian Stable distribution, one should install several packages:
$ sudo apt-get install clamav-daemon
[...]
$ sudo freshclam
[...]
$ sudo service clamav-daemon restart
[...]
ComodoCAVL - GNU/Linux
Comodo Antivirus for Linux can be downloaded from the Comodo’s download page. The following instruction enable
to install the Debian package. By default, the binaries are installed in /opt/COMODO/ directory. As ComodoCAVL
is not packaged for the current Debian Stable distribution, we must install it manually:
$ sudo apt-get install binutils
$ ar x cav-linux_1.1.268025-1_iXXX.deb
$ sudo tar xvf ~/data.tar.gz -C /
[...]
Updates for the database can be performed with the following command:
$ XAUTHORITY="$HOME/.Xauthority" sudo /opt/COMODO/menu/comodo-updater
[...]
Note: Dependencies to update the database
To be able to update the database using the updater provided with Comodo Antivirus for Linux, some dependecies
needed for the graphical interface may be missing from the distribution. On Debian Stable system, one can install
them with:
$ sudo apt-get install libqt4-sql libqt4-network libqtgui4
EsetNod32 - GNU/Linux
One can request a trial version of Eset Nod32 Business Edition for Linux on the Eset download page. Once downloaded, the anti-virus can be installed with the following commands on Debian. Just follow the typical installation on
the GUI:
$ sudo chmod u+x eset_nod32av_64bit_en.linux
$ sudo apt-get install libgtk2.0-0 libc6-i386
$ sudo ./eset_nod32av_64bit_en.linux
[...]
Binaries should be installed in /opt/eset/esets directory. Updates are performed from the GUI:
$ XAUTHORITY="$HOME/.Xauthority" sudo /opt/eset/esets/bin/esets_gui
Note: Disabling the antivirus daemon
32
Chapter 3. Manual Installation
IRMA Documentation, Release 1.5.3
To avoid the anti-virus to protect your system at startup, we deliberately disabled the script used to launch the anti-virus
early at boot:
$ sudo service esets stop
$ sudo mv /etc/init.d/esets /etc/init.d/esets.disable
F-Prot - GNU/Linux
A copy of F-PROT anti-virus for Linux workstations is available on the F-PROT download page.
The binaries should be installed in /usr/local/f-prot to make the python application detect it automatically.
$ sudo tar xvf fp-Linux.x86.32-ws.tar.gz -C /usr/local/
To launch an update, a configuration step is mandatory:
$ sudo cp /usr/local/f-prot/f-prot.conf.default /etc/f-prot.conf
An update is launched with:
$ sudo ./fpupdate
ERROR: ld.so: object 'libesets_pac.so' from /etc/ld.so.preload cannot be preloaded:
˓→ignored.
[...]
Note: Error
If you see an error message like:
DownloadingWarning: Network - Connection failed (18), trying again...
Downloading updateError: Update - Bad mergefile
Just relaunch the script.
Note: Dependencies to update the database
To be able to update the database using the updater provided with Comodo install them with:
$ sudo apt-get install libc6-i386
McAfeeVSCL - GNU/Linux or Microsoft Windows
A free evaluation of McAfee VirusScan Command Line can be downloaded from the editor download page.
The binaries should be installed in /usr/local/uvscan/ on GNU/Linux system and must be installed in
C:\VSCL on Windows Systems. Let us note that updates must be performed manually. Anti-virus databases and
engines can be downloaded here.
After downloading McAfee Virus Scan archive, create /usr/local/uvscan and extract the archive in it:
3.3. Probe
33
IRMA Documentation, Release 1.5.3
$ sudo mkdir /usr/local/uvscan
$ sudo tar xvf vscl-XXX.tar.gz -C /usr/local/uvscan # replace using your values
$ sudo chmod +x /usr/local/uvscan/uvscan
Extract also, using unzip program, the database:
$ sudo unzip avvepo7536dat.zip -d /usr/local/uvscan
$ cd /usr/local/uvscan
$ sudo unzip avvdat-XXXX.zip
Sophos - GNU/Linux or Microsoft Windows
A free evaluation of Sophos Endpoint Antivirus can be downloaded from the editor download page. You should receive
by email a username and a password to authenticate for updates.
It should be detected automatically by IRMA if the anti-virus is installed in its default installation directory.
On GNU/Linux:
• Download the archive for Linux, then execute:
$ tar zxf sav-linux-9-i386.tgz
$ ./sophos-av/install.sh /opt/sophos-av --acceptlicence --autostart=False -˓→enableOnBoot=False --automatic --ignore-existing-installation
$ /opt/sophos-av/bin/savconfig set EnableOnStart false
$ /opt/sophos-av/bin/savconfig set PrimaryUpdateSourcePath "sophos:"
$ /opt/sophos-av/bin/savconfig set PrimaryUpdateUsername "<your_username_from_email>"
$ /opt/sophos-av/bin/savconfig set PrimaryUpdatePassword "<your_password_from_email>"
$ /opt/sophos-av/bin/savupdate
Kaspersky - Microsoft Windows
A free evaluation of Kaspersky Internet Security can be requested on the editor download page. It should be detected
automatically if the anti-virus is installed in its default installation directory.
Symantec - Microsoft Windows
The procedure to install a trial version of Symantec Endpoint Protection is particularly tedious. We will not document
its installation.
G Data - Microsoft Windows
A
trial
version
of
G
Data
is
available
on
the
editor
download
page
<https://www.gdata.de/kundenservice/downloads.html>. It should be detected automatically if the anti-virus is
installed in its default installation directory.
VirusTotal - GNU/Linux or Microsoft Windows
The VirusTotal analyzer can be installed easily by downloading the python packages it depends on and by modifying
its configuration file. From the installation directory, one can execute:
On GNU/Linux:
34
Chapter 3. Manual Installation
IRMA Documentation, Release 1.5.3
$ pip install -r modules/external/virustotal/requirements.txt
[...]
then update configuration file located at modules/external/virustotal/config.ini.
Note: Meaning of the fields in the configuration file
Section
VirusTotal
VirusTotal
Option
apikey
private
Type
string
boolean
Default
Description
api_key used to query VirusTotal API
use private api (need a private api key)
NSRL - GNU/Linux
The National Software Reference Library can be downloaded on the NIST’s web page. The provided files are stored
in the RDS (Reference Data Set) format. To use this analyzer, one must build first the database. We use LevelDB as
fast key-value storage library.
To build the dabatase, one must install first the dependencies:
$ pip install -r modules/database/nsrl/requirements.txt
A (not optimized and very slow) helper script is provided to build the database:
$ mkdir /home/irma/leveldb
$ python -m modules.database.nsrl.nsrl
˓→db
$ python -m modules.database.nsrl.nsrl
˓→leveldb/mfg_db
$ python -m modules.database.nsrl.nsrl
˓→leveldb/prod_db
$ python -m modules.database.nsrl.nsrl
˓→file_db
create -t os NSRLOS.txt /home/irma/leveldb/os_
create -t manufacturer NSRLMfg.txt /home/irma/
create -t product NSRLProd.txt /home/irma/
create -t file NSRLFile.txt /home/irma/leveldb/
Finally, one must indicate to the analyzer where to find the files for the database by filling the configuration file located
at modules/database/nsrl/config.ini.
Note: Meaning of the fields in the configuration file
Section
NSRL
Option
nsrl_os_db
nsrl_mfg_db
nsrl_file_db
nsrl_prod_db
Type
string
string
string
string
Default
Description
location of the OS database
location of the Manufacturer database
location of the File database
location of the Product database
Note: Error
If you see an error message like:
fatal error: Python.h: No such file or directory
Then you’ll need to install python-dev package (for Debian like systems).
3.3. Probe
35
IRMA Documentation, Release 1.5.3
Note: leveldb.LevelDBError: IO error: /home/irma/leveldb/file_db/LOCK: Permission denied
If you encounter this problem, you likely have a problem with unix permissions. Please ensure that the folder is owned
by the user running the probes. On supervisord-based installation (default for vagrant/ansible scripts), the folder
owner should be set to nobody. For init.d-based installation, it should be irma instead.
StaticAnalyzer - GNU/Linux or Microsoft Windows
The PE File analyzer adapted from Cuckoo Sandbox can be installed easily. One need to install the python packages
it depends on. From the installation directory, one can execute:
On GNU/Linux:
$ pip install -r modules/metadata/pe_analyzer/requirements.txt
[...]
On Microsoft Windows, you need cygwin to successfully install the requirements (see python-magic documentation
for installation details):
$ C:\Python27\Scripts\pip.exe install -r modules/metadata/pe_analyzer/requirements.txt
[...]
Yara - GNU/Linux or Microsoft Windows
Please refer to modules/metadata/yara/README.md file for the documentation.
Guide on Debian (credits to Carbonn)
1. Installing dependencies
$ sudo apt-get install libtool automake bison
2. Installing Yara python modules
$
$
$
$
$
$
$
$
git clone https://github.com/plusvic/yara.git
autoreconf -i --force
./configure
make
sudo make install
python setup.py build
sudo python setup.py install
sudo ldconfig
3. Configuring for IRMA
$ mkdir /opt/irma/yara_rules/
$ cat /opt/irma/yara_rules/yara_rules.yar << EOF
# Insert rule below inside the file
rule silent_banker : banker
{
meta:
description = "This is just an example"
36
Chapter 3. Manual Installation
IRMA Documentation, Release 1.5.3
thread_level = 3
in_the_wild = true
strings:
$a = {6A 40 68 00 30 00 00 6A 14 8D 91}
$b = {8D 4D B0 2B C1 83 C0 27 99 6A 4E 59 F7 F9}
$c = "UVODFRYSIHLNWPEJXQZAKCBGMT"
condition:
$a or $b or $c
}
EOF
Installation Checks
Celery Workers
Before going further, you should check that the python application manages to communicate with the RabbitMQ server
and the Redis server through Celery. To ensure that, from the installation directory, execute the Celery worker:
On GNU/Linux:
$ cd /opt/irma/irma-probe/current
$ ./venv/bin/celery worker --app=probe.tasks
$ celery worker --app=probe.tasks:app --workdir=/opt/irma/irma-probe/current
WARNING:root: *** [plugin] Plugin failed to load: Kaspersky miss dependencies: win32
˓→(PlatformDependency).
WARNING:root: *** [plugin] Plugin failed to load: Sophos miss dependencies: win32
˓→(PlatformDependency).
WARNING:root: *** [plugin] Plugin failed to load: Symantec miss dependencies: win32
˓→(PlatformDependency).
WARNING:root: *** [plugin] Plugin failed to load: NSRL miss dependencies: leveldict
˓→(ModuleDependency). See requirements.txt for needed dependencies
WARNING:root: *** [plugin] Plugin failed to load: VirusTotal miss dependencies: virus_
˓→total_apis (ModuleDependency). See requirements.txt for needed dependencies
WARNING:root: *** [plugin] Plugin failed to load: StaticAnalyzer miss dependencies:
˓→pefile (ModuleDependency). See requirements.txt for needed dependencies
----------------- **** ------- * *** * --- * - **** --- ** ---------- ** ---------- ** ---------- ** ---------- *** --- * ---- ******* ------ ***** ------------------
celery@irma-probe v3.1.13 (Cipater)
Linux-3.2.0-4-amd64-x86_64-with-debian-7.6
[config]
.> app:
.> transport:
.> results:
.> concurrency:
[queues]
.> ClamAV
.> ComodoCAVL
.> EsetNod32
.> FProt
.> McAfeeVSCL
probe.tasks:0x1e18750
amqp://probe:**@brain:5672/mqprobe
redis://brain:6379/1
1 (prefork)
exchange=celery(direct)
exchange=celery(direct)
exchange=celery(direct)
exchange=celery(direct)
exchange=celery(direct)
key=ClamAV
key=ComodoCAVL
key=EsetNod32
key=FProt
key=McAfeeVSCL
[2014-08-19 18:25:06,469: WARNING/MainProcess] celery@irma-probe ready.
3.3. Probe
37
IRMA Documentation, Release 1.5.3
The equivalent command on Microsoft Windows is:
$ C:\Python27\Scripts\celery.exe worker --app=probe.tasks:app --workdir=C:\irma\irma˓→probe\current
[...]
If your Celery worker does not output something similar to the above output, you should check twice the parameters
in the application configuration file you are using. Let us note that the Celery worker gives us useful information on
the analyzer that are enabled. As each analyzer connects to a dedicated queue, we can deduce, in this example, that
the analyzers ClamAV, ComodoCAVL, EsetNod32, FProt and McAfeeVSCL are enabled and ready to serve.
SFTP accounts
Defaut File Transport Protocol since v1.4.0 is now SFTP, you can check whether the configured account is valid.
$ sftp <user>@<hostname of the brain>
FTP-SSL accounts
Additionnally, if you have configured IRMA to use FTP-ssl, you can check whether the configured account is valid.
On Debian, this can be done with the ftp-ssl package:
$ sudo apt-get install ftp-ssl
[...]
$ ftp-ssl <hostname of the brain>
Connected to brain.
220---------- Welcome to Pure-FTPd [privsep] [TLS] ---------220-You are user number 1 of 50 allowed.
220-Local time is now 18:55. Server port: 21.
220-This is a private system - No anonymous login
220-IPv6 connections are also welcome on this server.
220 You will be disconnected after 15 minutes of inactivity.
Name (brain:root): probe-ftp
500 This security scheme is not implemented
234 AUTH TLS OK.
[SSL Cipher DHE-RSA-AES256-GCM-SHA384]
200 PBSZ=0
200 Data protection level set to "private"
331 User probe OK. Password required
Password: probe-ftp-password
230 OK. Current directory is /
Remote system type is UNIX.
Using binary mode to transfer files.
ftp>
38
Chapter 3. Manual Installation
CHAPTER
4
Database migration
IRMA uses Alembic to manage and perform databases migration.
Note: Alembic is a useful tool to manage migration, but can’t surpass local engine implementation of SQL. As
SQLite doesn’t manage schema modifications such as ALTER_COLUMN, the whole migration system of IRMA
won’t support it. The preferred database engine is PostgreSQL.
You can still use SQLite, but you will be on your own for migrations.
Warning: Please note that most of the manipulations on this can and sometimes will alter your data. If you are
not sure about what you are doing, and even if you are sure, make backup.
Requirements
• Alembic
Content
Database migrations are managed in the frontend and brain IRMA components.
The files/directories used are:
alembic.ini
extras/migration/
+- env.py
+- script.py.mako
+- versions/
+- <revision_1>.py
39
IRMA Documentation, Release 1.5.3
+- <revision_2>.py
+- ...
Note: All the commands below will assert to be executed on top of this file system, as Alembic needs the alembic.
ini configuration file.
You could also use the -c <path_to_conf_file>.
Usage
Alembic manage a ‘revision’ for each database evolution. These revisions are used to upgrade or downgrade the
database schema.
The command:
$ alembic current
... shows the current revision of the database.
The command to get the history of the latest alembic migrations is:
$ alembic history --verbose
Create database from scratch with Alembic
Configuration and creating database
Alembic will use the information in the [sqldb] section of the configuration files (respectively config/
frontend.ini or conf/brain.ini for the repositories of the frontend or the brain components). Make sure
they are accurate.
The database must already exist. This step is quite simple, the SQL command usually being:
sql$ CREATE DATABASE <db_name>;
Update your schema with Alembic
If you use a virtualenv, activate it. Then enter:
$ alembic upgrade head
Alembic applies each revision one after the other. At the end of the process, if no error occurs, your database should
be updated.
Note: You can update the database one revision at a time, or up to a specific revision. See the revisions section for
further information.
40
Chapter 4. Database migration
IRMA Documentation, Release 1.5.3
If you already have a database WITHOUT Alembic
Alembic stores its current revision number in database. If your database doesn’t have this information, you are very
likely to encounter errors when using Alembic, as it will try to create already existing tables.
The easiest solution is to destroy your database and go for a fresh install.
Although, if you don’t want to lose your data, you could update the Alembic information manually.
You will need to:
1. Get the exact current Alembic revision of your database. Each migration file has a Revision ID in its header.
Investigate the successive revisions to know which one matches your current database state.
2. Once you known your Alembic revision, run:
$ alembic stamp <your_alembic_revision_number>
3. Your database is now synchronized with Alembic! You should be able to use Alembic to upgrade/downgrade
your database now. Be aware that if the revision number you provided is false, you could encounter massive
errors while attempting to upgrade/downgrade your database.
Generating a new revision
Creating a new revision can be done with the command:
$ alembic revision -m <revision_message>
This command produces a new <hash>_<revision_message>.py file in the extras/migration/
versions/ directory. This file contains two functions upgrade and downgrade, respectively used to upgrade
the database to the revision, or downgrade from it. These two functions are empty and must be completed with the
desired modifications (see the alembic documentation).
A revision could be produced automatically, from database metadata defined in the IRMA SQL objects description
through sqlalchemy, with the command:
$ alembic revision --autogenerate -m <revision_message>
These SQL objects are defined in:
• frontend/models/sqlobjects.py for the frontend,
• brain/models/sqlobjects.py for the brain.
Alembic scripts in IRMA repositories are already configured to use metadata defined in these files. You should be able
to use the --autogenerate option without further modifications.
Note: IRMA configuration allows to prefix table names through configuration. Our revision files use the function
<frontend_or_brain>/config/parser.py:prefix_table_name to generate table names rather than
keeping alembic-generated plain string names. A good practice would be to keep using this function in revision files.
Warning: Alembic easily detects changes such as adding/removing columns, but could be blind on thin, inner
modifications. Re-reading the auto-generated script is a strongly recommended step before actually performing
the migration.
See the alembic documentation for more information.
4.3. Usage
41
IRMA Documentation, Release 1.5.3
Warning: Database modifications using ALTER_COLUMN (such as changing the type of a column) can’t be
performed on SQLite databases. Be aware of this limitation if you absolutely want
to use migration scripts with this SQL engine.
Migrating between revisions
Once the revision is properly described, the migration is performed with:
$ alembic upgrade head
Alembic allows to migrate the database to any revision, relatively to the current revision or absolutely. Several examples:
$ alembic upgrade +4
$ alembic downgrade base
$ alembic upgrade <revision_number>+3
Tips and tricks
Note: Don’t trust Alembic too much. It is nothing more than a tool, without any comprehension on the code.
Cautiously read the revision scripts it generates.
Note: Database migration is hardly ever a painless step. Be sure to:
1. save your data before performing a migration,
2. test your application after the migration to ensure its compatibility with the new data schemes.
Note: With a PostgreSQL database, the Float type is tolerated but the real type name used by the database
is Real. It means that SQL objects described in sqlalchemy with Float columns will be properly applied in
database, but at each autogenerate revision, alembic will see Real type in database, against Float type in the
code metadata, and so will perform each time a useless alter_column from Real to Float. This problem could
be avoided (with PostgreSQL) by declaring Real instead of Float.
See this page for more information on PostgreSQL numeric types.
Note:
Alembic can’t directly deal with many somehow complex operations, such as type migration with no
trivial cast. In these cases, the operation must be manually described with a raw SQL command (which
could be database-dependent).
For instance, alembic can’t perform the migration from real to datetime:
> alembic.alter_column('table', 'column',
existing_type=sqlalchemy.REAL(),
42
Chapter 4. Database migration
IRMA Documentation, Release 1.5.3
type_=sqlalchemy.DateTime(),
existing_nullable=False)
...
because of an error a column "column" cannot be cast automatically to type
timestamp with time zone.
A proper migration for PostgreSQL would be (in Python):
> alembic.execute('ALTER TABLE "table" ALTER COLUMN "column" TYPE TIMESTAMP WITHOUT
˓→TIME ZONE USING to_timestamp(column)')
And the reverse code to downgrade the migration could be:
> alembic.execute('ALTER TABLE "table" ALTER COLUMN "column" TYPE REAL USING
˓→extract(epoch from column)')
Note: Rather than managing migrations directly with Alembic, we could generate SQL migration revision to be used
directly on database with the command:
$ alembic upgrade <revision> --sql > migration.sql
Note: Deleting a revision R is simple:
• downgrade the database to the revision before R-1 the revision you want to delete;
• if any, edit the script of the following revision R+1 and update the down_revision variable to match the
revision number of revision R-1;
• delete the script of the revision R you want to delete;
• upgrade your database.
The deleted revision want be applied any more.
4.4. Tips and tricks
43
IRMA Documentation, Release 1.5.3
44
Chapter 4. Database migration
CHAPTER
5
References
Disclaimer
IRMA is distributed as it is, in the hope that it will be useful, but without any warranty neither the implied merchantability or fitness for a particular purpose.
Whatever you do with this tool is uniquely your own responsibility.
License
IRMA source code is licensed under Apache License, version 2.0.
The full license text can be found below (Apache License, version 2.0).
Apache License, version 2.0
Authors
45
IRMA Documentation, Release 1.5.3
46
Chapter 5. References
CHAPTER
6
Frequently Asked Questions
Playing with tags
Tags are available in IRMA from version 1.3.0
Creating a tag
You could create tags by using the command line tools
>>> from irmacl.helpers import *
>>> tag_list()
[]
>>> tag_new("archive")
{u'text': u'archive', u'id': 1}
>>> tag_list()
[Tag archive [1]]
or directly from your terminal by using curl and posting a json with ‘text’ key:
$ curl -H "Content-Type: application/json; charset=UTF-8"
˓→tag>"}' http://172.16.1.30/api/v1.1/tags
-X POST -d '{"text":"<your
Note: There is currently no way to create a tag directly from the web IHM.
Tagging a File
Directly in web IHM, once you are on a file details page:
47
IRMA Documentation, Release 1.5.3
Just click the tag bar and you will see all available tags. You could add multiple tags.
It is also possible to add a tag through command line tools:
48
Chapter 6. Frequently Asked Questions
IRMA Documentation, Release 1.5.3
>>> from irmacl.helpers import *
>>> file_tag_add?
Signature: file_tag_add(sha256, tagid, verbose=False)
Docstring:
Add a tag to a File
:param sha256: file sha256 hash
:type sha256: str of (64 chars)
:param tagid: tag id
:type tagid: int
:return: No return
>>> file_tag_add("346ae869f7c7ac7394196de44ab4cfcde0d1345048457d03106c1a0481fba853",1)
Searching by tag
You could specify one or more tags while searching for files too:
choose your tag list then hit the search button:
or by command line:
>>> from irmacl.helpers import *
>>> file_search(tags=[1])
(1, [<irma.apiclient.IrmaResults at 0x7f079ca23890>])
SSL settings
Making RabbitMQ running SSL
6.2. SSL settings
49
IRMA Documentation, Release 1.5.3
Certificates generation
See rabbitmq detailled guide on how to generate server and clients certificates here
Update server settings on brain
Copy the RabbitMQ configuration template provided in ”./extras/rabbitmq” to “/etc/rabbitmq”
Restart RabbitMQ:
$ sudo service rabbitmq-server restart
Allow irma to use SSL with RabbitMQ
Note: Frontend configuration file is <irma dir>/config/frontend.ini
Brain configuration file is <irma dir>/config/brain.ini
Probe configuration file is <irma dir>/config/probe.ini
Update PORT in configuration file from 5672 to 5671. Except if you change the default port defined in the the
RabbitMQ configuration template provided in the precedent paragraph.
[broker_xxxx]
host = 127.0.0.1
port = 5671
Update activate_ssl switch in configuration file:
[ssl_config]
activate_ssl = yes
ca_certs =
keyfile =
certfile =
Put the SSL certificates (ca_cert, key_file, cert_file) in <irma dir>/ssl Update ca_certs, keyfile and
certfile in configuration file according to the filenames in ”./ssl”
[ssl_config]
activate_ssl = yes
ca_certs = <ca_cert_filename>
keyfile = <key_filename>
certfile = <cert_filename>
Note: If you are switching to ssl from an already running no_ssl version, please do the following on irma-brain
RabbitMQ server:
$
$
$
#
$
$
$
sudo rabbitmqctl stop_app
sudo rabbitmqctl reset
sudo rabbitmqctl start_app
create again the RabbitMQ vhosts, usernames and passwords:
sudo ./extras/scripts/rabbitmq/rmq_adduser.sh probe probe mqprobe
sudo ./extras/scripts/rabbitmq/rmq_adduser.sh brain brain mqbrain
sudo ./extras/scripts/rabbitmq/rmq_adduser.sh frontend frontend mqfrontend
50
Chapter 6. Frequently Asked Questions
IRMA Documentation, Release 1.5.3
How to debug
Switch debug log on
Configuration file for frontend, brain and probe is located by default in the config folder and is named respectively
frontend.ini, brain.ini and probe.ini.
To turn on debug log just add the following line:
[log]
syslog = 0
debug = 1
and restart all related applications.
To turn on SQL debug log (warning: its verbose) just add the following line:
[log]
syslog = 0
debug = 1
sql_debug = 1
and restart all related applications.
Debug a probe
Open a session on the probe machine and change directory to the irma-probe location. Try the run_module tool on a
file to see what analyzer is detected and what is its output on a file.
$ sudo su deploy
$ cd /opt/irma/irma-probe/current
$ ./venv/bin/python -m tools.run_module
[...]
usage: run_module.py [-h] [-v]
{Unarchive,StaticAnalyzer,ClamAV,VirusTotal} filename
[filename ...]
run_module.py: error: too few arguments
Here 4 probes are automatically detected. Now try one on a file:
$ ./venv/bin/python -m tools.run_module ClamAV requirements.txt
{'database': {'/var/lib/clamav/bytecode.cvd': {'ctime': 1458640823.285298,
'mtime': 1458640823.069295,
'sha256':
˓→'82972e6cc5f1204829dba913cb1a0b5f8152eb73d3407f6b86cf388626cff1a1'},
'/var/lib/clamav/daily.cvd': {'ctime': 1458640822.8932924,
'mtime': 1458640822.6692889,
'sha256':
˓→'9804c9b9aaf983f85b4f13a7053f98eb7cca5a5a88d3897d49b22182b228885f'},
'/var/lib/clamav/main.cvd': {'ctime': 1458640821.6972747,
'mtime': 1458640813.9771628,
'sha256':
˓→'4a8dfbc4c44704186ad29b5a3f8bdb6674b679cecdf83b156dd1c650129b56f2'}},
6.3. How to debug
51
IRMA Documentation, Release 1.5.3
'duration': 0.0045299530029296875,
'error': None,
'name': 'Clam AntiVirus Scanner',
'platform': 'linux2',
'results': None,
'status': 0,
'type': 'antivirus',
'version': '0.99'}
And check the output.
Debug Ansible Provisioning
To debug errors while provisioning (same goes with deployment) with following typical command:
$ ansible-playbook --private-key=~/.vagrant.d/insecure_private_key --inventory-file=.
˓→vagrant/provisioners/ansible/inventory/vagrant_ansible_inventory -u vagrant
˓→playbooks/provisioning.yml
Example output:
TASK [Mayeu.RabbitMQ : add rabbitmq user and set privileges] *******************
[DEPRECATION WARNING]: Using bare variables is deprecated. Update your playbooks so
˓→that the environment value uses the
full variable syntax ('{{rabbitmq_users_definitions}}').
This feature will be removed in a future release. Deprecation
warnings can be disabled by setting deprecation_warnings=False in ansible.cfg.
failed: [brain.irma] (item={u'vhost': u'mqbrain', u'password': u'brain', u'user': u
˓→'brain'}) => {"failed": true, "item": {"password": "brain", "user": "brain", "vhost
˓→": "mqbrain"}, "module_stderr": "", "module_stdout": "Traceback (most recent call
˓→last):\r\n
File \"/tmp/ansible_wKXoO5/ansible_module_rabbitmq_user.py\", line 302,
˓→in <module>\r\n
main()\r\n File \"/tmp/ansible_wKXoO5/ansible_module_rabbitmq_
˓→user.py\", line 274, in main\r\n
if rabbitmq_user.get():\r\n File \"/tmp/
˓→ansible_wKXoO5/ansible_module_rabbitmq_user.py\", line 155, in get\r\n
users =
˓→self._exec(['list_users'], True)\r\n
File \"/tmp/ansible_wKXoO5/ansible_module_
˓→rabbitmq_user.py\", line 150, in _exec\r\n
rc, out, err = self.module.run_
˓→command(cmd + args, check_rc=True)\r\n
File \"/tmp/ansible_wKXoO5/ansible_modlib.
˓→zip/ansible/module_utils/basic.py\", line 1993, in run_command\r\n
File \"/usr/lib/
˓→python2.7/posixpath.py\", line 261, in expanduser\r\n
if not path.startswith('~
˓→'):\r\nAttributeError: 'list' object has no attribute 'startswith'\r\n", "msg":
˓→"MODULE FAILURE", "parsed": false}
You could first increase ansible verbosity by adding -vvv option (-vvvv on windows for winrm debug), it will help
is the problem is linked to arguments.
$ ansible-playbook -vvv --private-key=~/.vagrant.d/insecure_private_key -inventory-file=.vagrant/provisioners/ansible/inventory/vagrant_ansible_inventory -u
˓→vagrant playbooks/provisioning.yml
TASK [Mayeu.RabbitMQ : add rabbitmq user and set privileges] *******************
task path: /home/alex/repo/irma-ansible/roles/Mayeu.RabbitMQ/tasks/vhost.yml:13
[DEPRECATION WARNING]: Using bare variables is deprecated. Update your playbooks
˓→so that the environment value uses the full
variable syntax ('{{rabbitmq_users_definitions}}').
This feature will be removed in a future release. Deprecation warnings can be
disabled by setting deprecation_warnings=False in ansible.cfg.
<127.0.0.1> ESTABLISH SSH CONNECTION FOR USER: vagrant
˓→
52
Chapter 6. Frequently Asked Questions
IRMA Documentation, Release 1.5.3
<127.0.0.1> SSH: EXEC ssh -C -q -o ForwardAgent=yes -o Port=2222 -o
'IdentityFile="/home/alex/.vagrant.d/insecure_private_key"' -o
˓→KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi˓→keyex,hostbased,publickey -o PasswordAuthentication=no -o User=vagrant -o
˓→ConnectTimeout=10 127.0.0.1 '/bin/sh -c '"'"'( umask 77 && mkdir -p "` echo $HOME/.
˓→ansible/tmp/ansible-tmp-1468570550.09-211613386938202 `" && echo ansible-tmp˓→1468570550.09-211613386938202="` echo $HOME/.ansible/tmp/ansible-tmp-1468570550.09˓→211613386938202 `" ) && sleep 0'"'"''
<127.0.0.1> PUT /tmp/tmpiysJ6l TO /home/vagrant/.ansible/tmp/ansible-tmp˓→1468570550.09-211613386938202/rabbitmq_user
<127.0.0.1> SSH: EXEC sftp -b - -C -o ForwardAgent=yes -o Port=2222 -o
˓→'IdentityFile="/home/alex/.vagrant.d/insecure_private_key"' -o
˓→KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi˓→keyex,hostbased,publickey -o PasswordAuthentication=no -o User=vagrant -o
˓→ConnectTimeout=10 '[127.0.0.1]'
<127.0.0.1> ESTABLISH SSH CONNECTION FOR USER: vagrant
<127.0.0.1> SSH: EXEC ssh -C -q -o ForwardAgent=yes -o Port=2222 -o
˓→'IdentityFile="/home/alex/.vagrant.d/insecure_private_key"' -o
˓→KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi˓→keyex,hostbased,publickey -o PasswordAuthentication=no -o User=vagrant -o
˓→ConnectTimeout=10 -tt 127.0.0.1 '/bin/sh -c '"'"'sudo -H -S -n -u root /bin/sh -c '"
˓→'"'"'"'"'"'"'"'echo BECOME-SUCCESS-rbeeckncuxenewcwkayivqiwvarchlrd; LANG=fr_FR.UTF˓→8 LC_ALL=fr_FR.UTF-8 LC_MESSAGES=fr_FR.UTF-8 /usr/bin/python /home/vagrant/.ansible/
˓→tmp/ansible-tmp-1468570550.09-211613386938202/rabbitmq_user; rm -rf "/home/vagrant/.
˓→ansible/tmp/ansible-tmp-1468570550.09-211613386938202/" > /dev/null 2>&1'"'"'"'"'"'"
˓→'"'"' && sleep 0'"'"''
failed: [brain.irma] (item={u'vhost': u'mqbrain', u'password': u'brain', u'user
˓→': u'brain'}) => {"failed": true, "invocation": {"module_name": "rabbitmq_user"},
˓→"item": {"password": "brain", "user": "brain", "vhost": "mqbrain"}, "module_stderr
˓→": "", "module_stdout": "Traceback (most recent call last):\r\n
File \"/tmp/
˓→ansible_Qo3lZl/ansible_module_rabbitmq_user.py\", line 302, in <module>\r\n
˓→main()\r\n
File \"/tmp/ansible_Qo3lZl/ansible_module_rabbitmq_user.py\", line 274,
˓→in main\r\n
if rabbitmq_user.get():\r\n File \"/tmp/ansible_Qo3lZl/ansible_
˓→module_rabbitmq_user.py\", line 155, in get\r\n
users = self._exec(['list_users
˓→'], True)\r\n
File \"/tmp/ansible_Qo3lZl/ansible_module_rabbitmq_user.py\", line
˓→150, in _exec\r\n
rc, out, err = self.module.run_command(cmd + args, check_
˓→rc=True)\r\n
File \"/tmp/ansible_Qo3lZl/ansible_modlib.zip/ansible/module_utils/
˓→basic.py\", line 1993, in run_command\r\n
File \"/usr/lib/python2.7/posixpath.py\",
˓→ line 261, in expanduser\r\n
if not path.startswith('~'):\r\nAttributeError:
˓→'list' object has no attribute 'startswith'\r\n", "msg": "MODULE FAILURE", "parsed
˓→": false}
˓→
In this particular case, verbose doesnt add much information as the problem is linked to ansible scripts. Lets go one
level deeper so. Ansible output the temporary script executed on guest (highlighted in previous code block), but delete
it just after execution. To further debug it we will set ansible to keep remote files and the debug session will now takes
place inside the guest.
$ ANSIBLE_KEEP_REMOTE_FILES=1 ansible-playbook -vvv --private-key=~/.vagrant.d/
˓→insecure_private_key --inventory-file=.vagrant/provisioners/ansible/inventory/
˓→vagrant_ansible_inventory -u vagrant playbooks/provisioning.yml
in debug log get the temporary ansible path to remote script:
/usr/bin/python /home/vagrant/.ansible/tmp/ansible-tmp-1468571039.87-134696488633275/
˓→rabbitmq_user
Log in to remote machine and go to the temporary ansible dir. Explode the compressed script and run it locallly:
6.3. How to debug
53
IRMA Documentation, Release 1.5.3
$ vagrant@brain:~/.ansible/tmp/ansible-tmp-1468571039.87-134696488633275$ ls
rabbitmq_user
$ vagrant@brain:~/.ansible/tmp/ansible-tmp-1468571039.87-134696488633275$ python
˓→rabbitmq_user explode
Module expanded into:
/home/vagrant/.ansible/tmp/ansible-tmp-1468571039.87-134696488633275/debug_dir
$ vagrant@brain:~/.ansible/tmp/ansible-tmp-1468571039.87-134696488633275$ ls debug_
˓→dir/
ansible
ansible_module_rabbitmq_user.py
args
$ vagrant@brain:~/.ansible/tmp/ansible-tmp-1468571039.87-134696488633275$ python
˓→rabbitmq_user execute
Traceback (most recent call last):
File "/home/vagrant/.ansible/tmp/ansible-tmp-1468571039.87-134696488633275/debug_
˓→dir/ansible_module_rabbitmq_user.py", line 302, in <module>
main()
File "/home/vagrant/.ansible/tmp/ansible-tmp-1468571039.87-134696488633275/debug_
˓→dir/ansible_module_rabbitmq_user.py", line 274, in main
if rabbitmq_user.get():
File "/home/vagrant/.ansible/tmp/ansible-tmp-1468571039.87-134696488633275/debug_
˓→dir/ansible_module_rabbitmq_user.py", line 155, in get
users = self._exec(['list_users'], True)
File "/home/vagrant/.ansible/tmp/ansible-tmp-1468571039.87-134696488633275/debug_
˓→dir/ansible_module_rabbitmq_user.py", line 150, in _exec
rc, out, err = self.module.run_command(cmd + args, check_rc=True)
File "/home/vagrant/.ansible/tmp/ansible-tmp-1468571039.87-134696488633275/debug_
˓→dir/ansible/module_utils/basic.py", line 1993, in run_command
args = [ os.path.expandvars(os.path.expanduser(x)) for x in args if x is not None
˓→]
File "/usr/lib/python2.7/posixpath.py", line 261, in expanduser
if not path.startswith('~'):
AttributeError: 'list' object has no attribute 'startswith'
You could now add debug to source files and properly understand where the problem is. In our example case, it is an
ansible problem related to module_rabbitmq_user present in 2.1.0.0 see github PR
How to migrate
Note: If you need help to connect to your box through ssh, see vagrant FAQ
This part is only useful to someone willing to manually upgrade from an older version of IRMA.
Install alembic
$
$
$
$
sudo su deploy
cd /opt/irma/irma-frontend/current
./venv/bin/pip install alembic
export PYTHONPATH=.:$PYTHONPATH
54
Chapter 6. Frequently Asked Questions
IRMA Documentation, Release 1.5.3
$ alembic history
430a70c8aa21 -> eb7141efd75a (head), version 1.3.0
2cc69d5c53eb -> 430a70c8aa21, version 1.2.1
<base> -> 2cc69d5c53eb, DB revision creation
from 1.2.1 to 1.3.0
Fix nginx configuration
Introducing multiversion API means python code should receive the api version parameter. in file /etc/nginx/sitesavailable/irma-frontend.conf replace:
rewrite ^/api/v1/(.+) /$1 break;
by:
rewrite ^/api/(.+) /$1 break;
and restart nginx
Migrate Database
First you should tell alembic you are at version 1.2.1:
$ ./venv/bin/alembic stamp 430a70c8aa21
then upgrade model and data:
$ ./venv/bin/alembic upgrade head
Regenerate IHM
to regenerate IHM do the following:
$
$
$
$
sudo su deploy
cd /opt/irma/irma-frontend/current/web
./node_modules/.bin/bower update
./node_modules/.bin/gulp dist
Its done.
API documentation
There is a dynamic documentation for IRMA API available on your instance
It allows you to read documentation but also try request and see server response. Give it a try.
6.5. API documentation
55
IRMA Documentation, Release 1.5.3
You could see detailed information about one specific API route:
and by clicking on the Try it button, see the server response:
56
Chapter 6. Frequently Asked Questions
IRMA Documentation, Release 1.5.3
Connect to a vagrant box through ssh
If you don’t already have it download vagrant insecure_private_key
$ wget https://raw.githubusercontent.com/mitchellh/vagrant/master/keys/vagrant -O
˓→insecure_private_key
Then change rights on the key otherwise ssh will complains and connect to your vagrant box
$ chmod 700 insecure_private_key
$ ssh vagrant@172.16.1.30 -i insecure_private_key
Enable SSL using OpenSSL in ansible scripts
If you want to activate SSL on the frontend server, you’ll need:
• modify frontend_openssl variables in group_vars/frontend:
frontend_openssl: True # Default is false
frontend_openssl_dh_param: # put the DH file locations
frontend_openssl_certificates: [] # an array of files {source, destination}
# to copy to the server
• Uncomment (and customize) the nginx_sites variable in the group_vars/frontend, a commented example is available.
Then, provision or re-provision your infrastructure. Ansible will only change file related to OpenSSL and Nginx
configurations.
6.6. Connect to a vagrant box through ssh
57
IRMA Documentation, Release 1.5.3
Speed up your Vagrant VMs
Install this softwares:
• vagrant-cachier (more info on vagrant-cachier)
$ vagrant plugin install vagrant-cachier
• vagrant-vbguest (more info on vagrant-vbguest)
$ vagrant plugin install vagrant-vbguest
58
Chapter 6. Frequently Asked Questions
CHAPTER
7
Resources
• Project website
• IRC (irc.freenode.net, #qb_irma)
• Twitter (@qb_irma)
59
IRMA Documentation, Release 1.5.3
60
Chapter 7. Resources
CHAPTER
8
Screenshots
Command Line Interface
A sample script can be found in frontend repository. Add your own frontend address before testing it.
61
IRMA Documentation, Release 1.5.3
Web Interface
Some screenshots of the irma user interface shipped with frontend package.
62
Chapter 8. Screenshots
IRMA Documentation, Release 1.5.3
8.2. Web Interface
63
IRMA Documentation, Release 1.5.3
64
Chapter 8. Screenshots
IRMA Documentation, Release 1.5.3
8.2. Web Interface
65
IRMA Documentation, Release 1.5.3
66
Chapter 8. Screenshots
Was this manual useful for you? yes no
Thank you for your participation!

* Your assessment is very important for improving the work of artificial intelligence, which forms the content of this project

Download PDF

advertising