Fortinet, Inc. FortiMail-100™ Anti-spam Effectiveness and Feature

Fortinet, Inc. FortiMail-100™ Anti-spam Effectiveness and Feature
No. 207258
December 2007
Fortinet, Inc.
Anti-spam Effectiveness and Feature
Comparison vs. Barracuda Networks Spam Firewall 200
Premise: E-mail security appliances
designed to thwart phishing attacks,
viruses and spam must be able to
deliver exceptionally high rates of spam
blockage combined with low levels of
“false positives” and “false negatives”
to be effective. Those E-mail security
products should have a robust set of
features that make them easy to deploy
and configure, plus offer a simple licensing model for enterprises and service
ortinet, Inc. commissioned The
Tolly Group to measure the
effectiveness of the company’s
FortiMail-100™ multi-layered
E-mail security appliance at blocking
spam and virus messages.
Tolly Group engineers tested the
spam detection effectiveness of the
FortiMail-100 against a Barracuda Networks Spam Firewall 200. In accordance with The Tolly Group’s Fair Testing Charter, Barracuda was invited to
review the test methodology, offer suggestions for its product and comment on
its results.
Engineers measured the percentage of
spam blocked, the number of “false
positives,” “false negatives” and virus
messages detected per product. Engineers also validated a number of E-mail
security features and deployment
Tests were conducted in October and
November 2007.
© 2007 The Tolly Group
Test Highlights
Blocks 99.87% of more than 21,000 inbound messages
containing spam
Generates 2/3 fewer false positives and 2/3 fewer false
negatives (missed spam) than the Barracuda device tested
Offers more features and functionality over the Barracuda
Spam Firewall 200
%&#$# #%
%&#&% #%
&!! #% #&%$% # # "&#%
&!! #% #$#'# !#% &!! #% ##$!#% ! *%
&!! #% #!#&$# )
&$% +#
&!! #% #!#&$# )! *
&!! #% #%#%%#
&!! #% #
#% &!! #% #%(* !#% &!! #% # &#
$$$ $! *%#$
Source: The Tolly Group, November 2007
Figure 1
Page 1
Fortinet, Inc.
Tests show that the Fortinet
FortiMail-100 is more accurate at blocking spam than
the Barracuda Networks
Spam Firewall 200 and is
more effective at combatting
“false positives” and “false
Service providers and enterprise
network architects can benefit from
adoption and deployment of integrated tools that offer protection
from E-mail abuse. The FortiMail100 integrates support for antispam, anti-virus, malware prevention and E-mail policy enforcement
into a single product.
This test focuses on FortiMail100’s anti-spam/anti-virus
effectiveness, particularly when
compared to the Barracuda Networks Spam Firewall 200.
FortiMail-100 accurately detected
21,179 spam messages and three
virus messages, while the Barracuda device detected 10,825 spam
and 21 virus messages. The delta in
the total messages processed is due
to differences in how each product
blocks spam. The Barracuda product immediately blocks all messages from a source IP address that
is blacklisted. The FortiMail-100
uses session limit and temporary
failure reply messages to validate
the sender. These temporary failure
reply messages are often ignored
by spammers but, occasionally,
they are replied creating a higher
number of total inbound messages.
The Tolly Group examined a variety of features and functions available on the FortiMail-100 and to
some extent on the Barracuda
Spam Firewall 200. (See Fig. 1.)
Built-in Storage for
Local Quarantine
Tolly Group engineers verified that
© 2007 The Tolly Group
Spam Detection Percentage of FortiMail-100 vs.
Barracuda Spam Firewall 200 Appliance
Based on 21,179 and 10,825 Live Corporate
Spam Messages Received*
% of spam messages
blocked correctly
* NOTE: Both devices were tested in a 24-hour period from 6:00 p.m. to 6:00
p.m. FortiMail-100 was tested from October 24th to October 25th. The Barracuda Spam Firewall 200 was tested from November 1st through November 2nd.
Source: The Tolly Group, November 2007
Figure 2
/,++. !-0)+%,*'$+!0&&
)-&().( !,,#!,!,-! /4",0"0"-,/$+&%/22&$4,8
++. *'$+!0&&
&"$)0,"4'/2- )&/24*"*,7"34&34&%%52*.(4)&%"4&3/'$4/#&2
.%'2/-0-4/0- )&42"''*$*.$,5%&%2&",7/2,% /,,82/50
Source: The Tolly Group, November 2007
Figure 3
Page 2
Fortinet, Inc.
the FortiMail-100 provides built-in
storage space — up to 250 GB —
11GB of which was used to save
locally quarantined messages that
were considered spam. The Barracuda Spam Firewall 200 does not
support local quarantine capabilities; the system does allow IT
administrators to setup an external
quarantine directory.
Server Mode
The Tolly Group examined that
server mode deployment is supported by the FortiMail-100 but not
by the Barracuda device. Server
mode enables the FortiMail-100 to
act as an E-mail server in the network to provide anti-spam and
anti-virus filtering, along with
E-mail services.
Transparent Mode
The Tolly Group confirmed that
inline mode, or transparent mode,
is supported by the FortiMail-100
but not by the Barracuda Spam
Firewall 200. Transparent mode is
used to perform spam filtering and
anti-virus scanning without modifying the MX records and network
topology in the existing infrastructure.
Support Per User/
Mailbox Quarantine
Engineers verified that the
FortiMail-100 supports a local peruser or per-mailbox quarantine
directory in the system. This allows
IT administrators to access an individual quarantine directory of each
user in the network easily instead
of having a single quarantine directory for all users. The Barracuda
Spam Firewall 200 does not provide local quarantine directory per
user or per mailbox. Barracuda
does support external quarantine
Customizable E-mail
Engineers verified that the
FortiMail-100 allows IT administrators to change the appearance of
the FortiMail-100 system by customizing the titles, company logos
© 2007 The Tolly Group
and company’s URL. The Barracuda Spam Firewall 200 does not
provide this feature.
Policy Management
The Tolly Group verified that the
FortiMail-100 provides policy control on a per-mailbox basis, as well
as on a domain basis. The Barracuda Spam Firewall 200 does not.
E-mail Anti-spam
Support for Ethernet
Interface Redundancy
Tolly Group engineers verified
whether each anti-spam/anti-virus
system provides multiple Ethernet
interfaces that offer the flexibility
to assign Ethernet interfaces to run
the anti-spam/anti-virus filters and
separately, assign an Ethernet interface as a management port to
remotely control the system. The
FortiMail-100 provides four 10/100
Ethernet interfaces, while the
Barracuda Spam Firewall 200
offers just one Ethernet interface.
Gateway Mode
The Tolly Group’s hands-on
evaluation shows that both the
FortiMail-100 and Barracuda Spam
Firewall 200 support gateway mode
deployment. This enables either
device to act as a gateway in front
of backend E-mail servers to filter
incoming messages for viruses and
LDAP Recipient
Engineers verified whether each
anti-spam/anti-virus system supported LDAP recipient verification
with The Tolly Group’s corporate
mail server. This feature minimizes
the amount of inbound messages
received by the back-end mail
server by verifying whether the
recipient address exists or not in the
network. FortiMail-100 does support this feature; however, Barracuda Spam Firewall 200 does not.
SMTP Policy
The FortiMail-100 supports configurable SMTP filters based upon
policy attributes. This enables
administrators to set incoming
Product Specifications
Vendor-supplied information not
necessarily verified by The Tolly Group
Fortinet, Inc.
Multi-layered Email Security
Anti-spam, Anti-virus,
Anti-phishing and spoofing
Inbound and outbound
email filtering
Anti-spam filters include:
Global Sender IP
SMTP protocol and
session filters
Image Analysis
Bayesian Filtering
PDF file scanning
Heuristic rules filtering
Localized sender
Deep header scanning
No per user/mail box
antispam and antivirus
Built-in email quarantine
and content archiving
Built-in email server for
small business deployment
Multiple Tiered
Administration Domain
For more information contact:
Fortinet, Inc.
1090 Kifer Road
Sunnyvale, CA 94086
Phone: (408) 235-7700
Fax: (408) 235-7737
Page 3
Fortinet, Inc.
session rates and administer policy
filtering. The Barracuda Spam
Firewall 200 also offers SMTP
policy configuration.
Spam Blockage
Engineers measured the products
tested to detect incoming spam
accurately and to block spam messages correctly. (See Figure 2.)
Tests show that out of 21,179
spam-based and virus-laden messages received, the FortiMail-100
correctly blocked 21,151 (99.87%)
of the spam messages. Both devices
were tested with live E-mail messages to ensure that reputation and
session-based blocking functioned
The Barracuda Spam Firewall 200
handled 10,825 total inbound spam
FortiMail-100 Anti-Spam Filtering Results
Based on 21,598 Incoming Live Corporate Messages Received
and virus messages, and detected
10,731 of those as spam. It correctly blocked 99.13% of the spam
messages. (See Figure 3.) This
demonstrates that the FortiMail100 is more accurate and effective
at blocking spam as the Barracuda
Spam Firewall 200.
“False Negatives”
When an anti-spam product misses
incoming spam E-mails, treating
them as “legitimate,” it generates
“false negatives.”
With 21,179 spam messages sent,
engineers found 28 false negatives
on the FortiMail-100, generating a
false negative rate of 0.13%. The
Barracuda Spam Firewall 200, with
10,825 total spam messages,
yielded 94 false negatives for a
false negative rate of 0.87%. — 3X
more false negatives.
“False Positives”
Spam blocked
Legitimate messages correctly identified
Spam missed (False negatives)
False positives
Figure 4
Source: The Tolly Group, November 2007
Barracuda Spam Firewall 200
Anti-Spam Filtering Results
Based on 11,468 Incoming Live Corporate Messages Received
Spam blocked
Legitimate messages correctly identified
Spam missed (False negatives)
False positives
© 2007 The Tolly Group
The 44 E-mails identified as false
positives by the Barracuda Spam
Firewall 200 were legitimate newsletters that Tolly Group engineers
normally receive. These E-mails
were identified as “spam” by the
Barracuda Spam Firewall’s Intent
Analysis blocklist based on a spam
scoring setup – which was the
default. After testing concluded,
Barracuda Networks eliminated the
newsletter domains from the blocklist.
On the FortiMail-100, 11 of the 15
E-mails identified as false positives
by the FortiMail-100 were newsletters with sender IPs blacklisted.
Fortinet has taken the action to
re-evaluate those IP addresses in
their database to prevent unnecessary false positives.
Source: The Tolly Group, November 2007
When an anti-spam product incorrectly identifies legitimate incoming E-mails as “spam” and then
blocks those messages, the result is
“false positives.” For the FortiMail100 and Barracuda Spam Firewall
200, engineers identified 15 and 44
“false positives,” out of 1,757
quarantined messages and 2,229
quarantined messages, respectively
for each platform. (See Figures 3, 4
and 5.)
Figure 5
Page 4
Fortinet, Inc.
Both the FortiMail-100 and the
Barracuda Spam Firewall 200 utilize their own databases to determine if the sending IP address is
reputable. Based on that information, the products can block the
vast majority of spam messages at
the connection level without incurring the overhead of receiving the
message and passing it on to the
next level of spam detection. The
FortiMail-100 supports an automatic spam submission service to
enhance the content of the reputation database service. A whitelist
feature also is supported in the
event a receiver still wishes to
receive E-mails from a blacklisted
Methodology &
Tolly Group engineers tested Fortinet’s FortiMail-100 E-mail security
platform version (3.00 Build 143,
071019) equipped with four 10/100
Ethernet ports and 250 GB (gigabytes) of storage capacity, against a
Barracuda Networks Spam Firewall 200 version (
equipped with one 10/100 Base-T
Ethernet port and 1 GB of storage.
Tolly Group engineers tested both
platforms with a live E-mail stream
of messages in order to test each
device capabilities and behavior
when they are deployed in a live
network. This way, all inbound
messages were kept intact without
modifying sender information and/
or SMTP session state.
firewall to route all incoming mail
traffic through the appliance. (See
Figure 6.)
Both platforms used out-of-the-box
configurations for anti-spam filtering. Engineers enabled the Quarantine feature of both products tested
to quarantine spam.
Tolly Group engineers also used the
LDAP query feature available on
the FortiMail device to run recipient verification on The Tolly
Group’s Active Directory server.
This feature was not available on
the Barracuda device. Engineers
made sure that Internet access was
made available to each platform to
download any newly available antispam and anti-virus definitions, or
new firmware updates.
Testing was conducted in succession, meaning that engineers first
deployed the FortiMail-100 and
then switched the platform to the
Barracuda Spam Firewall 200.
Each platform processed all inbound messages for a 24-hour
period. Inbound messages that were
identified as spam were tagged as
“spam” and saved into a Quarantine directory for engineers to
verify. Other E-mails were delivered to The Tolly Group’s corporate
mail server.
Anti-Spam Blocking
Once messages were scanned by
each solution, some of the messages were tagged as “spam” and
Test Bed
Tolly Group engineers deployed
the FortiMail-100 and the Barracuda Spam Firewall 200 E-mail
security platforms in The Tolly
Group’s corporate network each as
the main E-mail security gateway
for anti-spam and anti-virus filtering. Engineers configured the
FortiMail-100 in “gateway” mode
by connecting one of the ports to
the “public network” and the second port to the “private network.”
The Barracuda device has only one
Ethernet interface so it had to be
connected to the private network
side and engineers configured the
During the manual classification,
engineers determined whether messages were “unsolicited” or not by
checking the following criteria:
porn, sex, prescription drugs (i.e.
Viagra), gibberish language, “no
page found” links, no content, etc.
Engineers considered a message to
be a “false positive” whenever a
legitimate message was found in
the quarantine directory. Regarding
“false negatives,” engineers manually checked each employee’s
mailbox for unsolicited messages
that were not classified as spam.
In the case for “false negatives,”
engineers manually checked each
employee’s mailbox in Exchange
Server for any possible spam/virus
messages. Given that the entire
organization was involved in this
process directly or indirectly, there
was an inevitable possibility that
engineers missed some spam messages. For instance, an employee
could accidentally delete the spam
messages before the engineers
checked out the mailbox.
Even if there was error in the “false
negative” result, the error rate cannot possibly exceed 50%. This results in increasing about 15 more
“false negatives” as shown in Figure 3. The Tolly Group considers
that this possible error ratio would
not have any noticeable impact on
our main findings.
Test Bed Diagram
Source: The Tolly Group, November 2007
© 2007 The Tolly Group
filtered out in a quarantine directory to allow engineers to verify
manually whether the message
blocked was spam or not.
Figure 6
Page 5
Fortinet, Inc.
Fair Testing Charter™
Interaction with Competitors
The Tolly Group contacted Barracuda Networks
in October 2007 and invited the company to participate in the test. Barracuda Networks was
invited to review the test plans, the product levels and configurations of the company’s product and to review and comment on
results specific to tits Barracuda Spam Firewall 200. A Barracuda
Networks representative provided insight into tuning the configuration and updating firmware of the Barracuda Spam Firewall 200.
Barracuda Networks reviewed the results of its Barracuda Spam
Firewall 200. Barracuda Networks confirmed the accuracy of the
results in December 2007.
For more information on this process, please see:
The Tolly Group is a leading global provider of thirdparty validation services
for vendors of IT
components and
The company is based in Boca
Raton, FL and can be
reached by phone at (561)
391-5610, or via the
Internet at,
Device Under Test Specifications
Anti-Virus version
Fortinet, Inc.
3.00 Build 143,
Fortinet Anti-Virus Engine (Ver. 2.91, FortiGuard Anti-Virus Definition (Ver. 8.365)
Barracuda Spam
Firewall 200
© 2007 The Tolly Group
Page 6
Was this manual useful for you? yes no
Thank you for your participation!

* Your assessment is very important for improving the work of artificial intelligence, which forms the content of this project

Download PDF