Anti-Virus Comparative Proactive/retrospective test (on-demand detection of virus/malware)

Anti-Virus Comparative Proactive/retrospective test (on-demand detection of virus/malware)
Anti-Virus Comparative - Proactive/retrospective test – February/May 2010
Anti-Virus Comparative
Proactive/retrospective test
(on-demand detection of virus/malware)
Language: English
February/May 2010
Last revision: 5th June 2010
www.av-comparatives.org
–1-
www.av-comparatives.org
Anti-Virus Comparative - Proactive/retrospective test – February/May 2010
www.av-comparatives.org
Content
1. Introduction
3 2. Description
3 3. Test results
4 4. Summary results
7 5. False positive/alarm test
7 6. Certification levels reached in this test
8 7. Copyright and Disclaimer
9 –2-
Anti-Virus Comparative - Proactive/retrospective test – February/May 2010
www.av-comparatives.org
1. Introduction
This test report is the second part of the February 2010 test1. The report is delivered begin of June
due the high-required work, deeper analysis and preparation of the retrospective test-set.
Many new viruses and other types of malware appear every day, this is why it’s important that AntiVirus products not only provide new updates, as often and as fast as possible, but also that they are
able to detect such threats in advance (also without executing them) with generic and/or heuristic
techniques. Even if nowadays most Anti-Virus products provide daily, hourly or cloud updates, without
heuristic/generic methods there is always a time-frame where the user is not reliably protected.
The products used the same updates and signatures they had the 10th February, and the same highest2
detection settings were used as in February. This test shows the proactive detection capabilities that
the products had at that time. We used new malware appeared between the 11th and 18th February
2010. The following 20 products were tested:
• avast! Free3 Antivirus 5.0
• Kingsoft AntiVirus 2010
• AVG Anti-Virus 9.0
• McAfee AntiVirus Plus 2010
• AVIRA AntiVir Premium 9.0
• Microsoft Security Essentials 1.0
• BitDefender Anti-Virus 2010
• Norman Antivirus & Anti-Spyware 7.30
• eScan Anti-Virus 10.0
• Panda Antivirus Pro 2010
• ESET NOD32 Antivirus 4.0
• PC Tools Spyware Doctor with Antivirus 7.0
• F-Secure Anti-Virus 2010
• Sophos Anti-Virus 9.0
• G DATA AntiVirus 2010
• Symantec Norton Anti-Virus 2010
• K7 TotalSecurity 10.0
• Trend Micro AntiVirus plus AntiSpyware 2010
• Kaspersky Anti-Virus 2010
• Trustport4 Antivirus 2010
2. Description
Anti-Virus products often claim to have high proactive detection capabilities – far higher than those
reached in this test. This is not just a self-promotional statement; it is possible that products reach
the stated percentages, but this depends on the duration of the test-period, the size of the sample
set and the used samples. The data shows how good the proactive detection capabilities of the scanners were in detecting new threats. Users should not be afraid if products have, in a retrospective
test, low percentages. If the anti-virus software is always kept up-to-date, it will be able to detect
more samples. For understanding how the detection rates of the Anti-Virus products look with updated signatures and programs, have a look at our regular on-demand detection tests. Only the ondemand detection capability was tested. Some products may be had the ability to detect some samples e.g. on-execution or by other monitoring tools, like behaviour-blocker, etc. Those kinds of additional protection technologies are considered by AV-Comparatives in e.g. dynamic tests.
1
http://www.av-comparatives.org/images/stories/test/ondret/avc_report25.pdf
except AVG, AVIRA, F-Secure and Sophos; see comments in the February 2010 test report or on page 6
3
Avast Software decided to participate in the tests with their free product version
4
Based on two engines (AVG and Bitdefender)
2
–3-
Anti-Virus Comparative - Proactive/retrospective test – February/May 2010
www.av-comparatives.org
3. Test Results
Note: If you are going to republish those results, it is compulsory to include a comment that products
use also additional protection features (like behavior-blockers, etc.) to protect against completely
new/unknown malware. As described on previous and next pages, this test evaluates only the heuristic/generic detection of the products against unknown/new malware, without the need to execute it.
–4-
Anti-Virus Comparative - Proactive/retrospective test – February/May 2010
www.av-comparatives.org
The below table shows the proactive on-demand detection capabilities of the various products, sorted
by detection rate. The given awards (see page 8 of this report) are based not only on the detection
rates over the new malware, but also considering the false alarm rates.
The retrospective test is performed using passive scanning and demonstrates the ability of the products under test to detect new malware proactively, without being executed. In retrospective tests „inthe-cloud” signatures are not considered, as well it was not considered how often or how fast new
updates are delivered to the user, as that is not the scope of the test.
As it can be seen above, most products are able to detect a quantity of completely new/unknown
malware proactively even without executing the malware, using passive heuristics, while other protective mechanisms like HIPS, behavior analysis and behavior-blockers, etc. add an extra layer of protection.
We tried to include in the test-set only prevalent real-world malware that has not been seen before
the 10th February 2010 by consulting telemetry / cloud data collected and shared within the AV industry. Consulting that data was quite interesting for us, as it showed that, while some vendors had
seen some malware already many months or even years ago, the same malware hashes appeared in
some other vendors clouds only recently.
–5-
Anti-Virus Comparative - Proactive/retrospective test – February/May 2010
www.av-comparatives.org
Nowadays, hardly any Anti-Virus products rely purely on “simple” signatures anymore. They all use
complex generic signatures, heuristics etc. in order to catch new malware, without needing to
download signatures or initiate manual analysis of new threats. In addition, Anti-Virus vendors continue to deliver signatures and updates to fill the gaps where proactive mechanisms initially fail to
detect some threats. Anti-Virus software uses various technologies to protect a PC. The combination
of such multi-layered protection usually provides good protection.
To avoid some frequent questions, below are some notes about the used settings (scan of all files etc.
is always enabled) of some products, whereas highest settings were not used on vendors request:
F-Secure, Sophos: asked to get tested and awarded based on their default settings (i.e. without using
their advanced heuristics / suspicious detections setting).
AVG, AVIRA: asked to do not enable/consider the informational warnings of packers as detections.
–6-
Anti-Virus Comparative - Proactive/retrospective test – February/May 2010
www.av-comparatives.org
4. Summary results
The results show the proactive (generic/heuristic) on-demand5 detection capabilities of the scan engines against new malware. The percentages are rounded to the nearest whole number. Do not take
the results as an absolute assessment of quality - they just give an idea of who detected more, and
who less, in this specific test. To know how these anti-virus products perform with updated signatures, please have a look at our on-demand tests of February and August. Readers should look at the
results and build an opinion based on their needs. All the tested products are already selected from a
group of very good scanners and if used correctly and kept up-to-date, users can feel safe with any of
them. Below you can see the proactive on-demand detection results over our set of new malware appeared within about one week:
ProActive detection of new malware:
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
Trustport, Panda
G DATA
Kaspersky, Microsoft
AVIRA
ESET NOD32, F-Secure
BitDefender, K7, eScan
Symantec
McAfee
AVG
Sophos
Avast
Norman
Trend Micro
PC Tools
Kingsoft
63%
61%
59%
53%
52%
50%
43%
38%
34%
32%
29%
27%
26%
17%
11%
5. False positive/alarm test
To better evaluate the quality of the detection capabilities, the false alarm rate has to be taken into
account too. A false alarm (or false positive)6 is when an Anti-Virus product flags an innocent file to
be infected when it is not. False alarms can sometimes cause as much troubles like a real infection.
The false alarm test results were already included in the test report Nr. 25. For details, please read the
report available at http://www.av-comparatives.org/images/stories/test/fp/avc_report25_fp.pdf
5
6
Very few false alarms (0-3):
eScan, F-Secure, Bitdefender, Microsoft, ESET
Few false alarms (4-15):
Sophos, Kaspersky, G DATA, PC Tools, Trustport, AVG, Avast,
Symantec, AVIRA
Many false alarms (over 15):
Trend Micro, Panda, McAfee, Norman, Kingsoft, K7
this test is performed on-demand – it is NOT an on-execution/behavioral test.
All discovered false alarms were already reported to the vendors in February and are now already fixed.
–7-
Anti-Virus Comparative - Proactive/retrospective test – February/May 2010
www.av-comparatives.org
6. Certification levels reached in this test
We provide a 3-level-ranking-system (STANDARD, ADVANCED and ADVANCED+). Overviews of levels
reached in previous main tests can be found on our website7.
The following certification levels are for the results reached in the retrospective test:
CERTIFICATION LEVELS
PRODUCTS
TrustPort
G DATA
Kaspersky
Microsoft
AVIRA
ESET NOD32
F-Secure
BitDefender
eScan
Panda*
K7*
Symantec
AVG
Sophos
Avast
McAfee*
Norman*
Trend Micro*
PC Tools
Kingsoft*
*: Products with “many” false alarms were rated according to the below award system:
7
0‐10% Proactive Detection Rates 10‐25% 25‐50% 50‐100% None ‐ Few FP tested STANDARD ADVANCED ADVANCED+ Many FP tested tested STANDARD ADVANCED http://www.av-comparatives.org/comparativesreviews/main-tests/summary-reports
–8-
Anti-Virus Comparative - Proactive/retrospective test – February/May 2010
www.av-comparatives.org
7. Copyright and Disclaimer
This publication is Copyright © 2010 by AV-Comparatives e.V. ®. Any use of the results, etc. in whole
or in part, is ONLY permitted after the explicit written agreement of the management board of AVComparatives e.V., prior to any publication. AV-Comparatives e.V. and its testers cannot be held liable
for any damage or loss, which might occur as result of, or in connection with, the use of the information provided in this paper. We take every possible care to ensure the correctness of the basic data,
but no representative of AV-Comparatives e.V. can he held liable for the accuracy of the test results.
We do not give any guarantee of the correctness, completeness, or suitability for a specific purpose of
any of the information/content provided at any given time. No one else involved in creating, producing or delivering test results shall be liable for any indirect, special or consequential damage, or loss
of profits, arising out of, or related to, the use or inability to use, the services provided by the website, test documents or any related data. AV-Comparatives e.V. is an Austrian Non-Profit Organization.
AV-Comparatives e.V. (June 2010)
–9-
Was this manual useful for you? yes no
Thank you for your participation!

* Your assessment is very important for improving the work of artificial intelligence, which forms the content of this project

Related manuals

Download PDF

advertising