Today (Feb 14th), NSS Labs issued a report, which purported to test the effectiveness of endpoint security products from various IT security companies. We find that the “Advanced Endpoint Protection 1.0” test does not provide an accurate approach to testing endpoint security products in real world scenarios. Further, several other security vendors, including some who performed well on the test, do not agree with the test methodology or the results.
We believe that the purpose of a good test is to provide accurate data so the reader can be truthfully informed and therefore knowledgeable about the resulting performance of the products they are researching. When a testing organization does not adhere to industry standards when testing a product, it renders the results useless for this purpose.
Below are the key reasons why we disagree with the NSS Labs test and believe that it should not be regarded as a true independent 3rd party test:
- The test was conducted in an unrealistic testing environment meaning that significant detection mechanisms of the products were not correctly tested
- NSS repeatedly missed self-imposed deadlines for informing vendors about the test parameters and details
- NSS obstructed our attempts to obtain the sample sets for validation and verification
- Repeated requests for information went ignored until shortly before to publishing, leaving no time to give feedback on the flawed methodology
ESET regularly participates in and fully supports independent 3rd-party testing. For instance, performing consistently well in the real world testing undertaken in SE Labs' Enterprise Anti-Malware Protection test.
We believe good testing improves our products and is essential in providing customers with accurate information. It is paramount to establishing trust and earning credibility in the cybersecurity sector and therefore we feel it is incumbent upon us to convey our disagreement with NSS Labs test methodology and implementation in this case.
-ESET