AMTSO, a cybersecurity industry testing standard community, has published its first Guidelines for Testing of IoT Security Products.

The guidelines cover principles for the testing of IoT security products providing recommendations on test environment, sample selection, testing of specific security functionality, and performance benchmarking for testers.

General principles include all tests and benchmarks focusing on validating end results and performance of protection delivered, instead of how products function at the backend.

The guidelines provide guidance for challenges with choosing the right samples for IoT security solution benchmarking. For a relevant test, testers need to select samples that are still active, and that actually target the operating systems smart devices are running on. 

IoT security solutions work very differently from traditional IT-based cybersecurity products when it comes to detection and actions to be taken. For example, some solutions will simply detect and prevent a threat without notifying the user. The guidelines suggest to use threats with admin consoles that can be controlled by the tester or to use devices where the attack will be visible if conducted. 

The guidelines also cover the recommended IoT test environment. In an ideal case, all tests and benchmarks would be executed in a controllable environment using real devices. However, the setup can be complex, and if the tester decides against using real devices in the testing environment, it is advised that they should validate their approach by running their desired scenario with the security functionality of the security device disabled and checking the attack execution and success. 

For testing of specific security functionality, the guidelines embrace advice on different attack stages, including reconnaissance, initial access, and execution. They outline the possibility to test each stage individually vs going through the whole attack at the same time. Choices on this should be documented in the testing methodology.

The guidelines also cover IoT cybersecurity performance benchmarking, covering considerations such as suggesting to differentiate between various use cases such as consumers vs businesses, or the criticality of latency or reduced throughput per protocol, which depends on its purpose. 

The guidelines were approved by the AMTSO membership in June 2022.

AMTSO is a cybersecurity industry testing standard community, consisting of over 60 security and testing member companies from around the world. The organisation offers a platform for knowledge-sharing and collaboration on objective standards and best practices for anti-malware testing and assessment of other cybersecurity products.