What's a Runner"
Broadly speaking, in test automation, a runner (a.k.a. a test runner) is software component responsible for executing automated test cases and reporting results. Runners commonly orchestrate the process of running tests: discovering which tests to run, setting up the necessary environments, executing tests, and collecting and reporting outcomes, including passed, failed, or skipped tests and errors encountered.
OpenTAP Runners
A runner in the context of OpenTAP is a specialized plugin or component that facilitates the execution of test plans, often in distributed or remote environments. Runners act as agents that can execute OpenTAP test plans on different machines or platforms, enabling automation workflows to scale beyond a single local system.
In particular, under OpenTAP, a runner supports remote starting and control of OpenTAP sessions. These sessions can create, load, and execute test plans, modify test plan and test step properties, adjust OpenTAP settings, and provide logs and results.
OpenTAP Runners are implemented using NATS. NATS stands for Neural Autonomic Transport System, and is an open source messaging system or message-oriented middleware, a project under CNCF (Cloud Native Computing Foundation).
Why Are Runners Necessary?
Runners are necessary for several key reasons:
Distributed Execution: In complex test environments, such as those used in large-scale network testing or manufacturing, tests need to be executed across multiple devices, systems, or geographic locations. Runners enable this by acting as distributed agents that can run tests remotely.
Scalability: By using multiple runners, organizations can execute large batteries of tests in parallel across different platforms, dramatically reducing the total time required for comprehensive testing. For example, in the 6G-SANDBOX project, using multiple runners reduced a month-long manual testing process to just 24 hours.
Remote Access: Runners make it possible to access and control devices under test (DUTs) that are not physically co-located with the main test controller, supporting remote and automated workflows. With cloud-based commercial environments (KS8500B, etc.), there is no such thing as "local" execution: all execution stations are remote, making runners an absolute necessity. With premises products (KS8400B, etc.) and with command-line invocation of OpenTAP, runners enable remote/distributed test runs.
Automation and Orchestration: Runners integrate with the OpenTAP core engine to automate the setup, execution, and teardown of experiments, ensuring repeatability and reliability in test execution.
Are Runners Available as Open Source?
Yes, OpenTAP runners are available as open source. The OpenTAP core engine and its plugins, including runners, are developed and shared openly to encourage community contributions and extensibility. The entire OpenTAP project, including its core engine and plugins, is hosted on GitHub and distributed under an open source license. This open approach allows users to adapt, modify, and extend runners and other components to fit their specific test automation needs.
In summary, runners in OpenTAP are open source components designed to facilitate distributed test execution, and they leverage open source technologies like NATS for communication.
Learn more about creating and using runners from the OpenTAP Runner Documentation pages.