Like any software project or product, the OpenTAP test automation platform benefits from rigorous test and quality assurance (QA) as part of its build and release process. Recently, the OpenTAP.io team spent time with KeySight QA and Release Manager Sebastian Pop-Vlaic, chatting about how the OpenTAP project team and the Keysight organization go about testing OpenTAP.
Interview
OpenTAP: What is our general approach to testing and QA of OpenTAP? How many test cases are in the test code base – regression tests, etc.?
Pop-Vlaic: Before each OpenTAP build reaches the QA Team, over one-thousand (1,000+) unit tests are executed as part of the build process for each OpenTAP version. In addition to these unit tests, an extensive 83-step test plan runs across all three supported operating systems: Windows, Linux, and Mac. During this process, we also verify that no breaking changes have been introduced.
The QA Team then verifies each ticket associated with the current milestone, whether it involves a feature, bug fix, or enhancement. Once all tickets have been validated, a complete test run, consisting of 23 manual tests, is conducted on both Windows and Linux (with Mac support currently in progress). Since the SDK plugin is released concurrently, it undergoes a separate full test run consisting of 52 manual tests.
It’s worth noting that the SDK's full test run includes more tests than OpenTAP's. This is because, on its own, OpenTAP is limited in testability. However, by utilizing its plugins, we gain access to more OpenTAP functionality, enabling broader testing.
The OpenTAP full test run covers its core commands like "tap run" and "tap package," various features including logging functionality and its limits, and also includes installer testing. OpenTAP installers are created using an OpenTAP plugin called "SmartInstaller," and we validate this process with a dedicated 26-test case run.
Most OpenTAP plugins are stored in the public repository at https://packages.opentap.io/. The repository itself is an OpenTAP plugin and is tested separately using a 46-test case plan to validate features that cannot be tested in other scenarios.
OpenTAP: How does OpenTAP factor into testing Keysight commercial products?
Pop-Vlaic: OpenTAP is typically released alongside the PathWave Test Automation suite (KS8400), where a significant portion of OpenTAP testing is conducted. With a user interface available, we can test many OpenTAP features alongside all UI features within the Test Automation suite. This full test run includes over 700 test cases.
Finally, after OpenTAP is validated and released, a Post-Release test suite is executed to confirm correct versioning, installer deployment, and functionality.
OpenTAP: Is there a publicly available test suite for OpenTAP?
Pop-Vlaic: The unit tests that are executed for OpenTAP are part of the open source project and can be accessed here and here
The QA test suites that we use are in Test Rail, a test case management application. These are not public and currently I cannot give out access.
OpenTAP: Do we eat our own dog food, that is, use OpenTAP to test OpenTAP?
PopVlaic: Somewhat yes, but it's not the only thing in our diet.
Most of the products of the Automation team are by themselves OpenTAP plugins (like Runner, Repository, KS8400 Editor) and OpenTAP is used to build these plugins.
We have a feature available "tap package test" which is used in our Demonstration plugin. Tests are implemented within the plugin, and by running the "test" command, the plugin can be verified that it's functional and compatible with OpenTAP.
OpenTAP: How do we handle community bug reports? What are our metrics?
Pop-Vlaic: The development team monitors the OpenTAP forums and handles communication with users. QA gets involved when user reported issues need to be reproduced and reported.
Sebastian Pop-Vlaic
Sebastian graduated in 2008 from the Technical University of Cluj-Napoca with a bachelors in Telecommunications. That same year, he started his Quality Assurance (QA) career at Wirtek Romania, a local IT company in Cluj-Napoca, where he worked for 15 years, first as a hardware product tester for RTX then moving to software testing, both manual and automated, on various web-based projects. In 2019 he began performing QA on the Test Automation project, OpenTAP.
In the summer of 2024, Sebastian went to work directly with Keysight as QA and Release Manager. Today he is responsible for both the quality of the products by Keysight using OpenTAP as for driving improvements in processes related to product quality and team productivity.