Conformance testing is required to ensure that a device meets the specified aspects of a given standard. In today's marketplace, wireless conformance standards are paramount for both consumer and industrial devices, ensuring that those mobile phones, tablets, cars, and a range of other wireless clients communicate reliably over LTE, GSM, W-CDMA, 5G and also 802.11 networks. Before a device is launched commercially, it must meet all the required specifications defined for the specific technology.
Recently, KeySight field engineers met with a company that produces mobile-wireless infrastructure equipment. The testing team there had been using a legacy in-house testing platform for several years, but with each new device they released, the testing framework required unwieldy amounts of customization and updating to handle the requirements of evolving functional and conformance testing. The legacy bespoke testing platform could not keep up with agile software life-cycles, with rapidly increasing functionality in each new product release and across new product lines, and with ever-more stringent conformance tests.
When the Keysight team met with development and testing teams, the customer told KeySight that their testing efforts were falling further and further behind. Indeed testing had become the main bottleneck in their product life-cycles.
But what to do?
Background
The customer was very experienced in designing wireless access points but was encountering incrementally greater challenges in testing them. This problem came to a head when they needed to perform both functional testing and wireless conformance tests.
Moreover, the customer faced two steep learning curves if they wanted to move forward:
Minimal programming skills in both C# and Python to build the out the necessary tooling
Limited expertise in RF conformance testing (vs. functional product testing)
Testing Domain and Emphasis
The main wireless protocol of the Device under test was IEEE 802.11ac-2013, which is sharing 5GHz spectrum - and mandatory specifications to control included 256-QAM modulation support alongside with 160 MHz channel banwidth. An additional technology stack was based on cellular SIM-connectivity, so 3GPP conformance testing was also of great interest with test cases derived from TS 36.521-1 specification.
Their legacy testing practice was design to support manually-invoked tests across the product lifecycle and obviously needed automation. Drivers for automating included speed and repeatability.
Moreover, the customer realized that having a single hardware toolset to control each stage of their test operations was highly desirable. But the specific requirement (based on throughput needs) was an ability to customize the number of tests at each stage: heavily testing after the development phase, shifting to less intensive RF qualification but more functional KPI testing after Functional QA, leaving only mandatory RF/calibration testing for final QA.
Test Code Base
The customer test code base consisted of
C#/C++ based application originally supplied by their wireless chipset manufacturer for device calibration. These tests ran using AT-commands over a COM port.
Functional test automation code written in-house in Python - simple IP-throughput testing and higher-level verification routines for SSID and MAC control
Tests were applied manually at each stage of the product lifecycle, causing increasing delays.
Introducing OpenTAP
The company's development and test teams were not comfortable walking away from a functional testing solution that had served them across half a dozen years and through countless releases, even if it was now their principal headache. The KeySight team invited them to sit down for a Q&A session to address their main concerns about a possible move to OpenTAP.
| Customer Questions | OpenTAP Responses |
| Does my team need to learn tons of VISA libraries to control instrumentation? | There are dozens of off-the-shelf instrument plugins for OpenTAP, from the OpenTAP ecosystem, including those from KeySight, so your learning curve doesn't have to be a steep one. |
| What if we have to use both Python and C#? Our chipset test code utilizes C# and our access point functional control code is in Python | Both languages happily co-exist in one test plan and OpenTAP features SDKs for both languages. |
| OpenTAP is new to our team. Who can provide support for the platform, especially with debugging? | OpenTAP is open source with a worldwide user/developer community, ready to help via the OpenTAP Forum, the project website and the project repository. Moreover, KeySight offers commercial support for the company's test instruments built with OpenTAP. |
| How about examples of test steps and test code? | There are plenty of examples of other programs created by engineers at KeySight and in the OpenTAP community |
| What about support for RF conformance testing? | KeySight conformance test equipment (e.g., the P7000A Base Station Measurement Automation Solution) interoperates with OpenTAP out-of-the-box. |
| This sounds like it could get expensive - we're on a tight budget | The OpenTAP engine, C# SDK and Python SDK are all open source, check out the project today! |
| Wow, OpenTAP sounds almost too good to be true - how can we get started? | Visit OpenTAP.io and dig into the documentation at Getting Started |
Conclusion
Even in the hardware-centric world of network infrastructure and of testing its various components and the clients on those networks, software has become the central focus. All too frequently, testing software is also the main bottleneck in product delivery.
Real-world organizations (including the one described in this blog) are happily discovering that OpenTAP can help streamline testing during development and functional and conformance testing that follows. If your organization is looking accelerate software and hardware test, to make testing more reliable and repeatable, and to scale your test bench to match increasing product complexity, check out OpenTAP.