Open Source Resources for Test Automation

The mission of the OpenTAP Blog is to make connections between test automation and open source software, first and foremost around the OpenTAP platform. But the universe of open source test automation extends beyond our favorite project and includes other test automation platforms and technologies that can complement and extend OpenTAP.

Popular Open Source Test Automation Platforms

There are myriad self-styled "test automation" platforms and frameworks that are developed and distributed as open source. Their use cases cover a broad swatch of functionality, especially testing of web apps, and for the most part, do not compete directly with OpenTAP.

Appium

Appium is an open source project and ecosystem of related software, designed to facilitate UI automation of many app platforms, including mobile (iOS, Android, Tizen), browser (Chrome, Firefox, Safari), desktop (macOS, Windows), TV (Roku, tvOS, Android TV, Samsung), and more!

Appium is implemented in Javascript and distributed under the Apache version 2.0 license.

Key Features

• Uses standard automation APIs

• Supports multiple programming languages

• Ample community resources for troubleshooting and learning

• Extensible via drivers and plugins in Node.js packages

Visit the Appium project repository at https://github.com/appium/appium

Cypress

Cypress is an open source end-to-end testing framework designed for modern web applications. Cypress is known for its developer-friendly approach and real-time reloading features. Cypress is implemented in TypeScript and distributed under the MIT License.

Key Features

• Real-time reloading during test execution

• Easy setup with no dependencies on other tools

• Strong documentation and community support

• Extensible with Cypress plugins

Visit the Cypress repository at https://github.com/cypress-io/cypress.

Gauge

Gauge, a relatively new test-automation tool, is lightweight and cross-platform. it's built on a plugin architecture, facilitating use with multiple languages, IDEs, and ecosystems. Gauge is implemented in Python and distributed under the Apache 2.0 license.

Key features

• Easy setup; you can get a framework up and running with a single command

• Simple language syntax

• Supports modular architecture with myriad extended plugins

• Creates text documentation in simple Markdown

Visit the Gauge repository at https://github.com/getgauge/gauge

JUnit

JUnit is a popular open source framework for unit testing in Java applications. It is widely used for testing individual components to ensure they function correctly. JUnit 5 is the most current version of the framework, which is implemented in Java and distributed under the EPL version 2.0 license.

Key Features:

• Annotations for test organization

• Integration with IDEs like Eclipse and IntelliJ IDEA

• Supports test-driven development (TDD) practices

Visit the JUnit repository at https://github.com/junit-team/junit5/

Robot Framework

Robot Framework is an open-source automation framework that implements a keyword-driven approach for acceptance testing. It supports various use cases, including mobile, desktop, and web test, with applications in acceptance testing, acceptance test driven development (ATDD), and robotic process automation (RPA).

Robot Framework is implemented in Python and distributed under the Apache 2.0 license.

Key Features:

• Integration with Selenium and Appium

• Provides rich integration of APIs, generic text libraries, and tools

• Integrates with Jython (Java) and IronPython (.NET)

• Supports multiple operating systems.

Visit the Robot Framework repository at https://github.com/robotframework/robotframework

Selenium

Selenium is an open source tool for automating test of web applications. Selenium is implemented primarily in Javascript and licensed under the Apache Version 2.0 license. It supports programming and extension in Java, Python, and C#, and allows cross-browser testing across major browsers like Chrome, Firefox, and Safari.

Key Features:

• Highly customizable with integration of varied APIs and languages, including Java, Python, .NET, C#, and Ruby

• Cross-browser testing capabilities

• Integration with CI/CD tools like Jenkins

• Supports parallel test execution

Selenium is actually a collection of sub-projects that include

• WebDriver – drives a browser natively (like an actual user would), locally or remotely

• IDE – a browser extension for recording and playing back test sequences

• Grid – test distribution for parallel testing across browsers and machines

Visit the project repository at https://github.com/SeleniumHQ/selenium

Comparing OpenTAP to Other Open Source TA Projects

The following table provides a very cursory comparison of the main attributes of OpenTAP and the test automation tools called out in the previous section

Use Cases –Most of the projects listed here target web-based test automation. Only JUnit and Robot accommodate other use scenerarios, Java unit testing and general test respectively. By contrast, OpenTAP is designed to run natively on desktop and/or embedded systems, enabling a very range of hw and sw test scenarios. It also runs in the cloud with Keysight commercial products, like the KS8500.

Execution Environments – Web app test by definition needs to run in a browser (or in some cases, on a web server). JUnit can execute wherever a Java Virtual Machine is present (in a browser or almost anywhere else). Robot and OpenTAP can run natively on Windows, macOS and Linux systems.

Implementation – Browser-based execution requires implementation in Javascript (or TypeScript) and can also support execution of Java code. Implementation in Python or C# facilitates running the test engine natively on desktops and embedded systems, and even in the cloud.

License – All of the OSS test automation platforms listed here employ either MIT or Apache licenses for distribution, facilitating modification and redistribution without obligation to expose high-value internal code.

Plugins – all the test automation projects listed here have some sort of plugin architecture.

Extension – The ability to extend the functionality of a test automation platform is key to applications in h/w and s/w QA. OpenTAP is extensible via plugins in C#, Python and .NET.

Lines of Code (LoC) – the size of an open source project can indicate its range of functionality and also its maturity (more functionality aggregated over time). It can also result from cramming too much functionality into a single project (bloat). The largest project described here is Selenium, at 664 KLoC, which actually includes a web driver, an IDE and a distributed test framework. OpenTAP at 97.3 KLoC, and Gauge at 35.5 KLoC, are the smallest projects, but this value does not include the many plugins and composition tools that accompany both.

Contributors – The number of contributors indicates the current size of the developer community and so the health of the project.

Activity – Activity is indicative of the number of commits over the last twelve months (as measured by OpenHub.net).

In-House / Roll-Your-Own Test Automation

The projects described in this blog offer software and hardware test teams a wide range of functionality, off-the-shelf, with many options for customization and extension. Despite the ready availability of these open source options (and a number of closed/proprietary platform, a very large cohort of test teams still opt for building their own testing platforms and tools.

While there are a few (perceived) advantages to rolling-your-own test automation software (e.g., control, local expertise, etc.), open source platforms offer comparable advantages and others:

• Ability to customize to meet local requirements

• Shared development and maintenance responsibility

• Community support and often also commercial support

• Lower cost to acquire and maintain (vs. rolling your own, even over time)

• Faster time to functionality

• Reduced risk of losing expertise through local attrition

• Interoperability with third party hardware and software

Extending and Customizing with Open Source

The languages and frameworks most commonly employed to extend the functionality of test automation platforms are in most cases, open source projects themselves:

• .NET – the free, open source, cross-platform framework for building modern apps and powerful cloud services, sponsored by Microsoft.

• C# – while the C# language is a part of commercial offerings from Microsoft, the language itself is supported by FOSS compiler tools.

• Java – the Java language and JVM runtime are open source projects, sponsored by Oracle, with a global contributor and user community.

• Javascript – the most popular language and run-time engine for creating custom web pages and web apps, features an ecosystem of multiple open source execution engines: V8, JavaScriptCore, SpiderMonkey.

• Node.js – an open source, cross-platform JavaScript runtime environment that lets developers create servers, web apps, command line tools and scripts.

• Python – the Python language and myriad libraries were all created and are maintained as vibrant open source projects.

Conclusion

Open source software projects offer today's test engineers a wide range of powerful and innovative options for automating and streamlining software and hardware test. Even proprietary test automation platforms often benefit from using open source plugins and tools to extend their functionality. The free distribution paradigm of open source software makes it easy to download and evaluate one (or more) test platforms. Might we suggest taking OpenTAP for a test drive today?