Cost Calculation
Calculating and comparing actual costs can be quite challenging. The following sections elaborate considerations for comparing costs for in-house, commercial/proprietary and open source test automation
In-house Test Automation
Cost calculation for legacy in-house test automation platforms is seldom a simple task:
Those platforms usually evolved organically to accompany a product or product line
Costs for in-house test automation are usually embedded in software and hardware development budgets; if itemized at all, they fall into product test and QA/QC budgets
Within QA/QC budgets, the cost of developing and maintaining a test automation platform is almost never itemized because it’s not part of the product specification
Despite substantial investment needed for maintenance and tracking of new product releases, such incremental costs are seldom called out either
Open Source
In comparing the costs for integrating and deploying open source vs. commercial software, it’s good to consider that open source helps you avoid vendor lock-in, provides greater choice and flexibility and most often offers lower costs over its lifetime. Note that commercial open source – software built on open source but obtained from and supported by dedicated vendors – provides a “best of both worlds” option, comprising open source-based products and services available with formal support, service-level agreements, indemnification and other benefits.
Actual costing examples for open source test automation acquired from community sources are laid out in the Appendix.
Commercial / Proprietary
COTS test automation software, and more recently SaaS offerings, superficially offer the simplest cost calculation. Acquisition and support are quotable line items from vendors, with possible volume-based and negotiated discounts. Outlays to acquire and maintain commercial/proprietary software, however, usually run significantly higher than those for open source software, although not as high as costs for custom development.
Cost Calculation Tools
CoCoMo – Cost Constructive Model
CoCoMo (Constructive Cost Model) is a popular software cost estimation model. CoCoMo can estimate or predict the effort required for a project, total project cost and scheduled time for the project. This model depends on the number of lines of code for software product development.
Other Models and Tools
Wideband Delphi / Expert Judge Model
Past Project Model
The last model, based on past projects undertaken in your organization, usually offers the greatest benefit with the simplest most concrete type of data gathering. Even if prior projects did not meticulously manage metrics , you can still recover information about partial or dedicated headcount (FTEs) and other line items from departmental and project budgets.
Conclusion
This white paper amply illustrates that TCO calculation for test automation software, while non-trivial, is hardly intractable. As for all exercises that bridge technical and financial areas, the devil lies in the details: the more comprehensive the parameters considered, the more accurate the costing calculation will be.
The results of the TCO calculations laid out here are perhaps foregone conclusions:
The main contributors to this ordering are
Acquiring OSS is the least costly with more options for obtaining and integrating test software.
Planning, development, maintenance and support of in-house solutions are extremely costly.
OSS and commercial/proprietary costs are not further apart because organizations using test automation must still engage in planning, migration, integration, maintenance and support.
The delta between OSS and commercial/proprietary is occluded by this analysis as both are overshadowed by the high cost of in-house development. Considering only OSS and commercial/proprietary, open source is at least 28% less costly.
The authors encourage development and test organizations to decompose and analyze their actual costs for legacy in-house platforms to achieve the most realistic TCO comparison.
Try It Yourself
We've attached a TCO spreadsheet based on CoCoMo calculations. It doesn't include every possible parameter, but it's a good place to start thinking about your TCO.