What’s the Most Cost-Effective Mobile Test Automation Tool at Scale? 🤔💭

Factor #1 – Device Needs

When you test on physical devices, you can be confident that your application will behave as expected in real-world scenarios. This can be important for user satisfaction and app performance. However, the costs associated with physical device testing can be high. 

High-quality tools that allow for mobile emulator testing help balance these costs while increasing test coverage and frequency. Choosing a tool that enables proven reliability with emulator testing can significantly help manage budgets, especially in large-scale projects.

Factor #2 – Setup Complexity

The setup and integration of a complex testing automation tool into an existing platform and workflow can be a major budget drain—in terms of increased potential for errors and lags. 

Therefore, to maximize ROI, a tool’s setup should minimize downtime, accelerate development cycles, and reduce the margin for error. 

Factor #3 – User Experience

A testing automation tool’s intuitiveness and ease of use significantly affect project efficiency. A user-friendly interface is easy for anyone on the team to pick up, reducing training time and minimizing errors. As a result, the testing process is smoother, faster, and more affordable.

Factor #4 – Sharding/Parallelization

The ability to run tests in parallel or distribute them across different environments can greatly reduce testing time. With efficient sharding and parallelization capabilities, tests are completed faster. Speed is essential for agile development environments and quick iterations.

In contrast, a tool with built-in parallel test execution capabilities can cut testing time by over 50%, enabling faster feedback loops and quicker iterations without requiring any additional setup. 

Factor #5 – Handling Flaky Tests

Flaky tests undermine the reliability of test results. This unreliability often leads teams to overlook important problems and introduces unnecessary development delays. Therefore, a tool’s ability to manage and minimize flaky tests is a crucial element in maintaining testing process integrity and ROI.

Analysis—Firebase Test Lab

Tool Overview

Firebase Test Lab is a cloud-based service where developers can test Android and iOS apps on various devices and settings without the need for an individual testing setup. 

Hosted in Google’s data centers, Firebase Test Lab provides access to both virtual and physical devices, allowing for app testing under different conditions and on various hardware. It simplifies the testing phase by automatically finding crashes, creating detailed reports, and providing performance insights.

Device Needs

Firebase Test Lab’s cloud-based infrastructure allows developers to test mobile apps on a variety of devices to ensure compatibility and functionality across different models and operating systems. 

It’s particularly powerful for testing on physical devices, especially iOS applications. However, this power comes at a premium, costing $5 per device per hour. 

When it comes to extensive testing sessions at scale, this rate can quickly add up and can be prohibitive in a continuous testing scenario.

Additionally, Firebase can only test physical iOS devices—no virtual or simulated environments. As we discussed, virtual environments often allow for quicker and more flexible testing scenarios, making this limitation unfortunate for large-scale jobs.

Setup Complexity

Firebase Test Lab setup involves integrating with multiple tools and navigating the complexities of the gcloud CLI. This process can be cumbersome, adding significant overhead and potentially complicating the testing process for teams whose members lack technical expertise.

• Integration and Customization: Although a setup process involving the gcloud CLI and tools like Flank may seem complex, it offers substantial control and customization for testing environments. This can be particularly advantageous for teams already integrated into the Google ecosystem, enabling detailed automation within CI/CD pipelines.
• Seamless Google Services Integration: Firebase Test Lab integrates well with Google’s other services. This familiarity creates a simple workflow for users of Google’s cloud products by reducing the need to manage multiple systems. The integration supports scalability, making it easier to expand testing efforts without substantial additional overhead.
• Valuable Learning Curve: The initial learning curve required to set up Firebase Test Lab is steep. However, some teams consider it a long-term investment. Once mastered, it allows for highly customizable and automated test scenarios. Google also provides expansive documentation and support to help overcome setup challenges.

Despite some initial complexities, Firebase Test Lab’s robust integration features and scalability make it a strong contender for organizations vested in the Google Cloud infrastructure that need detailed control over their mobile app testing.

User Experience

Feedback from Firebase Test Lab users is mixed. While the tool supplies extensive test coverage and performance insights, many users find it challenging to navigate, especially those new to Google’s ecosystem. 

The need for multiple tool integrations and the dependency on the Google Cloud platform can be a barrier to quick adoption and ease of use. However, those familiar with Google services may appreciate the seamless integration with other Google tools.

Sharding/Parallelization

Firebase Test Lab’s support for sharding is limited:

• CLI Limitations: The CLI doesn’t allow you to set more than one parallel execution, which is a significant drawback for large-scale testing scenarios.
  
• Sharding Algorithm: The sharding algorithm borrowed from Orchestrator is not optimal. It can result in significant differences in execution time among different shards, leading to inefficiencies.
  
• Additional Tooling Required: To improve sharding, Firebase Test Lab requires integration with Flank and Google Cloud Storage, which adds to the cost and complexity of setup. Flank allows for better sharding, but the integration process is not straightforward and means additional expenses.

Handling Flaky Tests

Flaky tests lead to non-deterministic and often repetitive test executions, which can skew test reliability and increase testing times and costs. They’re a common challenge in software testing and an unfortunate weakness for Firebase Test Lab.

In Firebase, the approach to managing flaky tests involves rerunning the entire test suite when inconsistencies occur rather than isolating and rerunning only the failed or flaky tests. This method ramps up time and resource consumption, particularly in large-scale projects where test suites are extensive.

Each test rerun means additional time and charges, especially with physical devices, which are charged per device per hour. 

Remember, Firebase setup is already complex, requiring tools like Flank and Google Cloud Storage to enhance other functionalities such as sharding. These additional reruns further complicate budgeting and operational efficiency.

• Technical Limitations and User Feedback: The inability to target flaky tests for reruns is a limitation that leads to increased frustration among developers and testers. In fact, user feedback often points to flaky test handling as a significant inefficiency, with teams expressing concerns over time wasted on unnecessary reruns that could’ve been avoided with more sophisticated flaky test handling mechanisms.  Such limitations are particularly challenging in continuous integration environments where speed and efficiency are major concerns.
• Potential for Improvement with Additional Tools: While Firebase Test Lab itself does not offer advanced flaky test isolation, integrating with additional tools might give some relief. However, doing so requires additional investment in setup and maintenance, adding layers of complexity and potential points of failure in the testing pipeline.

Analysis—Marathon Cloud

Tool Overview

Marathon Cloud is a newer testing tool developed specifically for speed and efficiency in modern, fast-paced development environments. The service allows users to run thousands of tests simultaneously, completing them all within just 15 minutes and starting at $2 per device hour. 

Marathon is tailored for developers and companies looking to rapidly test and validate software across various scenarios, significantly cutting down on the time and resources typically required for extensive testing procedures.

Device Needs

Marathon Cloud supports emulator-based testing, providing a cost-effective solution for larger-scale project requirements. While it doesn’t support physical device testing, its emulators are exceptionally accurate and help to significantly lower overall costs.

User Experience

Marathon Cloud is often praised for its user-friendly interface and versatile CLI tool, which cater to less technical users and seasoned developers equally. 

The dashboard is commonly referred to as “intuitive,” making it easier for teams to integrate and manage testing processes. This ease of use translates to higher productivity and a smoother integration into existing development pipelines.

Watch our webinar to learn more on how different tools help improve your testing efficiency:

Sharding/Parallelization

Marathon Cloud is equipped with advanced mechanisms for test distribution and real-time load balancing, allowing for efficient sharding and parallel test execution. 

This built-in functionality means that teams can achieve high throughput without the need for external tools or complex configurations—a massive advantage in any large-scale, complex scenario.

Handling Flaky Tests

Marathon Cloud has built-in, automatic functionality for dealing with flaky tests, including advanced preventive analytics and fine-grained corrective retries. 

For example, rather than rerunning an entire test suite, Marathon isolates the flaky behavior, identifies the issue, and then executes immediate retries only for the failed tests.

This approach results in a higher degree of test reliability and reduces the time team members spend dealing with inconsistent test outcomes. 

These features save enormous amounts of time and reduce the cost impact of handling flaky tests, significantly improving the overall ROI of the testing process.

Use-Case Scenario Comparison

As you can see, Firebase Test Lab and Marathon Cloud differ significantly in their approaches to testing automation at scale. Now, let’s examine a few hypothetical scenarios to see how ROI might factor into the equation.

Device Needs Scenario

A startup is developing an app intended for a global audience with a wide range of devices and operating systems. Marathon Cloud would be the best choice here. Its extensive emulator testing capabilities will allow the testing team to ensure broad coverage for diverse markets while keeping costs within budget 

Setup Complexity Scenario

A large enterprise with an established CI/CD pipeline is on a tight time-to-market deadline. As a result, it needs a testing tool capable of integrating without major project disruption. 

Firebase Test Lab has the necessary control and customization for this project, but it’ll require additional configurations and integrations that could introduce complexities and delays. 

In contrast, Marathon Cloud could save weeks of engineering time, enhancing ROI by speeding up time-to-market.

UX Scenario

A team with members having varying levels of technical expertise is tasked with maintaining high testing standards for a new software product. 

Marathon Cloud, with its user-friendly UI and versatile CLI tool, enables beginners and experienced developers on the team to perform tests effectively, reducing the learning curve and boosting team productivity.

Sharding/Parallelization Scenario

A large software company is in a release cycle and needs to run thousands of tests daily. If the team chooses Firebase Test Lab, they’ll need to implement Flank, a massively parallel Android and iOS test runner, since the platform otherwise has no sharding capabilities.

Using Marathon Cloud, with its built-in sharding/parallelization functions, would allow the team to run its thousands of tests daily right from the gate, saving time and money.

Flaky Test Scenario

An app development team frequently encounters flaky tests that result in inconsistent test outcomes, leading to repeated test cycles and wasted developer time. 

Marathon Cloud’s preventive and corrective retries functions enforce robust strategies for handling these flaky tests. Using it for this project would significantly reduce inefficiencies, deliver more reliable testing outcomes, and more effectively allocate resources.