The visual testing landscape for mobile test automation has only gotten more complex over the past few years, but so have the test automation tools that are now available. We engineers all have our preferred methods for getting optimum results, but when you are spoiled for choice, which visual testing tool is going to suit your style the best?
Before we get started, introductions are needed: I’m Daria Krasulia, an iOS SDET, developer with significant experience performing visual test automation for Native iOS apps across several companies.
In this article, I will discuss the role visual testing plays and how it should be used when evaluating what users see when they use your app. I will also share my initial thoughts about five test automation tools that have risen to popularity within the visual testing community–the good, the bad, and the “maybe useful, depending on your needs.”
What is Visual Testing?
For those of you who are new to visual testing or just need a refresh, this part of the app development process is when developers like me evaluate an app’s visuals page by page to locate any anomalies, errors, or UI visual defects. I believe that as the field of visual testing has matured since 2010, it has grown far beyond simple image comparisons and finding visual defects.
Why Do You Need Visual Testing?
Unlike deep link testing, cross device testing, performance testing and other types of testing, visual testing focuses on verifying that the user interface of an application appears correctly to the end user across different devices and browsers. Visual testing for mobile apps is necessary to locate and fix any errors in how the app presents visual information to a user. If the UI is unintuitive, missing critical information, or does not live up to the needs of the intended user, the app is likely to fail. To ensure maximum user-friendliness and effectiveness, every aspect of an app’s visual presentation needs to be perfect for the purpose for which it is intended.
However, testing EVERY aspect of an app’s visuals can be an arduous task for any developer simply based on the time required. Since there are just not enough hours in the day, we turn to test automation tools to help get the process done under reasonable deadlines.
Other Ways to Use Visual Testing Automation
The visual test automation tools can do much more than validating correct imagery and mobile app functionality. Here are some other aspects to consider when performing your visual tests:
Localization: Finding localization issues can play a much bigger role in the process for finding errors in language translations or missing translated strings that we simply don’t have the bandwidth for, hence the necessity for automation in visual testing.
Accessibility: Mobile visual testing automation also makes it much easier for us to scan for accessibility issues so that our mobile apps can cater to users with varying visual or perception disabilities. Designing imagery and other on-screen visuals with accessibility in mind ensures more inclusivity and useability for all customers.
Figma Design and Implementation: As every developer knows, the road between app design elements, particularly those in Figma, and the implementation of those design elements into an app’s UI does not always translate well. An automated visual testing solution is ideal for finding such translation issues quickly to keep both the designers and engineers happy.
Daria the Testing Tools Connoisseur
As a developer, I’m always on the lookout for the best test automation tools that will make my job much more efficient. Recently, one of Engenious’s clients tasked us with heavily researching and testing out the many visual testing tools out there to see which ones were the most effective. After spending roughly a month diving in and trying out multiple tools, I can confidently say that I have come up with a comprehensive list of the top five visual testing tools that you should consider to enhance your mobile testing automation efforts.
Dive into the story with our video – no reading required!
Evaluation Criteria
Every method has its madness, and so here is mine! I developed this set of criteria in which to base my professional opinions on for each testing tool evaluated:
- Communication
- Documentation
- Mobile and Web Support
- AI/ML
- Standalone VS. CI
*These criteria will use a 5-star rating system for your reference (⭐⭐⭐⭐⭐)
PLEASE NOTE: The following is a comprehensive list of tools I tested using my own criteria for what constitutes an effective tool for mobile test automation and not web testing. These findings are based on my own experiences using each tool, and the tools in this list are in no particular order. Engenious is not affiliated with any of the testing tools nor their creators in any way. I encourage you to get the trial versions of the visual testing tools mentioned below and experiment with them to see what will work best for you and your particular business needs.
Percy (BrowserStack)
FINAL SCORE: ⭐⭐⭐
General Information:
- Simple and clear UI/UX
- Easily integrates with CI/CD processes
- Code changes automatically prompt tests
- Cloud-based solution for easier management and flexibility when accessing tests, but needs constant connection to Internet
- Pricing (Mobile): $0.018 – $0.0339 per screenshot, per month
Communication: ⭐⭐⭐⭐
- Trial version includes up to 1,000 FREE screenshots monthly
- Video tutorials are available to familiarize yourself with the platform, but can be time-consuming
- Company support offers LiveChat with near instantaneous replies to queries
Documentation: ⭐⭐⭐
- Necessary information is widely available, but is difficult to navigate to find specific answers
- Specific information may be hard to find due to it being spread across multiple topics
- Documentation UI was not as intuitive as I would have liked
Mobile and Web Support: ⭐⭐⭐⭐⭐
- XCUITest support is available
- Support for a large number of frameworks is available for both mobile and web testing
AI/ML: ⭐⭐⭐⭐
- Automated visual defect detection is available
- Percy AI structure focuses on human-visible deviations and minimizes false positives to accurately detect visual deviations, especially for dynamic elements
- These features were not verified due to the difficulty of uploading a project to make the comparison of AI visual defect detection
Standalone VS CI: ⭐⭐⭐
- Percy is claimed to be able to perform locally and with CI-configured projects
- I encountered a number of errors when trying to upload a project (took nearly four attempts); after contacting support about my upload issue, I was told that it would not work without using CI/CD configuration
- Taking screenshots of the project was not possible
Testim (Tricentis)
FINAL SCORE: ⭐⭐⭐
General Information:
- Clean interface that emphasizes codeless automation, which can be limiting for more advanced developers
- Option for quick test creation with ability to record and playback user actions
- Integrates with multiple development tools, CI/CD systems, and code version control systems
- Focused on Record and Playback functionality, which can be limiting to more advanced developers
- Pricing:
- 500 grid runs start at $450 monthly
- 1000 grid runs start at $1,000 monthly
Communication: ⭐⭐⭐⭐
- Informative notification-based onboarding process from Support team
- NO native visual testing tool available; requires Applitools integration to use visual testing automation (available in the Premium plan ONLY)
- 15-day FREE trial period includes both real and virtual devices, but has limited functionality
Documentation: ⭐⭐⭐⭐⭐
- Well-structured documentation interface with intuitive information organization
Mobile and Web Support: ⭐⭐⭐⭐⭐
- Mobile and web testing are available
- Code automation is NOT available due to focus on codeless automation
- Supports both iOS and Android apps
- Good for beginners without heavy programming knowledge, but not more advanced programming language input
AI/ML: ⭐⭐⭐⭐⭐
- AI-enabled user analysis to generate test scripts
Standalone VS CI: ⭐⭐⭐⭐⭐
- Local testing available
- CI/CD integration available with both trial and paid versions
- To integrate with iOS APP, upload an .ipa or .app file to Testim and start recording on either a virtual machine or a physical one (edit recording if necessary)
LambdaTest
FINAL SCORE: ⭐⭐⭐⭐
General Information:
- Easy-to-use UI
- Quickly create visual tests
- NO support for screenshots; external screenshots must be uploaded separately
- Pricing:
- $79 – $959 annually
- $109 – $1439 monthly
Communication: ⭐⭐⭐⭐⭐
- Trial version includes up to 2,000 FREE screenshots monthly
- Contact support team directly through dialog box
- Quick problem resolution (solved my issue within 15 minutes with phone call)
Documentation: ⭐⭐⭐⭐⭐
- Well-structured documentation with intuitive information organization and very user-friendly interface
Mobile and Web Support: ⭐⭐⭐
- NO native XCUTest framework available via SDK
- API interactions only
- Web testing framework integration through Java, Node, CY, and Webdriver I/O
AI/ML: ⭐⭐⭐
- No evidence of AI use or integration
Standalone VS CI: ⭐⭐⭐
- NO local or CI testing available
- GitHub integration
Applitools
FINAL SCORE: ⭐⭐⭐⭐
General Information:
- Fast and efficient automation
- Eagle-eyed AI image detection for even the smallest of differences
- Wide acceptance of various programming languages and frameworks
- Cloud-based system with available analytics and reporting
- Pricing: $20,000 – $120,000 annually (unconfirmed since pricing is not made public)
Communication: ⭐⭐⭐⭐⭐
- Trial version includes 100 checkpoints per month for one user
- Onboarding includes demo
Documentation: ⭐⭐⭐⭐⭐
- Well-structured documentation with intuitive information organization and very user-friendly interface
Mobile and Web Support: ⭐⭐⭐⭐⭐
- Supports native mobile frameworks such as Appium (C#, Java, Python, Ruby), Espresso, and XCUI
AI/ML: ⭐⭐⭐⭐
- AI-enabled automatic change detection, but AI results can very and still require human checking to ensure accuracy
- Intelligent image comparison
- Disregard dynamic elements
- Reduction in false positives
Standalone VS CI: ⭐⭐⭐⭐⭐
- Supports CI integration
- Ultra-fast test cloud available for local testing
- iOS APP integration available
pCloudy.com
FINAL SCORE: ⭐⭐⭐
General Information:
- Great for image comparison
- Pricing:
- $129 – $1,799 monthly
- $348 – $4,857 quarterly
- $1,084 – $15,122 annually
Communication: ⭐⭐⭐
- Support from team is lacking with slow replies
- NO support on weekends or holidays
- NO information on trial versions
Documentation: ⭐⭐⭐⭐
- Good documentation with intuitive information organization, user-friendly interface, and concrete examples
- Confusion over which API calls are included in the trial version versus the premium plan
Mobile and Web Support: ⭐⭐⭐
- Visual testing ONLY supports PNG image files
- NO code integration
- Visual testing can ONLY be completed by using pre-built commands or through API
AI/ML: ⭐⭐⭐
- AI supports functional testing of applications
- No evidence of AI supporting visual testing
Standalone VS CI: ⭐⭐⭐
- NO support for local or CI/CD integration
- iOS APP visual testing possible through API
- No device booking in trial version
As you can see from my findings, there is not really a clear winner in this competition. Each testing tool brings something different to the table that makes it stand out, for better or worse. Ultimately, it comes down to your personal preference and your existing experience with these apps or similar solutions.
To see my full talk about these visual testing tools, check out the webinar “What Should I Use for Visual Testing in 2024?
Have you had any experience with any of the visual testing tools mentioned above, or have valuable insight to share about other testing tools not mentioned here? Please share your experiences and knowledge with the community on our Discord channel!
Don’t miss out on our next webinar From QA to iOS Developer in 6 Months: Mastering iOS Test Automation with XCUItest happening soon! Remember, all registrants have access to the recording, so sign up today and reserve your spot!
