When I look back at my career in test automation, there are three distinct periods, or “waves” that come to mind.
Test Automation: First Wave
The first wave is filled with some good old-fashioned vendor tools like WinRunner, Silk Test, and QTP. In my eyes, these solutions started it all and set the stage for future testing automation innovations like Selenium.
Test Automation: Second Wave
Selenium began the second wave of test automation, focusing more on developers and programming best practices when creating automated tests.
But truth be told–in the quiet times when they think no one is listening–you can hear testers whispering the same curses they did about vendor tools: flaky tests and maintenance driving them crazy.
The current buzz these days is around AI and Machine Learning. Companies are rushing to create tools they can pitch as “AI-driven.” In fact, at a recent Google conference CEO Sundar Pichaiopened the event by stating that “We’re moving from a mobile-first to an AI-first world.”
We’re moving from a mobile-first to an #AI first world~Sundar Pichai
Here are seven newer “AI”-based tools that I think will take us to the next stage of test automation–the Third Wave. (Also, check out the Automation Guild Online conference for some awesome sessions we will have on AI test automation. You’ll also get a chance to ask question to many of the vendors mentioned in this article)
Test Automation: Third Wave Tools
Here are just a few of the “third wave” automation tools I’ve seen in the market. One of the main features of this software is that many of them are leveraging machine learning and AI-assisted technology.
- Applitools
- SauceLabs
- Testim
- Sealights
- Test.AI
- Mabl
- ReTest
To learn more about these solutions, be sure to check out the Automation Guild 2018 online conference, where many experts from these vendors will be joining us for an epic event of automation awesomeness.
Applitools
Applitools was one of the first tools in the third wave that I got my hands on, and it made me start believing that a new way of testing was possible.
When I first heard about visual validation testing, which uses a sophisticated algorithm to out potential bugs in your application without you explicitly calling out all the elements, I thought it must be B.S.
After speaking with founder Adam Carmi, however, and checking out Applitools for myself, I became a believer.
I discovered there really are no visual processing settings, percentages or configurations that need to be set up to create visual testing with Applitools.
The algorithm is entirely adaptive, and I can only imagine where they’ll take the technology as AI and machine learning advances even further.
I recently saw an Applitools demo, and on their roadmap were some cool features they plan on adding on top of their existing machine-learning technology.
Possible AI Type Features:
- Leveraging ML/AI-based for automated maintenance (being able to group together similar groups of changes from different pages/browsers/devices)
- Modifying their comparison algorithms to be able to discern what changes are meaningful/noticeable
- Being able to automatically understand which changes are more likely to be bugs vs. desired changes and prioritize diffs
A lot of these things are too early to see in action, but looking at the roadmap will give you a sense of how much AI is now being incorporated into test tool company’s roadmaps.
Sauce Labs
Of course, Sauce Labs were one of the first players in the cloud-based test automation space, but with all the data they currently have access to they’re in a great position to leverage machine learning and come up with some cool insights.
That was one of the points that came up during the 2017 SauceCon conference. During the keynote, CEO Charles Ramsey displayed a slide that showed how we’ve gone from mainframe all the way to iOT, as well as things like artificial intelligence, machine learning, and deep learning.
It’s obvious that Charles believes the use of known pattern matching and different AI technologies can be powerful within testing.
Which got me thinking — with Sauce Labs running over a million and a half tests a day, they have a virtual treasure trove of data that can be used to help their customers become better testers.
I definitely foresee Sauce adding more intelligence into their analytics that will proactively help customers improve their test automation.
Testim
Testim tries to leverage machine learning to speed up the authoring, execution and most importantly the maintenance of automated tests. Their goal is to help you to start trusting your tests.
Testim focuses on reducing your flaky tests and test maintenance, which they see as one of the most significant challenges for most organizations.
Oren Rubin, co-founder of Testim, mentioned in a recent TestTalks interview that the firm’s main goal is to help liberate test automation from the exclusive realm of developers and make it simple enough for anyone on the team to create. After speaking with Oren I got a sense that Testim is well on its way to achieving that goal.
Sealights
Sealights is a Cloud-based platform. We all know that developers and QA–both managers and engineers–are super busy these days using CI and CD practices, where they have frequent releases and not enough time to test the entire application multiple times.
That’s one of the main reasons Sealights was created.
With their machine learning-like technology that analyzes both your code and the tests that run against it, it lets you know exactly what your tests are covering and what they’re not. But when Sealights says “tests,” they don’t only mean unit tests; they mean any kind of test, from functional, manual, performance, you name it.
“Quality Risks” is even a more exciting insight they provide, as it focuses the user’s efforts on the things that matter, by letting him or her know exactly which files/methods/lines have changed in the last build that wasn’t tested by a specific test type (or any test type). Once you know that, you can easily ensure that untested code will not reach production before undergoing, at the very least, a minimal validation.
As we move toward CI/CD, dashboarding becomes critical.
If you’re like most companies, everything today is within your CI/CD, but often this data is not visible or accessible for consumption by your teams.
Sealights makes it easy to create a quality dashboard that everyone will see. So for every build, you’ll be able to understand what was tested, what the status and coverage were, and whether it’s improving, decreasing, or has quality holes or not.
Test.AI
Test.AI is billed as a tool that will add an AI brain to Selenium and Appium. It was created by Jason Arbon, co-author of How Google Tests Software and the founder of appdiff. Tests are defined in a simple format similar to the BDD syntax of Cucumber, so it requires no code and no need to mess with element identifiers.
The AI identifies screens and elements dynamically in any app and automatically drives your application to execute test cases.
It’s smart enough to know that if an element ever changes it can adjust and identify it with you having to make any manual changes. This tool is still in beta, but I was given a sneak peek of it by Jason.
I’m excited to learn more about it, as well as the potential of all these test tools.
Mabl
Mabl is similar to Test.AI. Mabl started by a bunch of ex-Google employess runs functional tests against your apps or website. In Mabl terminology, you “train” your tests to interact with your applications. When you’re done recording, your trained tests will run at a predetermined amount of time and alert you.
Their website makes three main promises:
- Eliminates flaky tests–like the other AI-based test automation tools, Mabl can automatically detect whether elements of your application have changed, and dynamically updates the tests to compensate for those changes.
- Mabl can continuously compare test results to test history to quickly detect changes and regressions, resulting in more stable releases.
- Mabl helps identify and surface problems quickly, alerting you to possible impacts before they impact your customers.
ReTest
Use an artificially intelligent monkey to fully automatically test your application. That’s how ReTest markets itself.
ReTest claims to be different from other test automation tools because it was built specifically with testers in mind.
It also stems from an artificial intelligence research project, so it tries to bake that AI intelligence into their tool, effectively eliminating the need for their users to possess any programming skills.
They, like some of the other tools on this list, also avoid having to select element IDs to work with when creating a script. ReTest also automatically takes care of wait times.
If you want to try out ReTest, they offer an excellent step-by-step demo post.
So is AI/Machine Learning Just Hype?
Clearly, AI/Machine Learning is the latest buzz word currently being used in the testing industry. But is it real, or just hype?
Only time will tell if the third wave will finally fulfill the promise of reliable, easy-to-maintain test automation for all.
Source: https://www.joecolantonio.com/2017/11/07/7-innovative-ai-test-automation-tools-future-third-wave/