Tosca & AI ...GenAI, a revolutionary dimension in testing or just a hype? (part 1)


What does, Robotics , Neural Networks, Machine Learning and Artificial Intelligence, all have in common? They are all sub-areas within computer and information science, with the following important and common aspects: Intelligence, Learning and Automation. All these 3 key aspects will be discussed in this blog and in the next blog (part 2) by Otman Zemouri. 

The information contained in this blog (and also previous blogs) draws on his experiences as a test consultant and is intended for educational purposes only. 

Let’s start with a disclaimer! The information contained in this blog (and my previous blogs) draws on my experiences as a test consultant and is intended for educational purposes only. It should not be considered a substitute for professional advice or an endorsement of any specific test tool. While I've used and evaluated various tools throughout my career, the information presented here is also based on publicly available resources. It's crucial to conduct your own in-depth research and evaluation of any test tool before implementing it in your projects. The best tool for you will depend on your specific needs and requirements. This blog is completely independent of any test tool vendor mentioned. There is no intention to promote or market any specific product or service. My goal is to provide a balanced and informative view of various test tools for educational purposes.

Now back to my blog! What does, Robotics , Neural Networks, Machine Learning and Artificial Intelligence, all have in common? They are all sub-areas within computer and information science, with the following important and common aspects: Intelligence, Learning and Automation. All these 3 key aspects will be discussed in this blog and in the next blog (part 2). 

AI History

If we are to believe the media, then "AI" has conquered the world.... But is that really the case? Since the global introduction of ChatGPT to the public (November 2022), GPT* technology has become more accessible and the innovative trend to look out for. Terms such as Large Language Model (LLM) and Generative AI (GenAI) are discussed. If we zoom in on the history of AI, it is really nothing more than the result of successful mathematical research in computer science over the years. Is it a hype? We all know that computer science is based on mathematics and logic, and this "basis" consists of so-called algorithms. The word "algorithm" sounds innovative, but it has been around since time immemorial and is derived from the surname of the 9th-century Arabic-Persian mathematician Muhammad ibn Musa "al-Khwarizmi", one of the most important founders of algebra(ic mathematics).

There are many common definitions about GenAI, a derivative of AI. But when we try to explain GenAI, we are talking about GenAI models that "learn" patterns and structures from (input) data and transform them, in order to generate new data with similar characteristics. Sounds very abstract, but in short, it is a model that generates new content based on what the model has learned from existing content.

As mentioned earlier, GPT is not new, as technology is evolving, and LLMs are getting smarter all the time. In 1966, "a major breakthrough in language models" occurred with the introduction of the world's first chatbot Eliza, by MIT researcher Joseph Weizenbaum. One of the highlights in the evolution of language models since Alan Turing's groundbreaking research in theoretical computer science. Today, GenAI models such as GPT-3, GPT-4, or GPT-5 can not only mimic existing language patterns, but also modify or create entirely new forms of content and generate more realistic and coherent text, images, and videos. This opens up a wide range of possibilities, from generating realistic dialogue for chatbots to curating creative content such as poems, scripts, and even filmographic masterpieces. However, it's important to realize that GenAI models are still constantly evolving. These models can sometimes generate nonsensical content, biased training data, or (test) data from unverified studies or other sources. So be sharp and careful in the use of (artificially) generated data!

An interesting tool with AI capabilities

In my previous blog I discussed about the benefits of using AI in test automation, Testim compared to Tosca. This year, Tricentis presented their new AI solutions, and as you can see in figure 1 below, Tosca doesn't have any GenAI features yet, Testim does. 

Tricentis AI
Figure 1.  Machine Learning in Tosca

If we are going to compare Tosca with another testing tool that uses GenAI, then the question arises whether that is a fair comparison? I will come back to this later in this blog.

Let's take a closer look at a testing tool with interesting AI features! As a test consultant, I've had the opportunity to work with a wide range of testing tools over the years. One of the "tools" that caught my attention was Applitools. The company of the same name calls it "The All-In-One Next Generation AI-Powered Testing Platform".  Is it a tool at all? When I ask the question to Gemini, a GenAI model from Google, the following answer appears, shown in Figure 2 below.

Figure 2. Google's Gemini

It says: "Applitools is an AI-powered test automation platform. Unlike traditional tools that check every pixel, Applitools uses AI to analyze user interfaces like a human would, finding visual bugs and ensuring your app looks good across browsers and devices." But if I ask the following question "What is Applitools?", then Gemini gives a different answer, see figure 3 below. Note the difference between lowercase and uppercase letters and the different drafts.

Gemini prompt
Figure 3. Gemini "prompt"

Two different answers to roughly the same question... Strange? Actually, no. The process or technique behind this GenAI model and other LLM models explains why. If you look at the description in Figure 3, you'll see the word "prompt". The Windows purists among us know that "prompt" is also a command in the command prompt or command line interface (CLI). But if we relate this to GenAI, then "prompt" is nothing more than a natural language text describing a specific task. The science behind this is called "Prompt Engineering", in short, a process of structuring data (e.g. text) that can be interpreted and understood by a GenAI model. With that as a starting point, let's take a closer look at Applitools and analyze it. 

If you go to the Applitools website, and navigate tab/section "Platform", you will notice that Applitools offers 3 platforms or products (see figure 4). 

Applitools platform
Figure 4. Applitools Platform Overview

The "Autonomous" and "Preflight" platforms can reasonably be compared to Microsoft’s Copilot and Testim, but with two different purposes and on different levels. To keep things simple and easy, we'll first focus on Applitools "Eyes", a tool for Visual Testing*** powered by an AI engine. From a "testing perspective" and in the context of the so-called test pyramid, this means that we focus on testing at the (G)UI level. But how do I get started with Applitools? First of all, you need to register yourself, as it is a cloud testing platform (see figure 5).

Figure 5. Registration with Applitools

Once you've successfully completed the registration process, the next stage is to configure your account. This is shown as configuration windows in Figures 6 and 7 below, respectively.

Applitools next step
Figure 6. Steps on How to Configure the Applitools Eyes Account
demo Applitools test
Figure 7. Applitools Eyes (Getting Started)

After completing the configuration steps of your user account, you can start a new project immediately. You can choose to start with a completely new test project or an existing (test) project, integrating "Applitools Eyes" into your project. Applitools supports multiple SDKs (and IDEs), see the citation below. Personally, I choose to integrate the "Applitools Eye" API into my own test project. To find out how "Eyes" works in practice, let's start with a scenario.

A scenario: Acompany

In this scenario, you are hired by a large fictitious company "Acompany". As a test consultant, you will help structure the current testing process. After thorough research, it turned out, among other things, that support is needed for setting up a test automation framework. To address the identified need for improved regression testing, it was further found that the front-end development teams primarily use Python. These insights form the basis for the decision to develop a test automation framework that uses Python, Selenium and Applitools Eyes.

Your additional analysis has also shown that the current testing process is needed for a more robust solution, especially around regression testing at the (G)UI level. To improve the possibilities for visual testing***, Applitools Eyes will be integrated as an API within this framework. Subsequently, you communicated the following (software) requirements for a POC to the project organization:

  • Software
    • Python 3: A recent version of Python 3 for utilizing Python-based libraries.
    • PyCharm: This IDE will provide a user-friendly environment for developing and managing Python test scripts.
    • Applitools Account: Register for a free Applitools account to access the Eyes API for visual verification.
    • Web Browser (Chromium based): Chrome is the most used browser with Applitools, but other modern browsers are also compatible.
    • Web Driver: A web driver, like ChromeDriver for Chrome, allows your tests to interact with the browser and perform actions.
    • Selenium: This web automation framework will be the foundation for the test automation scripts, interacting with web elements and navigating the application under test.
  • Project configuration:  
    • Python Virtual Environment: Creating a virtual environment using tools like “venv” or “virtualenv” is highly recommended. This isolates project dependencies and avoids conflicts with other Python projects on your system.
    • Package Manager: "pip” is the default package manager for Python, to install required libraries and packages.

After Acompany's project organization has approved the above, you as a test consultant have set up 

Pycharm ide project
Figure 8.  Selenium Python Test Automation Project

All tests are developed in the "tests" folder and the first (visual) regression tests are written in the "acompany" subfolder. The "" file shown in the image above is the "skeleton" of the Python project. It is a special file that is used to define fixtures that can be shared with all test files in that directory and child subdirectories. Fixtures or definitions defined in the "" file can be used by any test in the "package" without an "import" rule, as the "Pytest" compiler will automatically identify these packages. By the way, the above project is based on a GitHub project of Applitools (see source). So much for the "Acompany" scenario, we will come back to this in the second part of this blog!

Tosca & GenAI

But what about Tosca and GenAI? At the outset, it became clear that Tosca will only have Machine Learning features for the time being (see figure 1). If I ask Gemini or Copilot to create a Python test project in Tosca Commander, that is logically an irrelevant request and an incorrect question. While Gemini and Copilot are very useful tools, creating test projects in Tosca Commander requires a completely different approach. There have been initiatives to incorporate GPT technology into Tosca, but this has not yet resulted in a fully functioning GenAI feature in Tosca Commander. But let's take a look at the possibilities!

Let's say we want to use Microsoft's Copilot to design a test case to test the registration functionality of a particular webshop. While Copilot is a powerful LLM model, it currently does not integrate directly with Tosca Commander for designing test cases, for example. Tosca uses its own "add-in" for this, namely TestCaseDesign. However, Copilot can still be a valuable asset in the design process of test cases in Tosca. But how? See some use cases below!

  • Brainstorming Test Cases: Describe the functionality you want to test and ask Copilot to generate different scenarios, "happy" and "unhappy" flows. This can help identify any potential issues that you may have overlooked.
  • Test Data Generation: If your test cases require specific data entry, Copilot can help generate relevant test datasets based on your requirements.
  • Natural Language Requirements to Test Cases: Provide Copilot with "user stories" or natural language descriptions of the functionality. This can help translate these into a more structured format that is suitable for Tosca Commander.
  • Requirements Analysis: Use Copilot to analyze your requirements documents and user stories. Copilot can identify key functionalities, user interactions, and potential "edge cases" (exceptions) that need to be addressed in your test cases.
  • Test Case Inspiration: Describe the functionality you want to test in Copilot and get ideas for test steps by asking Copilot to generate examples or variations of the functionality.

But there are some important limitations too. 

  • No Direct Integration: Directly, Copilot cannot create, generate, or import test cases into Tosca Commander.
  • Manual Effort Required: Identified test cases must be manually created in Tosca Commander.

While Copilot can't directly replace Tosca's "TestDesign" features, it can still be a valuable tool for test inspirations, needs analysis, and test data generation during the TestCaseDesign process.

Are you curious about the sequel and how "Acompany" will succeed in setting up a test automation framework? Keep an eye on my next blog (part 2)!

Abbreviations used in this blog:

  • AI (Artificial Intelligence)
  • API (Application Programming Interface)
  • GenAI (Generative Artificial Intelligence)
  • GPT* (Generative Pre-training Transformer)
  • GUI (Graphical User Interface)
  • IDE (Integrated Development Environment)
  • LLM (Large Language Model)
  • MIT** (Massachusetts Institute of Technology)
  • POC (Proof of Concept)
  • SDK (Software Development Kit)
  • SUT (System Under Test)

Sources used in this blog:


These blogs appeared in the series
"Do you know Tricentis Tosca..."

  1. ...can run Selenium scripts?
  2. ...can create Load scripts?
  3. ...can work with CI pipelines? (part 1)
  4. ...can work with CI pipelines? (part 2)
  5. ...has made CI/CD integration even easier? (part 1)
  6. ...has made CI/CD integration even easier? (part 2)

Published: 23 April 2024
Author: Otman Zemouri