Introducing Katalon Analytics for Test Automation Analysis & Quality Visualization

Test execution reports are not only one of the most important feature for any automation framework but a valuable deliverable to track dynamic changes over the project timeline. To add value to the stakeholders, automation teams usually rely on third-party libraries to generate visual reports & dashboard which are time-consuming and unreliable sometimes.

Unlike other automation tools, Katalon Studio provides a rich UI for viewing test execution reports as a trusted, built-in feature without any configuration of third parties. The native reports provide the “snapshot” information of each test execution but limited to analyzing data to help make smart decisions about the quality of the application under test.

0

Offering a solution to this problem, Katalon Studio is going far beyond the basic by introducing Katalon Analytics that brings users to the next level of test reporting with analytics and visualization on aggregated execution historical data.

What is Katalon Analytics?

Katalon Analytics is an intelligent analytics platform with visual and filtering tools to give you a better understanding of your test executions of multiple projects in real time. The Analytics translates test execution results into actionable data which empowers you with the knowledge to pivot your automation strategy, focus on the high-risk areas of your product, detect flakiness tests and to have a better control on the quality.

A product owner could utilize the test results to the fullest only when it is structured and categorized such that the information facilitates decision making and improve the quality of the product. For the purpose of effective analysis, we need to have all the test execution results in one central repository to form a single source of truth for the test. With Katalon Studio v5.0, the test results can be uploaded automatically to Katalon Analytics server to form a seamless solution to provide the benefits of having the full history at your fingertip to be analyzed and filtered in any way you need it.

1

How Katalon Analytics helps your Test Automation Process?

Flaky tests are noteworthy, sometimes your tests fail unexpectedly for various reasons which could be data synchronization, improper test data management or having the dependency on other tests. As your test suites and functionalities of application grow, flaky tests become harder to debug. There are cases that tests that run perfectly fine in series might get flaky when executed concurrently. Automation test is a vital component to detect new errors introduced as a result of new features development but if tests failure rates are high, time and effort spent in the analysis and validation process lead to unacceptable time loss which deceives the purpose of automated regression tests. Katalon Analytics helps you solve this problem by providing a flaky analysis on your test executions with visualization to easily identify this particular problem.

2

Katalon Analytics also helps you to optimize your code and reduce the regression rate. Execution time mainly depends on how you have designed & configured your tests, Katalon Analytics helps you to identify and benchmark the slowest performing functions by providing an analysis to optimize your test executions.

Benefits of Katalon Analytics:

  • Katalon Analytics is free to use with intuitive charts & reports to interactively navigate through vast amounts of log data
  • Eliminates the need for manually generating reports, reducing the possibility of data
  • Using Katalon Analytics pre-built aggregations and filters, you can easily run a variety of analytics
  • Updating and Collaborating test results in Real Time
  • Easy Distribution, Dashboards can easily share dashboards and reports with anyone on the team
  • The calculated execution time of each test cases
  • Time Series Analytics, the Total execution history of the test case and many more features
  • Log Analytics can easily gain valuable insights about the tests executed provided 3with screenshots and videos

How to start with Katalon Analytics

No code, No installations, it is quite simple to synchronize your test results from Katalon Studio to Analytics server. Setting up your project settings with Katalon Studio v5.0 and you and your team members can view all of the project test executions by access Katalon Analytics using your registered Katalon account.

Whenever you execute your test suites, Katalon Studio automatically uploads the test results to the server along with logs, screenshots and execution videos. Katalon allows users to create multiple projects inside the Analytics dashboard and also from Katalon Studio. It also allows users to share the dashboard with the team members just by sending an invitation. Dashboard not only provides us charts but also gives detailed information of the entire test case history such as a number of times executed, the performance of the test case, console logs, status of the tests. Using the dashboard one can easily analyze the results from the test suite level to test case level.

Katalon allows users to upload their test results to the dashboard in several ways. If the user does not want to sync the results automatically, then they can upload the results from the Reports section. By clicking ‘Upload‘ button under Katalon Analytics label, results will be uploaded to Katalon Analytics dashboard. The other way to upload the results is to execute a simple command by providing the project ID mapped in the dashboard. For more information, refer to the documentation.

You can give a try by downloading the latest version on Katalon v5.0.  Katalon Analytics in current in Beta stage and the team is working on additional quality intelligent reports that will be available in future releases. Please submit your feedback at the community forum to make the tool more helpful.

Advertisements

What makes Katalon Studio more powerful than any other Selenium based Frameworks?

As a test automation engineer, we should always consider a few cornerstones before employing a tool to automate our products. As you know the primary benefits of test automation are reusability, scalability, improving test coverage, time and effort saving. Quite many of us say, using Selenium integrated with other third-party tools would achieve all the above cornerstones. But still, we have a few drawbacks. Let’s see how Katalon Studio turns to a magic wand solving test automation difficulties.

The challenge to Selenium/Appium based Frameworks

Let’s say we need to automate a modern application which has web UI, a mobile app and web services. Applying test automation for this application takes much time and effort for coding to automate test scenarios, publishing content from the web UI and verifying the same content in the database & from web services, the same process is required for the mobile app on Android and iOS platforms. Making this more challenging, open-source automation solutions like Selenium require various third party tools needed to be integrated and configured.

This challenge is simply addressed by employing Katalon Studio, you can easily automate web, mobile, web services and execute the scripts on your desired platform. Trust me, people out there are taking weeks to automate the web, mobile apps, and API where Katalon Studio can do it for you in a few hours.

Technical Skills

Selenium, a functional automation API which needs technical skills to automate applications. Even though Selenium is a powerful, free and open source tool, the time we invest in learning and setup the development environment for automation would take months. Moreover, it supports only web-based applications. To automate mobile apps, we need to integrate Appium libraries into the framework. So not every tester can automate unless they have advanced coding skills. Isn’t this a drawback when we cannot automate basic functionalities of an application without coding?

Katalon Studio makes it possible to automate most of the common functionalities of web and mobile apps without the coding experience. This tool enables everyone to automate with ease and efficiency by leveraging built-in features, you can even start automating the application right from the first sprint.

Record and Play on Popular Browsers and Mobile Apps

Test recording is the fastest way for users to automate applications. Identifying objects in applications is time-consuming and painful. It is even a nightmare to update them when applications are changed.

Unlike Selenium, Katalon Studio utility supports record and run the same tests on multiple browsers. In addition, Katalon Studio is intelligent enough to handle dynamic web elements by modifying the test object properties with its built-in methods (Add/Remove Object Property) at runtime.

Hold on!! Not just recording your actions performed on web applications, using Katalon Studio you can also record your mobile app actions. Isn’t it cool when you record your mobile app actions and organize test objects with ease? With Selenium, this is not possible without using third-party libraries.

Refer: https://docs.Katalon.com/display/kd/recording+webui+test

0

Save the Investment in Designing an Automation Framework

The toughest job in automation is to design a framework satisfying specific requirements of an application. Nowadays Selenium with page-object model, data-driven framework, and TestNG has opted as an automation solution for many of us. There is nothing wrong to implement the framework yourself, but time and cost are the most important factors to be considered. We cannot take 3 weeks to design a framework when the testing phase ends in 5 weeks.

This is why Katalon Studio empowers teams to collaborate and produce high-quality applications using its well-structured framework and built-in features.

Page-Object Model Framework

Katalon Studio enables the users to streamline the capturing process to store and manage the page objects using the project structure and then organize them in specific classes to reuse it later. With other Selenium open source wrappers, you can only use the XPath on the code level with the result that you would have to maintain your test scripts upon the change of the test application. And to capture the page object we need to rely on other tools such as Firebug/Firepath to manually identify the web elements, and this process is tedious and painful to organize the page object pattern when the requirements get changes.

So what can be the solution? How can we save time?

Just chant Katalon Studio, this will solve the problem and save you weeks of time. When we are designing a framework we must assume that they are going to be executed, analyzed and maintained by other resources as well. Katalon Studio itself has a built-in framework which is well organized. Using its object repository feature, we can achieve reusability and easy maintenance. The most interesting features, the spy utility captures the web and mobile objects just with one key press. Katalon Studio is intelligent enough to capture your element with all possible locators. So if we are dealing with dynamically changing elements, we can use our desired locator just by selecting it. This saves huge time when we are supposed to capture mobile elements.  Once you capture the page objects all you need to do is click on the button ‘Add to Object Repository’. This would keep all your objects organized within your project explorer which can be reused & maintained with ease. You can easily modify the object properties from the user interface.

Refer: https://docs.Katalon.com/display/kd/web+object+spy

1

Built-in Data Driven Framework

It is always tricky to implement data-driven testing with Selenium. You have to be a good programmer to write the code that fetches the data from spreadsheets or CSV file. Furthermore, to retrieve the data from the database, you need to have a proper JDBC connection configured with the desired database. Can you imagine how much of the time we spend on coding to parse test data in spreadsheets (Excel/CSV)?

Katalon Studio shipped with a built-in data-driven framework. Using Katalon Studio data files we can create more flexible and secure tests that can run on multiple configurations. Data can be parsed easily from Excel sheets, CSV files and even from the database. When we prefer Excel sheets, Katalon Studio gives us an option to select the columns and rows to be imported which becomes easy for a tester and we can quickly use the test data wherever we need. In addition to external data sources, Katalon Studio also has internal data storage.

Refer: https://docs.Katalon.com/display/kd/manage+test+data

2

TestNG & Exception Handling

TestNG with Selenium API is a common automation solution giving us options to group our tests, parallel execution, cross-browser execution and defining and executing dependency tests. But to put all these in place we need technical skills. We know how tough it is to code, maintain and execute the test scripts in multiple browsers in parallel, not to mention custom coding to handle exceptions.

For cross browsers and platforms testing, Katalon Studio provides built-in features to execute test scripts in multiple browsers in parallel or sequential by creating the test suites to be automated and select the environment configurations in which we want to execute. The tool provides flow control ability which allows us to manage exception handling and dynamically define the execution procedure. Another interesting feature, Katalon Studio supports remote server execution with a few possible clicks. No more xml configuration files and coding, all these features come handy with Katalon Studio.

Refer: https://docs.Katalon.com/display/kd/execute+a+test+suite+collection

3

API Testing

Generally, when we need to test our web services, we may employ tools like SOAPUI or we may code it using restful services to send a request to the server and parse the response to validate it.

Employing Katalon Studio would avoid coding part, it has built-in keywords for both rest and soap web testing services. To parse and validate the response, we also have many built-in keywords for the assertion. If we employ a tool like SOAPUI free version, we cannot run it through automation and need coding to generate the report. Katalon Studio allows us to append our web-services tests in a test suite to trigger whenever we need it and a report will also be generated and auto emailed to the recipient list provided.

4

Logging and Debugging

When we use Selenium frameworks we need to integrate third party libraries to put logs for debugging purpose. And it’s always a mundane work to enable and disable logs for every function and model. Katalon Studio by default enables logs for every test step in the test case. We don’t need to write any code further to put some logs. This tool has log viewer section where we can view the logs or we can also check them in the reports. Coming to debugging, Katalon Studio has a good debugging feature which allows us to debug and solve the exceptions we face.

Rich Reporting Framework

Though we design perfect test scripts we still need a good report to share the automation status and test coverage. In Selenium we need to call third-party libraries to generate a report. And to generate the report in various formats it requires coding and designing skills. By default, Selenium API won’t give us any report unless the TestNG framework is integrated.

Katalon Studio easily generates HTML, CSV or PDF reports providing an overview of how our automated testing efforts are being progressed by appended with screenshots and logs.

5

ALM Integrations

Selenium needs third party tools to integrate with ALMs like JIRA, Slack…  and custom coding to execute Selenium test scripts in the cloud environment (Sauce Labs, BrowserStack, Kobiton…).

Solving this issue, Katalon Studio gives us native integration with ALM like JIRA, qTest to link the test execution to the requirements and defects. Katalon Studio also supports CI tools like Jenkins, TeamCity for continuous integration and plugins with cloud services.

6

Here is the summary of what Katalon Studio can do to address the limitation of Selenium

  • Can eliminate the need for user-defined frameworks and expensive programmers to build test automation.
  • Can create test scenarios that accurately simulate real business workflows much faster & simpler.
  • Can Create, Manage and Use test data securely with ease
  • Can easily automate Web Apps, Mobile Apps, Web APIs without programming skill.
  • Can execute test scripts in parallel and cross browsers without programming skills.
  • Empower your Agile Teams with Continuous Testing by Continuous Integration.
  • Simulate complex test workflows easily with looping, conditional statements and control flow commands with built-in keywords.
  • Can easily automate dynamic UI & frequently changing content with ease.
  • For programming lovers, Katalon Studio allows us to code to achieve complex functionality using Script Mode.
  • Katalon Studio eliminates the need for wait statements even for inconsistent page loads and AJAX with its built-in keywords for synchronization.

Katalon Studio is available for testers at no costs. Download Katalon Studio at http://www.katalon.com/

Katalon Studio for Test Automation – Simple & Powerful

Recently I came to know a Tester/Developer friendly Test Automation tool Katalon Studio. I am really impressed with the provided features of Katalon Studio 4.5. I have seen all features which are required for Test Automation. I can say this is a comprehensive Test Automation Tool, a cluster of Selenium, Appium, QTP, SOAPUI for both Web & Mobile Apps testing.

What is Katalon Studio

Katalon Studio is a Free powerful test automation toolset for testers & developers solving many challenges we face in Test Automation. This tool can be used for Functional Testing (Web & Mobile), Webservices Testing as well. Katalon Studio hides all technical complexities and provides us a powerful & friendly UI with the manual mode. A tester can simply drag-drop, select built-in keywords and automate test cases with ease. Katalon comes in handy with all necessary software Android SDK, Web drivers to drive browsers, and other required dependencies.

Recently I came to know a Tester/Developer friendly Test Automation tool Katalon Studio. I am really impressed with the provided features of Katalon Studio 4.5. I have seen all features which are required for Test Automation. I can say this is a comprehensive Test Automation Tool, a cluster of Selenium, Appium, QTP, SOAPUI for both Web & Mobile Apps testing.

What is Katalon Studio

Katalon Studio is a Free powerful test automation toolset for testers & developers solving many challenges we face in Test Automation. This tool can be used for Functional Testing (Web & Mobile), Webservices Testing as well. Katalon Studio hides all technical complexities and provides us powerful & friendly UI with the manual mode. Tester can simply drag-drop, select built-in keywords and automate test cases with ease. Katalon comes in handy with all necessary software Android SDK, Web drivers to drive browsers, and other required dependencies.

0 (1)

Why should I use Katalon for Test Automation

  • First of all it is a Freeware. Freely available to the testing community
  • Easy installation, Configuration is not at all a challenge
  • Built-in project templates
  • Record & Playback feature
  • Limited coding skill required
  • Built-in object repository,object identification
  • Fully supports Web, Android, iOS and API testing
  • Database Testing can be done (Coding skill required)
  • Supports Java/Groovy languages
  • Supports Data-driven Testing
  • Experienced programmers can create and execute sophisticated test scripts.
  • Can be executed in Chrome, Firefox, IE, Headless Browsers..
  • Can be integrated with GIT, JIRA, Jenkins, qTest, Teamcity, Kobiton, Slack
  • Detailed Reporting Structure
  • Katalon Community Support for questions and updates
  • Detailed documentation & user manual

Here you can find the comparison of tools with Katalon.

Who doesn’t love the tool which helps testers with no programming experience to automate their applications just by using recording and built-in keywords to create and execute test scripts easily and faster. I can say this framework is very sophisticated, you can also tag test cases into multiple test suites (sanity, regression). You can import Test Data from your local system or database or you may create data internally within the IDE. We can declare global variables to validate in test cases. Also, reports get generated with detailed information (HTML, CSV formats).

0 (2)

I am not only referring to Web Applications, we can use same Katalon IDE to automate Mobile Apps with ease. We just need to install Appium in our machine and configure the path in settings with few clicks. You can simply automate the app using built-in mobile keywords. Anyway, Katalon allows you to write custom code depends on your requirement.

0 (3)

I used to buy hours to create a Page Object Model in Selenium. But with Katalon Object Spy a built-in feature which helps us to identify the objects much faster and easier. It’s similar to Object Spy in QTP. Now, this is really a time-saving feature.
0 (4)

We can also add identified objects to our repository with one click, which needed for reusability, easy maintenance, easy debugging.

I generally use SOAPUI or Custom code in Java to test the API’s, now with Katalon it is so simple to test the API and capture the report in detail. This supports both REST & SOAP.

0 (5)

As I said, this can be integrated with your GIT repository, CI(Jenkins), Test Management tools like Jira, qTest. I believe we can always code to integrate tools like Sikuli, AutoIT which depends on our requirement.

Who doesn’t like it when all these powerful & useful features come in handy. This is an All-in-one tool. Would you like to try it? You can download it from here. You can always share your opinions/feedback/suggestions by posting a comment

I should thank Katalon Team for bringing this powerful tool as a Freeware to the testing community.

Other References

Katalon vs Selenium

Katalon Documentation

Katalon Tutorials