What technical information do you like to capture?

Discussions
, ,
1

Hello again lovely people!

I’m keen to learn about what sort of information people capture in their automation frameworks to help them report details to their team and assist the debugging of flakey checks / bugs in a system. Specifically I’d like to know:

What technical information do you like to capture

Anything shared would be great to add to the course I’m working on to show how we capture different technical information and why it’s useful.

3 Likes
2

Lately, we’ve focussed on pushing all important metrics and traces to our telemetry tool and have analysed its output there.

Two core aspects

  1. Pushing build data and analyse what step takes too long and optimise there.

  2. All test runs are reported directly there as well. With lots and failures. This way we are able to analyse which component is unstable and are able to derive meaningful information and insights.

Keen to see what others do!

3 Likes
3

I’ve had a think of what information could be relevant and narrowed it down to a couple of key areas specifically: the system under test, the user, the device, and the test itself. (Perhaps there are more, but that’s what I came up with over a cup of tea!)

Here’s some specific things I came up with:

Recording network requests, particularly useful for debugging overnight test run issues related to the API (specifically I’ve used Sauce Labs Network Traffic Capture). It’s very useful information, for example if I couldn’t replicate an issue found during the overnight tests due to some unknown variable, I still had useful information to investigate the bug further as I was able to understand if it was or wasn’t an issue involving the API.

The configuration of the user. In my experience I have logged this implicitly, for example when key entry steps are recorded, it shows the exact user details inputted during the test. Alternatively, there can be general user details mentioned in the Test Name / Step Definitions (which mention if it’s a new user, registered user, admin, etc.)

General technical information about the test session which is useful for debugging: configuration of device, operating system, app version, base URL, browser & version. This information could be made available in the test report, found in the capabilities/configuration file, or on the cloud session on Sauce Labs or BrowserStack.

After a test failure, outputting the stack trace with information on what action failed (click, double-click, drag, etc.) with relevant screenshot or video (with timestamp) as it illustrates the exact problem area :grin:

2 Likes
4

product build numbers and which branch they are from, is like, super invaluable, easy to include but often omitted.

1 Like
5

Something that I find really useful in UI automation is screen recording. Screenshots definitely helps but screen recordings are even more helpful.

1 Like
6

Here’s what I was able to come up with:

  • Environment Information: Details about the test environment setup eg. Operating system, browser version, and device model.
  • Test Steps and Actions: Sequence of steps taken in the test and their outcomes, eg. Step-by-step actions leading up to a failure.
  • Resource Usage: Metrics on CPU, memory, and disk usage during tests.
  • Historical Trends: Analysis of test results over time to identify patterns
  • Test Priority
  • Automated Test ID: The unique identifier for each test for easy tracking.
  • Failed Assertions Details: Specific details on which assertions failed during the test.
  • Data Used for Testing: Details about the test data or parameters used during the test.
  • Version Control Information: Details about the version of the application or component tested.
  • Test Annotations: Notes or tags added to tests for additional context. Example: Tags indicating which sprint or feature the test is associated with.
1 Like
7

I wrote an article about logging your autotests/scripts for debugging Writing Your Own Logs - DZone. It isn’t particularly about capturing technical information within the autotest framework, but anyway, I believe it’s quite relevant to the topic. I think that it’s crucial to collect as much data as possible - all actions, entity IDs, test data, and sometimes even timestamps, as well as response codes or the entire content, payload, headers, response time, etc. + collecting all the application logs makes it easier to identify the issues and how it was reproduced, and to locate specific logs on the server or other microservices/3rd party services

1 Like
8

At my company we faced (still continue to but far lesser now) the problem of flaky tests as well, to solve it we came up with the strategy to include as many logs as possible which are normally used by developers when they want to debug a manual test failure. This includes -

  1. WS Logs
  2. Network logs ( to make them meaningful we filter out only network calls owned by our product and remove all third party calls, also all failed calls are labelled as failed so they are easy to search) These further include call duration data, data for calls which were made but no response was received etc.
  3. Video recordings ( this is the best debugging tool you can give anyone, so many times we have figured out reason of flakyness from this)
  4. Screenshot of failure
  5. All logs which are printed in tests, although people might say this is not a good practice but I feel this helps locate the point of failure much faster.
  6. Some API calls we make in the tests to prepare user data, we log the data for network calls also separately so that developers have access to it if the test is failing at data creation level itself.
3 Likes