Any cypress shops also utilizing distributed tracing?

We (Tracetest) just added the ability to trigger a trace-based test from Cypress. This allows Cypress tests to be utilized as a trigger for the test (ie trigger the action based on a user interface UI flow). Once the test is run, you will see the full distributed trace from your front end system and the trace of all involved components in the backend. You can then apply assertions across the entire flow, verifying your frontend tests drive the proper actions in the backend. Documented the capability in this blog post: Trace-Based End to End Testing with Cypress and Tracetest.

Would love to talk to a few shops that are using both distributed tracing (ie Jaeger, Grafana Tempo, Honeycomb, Lightstep, AWS X-Ray, etc. and Cypress. Looking to how it resonants with real users, see if our instructions make since, and ask what capability we should add.

You can reach out to me here or on Linkedin: Ken Hamric - Tracetest.io | LinkedIn

2 Likes

It looks pretty interesting and I scanned through some of the information on the tracetest site.

But I think I need an ELI 5 understanding of how this is implemented and what it provides.

For example - most projects we in QA work on are not greenfield. Often there is a large body of development work in place by the time we are let in the door. Or even we are working on legacy products that already have a large body of tests and test infrastructure. E2E, Integration, unit, etc

With that in mind - what sort of a cost would I be presenting to management if I wanted to implement it? Would it require refactoring all of the existing code in order to update logging statements? Existing code as in: feature code, integration tests, etc?

Depending on that other questions would arise. Im probably exposing some terrible ignorance on my part with that, so I’l stop there :slight_smile:

1 Like

Heck… I had to look up ‘ELI 5’, so I will be the last person to cast a stone your way!

Trace-based testing, which is what Tracetest does (sort of like Postman does API testing) enables testing at a deeper level. It is really a form of white box or clear box testing. It allows you to not only verify the responses a web page UI sees via a tool like Cypress or the API response a tool like Postman checks, but it allows you to verify anything that should have occurred throughout the system as a result of the test. A few examples:

  • When a user logs in, we should cache his information in Redis so it is available on any subsequent call. Check to make sure this is happening.
  • When a user creates a new account, we should successful send them an email via Mail Chimp - fail the test if this does not happen.
  • All database calls invoked due to this process under test better happen quickly - fail the test if they take longer than 200 ms.

To do these deep tests, trace-based testing depends on your system being instrumented for distributed tracing. OpenTelemetry is the standard the industry is adopting, with all the big observability projects & vendors backing it (Jaeger, Datadog, Grafana, Honeycomb, etc).

To (finally) answer your question - the presentation to management would first be ‘we need to instrument our systems so we can produce distributed traces using a standards based approach, ie OpenTelemetry - it will help our SREs in a production environment get to the root cause of issues a lot quicker.’ (if you dont have it already).

Secondly, if you implement tracing (or already have it), the presentation is 'we have our apps instrumented and producing these great traces that we leverage in production when things go wrong (ie reactively), wouldn’t it be great if we could use the same data proactively and have automated tests for our critical flows before releasing.

This second step, using the trace in trace-based testing is super easy… the first step (ie getting started on observability) takes more of a commitment. Trace-based testing and Tracetest assume you have distributed traces.

Sorry for the long answer… let me know if I can clarify better.

3 Likes

Thank you very much for the informative answer!

Just before I was sent packing by a layoff, I was just getting involved in some observability efforts in my company. So I have something of a vocabulary but not the execution. And during my tenure as a QA analyst and then engineering team lead there I did a lot of rooting around in datadog logs as well as trying to make some sense out of grafana with our kubernetes pods.

I ask this because I want to better understand the technique and how it applies to legacy products where it might take a considerable effort to implement. I also want to understand how this will “find bugs” (because that s what many managers understand as a metric) and finally, I may (fingers crossed) have an opportunity to take on a role at a company where I would be in a position to influence this sort of work. If I get the role and can get managers and engineers excited about it.

Thank you for clarifying much of that for me

3 Likes

One great benefit to both QA and Devs with trace-based testing is the richness of the information that is available for a test run. You do not just see a pass or fail status and the normal results returned by a Cypress or API test, you also get the full distributed trace that was generated for that full test. We have a live demo environment that we are continuously running synthetic tests in (both API and Cypress based) that you can log into - if you look at the tests you will see the test runs under them - each has a full trace as part of its articfacts. As a QA tester, being able to say to the developer ‘this test failed these assertions, and here is a full trace from the execution’ will make everybody happier.

2 Likes

That is, from my experience, a great way to get developers on board with the idea.

2 Likes

I hadn’t heard of trace testing, so I am happy to have learned what it is, especially helpful the ELI5 version of it :wink:

2 Likes

We did a 30 minute session this morning that showed the Cypress integration in detail - would love to hear your thoughts: https://www.youtube.com/watch?v=k0d_ap5jatU

2 Likes

Thats very cool Ken! thank you for doing that. It answered a lot of questions for me in a nice bite sized session. Its likely a factor of me not having the time to try things out or dive deeply into the documentation…I would have liked to see the workflow for adding a test or implementing this in a legacy application where there is an extensive library of Cypress tests as well as separate integration tests.

I think I understand the new test workflow to be (assuming the infrastructure is set up)

  1. Create the new Cypress test using the before and after actions
  2. manually add spans to the tracetest cloud app
  3. profit

for onboarding into a legacy application with extensive existing tests:

  1. set up open telemetry
  2. instrument open telemetry in the application code (ctrl-c ctrl-v everywhere)
  3. Set up tracetest
  4. instrument tracetest in the existing cypress tests (ctrl-c ctrl-v everywhere)
  5. import/recreate integration tests as spans in tracetest cloud app

(obviously this doesnt have to happen all at once or halt all progress until this is all completed)

If this employment opportunity that Im currently pursuing works out Im definitely going to work to onboard this stuff.

1 Like

That is a good run down. 1 & 2 in your list (getting your application so it is setup & instrumented with OpenTelemetry) is the hard part. Once that is done, trace-based testing and Tracetest are easy to set up - minutes with experience… and we are glad to help if you have trouble.

2 Likes