Rosie, appreciate you hosting us! And thanks to all the attendees for the attention.
Below, I’ll post the consolidated responses for both answered and unanswered questions for easier review. As some of the questions overlap, I’ve broken the answers by 4 topics.
Topic 1 – “External support”
-Can we use this with an external/local model ? maybe based on URL and API key ?
-Could we train with our own data ? Does it leverage Rovo ?
Answer:
At the moment, you cannot bring any external model or train/fine-tune with any extra data, including your own. The AI features shown in this webinar are not connected to Rovo from Atlassian Intelligence, however, there is a separate project to allow Rovo to support Xray entities, more information on that will be shared in the future.
Topic 2 – “Access, Licensing, and Credits”
-In which version will we get access to this new AI feature ? was it 7.0.0 ?
-Is this only available in Xray cloud?
-Will the AI feature be available in Xray DC or cloud or both?
-Do we need Xray enterprise? Or will there be a separate license?
-is this something that would require a separate enterprise license? or included as part of the main xray enterprise team license
-How can I join the beta to experiment with this feature?
-How to find out more about the new AI feature in Xray. How to use it, pricing, privacy, details on the model, etc.?
-What do you recommend for getting started? How are people approaching that?
-How does the credit pricing work if we need more credits?
-How many AI credits does your example (28 tests) use up?
-How many base credits come with enterprise?
-Can we limit the number of test case suggestions generated (to control AI credit usage)?
-How about the risk of test case explosion which might kill the performance during generation or leads to a lot of review effort?
-Since this uses some kind of credits, how can we limit the AI to not generate 1000s of tests or get stuck in some loop and finish all the credits fast ?
-what performance consideration should we keep in mind when using xray Ai for thousands of test cases ?
-What would be the process of getting Xray API token? Would it be similar to getting a JIRA API token (as in would it be included in that token)
Answer:
The AI features are available only in Xray Cloud with the release of 6.9.0 version, which is now live! You can sign up for access to the beta using this form.
AI Test Case Generation is included in the Xray Standard license while AI Test Model Generation is included in Xray Advanced and Enterprise licenses; no separate license purchase needed. You can review the feature breakdown between Xray versions here.
To get started, you can check out the overview articles (this and this) and then just jump into the experimentation on the smaller set of requirements that you already have tests for, so that you can compare the output.
The demo shown during the webinar used 2 AI credits for Test Case Generation (1 for returning suggested topics, titles, and descriptions; 1 for returning the full test content).
For Test Model Generation, there is a separate limit described in the last part of the overview article - “AI Test Model Generation API has a limit of total requests on a monthly basis, 250 per user account (each AI action = 1 request; it resets on the 25th of each month).”. The webinar demo also used 2 credits - 1 for the initial bulk generation, 1 for the individual “Currency” parameter generation.
So, the credit consumption is not proportional to the number of returned entities, it only depends on the number of successful AI requests. With that said, the number of returned suggestions is capped at 60 tests per request for Test Case Generation and ~10 parameters (“softer” restriction) for Test Model Generation. That is intentional not just for performance purposes but also to make the reviews easier and more “concentrated”. You would not be in a position to generate 100s or 1000s of tests or parameters without explicitly and intentionally submitting multiple AI requests.
When it comes to exact credit limits per Xray version, we are currently more focused on encouraging usage and experimentation, so the numbers are “in flux”.
Xray API token is created separately in the Xray global settings (for the avoidance of doubt, it is not connected to the AI feature).
Topic 3 - “Feature Scope”
-What is the underlying model being used (if you can disclose) and assume the internal data is not used to train it?
-Garbage in Garbage out - will the quality of requirements impact the tests outcome, does the agent learn the domain using JIRA linked stories and epics etc?
-how about the results with a „wild“ system documentation?
-how do u manage test overlapping/duplicates?
-For generation - Will it focus on one item during generation? Or is AI able to consider content from referenced issues?
-If the requirements have linked Figma or figjam for UI workflows and designs - will AI be able to read and extract information to generate the manual tests? or any other docs
-Does it provide the edge cases and the negative tests if those are not mentioned in the requirements, Using AI tests generation
-Will it cover boundary value Test cases ,negative scenarios while generating
-Do u support gherkin syntax? if so, do u create reusable business logic steps?
-how does xray AI gather context for shared/basic steps (login, navigation, etc)
-Is the ‘callable’ functionality of tests still available? How?
-Can you parameterize tests?
-Is there a way to provide a template of existing testcase to follow.
-Is it possible to apply a particular test design method while generating the Tests?
Answer:
Currently, the underlying models are Amazon Nova Lite for Test Case Generation and Amazon Nova Pro for Test Model Generation, based on the benchmarking for these particular activities. We are also currently transitioning Test Case Generation to Google Gemini. Both Nova and Gemini are listed as sub-processors. You can learn more about that aspect and other security/privacy details in this article as well as the ones linked inside the AI feature dialogs in the tool itself.
The quality of input context still plays a large role in determining the quality of the output. Xray AI will not automatically “learn” from the broader Jira as we do not allow extra training or fine tuning. But you can link any necessary Epics, Stories, etc. to the generation task. One of the main goals for the beta period is to get more feedback on the AI performance “in the wild”, so we are looking forward to learning more about your successes or challenges with the hands-on experience.
Xray AI for Test Case Generation will consider all work items you explicitly share in the dialog, based on their relevance to the goal you specify in the “requirements” field. Only the text content from Summary and Description of those linked items will be processed - not attachments, other fields, Figma designs (or other complex embedded entities), links (external or to other Jira items), etc.
AI Test Model Generation will accept the upload of text-based formats but will ignore any content inside that is not text (e.g. in a pdf with text and images, text will be processed, images will be ignored).
The back-end Sembi iQ service that powers Xray AI features has multiple guardrails to prevent duplications (in addition to overall prompt engineering optimizations), however contextual redundancies could still happen. If they do, please let us know.
Edge cases, boundary testing, and negative scenarios are supported in both AI features. Test Case Generation can suggest them all without you explicitly including the instruction. Test Model Generation typically focuses on positive testing by default, so you can get boundaries out-of-the-box, but you may need to change the goal wording to get edges/negatives. You also need to be mindful of the requirements wording that potentially excludes negative variation - e.g. the goal of “Create tests that generate calculator estimates” can be interpreted as positive only by AI (because negatives would result in an error instead of the estimate).
Gherkin syntax is supported, however it currently does not leverage the BDD steps library for reusability, so you may get steps that perform similar actions but have different wording. Modular/call tests are not currently supported.
Parameterized tests are not currently in scope of Test Case Generation, however Test Model Generation is specialized in returning parameters and values, then the dataset from Test Case Designer could be imported into Xray (without Scripting in Test Case Designer).
It is not currently possible to provide the template (beyond determining the Manual vs Gherkin kind) or the recommended test design method.
Topic 4 - “Other Nuances”
-Once you have created tests, and if the requirements gets updated can you generate the tests for the updated part only?
-how about maintaining my test base? will it propose updates on existing test cases?
-Can Xray Ai leverage defect trends, requirement criticality or execution history to prioritize or generate more risk focused test cases automatically?
-will it give me a hint if I choose contradicting items for generation?
-would you recognize/flag contradictions in requirements
-Does it provide the ability to convert the manual tests to automated scripts(sorry if i missed that piece)
Answer:
When it comes to maintenance and updates, at the moment AI will not rework your test suite, so you would have to update the context in the first step, produce a new set of suggested tests, then figure out the optimal way to “merge” the two suites. You can try adjusting the content in the first step of the AI dialog to focus more/only on the updated requirements, so that the generated results are more aligned with them. However, you still won’t have “surgical” control over results.
Risk-based testing would not be automatic based on the analysis of defect trends, requirements criticality, etc. that exist “somewhere” in Jira. However, the wording of work items you link and the requirements field you specify can impact the prioritization of returned test cases.
There would not be a direct warning about the contradictions, it would fall onto the user to recognize that at the “review” stage for titles/descriptions and to exclude/edit items accordingly.
The features from this webinar will not handle the conversion of manual scripts to automated ones, but we are evaluating another feature that would tackle this use case, more information will be shared in the future.