What white papers are you reading about LLMs?

Discussions
, ,
1

During a This Week in Testing, @al8xr mentioned the following whitepaper:

Mutation-Guided LLM-based Test Generation at Meta

I hadn’t seen a whitepaper for a while and I wondered, are folks diving into the details of advances in Large Language Model (LLM) tech via whitepapers? If so, where do you get your LLM related whitepapers from?

And feel free to comment on this particular whitepaper if you like. Have you seen it before reading this post? What did you make of it?

2 Likes
2

I don’t read white papers much, but as I work in Fin-Tech I was exploring the use cases of LLM in finance and recently came across the below white paper, which focuses on the same:

I usually find such white papers in blogs and articles on different platforms.

3 Likes
4

Why

Reading whitepapers is hard, but beneficial.

At some point in your career, you realize that “there are no new or interesting topics” in testing or automation. (Especially when you have not transitioned to the leadership role). Here comes the whitepapers.

Whitepapers can help you by providing up-to-date research on the given topic. Personally, I am reading and collecting whitepapers on blockchain testing, but I am also reading on LLMs and testing from time to time.

Where to start

General guidelines on how to read papers - here.
Or a good video on the topic.

A few advices

  1. How to search for an interesting whitepaper: find one, go to references, and find even more papers on the topic!
  2. To search for a free whitepaper - use Google Foo and add ‘filetype:pdf’ to your search phrase
  3. Make a list of papers and read it one by one
  4. Some papers are more “general” one - so you can skip part of it
  5. Some papers contain math - so you might spend some time getting through it
  6. Make notes from papers! (There are a bunch of note-taking techniques)

Papers on LLMs

Papers that I added to my ToDo (after reading the paper I mentioned on TWiT session):

  • Testing Web-Enabled Simulation at Scale Using Metamorphic Testing
  • Automated Unit Test Improvement using Large Language Models at Meta
  • Software Testing Research Challenges: An Industrial Perspective
  • Assured LLM-Based Software Engineering
  • Observation-based unit test generation at Meta
  • The Oracle Problem in Software Testing: A Survey
  • What It Would Take to Use Mutation Testing in Industry—A Study at Facebook
  • ChatUniTest: A Framework for LLM-Based Test Generation
  • Mutation-guided LLM-based Test Generation at Meta

I plan to start making reviews on papers in my blog. Because I am collecting a lot of information and not sharing it with the world. This thing should be fixed :grinning:

4 Likes
5

Just forgot to mention.

For those of us who prefer listening to podcasts - you can use Notebook.LM service from Google and turn any whitepaper into a … podcast.

There are some limitations, though - it can’t work with graphs, diagrams, and other images.

But beware of relying solely on such an AI output. It may contain errors (like any software).

2 Likes
6

I m currently reading papers from arxiv, n have found it useful

3 Likes
7

I haven’t, yet, started reading on LLM-Based Test Generation. I would appreciate any feedback on the matter (articles, whitepapers…) has it’s on my top of things to lookout this year.
About META’s Mutation-Guided LLM-Based Test Generation paper, it’s a hard and long read. I’ve found an easier article that gives the high level view:

1 Like
8

The Google Deepmind Gemma Team released this on 12/03/2025:

Gemma 3 Technical Report

1 Like
9

Source: Simon Willison’s Weblog.

10

Thank you for your advice, Oleksandr! Especially regarding the list of papers on LLM

2 Likes
11

Generative AI Act II: Test Time Scaling Drives Cognition Engineering via Cornell University.