What kinds of questions do you think are still best answered by a community rather than AI?

Discussions
,
1

During my first webinar with the Continuous Testing Meetup, someone asked a question that stayed with me even after the session ended.

“If AI can answer almost anything in seconds, what kinds of questions should we still bring to a community?”

I shared my thoughts during the webinar, but I have been thinking about it ever since.

AI provides answers in a few seconds to a few minutes. Communities, on the other hand, take time, bring experience, diverse perspectives, disagreements, and real-world context.

So I’m curious:

What kinds of questions do you think are still best answered by a community rather than AI?

I would love to hear your perspective.

2

A massive over simplification but in my mind, ask AI to help you solve or give ideas to an individual problem or menial task. Don’t ask AI if your goal is to bring people together.

3

AI is for finding out what other people have decided to share with the world, while community is for finding things that are hidden in our own contexts.

I’m competing with Gary’s excellent compact answer, and finding it hard to beat here obviously. Still perplexed as to how the “appliance” got renamed from “search-agent” to “AI-agent” when it’s often just the same as how all cross-head screwdrivers came to be know as Philips ,throughout Europe, or how vacuums are called Hoovers in the USA.

4

Problem is with community members using AI to answer :slight_smile:

5

If the subject is user experience then understanding the emotional feedback from users will require human processing and not AI. Factual, statistical quant test data is where the AI can really help

6

“If AI can answer almost anything in seconds…”

Isn’t AI just regurgitating community answers (& other sources) and presenting them in a readable form?

I’d ask humans questions that have not been asked before or have a few answers which are not very helpful. I have seen AI get puzzled or hallucinate when I ask it about a recent phenomenon (which occurred in the past few hours or 1-2 days), especially one that has never occurred before. For such things, I just look online for news articles or reddit posts to see what humans say. It looks like AI gets befuddled in such situations probably because it has not yet ingested the recent data.

For the important things in life and work, I’d ask AI to get a list of things that I can investigate further (i.e. not accept it as the ultimate truth). I’d then ask humans the same thing to see what information and counter questions they provide. I’d also ask the humans about the points which AI raised and see if they make sense or not.

7

AI always sounds plausible but is not always correct. We need a community to validate learning.

8

For example: Claude counts this code as 8 test cases:

def check_bad_arguments(exe, out_dir, failures):
  target = str(out_dir / "bad.bmp")
  cases = [
    ["-f", target, "-w", "0", "-H", "5"],
    ["-f", target, "-w", "5", "-H", "5", "-b", "3"],
    ["-w", "5", "-H", "5"],
    ["-f", target, "-w", "5"],
    ["-f", target, "-w", "5", "-H", "5", "-p", "foo"],
    ["-f", target, "-w", "abc", "-H", "5"],
    ["-f", target, "-w", "2000000000", "-H", "2000000000", "-b", "8"],
    ["--nonsense"],
  ]
  for args in cases:
    result = run(exe, *args)
      if result.returncode != 1 or not result.stderr.strip():
      failures.append("bad args {} : expected exit 1 with an error, got {}".format(args, result.returncode))
  return len(cases)

and it’s only running naive negative tests, so it’s giving itself points because a data annotator who was paid to review this said it was a valid test case. It spat out 144 tests in total, and it seems to only have tested the “edges”, but not the positives in bitmap dimensions. Oh dear. It might be quick, but it reality it only halved my testing time, it did not eliminate it to near zero.

9

eight argument combinations can still be one kind of check if they all use the same oracle. the missing part in that example is a positive control: give it valid dimensions, then decode the output and check the actual width and height, not just the exit code.

for the negative cases, i’d also check that the error identifies the bad input and that no usable output is left behind. a community reviewer can ask what behavior is missing, not just whether the generated cases ran.