🤖 Day 7: Research and share prompt engineering techniques

[sabine.k (Sabine Kolics)]

Prompt Engineering (source:

Prompt Engineering | Definition und Beispiele - IONOS AT) are techniques and methods to optimizes prompt for Natural language Processing and Large Language Models, as GPT-3 or GPT4, based on Machine Learning. The goal is to get better more precise and specific answers, as it is very important how a question or statement is formulated, and as big impact of the quality and relevance of the answer provided by AI. Prompt Engineering for AI needs creativity and precision to fully understand the speech model, as the selection or order of word can have a big impact to the output. The same prompt can come to different results on different AI platforms. Therefore, Prompt Engineering needs to be processed individually for every AI test generator, AI image generator or AI video generator.

Typical prompt engineering process to develop a communication interface could look like:

(Source: Was ist Prompt Engineering? - Wissen kompakt - t2informatik )

• Data collection: First, you collect data to build the system. This data can come from various sources such as customer dialogues, chat logs or other similar sources.
• Data preparation: The collected data must be prepared and cleaned to remove invalid or unusable data and to make the system efficient.
• Model selection: Next, choose a suitable ML model to train the system. Here you can choose between different models such as regression models, neural networks or decision tree models.
• Model training: The selected model is trained on the prepared data to tune it to the task to which it is to be applied
• Deployment: Once the model is trained, it can be deployed to be used in a productive environment.
• Monitoring and optimization: Finally, the system is monitored and continuously optimized to ensure that it meets current requirements and provides a good user experience.

I looked at the following techniques source from https://www.promptingguide.ai/

Prompt technique 1: Zero-shot Prompting source: Zero-Shot Prompting | Prompt Engineering Guide<!-- -->

• How it works: e.g., you are directly prompting the model for a response without any examples or demonstrations about the task you want it to achieve. Some large language models have the ability to perform zero-shot prompting but it depends on the complexity and knowledge of the task at hand and the tasks the model was trained to perform good on.
• Potential impact: It can help to finetune models on datasets.

Prompt technique 2: Few-shot Prompting source: Few-Shot Prompting | Prompt Engineering Guide<!-- -->

• How it works: Few-shot prompting can be used as a technique to enable in-context learning where we provide demonstrations in the prompt to steer the model to better performance. The demonstrations serve as conditioning for subsequent examples where we would like the model to generate a response.

Prompt technique 3: Chain-of-Thought Prompting source: Chain-of-Thought Prompting | Prompt Engineering Guide<!-- -->

• How it works: chain-of-thought (CoT) prompting enables complex reasoning capabilities through intermediate reasoning steps. You can combine it with few-shot prompting to get better results on more complex tasks that require reasoning before responding.
• Potential impact: When applying chain-of-thought prompting with demonstrations, the process involves hand-crafting effective and diverse examples. This manual effort could lead to suboptimal solutions. Zhang et al. (2022)(opens in a new tab) propose an approach to eliminate manual efforts by leveraging LLMs with “Let’s think step by step” prompt to generate reasoning chains for demonstrations one by one. This automatic process can still end up with mistakes in generated chains. To mitigate the effects of the mistakes, the diversity of demonstrations matter. This work proposes Auto-CoT, which samples questions with diversity and generates reasoning chains to construct the demonstrations.

Auto-CoT consists of two main stages:

• Stage 1): question clustering: partition questions of a given dataset into a few clusters
• Stage 2): demonstration sampling: select a representative question from each cluster and generate its reasoning chain using Zero-Shot-CoT with simple heuristics

The simple heuristics could be length of questions (e.g., 60 tokens) and number of steps in rationale (e.g., 5 reasoning steps). This encourages the model to use simple and accurate demonstrations.

[vjmobby (Vijay Kumar)]

Prompt Engineering Techniques
Many advanced prompting techniques have been
designed to improve performance on complex tasks

1. Few-shot prompts
2. Chain-of-thought (CoT) prompting
3. Self-Consistency
4. Knowledge Generation Prompting
5. ReAct

The above five techniques are very interesting & I most liked is Chain-of-Thought (CoT) - Zero-Shot COT & Self-Consistency.

Chain-of-Thought (CoT)

Prompting can be further improved by instructing the
model to reason about the task when responding.

• This is very useful for tasks that requiring reasoning.
• You can combine it with few-shot prompting to get better results.

The odd numbers in this group add up to an even number: 4,
8, 9, 15, 12, 2, 1.
A: Adding all the odd numbers (9, 15, 1) gives 25. The
answer is False.
The odd numbers in this group add up to an even number: 15,
32, 5, 13, 82, 7, 1.
A: Adding all the odd numbers (15, 5, 13, 7, 1) gives 41.
The answer is False.

Zero-Shot CoT
Involves adding “Let’s think step by step” to the original
prompt.

I went to the market and bought 10 apples. I gave 2 apples
to the neighbor and 2 to the repairman. I then went and
bought 5 more apples and ate 1. How many apples did I
remain with?
11 apples

I went to the market and bought 10 apples. I gave 2 apples
to the neighbor and 2 to the repairman. I then went and
bought 5 more apples and ate 1. How many apples did I
remain with?
Let’s think step by step.
First, you started with 10 apples.
You gave away 2 apples to the neighbor and 2 to the
repairman, so you had 6 apples left.
Then you bought 5 more apples, so now you had 11 apples.
Finally, you ate 1 apple, so you would remain with 10
apples.

Self-Consistency

• Self-Consistency aims to improve on the naive greedy decoding used in chain-of-thought prompting.
• The idea is to sample multiple, diverse reasoning paths through few-shot CoT, and use the generations to select the most consistent answer.
• This helps to boost the performance of CoT prompting on
  tasks involving arithmetic and commonsense reasoning.

Example:-

Q: There are 15 trees in the grove. Grove workers will plant trees in the
grove today. After they are done, there will be 21 trees. How many trees did
the grove workers plant today?
A: We start with 15 trees. Later we have 21 trees. The difference must be the
number of trees they planted. So, they must have planted 21 - 15 = 6 trees.
The answer is 6.

[futurelady25 (Amruta Bhokare)]

System message

System message	User	Assistant
You’re an AI assistant that helps people find information and responds in rhyme. If the user asks you a question you don’t know the answer to, say so.	What can you tell about me, John Doe?	Dear John, I’m sorry to say,
But I don’t have info on you today.
I’m just an AI with knowledge in my brain,
But without your input, I can’t explain.
So please tell me more about what you seek,
And I’ll do my best to give you an answer unique.

Few-shot learning

System message	Few-shot examples
Assistant is an intelligent chatbot designed to help users answer their tax related questions.
Instructions

• Only answer questions related to taxes.
• If you’re unsure of an answer, you can say “I don’t know” or “I’m not sure” and recommend users go to the IRS website for more information.|User - “When do I need to file my taxes by?”
  Assistant - "In 2023, you’ll need to file your taxes by April 18th. The date falls after the usual April 15th deadline because April 15th falls on a Saturday in 2023. For more details, see When to file | Internal Revenue Service "

User - “How can I check the status of my tax refund?”
Assistant - "You can check the status of your tax refund by visiting Refunds | Internal Revenue Service "|

Break the task down

System message	User	Assistant
You’re an AI assistant that helps people find information.	You’ll read a paragraph, and then issue queries to a search engine in order to fact-check it.

---

PARAGRAPH
John Smith is married to Lucy Smith. They have five kids, and he works as a software engineer at Microsoft. What search queries should I do to fact-check this?

Now you’ll extract factual claims, and issue queries to fact-check them. When issuing a query, use the function SEARCH(“query”)

FACTUAL CLAIMS|1. John Smith is married to Lucy Smith
2. They have five kids
3. He works as a software engineer at Microsoft

QUERIES
SEARCH(“John Smith married Lucy Smith”)
SEARCH(“John Smith family size”)
SEARCH(“John Smith Microsoft software engineer”)|

[simon_tomes]

@lisacrispin

Don’t trust everything you read
Ask multiple times in multiple ways

This is powerful advice, @lisacrispin.

Thanks for amplifying Rachel Kibler.

[mikeharris]

What an interesting exercise

1. Tree of Thought

• https://www.promptingguide.ai/
• I like the Tree of Thought prompt because I often analyse applications as a tree. It works by building a tree and choosing the result from the tree that it reasons is correct. I tried the examples I found on these pages:
  • Tree of Thoughts (ToT) | Prompt Engineering Guide<!-- -->
  • GitHub - dave1010/tree-of-thought-prompting: Using Tree-of-Thought Prompting to boost ChatGPT's reasoning
• and ChatGPT answered the question correctly but Gemini did not! I could not find a link to Gemini about Tree of Thought
• Tree of Thought “substantially outperforms the other prompting methods” so its impact could be considerable. If it is more accurate we all may start to use it.

2. Zero-Shot Prompting

• In natural language processing models, zero-shot prompting means providing a prompt that is not part of the training data to the model, but the model can generate a result that you desire.
• I think that this method is probably what we all do mostly and so it has a wide impact.
• https://machinelearningmastery.com/what-are-zero-shot-prompting-and-few-shot-prompting/

3. Few-Shot Prompting

• If you cannot describe what you want but still want a language model to give you answers, you can provide some examples.
• I think that we also all probably do this quite a bit too! So it also has a wide impact
• https://machinelearningmastery.com/what-are-zero-shot-prompting-and-few-shot-prompting/

[maaret (Maaret Pyhäjärvi)]

I was on a (Finnish) Testing & AI conference yesterday, and picked up a local reference that I kind of like. I think you might too, so sharing 85 AI Prompts for Software QA Professionals by Prove.

Notion – The all-in-one workspace for your notes, tasks, wikis, and databases.

In the presentation, Antti Niittyviita shared the prompt of classic triangle exercise as image and felt like ChatGPT4 did well on identifying things based on picture. He also reminded the audience that these systems now are at their worst - they get better over time. So if their worst today is impressive, we may want to follow along to see them evolving.

As I am realizing “classic triangle exercise” holds more meaning to me than most people, I will add this. In book Art of Software Testing by Glenford Myers, 1979, he presented triangle exercise as evidence of how badly people do testing. He had done research giving the problem to programmers (time before the tester profession) and discovered that pretty much no one knew how to test it. This became classic as there is a series of books that criticize Myer’s approach, articles claiming better job with ISTQB-style equivalence class analysis, one most known and thorough being part of the book on Testing Object-Oriented Systems. I used to be a researcher and I read these things. So when someone drops them as “here’s how AI does it” I have a lot of people doing it in writing samples to compare against.

[indivicivet (I X)]

Concise Guide to Prompt Engineering Techniques

Technique 1: Iterative Refinement

• Example: Initial: “Summarize AI advancements.” → After AI response (too general): “Focus on AI in healthcare since 2023.”
• Impact: Refines AI outputs through feedback, achieving targeted and relevant responses.

A particular case that I’ve found works well with ChatGPT4 (although much less so with ChatGPT 3.5) is to point out things you dislike about the response. It seems like one of the main distinctions between current cutting edge LLMs and one generation back are how well they respond to negative feedback. Some thematic examples are:

“please refine it in the following way: make it shorter”
“give explicit examples for each case”
“remove all the pleasantries about and keep it to the point”

ChatGPT3.5 gives lower-quality responses to these refinements but ChatGPT4 seems to give me better responses.

Technique 2: Precision in Language and Context

• Example: “Translate ‘sustainable energy’ into French, for an academic paper.”
• Impact: Clarifies expectations, resulting in accurate and context-appropriate AI responses.

Technique 3: Leveraging Examples

• Example: “Write a product description like Apple’s iPhone 12 description, but for an eco-friendly smartphone.”
• Impact: Guides AI to produce outputs matching the style and format of the provided examples.

[sargam (Sargam Panwar)]

Greetings Everyone!
Earlier today I had no idea about what ‘Prompt Engineering’ is, but here are my insights after exploring a lot about this topic.

• Prompt technique 1: Zero-shot prompt
• How it works: Even if the tool has not been prompted to perform actions on certain words, still it will manipulate the input and give the output in related terms.
  For example: if we ask the tool to generate a summary for any specific article or blog, and even if it is not trained to do that, still it will provide a summary by manipulating the input accordingly.
• Potential impact: It can help the AI to train the models in such a way that , even if they are not specifically coded to perform an action, they would still be able to provide an output which can be used to get the exact output the user wants.

[poojitha-chandra (Poojitha Manam)]

@maaret

so sharing 85 AI Prompts for Software QA Professionals by Prove.

Thanks for sharing this informative article Maaret! 13 building blocks for prompts, 24 outcomes to ask for, 10 thinking frameworks really stood out for me along with the rest of the sections. Waiting for the version 3.0

[dover (Dover Hellfeldt)]

1. Adjust Large Language Models parameters. With adjustments on parameters is possible to increase confidence on outputs. For example, adjusting Temperature, “the lower the temperature, the more deterministic the results in the sense that the highest probable next token is always picked. Increasing temperature could lead to more randomness(poems).”
   See details here LLM Settings | Prompt Engineering Guide<!-- -->

2. Self-consistency to improve chain of thought reasoning. We can use this technique to improve the model to consistently respond the prompt based on how we present it our chain of thought - next time we ask a question, or we create a testing scenario, for example.
   “Although language models have demonstrated remarkable success across a range of NLP tasks, their ability to demonstrate reasoning is often seen as a limitation, which cannot be overcome solely by increasing model scale…In an effort to address this shortcoming, Wei et al. (2022) have proposed chain-of-thought prompting, where a language model is prompted to generate a series of short sentences that mimic the reasoning process a person might employ in solving a task. For example, given the question “If there are 3 cars in the parking lot and 2 more cars arrive, how many cars are in the parking lot?”, instead of directly responding with “5”, a language model would be prompted to respond with the entire chain-of-thought: “There are 3 cars in the parking lot already. 2 more arrive. Now there are 3 + 2 = 5 cars. The answer is 5.”. It has been observed that chain-of-thought prompting significantly improves model performance across a variety of multi-step reasoning tasks” .Reference: [2203.11171] Self-Consistency Improves Chain of Thought Reasoning in Language Models

[mahatheed (Mahathee Dandibhotla)]

The Prompt Engineering task is huge.Based on my analysis here is my response to this task.

1. Zero-shot Prompting

• How it works: Generates outputs without specific training or examples. Useful for quick answers to general questions.
• Potential impact: Allows AI to respond to novel tasks without specific coding.
• Resource: What is Zero Shot Learning in Computer Vision?

2. Few-Shot Prompting

• How it works: Provides a few examples to guide AI responses.
• Potential impact: Enables more accurate and context-specific outputs.
• Resource: Few-Shot Prompting Guide

3. Chain-of-Thought Prompting (CoT)

• How it works: Involves breaking complex tasks into smaller reasoning steps.
• Potential impact: Facilitates multi-step reasoning, improving AI’s problem-solving capabilities.
• Resource: Chain-of-Thought Prompting Guide

4. Role-based Prompting

• How it works: Assigns a role or persona to the AI.
• Potential impact: Helps guide AI to adopt specific perspectives or tones.
• Resource: Role-based Prompt Engineering

5. ReAct Prompting

• How it works: Involves breaking down tasks into a series of reasoning steps with specific actions.
• Potential impact: Useful for complex problem-solving tasks.
• Resource: ReAct Prompting

6. Self-Consistency Prompting

• How it works: Aims for consistent responses by focusing on context and coherence.
• Potential impact: Ensures outputs remain cohesive and on-topic.
• Resource: Self-Consistency Prompting Guide

7. Iterative Prompting

• How it works: Builds upon previous responses with follow-up questions.
• Potential impact: Useful for deeper exploration and clarifications.
• Resource: Iterative Prompting

8. Retrieval Augmented Generation (RAG)

• How it works: Accesses external knowledge to ground AI’s responses.
• Potential impact: Helps combat hallucinations and provides more reliable outputs.
• Resource: RAG Prompting Guide

Each technique has its unique strengths, and choosing the right one depends on the task’s complexity and desired output

[mahatheed (Mahathee Dandibhotla)]

Also watched this interesting and easy introductory video by @pricilla09 again.

If you’re a beginner recommend or suggest starting with this.

Simple explanation without confusion

Link: https://www.youtube.com/watch?v=fgJm4flXCzo

Notes

• Focusing on crafting efficient prompts to get better results from Generative AI and other Large Language Models (LLMs).
• The goal is to explore various techniques, tools, and best practices for effective prompt engineering.

Prompt Engineering Basics

• What is a Prompt?: A prompt is a group of texts, lines, or passages that instructs the AI or LLM to produce desired outputs. Effective prompts lead to better results.
• Prompt Engineering: The art of creating prompts that guide the AI to generate precise and accurate responses.

Prompt Techniques

• Instruction: Start prompts with verbs or action items to direct the AI. Examples include “summarize,” “condense,” or “draft.”
• Role Play: Instruct the AI to assume a specific role, such as a doctor or a superhero, to generate contextually appropriate responses.
• Examples: Provide examples to guide the AI towards the desired output.

Types of Short Prompting

• Zero-shot: No examples are provided, and the AI predicts the output based on its training.
• One-shot: One example is given, guiding the AI to produce the desired response.
• Few-shot: Multiple examples are provided, resulting in improved output quality.

Advanced Concepts in Prompting

• Chain of Thought: Involves breaking down a complex task into smaller steps, enabling AI to explain its reasoning and provide a more comprehensive answer.
• Self-Consistency: Providing diverse examples to encourage consistent and accurate responses.
• Tree of Thoughts: A recent framework that allows the AI to evaluate multiple paths, helping to find the most efficient or correct solution.

Tools for Prompt Engineering

• Prompts Royal: A tool for developing and generating prompts, especially helpful for beginners.
• OpenAI Playground: A platform for experimenting with different modes and models of OpenAI, allowing for prompt testing and refinement.

Best Practices for Prompt Engineering

• Be clear and specific, structuring prompts with examples.
• Avoid overload of information and be mindful of open-ended questions.
• Define constraints and output styles, such as word count and tone.
• Practice and refine your skills through continuous experimentation

[pricilla09 (Pricilla Bilavendran)]

The mindmap is mind blowing @mahatheed, thanks for sharing that

Some more resources along with the one used in my talk can be found here: GitHub - Pricilla09/TestFlix2023