I recently attended an insightful meetup hosted by Riverty and Ministry of Testing Berlin, focused on integrating AI into customer service and bug management processes. The event featured key presentations, including AI@Riverty: Agent Co-Pilot (MVP), led by Gregor Schumacher and Jan Eumann, and detailed insight by Anaïs van Asselt, Senior QA Engineer at Choco, about automating bug management with AI.
Session Highlights:
1. AI@Riverty: Agent Co-Pilot (MVP)
Presenter: Gregor Schumacher and Markus Peter
Focus: Introducing RAG (Retrieval Augmented Generation) for AI-powered customer support.
Data Preparation and Challenges:
Data Sources: Content (text, images, flowcharts) is pulled from Confluence.
HTML to Markdown Conversion: Confluence pages are converted into Markdown files to reduce noise.
Document Chunking: Files are split into chunks (approx. 1,000 tokens each), then embedded using the TEXT-3LARGE model to create vector embeddings.
Vector Database: These embeddings are stored in a vector database for efficient retrieval during customer support queries.
Challenges Faced:
Complex Tables: Markdown struggles to render complex tables, often losing data beyond the headers in the first chunk.
Image/Flowchart Context: RAG framework mainly relies on textual input, so extracting meaningful context from images and flowcharts remains challenging.
How the RAG Framework Works:
Query Input: The customer submits a question.
Orchestrator: Generates relevant context from the stored data.
LLM: Uses the context to generate a response.
Output: The system returns an answer to the customer.
RAG Testing Framework:
Input Preparation: Collect a dataset of questions and accurate answers (ground truth).
Generation and Comparison: Use Azure Promptflow to generate answers and compare them against the ground truth.
Analysis: Analyze results via Databricks, focusing on performance metrics like accuracy and error rates.
Discussion points on RAG limitations:
How does the system handle outdated vs. current data from Confluence?
Can the system track the history of responses for future reference?
Does the system have self-learning capabilities to improve responses over time?
2. Revolutionizing Bug Management at Choco with AI
Presenter: Anaïs van Asselt, Senior QA Engineer at Choco
Focus: Automating bug management processes using AI.
Context:
Choco, a B2B messaging app connecting restaurants and suppliers, emphasizes the importance of quick bug management in a fast-paced environment.
Key Automation Stages:
Bug Management Automation: Using AI to streamline the bug handling process.
Bug Creation Automation: Automating the generation of bug tickets based on customer reports.
Recommendation Automation: Offering recommended solutions or workarounds for known issues.
Bug Management Process Using Intercom, Zapier, and Slack:
New Bug Submission: A customer reports an issue through Intercom.
Zapier Automation:
Retrieves a list of existing bugs from Google Sheets.
Compares the new bug description with the database using OpenAI.
Automated Responses:
If the bug matches a known issue, Zapier suggests a solution or workaround.
If no match is found, a new bug ticket is created in Jira.
Slack Notification: The team is notified via Slack with details about the bug (either a match or a new issue).
Outcome:
This approach reduces the chance of missed bugs, improving response time and overall customer satisfaction.
Conclusion:
The Meetup highlighted innovative uses of AI in both customer service and bug management, providing insights that can enhance operational efficiency and offering potential improvements in our organizational workflows.