
By making a new protocol known as the Model Context Protocol (MCP) open-source, Anthropic made a major change in AI. By overcoming the limitations of traditional data integration techniques and addressing the recurrent problem of data isolation, this protocol aims to enhance the connections between data hubs and AI systems.

## Understanding Model Context Protocol (MCP)

The Model Context Protocol (MCP) follows a client-host-server architecture where each host can run multiple client instances. This architecture enables users to integrate AI capabilities across applications while maintaining clear security boundaries and isolating concerns. MCP uses a client-server architecture with an emphasis on local-first connections. This choice reflects current concerns around privacy and security, especially as AI systems increasingly access sensitive data. Built on JSON-RPC, MCP provides a stateful session protocol focused on context exchange and sampling coordination between clients and servers.

MCP is a structured approach that allows AI systems to communicate effectively with various data sources and tools. It consists of several key components:

- **MCP host**: Programs like Claude Desktop, IDEs, or AI tools that want to access resources through MCP.
- **MCP clients**: Protocol clients that maintain 1:1 connections with servers.
- **MCP servers**: Lightweight programs that each expose specific capabilities through the standardized Model Context Protocol.
- **Local resources**: Your computer’s resources (databases, files, services) that MCP servers can securely access.
- **Remote resources**: Resources available over the internet (e.g., through APIs) that MCP servers can connect to.

![](assets/model-context-protocol-architecture.webp)

## Integration multiple source for AI assistants

MCP significantly enhances AI assistants by enabling seamless integration with various data sources and tools. Here’s how:

- **Integration with data sources**: AI assistants can pull in relevant information from different databases, include platforms like Google Drive, Slack, GitHub, Git, Postgres, and Puppeteer.
- **Interaction with business tools**: MCP allows AI assistants to work directly with tools used in businesses, improving efficiency.
- **Interaction with local resources**: MCP server shares more than just data as well. In addition to resources (files, docs, data), they can expose tools (API integrations, actions) and prompts (templated interactions)

Examples of using the Claude Desktop as an MCP client ( MCP Host ) might look like:

```json
{
  "mcpServers": {
    "filesystem": {
      "command": "npx",
      "args": [
        "-y",
        "@modelcontextprotocol/server-filesystem",
        "/path/to/allowed/files"
      ]
    },
    "git": {
      "command": "uvx",
      "args": ["mcp-server-git", "--repository", "path/to/git/repo"]
    },
    "github": {
      "command": "npx",
      "args": ["-y", "@modelcontextprotocol/server-github"],
      "env": {
        "GITHUB_PERSONAL_ACCESS_TOKEN": "<YOUR_TOKEN>"
      }
    },
    "postgres": {
      "command": "npx",
      "args": [
        "-y",
        "@modelcontextprotocol/server-postgres",
        "postgresql://localhost/mydb"
      ]
    }
  }
}
```

## Perspectives and future developments

### AI agents and workflows are game-changers

AI agents are changing by the Model Context Protocol (MCP), which provides them with the resources they require to operate more effectively and intelligently. By allowing their seamless integration into different systems and data sources, MCP makes AI agents above their current capabilities of job automation and decision-making. MCP improves their performance in the following ways:

- **Speeds up tasks:** MCP makes it easier for agents to obtain the data they want, allowing them to finish tasks more quickly and without encountering data bottlenecks.
- **Handles growth easily:** MCP makes sure agents can easily adjust to new procedures or scale up to accommodate more users in order to meet evolving demands.
- **Makes better decisions:** MCP assists agents in making more informed decisions by providing them with up-to-date, precise data, which benefits both users and businesses.

MCP not only improves the performance of AI agents, but it also creates completely new opportunities for automation, creativity, and more intelligent business processes.

![](assets/model-context-protocol-agent-workflow.webp)

### MCP will be the new backbone of AI workflows

MCP that allows AI agents to connect with a variety of platforms and data sources using a standardized framework. With MCP, AI agents can transition from basic task automation to executing **complex, adaptable workflows**. This capability is a game-changer for industries looking to integrate AI seamlessly into [their operations](https://www.forbes.com/sites/janakirammsv/2024/11/04/why-agent-orchestration-is-the-new-enterprise-integration-backbone-for-the-ai-era/).

1. **Standardized interactions:** MCP gives AI a consistent approach to access and use data across platforms, local resource and more.
2. **Local-first architecture:** MCP prioritizes security, ensuring sensitive data is secured while providing frictionless processes.
3. **Open-source innovation:** MCP creates a robust context which supports continuous enhancement and innovation.

![](assets/model-context-protocol-mcp-approach.webp)

While there are several frameworks for developing AI agents, almost all of them rely on tools which make implement of the LLM's function-calling capabilities. Though it is the correct approach, this mechanism restricts the amount of tools and functions that the LLM may assign to a prompt or inquiry.

Anthropic's MCP significantly enhance AI agents' capabilities by allowing for direct, bidirectional communication with external systems. This development enables AI agents to retrieve real-time data from external databases, handle file systems, and connect smoothly with platforms such as GitHub. As a result, AI agents can do complicated activities independently, such as improving software development workflows, which increases productivity and reduces the need for manual intervention.

### Business adoption

One of the main challenges to enterprise adoption of AI is its integration. Every new AI tool that is implemented in a company usually needs a custom integration, which frequently leads to costly maintenance and time-consuming development. By offering pre-built interfaces for well-known enterprise platforms like GitHub, Google Drive, and PostgreSQL, MCP aims to simplify this process.

MCP is already being used by companies like Block and Apollo, showing how handy it is in real-world scenarios. Dev tool makers like Zed, Replit, and Codeium are also tapping into MCP to make their platforms better. Its popularity shows how it’s totally changing the game for how AI systems deal with and use data.

## Reference

- https://github.com/modelcontextprotocol
- https://www.anthropic.com/news/model-context-protocol
- https://github.com/modelcontextprotocol/servers/blob/main/src/github/index.ts
- https://medium.com/towards-applied-generative-ai/ai-agents-agentic-workflows-f558674ee18b
- https://www.metadocs.co/2024/11/29/anthropic-model-context-protocol-how-to-give-wings-to-local-mode/
- https://www.infoworld.com/article/3613143/anthropic-introduces-the-model-context-protocol.html
- https://salesforcedevops.net/index.php/2024/11/29/anthropics-model-context-protocol/


Explore Anthropic's Model Context Protocol (MCP) - a groundbreaking approach to enhancing AI assistants. Learn how MCP integrates with local and remote resources, enabling smarter interactions with tools and databases.

Intro to model context protocol


## Aider v0.63.0 release

- Support for Qwen 2.5 Coder 32B.
- `/web` command just adds the page to the chat, without triggering an LLM response.

Qwen 2.5 Coder 32B has been making waves in the past week. This could be a turning point for self-hosting models for coding without needing to call a third-party API. Additionally, the new `/web` command allows developers to easily input a URL as documentation context for Aider.

Here's an example of how to use the `/web` command. The developer provides the documentation URL for initializing Langfuse, which is used to trace API calls for a wrapper.

![](assets/digest-02-web-command.webp)

## OpenHands

OpenHands just took a huge step forward with _All Hands Online (Beta)_, a new hosted version of its AI software development tool. This means no more wrestling with setup—just go to app.all-hands.dev, and you’re good to go in seconds! OpenHands has already proven itself by solving over half of real GitHub issues on the SWE-Bench Verified benchmark, and now it’s easier than ever to get started.

![](assets/digest-02-new-ui-openhand.webp)

The hosted version also comes with a fresh, redesigned interface and smooth GitHub integration. With the GitHub integration, you can hook up your repos directly, clone projects, and push changes right from OpenHands, keeping everything in one place. And this is just the beginning—more features are on the way to make the tool even smarter, more customizable, and ready to tackle bigger challenges.

![](assets/digest-02-github-integrated-openhand.webp)

## Qwen2.5 Coder 32B

The Qwen team is excited to announce the open-source release of the Qwen2.5-Coder series, a set of highly capable AI coding models designed for practical, real-world programming tasks. Headlining the series is the Qwen2.5-Coder-32B-Instruct, which has set a new standard as the top open-source code model, rivaling the performance of commercial models like GPT-4o. With six model sizes ranging from 0.5B to 32B, Qwen2.5-Coder meets diverse developer needs, whether for lightweight tasks or complex code generation. The models excel in coding benchmarks like EvalPlus and BigCodeBench, offering robust performance across over 40 programming languages, making it a flexiable assistant for a wide range of coding needs.

![](assets/digest-02-qwen-bench-1.webp)

In addition to coding and code repair, Qwen2.5-Coder also demonstrates strengths in code reasoning and multi-language support, with impressive scores on benchmarks such as McEval and MdEval. This release is a big win for open-source developers who want a powerful, accessible alternative to proprietary models for code generation, repair, and reasoning tasks. Qwen2.5-Coder is also licensed under Apache 2.0, ensuring flexibility and ease of use. The team is looking forward to expanding its capabilities, exploring advanced reasoning models, and unlocking new possibilities in collaboration with the coding community.

![](assets/digest-02-qwen-bench-2.webp)

## Predicted outputs - OpenAI

Predicted Outputs let you speed up model responses by giving the model a heads-up about parts of the response you know won’t change. This is super handy when you’re making small edits to code or text since most of it stays the same. You just pass the existing content as a "prediction" to the model, and it can skip regenerating what it already knows.

For example, say you have a function that logs messages, and you want to tweak it to include the current date. Instead of rewriting the whole thing, you send the original function as a prediction. The model then quickly returns just the updated version, where it logs both the date and the message, saving time and effort.

### Original code

```jsx
function logMessage(message) {
  console.log("Log:", message);
}

export default logMessage;
```

### Prediction setup and code update prompt

In this case, we want to modify the `logMessage` function to include the date. Here’s how you’d set up the request with a `Predicted Output`:

```jsx
import OpenAI from "openai";

const code = `
function logMessage(message) {
  console.log("Log:", message);
}

export default logMessage;
`.trim();

const openai = new OpenAI();

const updatePrompt = `
Modify the logMessage function to log the current date and the message.
Return only the code.
`;

const completion = await openai.chat.completions.create({
  model: "gpt-4o",
  messages: [
    { role: "user", content: updatePrompt },
    { role: "user", content: code },
  ],
  prediction: {
    type: "content",
    content: code,
  },
});

// Inspect the response
console.log(completion.choices[0].message.content);
```

### Expected response

The model will likely return the following:

```jsx
function logMessage(message) {
  const date = new Date().toLocaleString();
  console.log("Log:", date, message);
}

export default logMessage;
```

In this example:

- `Predicted Outputs` speed up the response by leveraging the known structure of the original function.
- The only change made is the addition of the date logging, saving on both response time and computational resources for minor modifications.

Using predictions helps accelerate similar tasks, particularly when refactoring or adding minor updates to existing code or text documents.

## Reference

- <https://aider.chat/HISTORY.html>
- <https://qwenlm.github.io/blog/qwen2.5-coder-family/>
- <https://www.all-hands.dev/blog/announcing-all-hands-online-beta>
- <https://platform.openai.com/docs/guides/predicted-outputs>


Stay updated on the latest in AI tools for developers with this week’s digest, new command Aider, OpenHands, Qwen2.5 Coder 32B

AI digest #2 new command Aider, OpenHands, Qwen2.5 Coder 32B, predicted output


## Salesforce agentforce platform

Salesforce's Einstein GPT is a key component of their AI-driven capabilities. It serves as a user interface that connects Salesforce's core offerings (Sales Cloud, Service Cloud, Marketing Cloud, etc.) with different LLMs. Einstein GPT helps Salesforce customers generate natural language-based insights and operational support like automated responses, summaries, and content creation.

- **Customization and integration:** The agnostic approach allows users to customize AI to their specific organizational or industry needs. By supporting multiple LLMs, different businesses can pick models based on cost, performance, or specific functionality.
- **Multiple LLM support:** Einstein GPT can integrate not only with models like OpenAI’s GPT series but also with other LLM providers like Cohere, Anthropic, or Google’s AI. This means enterprises can leverage their LLM of choice with Salesforce.
- **Apex models layer:** Salesforce built a flexible AI layer called "**Apex models**" which interfaces between Salesforce applications and various third-party and in-house LLMs. The Apex models layer helps to filter data, improve operations, and produce generative outcomes, regardless of which LLM is used.
- **Einstein trust layer:** Security and trust are of utmost importance for Salesforce, which is why this approach ensures the safeguarding of customer data. The company needs to ensure that customer-specific, private information is secure when interacting with these language models. Salesforce allows customers to have control over where data is processed and stored while benefiting from advanced AI functionality.

![](assets/salesforce-agentforce.webp)

There are other AI models out there, like Google’s Vertex AI, Amazon SageMaker, OpenAI (you know, ChatGPT), Claude (from Anthropic), and a bunch more. These models can be trained to provide the best results for businesses when prompted correctly.

To train these LLMs effectively, you need a ton of data. Organizations with lots of data usually turn to data lakes, with names you might recognize like Snowflake, Databricks, BigQuery (from Google), and Redshift (from Amazon).

So, when you mix AI models with data lakes, you get data harmonization and generative AI that can really benefit the business. Salesforce’s LLM works with Data Cloud or can connect to other data lakes, making it super flexible for using GenAI with your Salesforce data.

![](assets/salesforce-data-platform.webp)

Data required for generating highly contextual responses may not always reside within Salesforce. As a result, **Salesforce provides flexible methods for interacting with external data sources** while ensuring that data is available to produce optimal results.

- **Data cloud ingestion** : One way to interact with external data is by **ingesting structured and unstructured data** from different sources into the Salesforce Data Cloud. This can be configured to occur on a scheduled basis, bringing data into Data Cloud for further processing and analysis. The ingestion process allows Salesforce to directly **move or copy data** from external systems into the platform.
- **Zero-copy access** : In cases where it's unnecessary or inefficient to physically move data into Salesforce, the **zero-copy architecture** is useful. Zero-copy refers to **"virtualizing" data** by connecting to and querying it in real-time without moving or replicating the data into the Data Cloud. This is a form of **data federation** where Salesforce links to external records but does not store them directly.
- **MuleSoft APIs** : For systems that require more customized integrations, **MuleSoft APIs** serve as connectors between Salesforce and external systems. By leveraging these APIs, you can **create real-time data pipelines** between external data platforms and Salesforce without the need for complex data transformations or manual processes. MuleSoft supports connectors for several major platforms, such as **Snowflake** (for order management systems, for example).

![](assets/salesforce-data-streaming.webp)

The **Atlas reasoning engine** uses a technique called **retrieval augmented generation (RAG)** to search and retrieve pertinent information from the connected data streams before generating a response based on user prompts. Here's how the search operates:

**RAG's search retriever** : This module is responsible for **retrieving relevant data** from the connected systems (e.g., Data Cloud, virtualized data sources). The retriever uses **search indexes** and configured filters to pull the most relevant data at the time of the request.

![](assets/salesforce-retrieval-model.webp)

## How Salesforce applies LLMs for CRM tasks

Salesforce integrates LLMs into its **customer relationship management (CRM)** system to automate, streamline, and enhance the performance of "knowledge-work" tasks across different CRM components. Here are some of the primary CRM use cases where LLMs are applied:

1. **Conversational AI and agent-assisted responses** :
   - **LLMs can power chatbots and virtual agents** that assist service agents by understanding customer inquiries and responding accordingly. Agent-assist models use LLMs to provide **recommendations for replies** or pre-populate responses that agents can refine and send.
   - LLMs are trained to understand CRM-specific vocabulary (e.g., customer service issues, billing inquiries, product questions) and adjust to the nuances of language used in different industries like financial services, health care, etc.
2. **Summarizing conversations** :
   - LLMs help automatically **summarize key points from customer interactions** (e.g., chat, email, or call transcripts) to reduce manual effort and support follow-up actions. This ensures that client-facing teams have a high-level view of client interactions without having to go through all communication manually.
3. **Automatic data entry & updating CRM records** :
   - One of the critical challenges in CRM is keeping records up to date. **Large Language Models automate the entry and updating** of CRM data, based on textual inputs such as customer emails, meeting notes, or even unstructured data.
   - For instance, LLMs can convert a support interaction into an updated customer profile or case ticket, ensuring that teams are constantly working with accurate and up-to-date information.
4. **Content generation** :
   - LLM models allow CRM users to automatically generate content like **email drafts, marketing messages, or knowledge base articles** . By integrating CRM context with generation capabilities (provided by LLMs like GPT-4 or Claude), the system can generate more relevant, personalized content for marketing or customer touchpoints.
5. **Sentiment analysis and predictive customer insights** :
   - LLMs are used for **sentiment analysis** —understanding the tone and emotional sentiment of customer feedback. This leads to predictive insights regarding customer churn risk, intent to buy, satisfaction levels, and areas of improvement.
   - Through **specialized prompt engineering** and training on CRM-specific data, LLMs can derive insights, trends, and predictions from historical customer interactions, enabling teams to be more proactive in managing customer relationships

## How Salesforce tests LLMs for CRM

Salesforce emphasizes the need for **extensive and rigorous benchmarking** when testing the performance of different LLMs in CRM tasks. [The blog](https://www.salesforce.com/blog/llm-benchmark-crm/) highlights several dimensions of testing LLMs for CRM environments, including metrics of **precision** , accuracy, response quality, and user satisfaction. Below are the main aspects Salesforce considers for testing and evaluation:

1. **Tasks-based benchmarking** :
   - Salesforce uses **task-specific benchmarks** that are aligned with CRM goals, such as:
     - **Content generation quality** (for tasks like marketing campaigns or email drafts).
     - **Response accuracy and fluency** (for conversational AI models).
     - **Data extraction and record update precision** (for automating data entry with LLMs).
   - The **LLM benchmarks are carried out under CRM-specific scenarios** where real-world customer data (e.g., case tickets, emails) is used to test the LLM’s capabilities in handling CRM tasks.
2. **Multi-phase evaluation** :
   - Salesforce splits the evaluation of LLMs into **several phases** to ensure robustness:
     - **Training phase** : The model is fine-tuned on CRM datasets (such as customer service transcripts, marketing copy, etc.).
     - **Validation phase** : Using real use cases, Salesforce evaluates how well the model can perform tasks like summarization, sentiment analysis, or auto-population of records.
     - **Live testing phase** : Once trained, partial deployment allows teams to run live testing in operational environments (e.g., on actual customer emails or service chats).
   - The models are also tested against **specific CRM domains and industries** , assessing whether they adapt well to customized use cases (e.g., understanding legal or medical terminology).
3. **Metrics for performance and reliability** :
   - Key performance metrics cited include:
     - **Accuracy & precision** : Especially for tasks like auto-generating case summaries or extracting key customer data from interactions.
     - **Fluency and coherency** : Tested in **generated content** like marketing emails and customer replies. This ensures that LLMs don’t generate awkward or off-brand content.
     - **Latency** : The speed at which the system responds with LLM-assisted features is a key measure, especially for use cases in customer service where real-time responses are needed.
     - **User satisfaction** : The final goal is providing **improved end-user outcomes** , so **agent satisfaction scores** and customer feedback on chatbot interactions help measure the model’s actual performance.
4. **Scenario-specific performance:**
   - Salesforce tests **representative CRM use cases** to benchmark how well each model adapts to those use cases. For example:
     - **B2B sales** : Evaluating if LLM-powered insights improve win rates for deals.
     - **Service agent responses** : The ability of LLMs to **reduce the average handling time (AHT)** by providing relevant contextual recommendations automatically.
     - **Customer engagement** : How well LLMs drive better click-through rates (CTR) in personalized marketing campaigns.
5. **Overall insights and model performance** :
   - From the benchmarking efforts, Salesforce found that the **top LLM models had a 63% average improvement** in CRM task success rate compared to traditional rule-based or older NLP systems. Some of the key insights include:
     - **Increased agent efficiency** : LLMs reduced case resolution time by an average of **20-30%** by reducing back-and-forth between departments or manual research.
     - **Improved customer response quality** : Models fine-tuned on CRM data demonstrated the ability to **reduce response errors by 25%** and provide more consistent, high-quality customer interaction outputs.
     - **Enhanced personalization in marketing** : LLM-generated content led to **a 15% increase in engagement** metrics (e.g., open rates, conversion rates) in marketing emails and campaigns.
6. **Feedback loops for model improvement** :
   - Salesforce emphasizes the role of **continuous improvement and real-time feedback loops** . Data from **live use cases** (e.g., how well autogenerated responses or CRM updates performed) is continuously fed back into the system to help the model adapt and improve its accuracy over time.

![](assets/salesforce-benchmark-design.webp)

![](assets/salesforce-benchmark-result.webp)

## Reference

- <https://www.salesforce.com/blog/llm-benchmark-crm/>
- <https://www.salesforceairesearch.com/crm-benchmark>
- <https://www.salesforceben.com/ai-wars-how-salesforces-agnostic-llm-approach-works/>


Salesforce is taking AI to the next level with large language models that make customer service smoother, sales more strategic, and insights faster. By automating routine tasks, these tools free up teams to focus on real connections with customers. The result? Happier customers, smarter sales, and big wins for businesses.

Salesforce use cases


This week's AI updates are all about making life easier for developers working with agentic tools. **Cline**, formerly known as Claude Dev, just dropped its v2.0.0 update with faster response times and real-time interactions, perfect for streamlining workflows. **OpenAI’s Realtime API** is a game-changer for real-time function chaining and voice-based coding with Ada. And finally, **Aider’s Architect/Editor** split simplifies complex code reasoning, making pair programming smoother than ever.

## [Cline - (prev. Claude Dev)](https://github.com/cline/cline)

The new **Cline v2.0.0** update is packed with cool upgrades. First off, "**Claude Dev**" has been rebranded as **Cline**, and it now works faster across different models. You get **real-time responses** streamed right into your editor, and there's a cancel button so you can stop things if **Cline** heads in the wrong direction. They also switched to using XML tags for tool interactions, which cuts down on requests by 40%, making everything smoother. Another, you can search and use any **OpenRouter** model easily, and the project now runs under the Apache 2.0 license.

![](assets/digest-01-cline-benchmark.webp)

## [Realtime API with o1 assistant](https://www.youtube.com/watch?v=vN0t-kcPOXo)

We founded it [github repository](https://github.com/disler/poc-realtime-ai-assistant) 2 week ago. The speaker talks about [**real-time APIs** provide by OpenAI](https://openai.com/index/introducing-the-realtime-api/) and the way communicate with **o1 assistant**. It's cool to see how OpenAI has made a game challenge for the coding assistant, letting software engineers now interact with LLMs using voice. Here are some key points from the talk:

- **Realtime API features**: The speaker explains how this API makes **real-time tool chaining** and **function chaining** possible, allowing Ada to handle complex tasks with great accuracy.
- **Technical insights**: They dive into how Ada uses different AI agents, with a focus on file management (CRUD operations) and testing out file-related AI agents.
- **Trade-offs and risks**: The speaker also talks about the trade-offs with using the Realtime API, pointing out that while there are risks, the benefits for engineers and developers are huge. He encourages devs to jump on board to stay competitive.

![](assets/digest-01-openai-realtime.webp)

## [Aider - separating code reasoning and editing](https://aider.chat/2024/09/26/architect.html)

**New approach from Aider:**

- **Architect role**: Generates a solution to the coding problem, focusing purely on reasoning. ( Use a strong reasoning model like o1-preview as your Architect. )
- **Editor role**: Converts the Architect's solution into specific, well-formatted code edits. ( Use a cheaper, faster model like gpt-4o as your Editor. )
- **Performance test**: Using o1-preview as the Architect with either DeepSeek or o1-mini as the Editor produced the SOTA score of 85%. Sonnet, GPT-4o and GPT-4o-mini all scored higher when used as an Architect/Editor pair.

Aider's new "Architect/Editor" feature is inspired by OpenAI’s o1 models. Instead of struggling with perfect code edits, it lets the LLM describe the solution freely, then passes that to a second LLM to generate code updates. This way, it's faster and smoother, making complex sources code easier to handle while still keeping that interactive, pair-programming feel.

```bash
pip install -U aider-chat

# Change directory into a git repo
cd /to/your/git/repo

# Work with Claude 3.5 Sonnet as the Architect and Editor
export ANTHROPIC_API_KEY=your-key-goes-here
aider --sonnet --architect

# Work with OpenAI models, using gpt-o1 as the Architect & gpt-4o-mini as the Editor
export OPENAI_API_KEY=your-key-goes-here
aider --model openrouter/openai/o1-preview --architect --editor-model openrouter/openai/gpt-4o-mini
```

![](assets/digest-01-aider-benchamrk.webp)


Stay updated on the latest in AI tools for developers with this week’s digest, featuring Cline's v2.0.0 update, OpenAI’s Realtime API, and Aider’s Architect/Editor split for enhanced coding workflows.

AI digest #1 Aider reasoning, OpenAI Realtime API, Cline - pre Claude-dev 


Yelp Inc. is a platform that helps users discover local businesses through reviews, ratings, and recommendations. Recently, they've integrated AI and large language models (LLMs) to improve content moderation, search capabilities, and user interactions with features like Yelp assistant.

## Key takeaways

- Yelp uses LLMs to catch inappropriate reviews, blocking 23,600+ bad ones in 2023.
- Yelp uses the CLIP model to accurately categorize and understand the content of photos.
- Yelp uses LLMs to summary highlight review.
- Yelp assistant helps users find service providers by using LLM with their ML system.

## Yelp contents as embeddings

### Text embeddings

Yelp’s platform has tons of user-generated content, like reviews, and to keep users trusting the site, they need to make sure inappropriate stuff (like hate speech, harassment, lewdness, or threats) gets spotted and removed. Relying only on human moderators isn’t enough, so they’ve turned to automated tools to help. They went with LLMs because these models are great at picking up on tricky, harmful language across different situations.

They mainly looked at how well LLMs can catch bad content like:

- Hate speech, which is offensive stuff aimed at people or groups based on things like race, gender, religion, or sexuality.
- Lewdness, including dirty jokes, pickup lines, asking for sexual favors, or sexual harassment.
- Threats, harassment, and other extreme personal attacks.

![](assets/yelp-toxic-content.webp)

Yelp put together a dataset ( you can find [this datasets at here](https://huggingface.co/datasets/Yelp/yelp_review_full), based on 5 star rating system review) of old inappropriate reviews to train their model. This dataset had labeled examples of really bad language. To make the model work even better, they used a few tricks like:

- **Scoring system**: Moderators rated how bad the inappropriate content was.
- **Sentence embeddings**: They used LLMs to find reviews that were similar to high-quality examples to bulk up the dataset.
- **Sampling techniques**: They adjust the dataset by over-sampling and under-sampling to boost recall, especially for rare types of inappropriate content. **They also used zero-shot and few-shot technique** to handle cases where there wasn’t enough data for certain categories.

**Yelp used a curated dataset and LLM model from HuggingFace to classify inappropriate reviews.** They evaluated model performance by visualizing sentence embeddings and fine-tuned the model to improve accuracy.

![](assets/yelp-embedding-vector.webp)

Since incorporating LLMs to help detect harmful and inappropriate content, it enabled Yelp moderators to proactively prevent **23,600+ reviews from ever publishing to Yelp in 2023**.

### Photo embeddings with CLIP model

Yelp uses business and photo embeddings to enhance data accessibility and improve recommendations, semantic search, and clustering.

1. **Business embeddings**: These are created by averaging the vector embeddings of the 50 most recent reviews of a business, representing its metadata. Before LLM trends grows, they are apply ML for this feature.
2. **Photo embeddings**: Yelp uses **OpenAI's CLIP model** to generate semantic representations of images. CLIP is a zero-shot model that pairs images with relevant text, helping classify photos more accurately with minimal data.

**Semantic Understanding:** CLIP is employed to generate semantic embeddings of images, enabling the system to understand and categorize the content of photos effectively. For example from Yelp, we observe that many **Interior** and **Exterior** photos get classified as **Other** by the CLIP model. Here are some examples for **Interior**.

![](assets/yelp-detect-background.webp)

**Category Identification:** The model classifies photos into predefined categories such as **Food**, **Drinks**, **Menu**. For example, Images labeled **Waffles** in Yelp dataset were considered misclassified as **Chicken Wings or Fried Chicken** by the CLIP model.

![](assets/yelp-category-food.webp)

Yelp's project involves generating new embeddings for its extensive data using models like CLIP. These embeddings (for reviews, photos, and metadata) allow Yelp to improve the breadth, depth, and accuracy of its content, making it more useful for internal teams. They plan to fine-tune the CLIP model to enhance photo embeddings and expand business embeddings by integrating multiple data types. With hundreds of millions of embeddings, different teams at Yelp are already leveraging this data to enhance their products and services.

## Review highlights and tagging

They’ve also improved the search experience with **AI and LLMs** (Large Language Models). These updates help you find exactly what you’re looking for by analyzing all the user-generated content on Yelp and giving you smarter, more relevant search results. They’ve even added a fun new **“Surprise Me”** feature that suggests places to eat when you’re not sure what you want, and new clickable tags to make narrowing down your search easier.

![](assets/yelp-highlight-summary.webp)

Another cool addition is how Yelp’s making reviews more engaging. You can now **add videos to your reviews**, making them more immersive and interactive. They’ve also added **new review reactions** (think thumbs up or similar) and **review topics** to help you write better, more organized reviews.

## Yelp assistant

Yelp is using Large Language Models (LLMs) in some cool ways to make things easier for both users and businesses. One of the main features powered by LLMs is **Yelp assistant**, a conversational AI tool that helps you find and hire service pros. You can just tell Yelp assistant what you need, and it’ll ask you follow-up questions, then match you with the best local pros for the job. It’s smart because it pulls from Yelp’s huge collection of business info and reviews, making sure you get the right fit.

![](assets/yelp-assistants.webp)

Yelp’s also got the **Yelp Fusion AI API**, which lets other companies integrate Yelp’s content into their own apps or platforms. So, if you’re on a different app and you ask something like, "Find a coffee shop with free Wi-Fi nearby," the LLMs will pull from Yelp’s data and give you solid recommendations, complete with reviews, ratings, and photos. It’s a way for third-party apps to give their users access to Yelp’s content with smart, natural language searches.

## Conclusion

In terms of user experience, Yelp's integration of AI and large language models (LLMs) has changed the platform by improving search intelligence, review insight, and content moderation effectiveness. Yelp is transforming the way businesses communicate with consumers by applying these advanced technologies. Yelp is using LLMs in a bunch of different ways, from helping user find service pros with LLM, to making search results smarter, to powering other apps with Yelp’s data. It’s all about making things easier, faster, and more personalized for users.

## References

- https://engineeringblog.yelp.com/2023/04/yelp-content-as-embeddings.html
- https://blog.yelp.com/businesses/new-yelp-business-features-august-2023/
- https://engineeringblog.yelp.com/2018/05/scaling-collaborative-filtering-with-pyspark.html
- https://engineeringblog.yelp.com/2024/03/ai-pipeline-inappropriate-language-detection.html
- https://www.yelp-press.com/press-releases/press-release-details/2023/Yelp-Introduces-New-Ways-to-Discover-and-Connect-with-Local-Businesses-and-Contribute-Helpful-Content/default.aspx


Yelp already had a machine learning platform before the big push for large language models (LLMs). Now, they’re using LLMs to level up their search and recommendation systems, making it easier for moderators and businesses to track down users. Let’s dive into how Yelp is making it work.

Yelp use cases


When you’re working with generative AI application, one thing that often gets overlooked is logging. Logging helps you keep track of what’s happening under the hood and gives you the insights you need to improve your model. Whether it's detecting errors or maintaining your AI runs smoothly, logging is fundamental. In this article, we'll look at why logging is important and how to use it to improve your LLM application.

## Roles of logging in LLM application

So, what’s logging? In simple terms, it’s about keeping a record of what happens between users and large language models (LLMs). This means saving both the questions users ask and the answers the model gives.

If you look at the image below, it shows how an LLM app works. Logging is a key part of this because it captures things like the model’s inputs, outputs, the current state, memory being used, and the prompts running. This helps us see the big picture and keep track of how well the system is doing.

![](assets/logs-pillar-sample-rag-system.webp)

## The impact of logging

### Enhancing user experience

Logging everything gives you a clear view of how users interact with your system. By tracking every query, output, and action, you can spot common issues, improve responses, and roll out updates that make the overall user experience smoother. The more you understand user behavior, the better you can tailor your AI to meet their needs.

### Improving model accuracy

Logs help identify where your model is underperforming. By analyzing logs of bad outputs or crashes, you can change system prompts, adjust configurations or parameter. Logging creates a feedback loop that helps you to detect faults and improve the model's accuracy.

### Faster debugging and issue resolution

When things go wrong - like a crash or a weird bug - logs are your find out then troubleshooting. By logging when a component starts, stops, or fails, you can track down the exact point where the issue occurred. This saves you tons of time in debugging, allowing you to fix problems quickly and keep the system running smoothly.

### Better decision making

Logs don’t just help with fixes - they also provide data to guide future decisions. By reviewing logs over time, you can see trends in how your AI performs, which features are working well, and where you might need to invest more effort.

![](assets/logs-pillar-sample-view-dashboard.webp)

## Techniques

### Context is everything

**Session logging**

Image it like keeping a record of everything a user and the model do during a session. You’re capturing not just the user’s input but also the LLM’s responses. Each response might even come with a score, showing how confident ( we can apply LLM-as-a-judge to evaluation each response ) the model was or how well it performed. This way, you can see patterns in what users are asking and how well the model is answering. If the same question keeps coming up or the scores for responses are low, it’s a signal that you might need to change system prompt or adjust parameter of the model.

![](assets/logs-pillar-session.webp)

**Adding contextual metadata**

Another key technique involves logging contextual metadata, such as the component used (e.g., "text_embedder") and the time taken for processing (latency). By including metadata, such as model type, request time, and user session details, it becomes easier to analyze performance across various scenarios. This metadata can also help segment user responses by device type, geography, or even specific time frames.

![](assets/logs-pillar-metadata-context.webp)

**Prompt management**

Prompt logging is important for keeping track of how well LLMs handle user inputs. By logging prompts, their responses, and scores, you get a clear picture of what’s working and what isn’t. It adding details like when the prompt was used or what device the user was on gives more context, so you can see how different factors affect performance. In short, logging makes it easy to fine-tune prompts and keep your LLM improving.

![](assets/logs-pillar-prompt-management.webp)

### Element in LLM application

**Model parameters**

Model parameters are the internal variables that the LLM adjusts during training to optimize its understanding and generation of language. Key parameters include:

- **Temperature**: Adjusts how creative or random the model's output is. Higher values = more randomness.
- **Max tokens**: Limits the length of the response generated.
- **Top-k sampling**: Controls how many token options the model considers for each word.
- **Top-p (Nucleus) sampling**: Ensures the model chooses from a smaller, more focused set of word options, based on probability.

![](assets/logs-pillar-llm-parameters.webp)

**Management agent**

Agents are like decision-makers in LLM systems. They take user input and decide how to handle it, often running multiple tasks to come up with a response. Logging the **input and output** of agents is key because it helps you track exactly what was asked and how the agent responded.

- **Debugging**: If something goes wrong (like incorrect task prioritization or tool selection), logs show exactly what input led to the error.
- **Optimization**: With logs, you can monitor how well the agent manages tasks, interacts with external tools, and adapts based on the output, helping you improve its performance.

![](assets/logs-pillar-management-agent.webp)

**Handling chain and step**

Chains involve calling multiple tools or agent to retrieve data. Each step relies on the previous one, which makes the whole process more complex. Here's how logging comes in handy at each step:

- **Retrieval**: The system retrieves relevant information, embedding it into vectors to improve accuracy. Logs help you see if the retrieval process worked and how well it pulled in the right data.
- **Generation**: The system generates a response based on the data retrieved. Logging here ensures you can trace how well the generated content fits the user’s query.
- **Multiple tools**: Embedding, retrieving, calling APIs, and parsing are all part of this chain. Each of these steps is logged so you can monitor how each function performed, catch issues, and debug easily.

![](assets/logs-pillar-tracing-chain.webp)

**Scoring the evaluation**

Logging scores after you run an evaluation is a smart move for keeping track of how well your AI is doing. Whether you're scoring things like accuracy, conciseness, or relevance, these logs give you a clear picture of what’s working and what needs improvement. It’s like having a report card for your model, and over time, you can see patterns and figure out where it might be falling short.

![](assets/logs-pillar-trace-score.webp)

## Analyzing logged data

### Visualization

Tools like dashboards, charts, and graphs help you make sense of the data quickly. You can monitor trends over time, see how users are interacting with your AI, or track response ratings. It’s super helpful when you need to share insights with your team.

Using monitoring tools also means you can keep an eye on performance in real-time. If something starts going sideways, you’ll catch it early and fix it fast, keeping everything running smoothly.

![](assets/logs-pillar-honeyhive-dashboard.webp)

### Feedback loops

Now, let’s talk about feedback loops. This is all about taking what you learn from your logs and turning it into action. But it gets even better when you bring humans into the mix. A **human-in-the-loop** approach means you’re not just relying on AI; you’re combining human judgment with machine learning. For instance, after a model update, if your logs show users aren’t loving the changes, a human can step in to analyze why and make adjustments. You can even use **human-annotated** data to fine-tune responses, making sure the AI is delivering what users actually need.

![](assets/logs-pillar-feedback-loop.webp)

## Conclusion

While logging might feel like a small detail in the bigger picture of generative AI, it’s actually a powerful tool. By observing user interactions and looking into the data, you could discover valuable insights that not only increase accuracy but also improve the user experience.

## References

- https://www.honeyhive.ai/monitoring
- https://neptune.ai/blog/llm-observability
- https://www.qwak.com/post/prompt-management
- https://humanloop.com/blog/human-in-the-loop-ai
- https://www.projectpro.io/article/llm-parameters/1029
- https://langfuse.com/docs/prompts/example-openai-functions
- https://www.evidentlyai.com/blog/open-source-llm-evaluation
- https://docs.smith.langchain.com/old/cookbook/tracing-examples/traceable
- https://medium.com/@simon_attard/leveraging-large-language-models-in-your-software-applications-9ea520fb2f34
- https://www.researchgate.net/figure/An-LLM-based-agent-autonomously-reasons-about-tasks-and-composes-external-tools-to_fig1_376401381


Logs are like the footprints of your LLM, tracking every move it makes. We will look at how logging can help you see beneath the top layer of a system, which can help you troubleshoot problems and better understand the system behavior.

Logging


When it comes to observability in Large Language Model (LLM) applications, metrics have significance delivering that these systems work correctly. Metrics provide information on both system performance and model efficiency, enabling developers and researchers to fine-tune their systems. In this article, we'll look at important metrics for monitoring and evaluating LLMs.

## System metrics

System metrics are essential for understanding the overall health and performance of your LLM application. Here are four key system metrics to keep an eye on:

- **Latency**: This metric indicates how long it takes for the system to react to a user query. Monitoring latency is important because it directly affects user experience. High latency can cause unhappiness, while low latency is often associated with a fast application.
- **Throughput**: The amount of requests that the system can handle in a given time period. High throughput is expected, especially in high-demand contexts, because it shows the system can handle multiple requests at once without decreasing performance.
- **Error rate**: This metric tracks the percentage of failed requests or errors generated by the system.A high error rate may indicate underlying issues that must be solved immediately to ensure customer trust and happiness.
- **Resource utilization**: Monitor CPU, memory, and disk utilization to discover bottlenecks and improve resource allocation. Understanding how resources are used can result in improved scalability and performance improvements.

| Metric Type          | Description                   | Importance                             |
| -------------------- | ----------------------------- | -------------------------------------- |
| Latency              | Time taken for a response     | Direct impact on user experience       |
| Throughput           | Queries handled per time unit | Essential in high-demand scenarios     |
| Error rate           | Percentage of failed requests | Indicates system reliability           |
| Resource utilization | CPU, memory, and disk usage   | Helps identify performance bottlenecks |

![](assets/metric-pillar-monitoring-dashboard.webp)

## Model metrics

Model metrics examine the performance of the LLM itself. We'll separate them into two sections: metrics for model-based scoring and metrics for retrieval-augmented generation (RAG) systems.

### Scoring based on the model

Evaluating the performance of an LLM requires specific metrics that quantify its output quality. Almost they are testing based on public dataset or benchmarks. Here are four key metrics used for model scoring:

- **Perplexity**: Perplexity measures how well a probability distribution predicts a sample. Lower perplexity indicates better predictive performance, making it a valuable metric for evaluating language models.
- **BLEU score**: The BLEU (Bilingual Evaluation Understudy) score is used to assess the quality of machine-generated text by comparing it to one or more reference texts. A higher BLEU score indicates a closer match to human-generated outputs.
- **METEOR**: This metric improves upon BLEU by considering synonyms and stemming, providing a more nuanced evaluation of generated text quality. Higher METEOR scores reflect better semantic meaning.
- **ROUGE**: ROUGE (Recall-Oriented Understudy for Gisting Evaluation) focuses on recall and is particularly useful for summarization tasks. It compares the overlap of n-grams between the generated text and reference texts.

| Metric Type | Description                           | Importance                           |
| ----------- | ------------------------------------- | ------------------------------------ |
| Perplexity  | Predictive performance measure        | Lower values indicate better models  |
| BLEU        | Quality comparison to reference texts | Higher scores reflect closer matches |
| METEOR      | Evaluates semantic similarity         | Enhances BLEU's effectiveness        |
| ROUGE       | Measures overlap in summarization     | Useful for content generation tasks  |

![](assets/metric-pillar-model-metric.webp)

### Scoring based on RAG systems

In retrieval-augmented generation systems, the effectiveness of information retrieval can be as important as the quality of generated text. Some metrics below help us understand the quality and precision of search engine.

- **Precision@K**: This measures the proportion of relevant documents within the top K results returned by the system. A higher Precision@K indicates that the system effectively retrieves relevant content, which is vital for generating accurate responses.
- **Recall@K**: Recall@K evaluates how many of the total relevant documents were retrieved. This metric helps ensure the system captures all necessary information, thus preventing critical data loss.
- **Mean Reciprocal Rank (MRR)**: MRR assesses the average rank of the first relevant result returned. A higher MRR indicates that relevant results appear earlier in the list, which enhances user satisfaction.
- **Normalized Discounted Cumulative Gain (NDCG)**: NDCG considers the position of relevant documents in the result list, providing a comprehensive view of ranking quality. High NDCG scores signify that relevant documents are prioritized, improving user experience.

| Metric Type                           | Description                                | Importance                         |
| ------------------------------------- | ------------------------------------------ | ---------------------------------- |
| Precision@K                           | Relevant documents among top K results     | Importance for content quality     |
| Recall@K                              | Proportion of relevant documents retrieved | Ensures no critical info is missed |
| Mean Reciprocal Rank                  | Average rank of the first relevant result  | Improves user satisfaction         |
| Normalized Discounted Cumulative Gain | Evaluates ranking quality                  | Enhances overall user experience   |

![](assets/metric-pillar-rag-metric.webp)

### Metrics for fine-tuning model

Fine-tuning models is an essential step for improving performance when the RAG technique cannot improve the behavior and predictability of the model.

- **Performance improvement**: This metric compares model performance before and after fine-tuning using various scores (e.g., BLEU, ROUGE). It provides a clear indication of whether the fine-tuning process was successful
- **Training time**: Monitoring the time taken for fine-tuning helps assess the efficiency of the training process. Reducing training time while maintaining performance is a key goal.
- **Overfitting rate**: The overfitting rate evaluates how well the model generalizes to unseen data after fine-tuning. A low overfitting rate indicates that the model has retained its ability to perform well across different datasets.
- **Loss reduction**: Tracking the loss function before and after fine-tuning gives insights into how well the model learns from the data. A significant reduction in loss indicates effective fine-tuning.
- **User feedback**: Gathering qualitative feedback from users can provide insights into perceived improvements in model performance, helping to complement quantitative metrics.

| Metric Type      | Description                                    | Importance                            |
| ---------------- | ---------------------------------------------- | ------------------------------------- |
| Performance      | Comparison of scores pre- and post-fine-tuning | Indicates success of fine-tuning      |
| Training time    | Duration of the fine-tuning process            | Critical for efficiency               |
| Overfitting rate | Generalization capability post-tuning          | Ensures model robustness              |
| Loss reduction   | Change in the loss function                    | Reflects learning effectiveness       |
| User feedback    | Qualitative assessment of model performance    | Provides context to quantitative data |

![](assets/metric-pillar-fine-tuning-metric.webp)

## Cost metrics

Finally, the operating system should mention cost and price of the amount of model to help us understand the behavior of the user when choosing the model. A balance between pricing and performance is good for we observability.

- **Pricing per request**: This metric reflects the cost associated with processing each user request. Understanding this is crucial for budgeting and resource allocation.
- **Token in/out**: Tracking the number of tokens processed (input and output) helps in understanding usage patterns and associated costs. Many third-party providers charge based on token counts.
- **Total time**: This metric aggregates the total time spent processing requests, which can be correlated with costs, especially in cloud environments where time translates to billing.
- **Resource costs**: Monitoring costs associated with cloud resources (e.g., CPU, storage) is essential for calculating total operational costs.
- **Service rate limits**: Understanding the rate limits imposed by third-party services helps in planning usage and avoiding unexpected costs or service interruptions.

| Metric Type         | Description                             | Importance                        |
| ------------------- | --------------------------------------- | --------------------------------- |
| Pricing per request | Cost per processed user request         | Important for budgeting           |
| Token in/out        | Count of processed tokens               | Affects overall cost              |
| Total time          | Aggregate processing time               | Correlates with operational costs |
| Resource costs      | Expenses linked to resource utilization | Essential for cost management     |
| Service rate limits | Limits set by service providers         | Important for usage planning      |

![](assets/metric-pillar-management-resource.webp)

## Conclusion

Knowing and implementing a robust set of observability metrics in LLM applications is important for making sure high performance and client happiness. Reviewing all the metrics mentioned in the article gives a lot of valuable insights into why each one is important and why we should be using them.

## Reference

- https://aman.ai/primers/ai/LLM/
- https://www.pinecone.io/learn/offline-evaluation/
- https://docs.smith.langchain.com/tutorials/Developers/observability
- https://konfuzio.com/de/limits-llms-retrieval-augmented-generation/
- https://sebastianraschka.com/blog/2023/optimizing-LLMs-dataset-perspective.html
- https://www.trulens.org/trulens/getting_started/core_concepts/feedback_functions/#large-language-model-evaluations
- https://kili-technology.com/large-language-models-llms/how-to-build-llm-evaluation-datasets-for-your-domain-specific-use-cases


Metrics give you the rundown on how your LLM’s performing. We will show how to use these metrics to identify issues, increase efficiency, and make changes for improved outcomes.

Metrics


## Introduction

### Importance of observability

Observability in AI systems, especially LLMs, is about understanding what’s happening behind the scenes. It’s essential for ensuring smooth operations, building user trust, and meeting compliance standards by monitoring performance, spotting issues, and staying accountable. As AI becomes more central to our lives, observability directly affects system stability and performance.

### Integrating observability early

The best advice is to integrate observability tools right from the start of your project. Delaying it can cause worse issues later on. Early integration helps catch issues before they escalate and sets foundation for scaling as your systems grow more complex.

![Three pillars in observability](assets/observability-circle.webp)

## The three pillars of observability

Understanding observability requires understanding its three pillars: **Metrics**, **Logs**, and **Traces**. Each plays a different role in creating a overview of your LLM application.

### Metrics

[Metrics](metric-pillar.md) are the foundation of AI observability, including system- and model-specific indications. System indicators like throughput and hardware usage are common, whereas model metrics like accuracy and hallucination rates are AI-specific. Cost tracking includes tracking query volumes and token usage. Using a combination of spot and extensive checks ensures complete monitoring.

### Logs

[Logging](logs-pillar.md) in AI applications ensures detailed records are maintained, enabling effective monitoring and debugging throughout the system’s operation. The golden rule of logging is to record everything: system parameters, queries, outputs, and component lifecycles. Effective logging needs consistent tagging and identification assignment for traceability.

### Traces

[Tracing](trace-pillar.md) in AI applications provides a full picture of the execution path, from query to response. It includes document retrieval, prompting, and model interactions, as well as time and cost estimates for each step. Visualization tools such as Langsmith provide simple trace representations.

## Benefits of LLM observability

Using LLM observability tools brings a range of benefits to business:

- **LLM performance:** Ongoing monitoring helps fine-tune LLMs, improving speed and accuracy.
- **Faster problem diagnosis:** Detailed logs and metrics make it easier to spot and fix problems fast, reducing downtime.
- **Cost savings:** Early detection of inefficiencies and better resource management can lower operating expenses.
- **Better explainability:** A clearer understanding of how LLMs work helps companies explain decisions, especially in regulated industries.
- **Increased reliability:** Proactive monitoring helps catch issues early, making LLMs more dependable.

## Challenges in LLM observability

Monitoring LLMs presents several challenges:

- **Model complexity:** LLMs are costly and complex, making them difficult to monitor and optimize effectively.
- **Third-party rate limits:** A lot of LLMs use third-party APIs with rate limits, which can slow down monitoring and make it harder to get real-time data.
- **Dynamic workloads:** LLM performance can change in response to shifting demands, requiring adaptive monitoring strategies.
- **Data privacy:** Ensuring data privacy when monitoring LLMs is important because businesses must meet legal requirements without sacrificing insights.

## References

- https://theblue.ai/blog/llm-observability-en/
- https://medium.com/@aiswaryasomanathan4/logging-traces-and-metrics-whats-the-difference-c796ea276c98


Observability in AI is all about understanding what’s going on inside complex systems. It gives you the tools - logs, metrics, and traces - to monitor, troubleshoot, and optimize how AI models and services run.

Observability in AI platforms


## What is tracing

Tracing is a way to keep track of, debug, and get a clear picture of how an LLM app is running. It gives a detailed snapshot of a specific action, like making a call to the LLM, formatting a prompt, or running a function.

A trace is just a bunch of actions, set up like a tree or graph. Each action is called a “span,” and it has its own inputs and outputs. The top-level action, known as the “Root Run” is the one that’s triggered by the user or app.

Tracing helps you see how well an LLM app is performing, including details like how long things take, how many tokens are used, and what the sequence of actions looks like. It’s great for finding and fixing errors, seeing the full path of a request, and improving overall performance.

There are different tools available for tracing LLMs, like [Klu.ai](http://klu.ai/), [LangSmith](https://docs.smith.langchain.com/), which can log all calls made to LLMs, agents, and other tools, showing you visual breakdowns of inputs, outputs, and even tracking errors and costs. Besides performance and debugging, tracing is also useful for figuring out where LLMs come from, which is getting trickier as more companies release their own models.

![](assets/trace-pillar-tracing-roadmap.webp)

## Why tracing is necessary

Tracing can help you track down issues like:

- **Application latency:** showing delayed LLM and Retriever invocations.
- **Token usage:** provides a breakdown of token usage with LLMs to highlight your most expensive LLM calls.
- **Runtime exceptions:** important runtime errors, such as rate limitation, are recorded as exception events.
- **Retrieved documents:** view all the documents retrieved during a retriever call, including the score and order in which they were returned.
- **LLM parameters:** view the parameters used when calling out to an LLM to debug things like temperature and system prompts.
- **Prompt templates:** determine which prompt template was used during the prompting step, as well as the variables used.

![](assets/trace-pillar-tracing-example.webp)

## Element in tracing

We should be making clear the difference between trace and span.

| **Attribute**       | **Trace**                                               | **Span**                                                                      |
| ------------------- | ------------------------------------------------------- | ----------------------------------------------------------------------------- |
| **Scope**           | Covers the entire lifecycle of a request                | Focuses on individual operations or steps                                     |
| **Level of detail** | High-level overview                                     | Detailed, includes specific metrics                                           |
| **Granularity**     | Includes multiple spans                                 | Captures single actions                                                       |
| **Primary use**     | Understanding overall application flow and dependencies | Debugging or optimizing specific components/tasks                             |
| **Data collected**  | Timeline of operations, parent-child relationships      | Duration, input/output, token usage, errors, attributes like provider, scores |
| **Examples**        | Full document retrieval process                         | Querying a database, calling an API, embedding query                          |

### Trace

Traces, also known as distributed traces, provide a view of a system by crossing agent, process, and function. Spans form the fundamental components of a trace.

A trace consists of a tree structure of spans, beginning with a root span that has no parent. This root span encapsulates the total time required to complete a task, representing a single logical operation such as adding an step to a get current weather. The root span serves as the foundation, with child spans branching off to provide more detailed information about specific subtasks or processes within the overall operation.

![](assets/trace-pillar-trace-explain.webp)

### Span

Span help define the main operations within LLM applications. These types of operations are broken down into different categories to keep things organized and easy to understand.

- **Chain (Workflow)**: This is like a roadmap of static steps, which can include things like retrieving data, embedding text, or making LLM calls.
- **Embedding**: This deals with embedding tasks, such as working with text embeddings, often used for making similarity-based queries or refining questions.
- **Retrieval**: In setups like RAG system, this fetches data from a vector database to give the LLM more context for better, more accurate responses.
- **LLM**: Calls to the LLM itself for things like generating text or getting inferences, often using various APIs or SDKs.
- **Tool**: External tool calls, like grabbing info from a weather API or using a calculator to get real-time data.
- **Agent**: In intelligent agent scenarios, this handles more dynamic workflows, making decisions based on LLM outputs.

![](assets/trace-pillar-span-explain.webp)

## Conclusion

Tracing lets you see what’s going on in your LLM app, from tracking performance to fixing errors and understanding token usage. It’s a simple way to debug and optimize everything from prompts to external tool calls.

## Reference

- https://www.datadoghq.com/blog/datadog-llm-observability/
- https://mlflow.org/docs/latest/llms/tracing/tracing-schema.html
- https://arize.com/blog/llm-tracing-and-observability-with-arize-phoenix/
- https://arize.com/blog-course/traces-spans-large-language-model-orchestration/
- https://www.linkedin.com/posts/aurimas-griciunas_llm-genai-llmops-activity-7250055380553084928-9XAA
- https://www.alibabacloud.com/blog/observability-of-llm-applications-exploration-and-practice-from-the-perspective-of-trace_601604


Tracing is like following your LLM’s journey, step by step. We will explain how tracing makes it easy to identify and address problems by allowing you to track the entire process.

Tracing

This guide offers a structured approach to evaluating and optimizing the integration of third-party Large Language Models (LLMs) into applications, ensuring alignment with business goals and user needs through detailed checklists and evaluation metrics.

Evaluation guidelines for LLM applications


## Introduction

Retrieval-Augmented Generation is challenging the world of traditional search engines and LLM information retrieval. Hybrid search, which combines the strengths of vector search and full-text search, offers a powerful solution for the lack of precision in vector search and meaning in full-text search. In this article, we will know how hybrid search approach and how to evaluate each search method through metrics in information retrieval.

## Search methodology in RAG system

### Vector search

Vector search is an simple way for finding data that looks at the context of search queries and data inputs rather than just matching text. First, we turn both the search query and a column from the dataset into numbers called vector embeddings. Then, we use methods such as **Cosine similarity** or **Euclidean distance** to calculates how near these numbers are. After that, we find the entries that are most similar to our query. Finally, we return the top k results that are closest to the query vector.

![Vector distance](assets/hybrid-search-vector-distance.webp)

However, in a large dataset, their suggestions might sometimes miss the keyword or point when you need something very specific. For instance, if you're looking for a detailed guide on planting tomatoes, they might bring you general gardening books that cover a bit of everything but not in-depth on tomatoes.

### Full text search

Imagine searching for "adventure" and "mystery" in a library of thousands of books - full-text search makes it possible! To make this magic happen, various algorithms come into play. **The inverted index** acts like a giant dictionary, mapping each word to the documents it appears in. **TF-IDF** ( term frequency-inverse document frequency ) considers the frequency and uniqueness of words to prioritize relevant documents. Boolean retrieval lets you combine search terms using logical operators like AND, OR, and NOT. Full-text search allows you to find exact keyword by enclosing words in quotes. These algorithms work together to make searching through large amounts of text quick, efficient, and accurate.

The problem with full-text search is that many documents might have the same keywords but entirely different content, leading to a heap of irrelevant results. For example, we have 10 articles on each topic: design pattern, design UI/UX, and design workflow. That means the word “design” will appear throughout the document, but the meaning of the document is different. 

### Hybrid search

#### When to use hybrid search

Imagine you're searching for something online, and you have two helpers: one who understands the exact words you use, and another who gets the overall meaning. This is what hybrid search is all about. It combine traditional full-text search with vector search, which understands the context. This combination tackles the limitations of each method, making your search experience both precise and contextually aware.

The best uses cases for hybrid search are those that require both contextual relevance and exact precision. To make sense, I give you scenario below:

We want to use hybrid search when dealing with complex queries requiring both specific keyword matches and understanding of the content.

Imagine you’re searching a database of scientific articles. A full-text search provides the precise instances of a term, such as "neural networks" but it might miss out on articles that discuss the concept without using the exact keyword. Vector search captures the context of the articles but might lack precision. By combining both methods, you get the best of both strengths: articles that mention your specific keywords and also cover the relevant context. Ranking these results helps bring the best, most relevant articles to the top.

![Hybrid search common flow](assets/hybrid-search-common-flow.webp)

#### Some limitations

While hybrid search offers many advantages, it's not without its limitations.

1. **Latency:** Since hybrid search runs both vector and full-text search algorithms, it can be slower than using just a vector search, especially when dealing with large amounts of documents.
2. **Context-length limitations:** Re-rank models can only handle a certain amount of text at once. It’s depend on the context-length window of a third-party model.
3. **Database support:** Not all vector databases support hybrid search. Ensure that your choose database has the necessary capabilities.

Each search method has its pros and cons, and the best approach really depends on your specific dataset and what you need. Whether you’re sorting through complex scientific articles or looking up exact product codes, picking the right search method is key to getting the most relevant and accurate results. So, always keep in mind: no single search method works for every situation, you've got to choose based on your dataset.

#### Apply ranking to get top quality documents

Imagine you have two lists of search results: one from vector search and one from full-text search. How do you choose the best documents from both? Two effective methods can help:

- **Re-rank model:** The re-rank model calculates the semantic match between the list of candidate documents and the user query, reordering them based on semantic match to improve the results of semantic sorting. The principle is to compute a relevance score between the user query and each candidate document and return a list of documents sorted by relevance from high to low

![Ranking model flow](assets/hybrid-search-ranking-model.webp)

- **Reciprocal rank fusion:** RRF is a straightforward and effective way to mix search results. It takes the rankings of documents from different search methods, uses a formula to calculate a combined score, and then re-ranks the documents based on these scores. This method makes sure that documents relevant to both search methods get prioritized.
  - **Combining different results**: When you perform a search, both vector search (which understands the meaning behind your query) and full-text search (which looks for exact matches) generate their own list of relevant documents. Each document in these lists is ranked based on how relevant it is to your query.
  - **Ranking and scoring**: The higher up a document is on the list, the better its score. So, if a document is #2 in the vector search but #5 in the full-text search, it still gets a good overall relevance score.

![Reciprocal rank fusion formula](assets/hybrid-search-rrf-method.webp)

## Organizing search strategy for LLM application

In this session, I will explain how we design and store retrieval settings in the Postgres database. This setting should be stored in the dataset table because each dataset has many documents that will accept the search method for all documents. We can have many datasets, and each dataset has a different search method.

```tsx
import { type RetrievalModel } from "~/types/retrieval-model";

// Here is example schema
export const dataset = createTable("dataset", {
  id: uuid("id").notNull().primaryKey(),
  retrievalModel: jsonb("retrival_model").$type<RetrievalModel>(),
  visible: boolean("visible").default(true),
  createdAt: timestamp("created_at").notNull().defaultNow(),
  createdBy: uuid("created_by").references(() => users.id),
  updatedAt: timestamp("updated_at"),
  updatedBy: uuid("updated_by").references(() => users.id),
});
```

The retrieval settings should have the properties below:

- searchMethod ( We define an enum type for three kinds of search method )
- topK ( The user can justify based on how many chunks they have gotten )
- similarityThreshold ( This property is used for vector search to calculate similarity )
- alpha ( The user can justify the full-text search result percentage more than others )

```tsx
import { SearchTypeEnum } from "~/model/search-type";

export type RetrievalModel = {
  searchMethod: SearchTypeEnum;
  topK: number;
  similarityThreshold: number;
  alpha: number;
};
```

To cover all search methods, we define the function retrievalSearch that can pass the props into the scope of the function. The return of each method has been formatted to combine into a hybrid search and render to the UI. 

```tsx
interface RankedResult {
  content: string | null;
  referLinks: string | null;
  referName: string | null;
  sourceType: number;
  vectorRank?: number;
  textRank?: number;
  rrfScore: number;
}

export async function retrievalSearch(
  type: SearchTypeEnum,
  topK: number,
  similarityThreshold: number,
  botId: string,
  msg: string,
  alpha: number,
): Promise<RankedResult[]> {
  switch (type) {
    case SearchTypeEnum.Vector:
      return await vectorSearch(botId, topK, similarityThreshold, msg);
    case SearchTypeEnum.FullText:
      return await fullTextSearch(botId, topK, msg);
    case SearchTypeEnum.Hybrid:
      return await hybridSearch(botId, topK, similarityThreshold, msg, alpha);
    default:
      return [];
  }
}

function calculateRRFScore(
  vectorRank: number | undefined,
  fullTextRank: number | undefined,
  alpha: number,
  k = 60,
) {
  if (vectorRank === undefined && fullTextRank === undefined) {
    throw new Error("Both ranks cannot be undefined");
  }

  const vectorScore = vectorRank ? 1 / (k + vectorRank) : 0;
  const fullTextScore = fullTextRank ? 1 / (k + fullTextRank) : 0;

  return (1 - alpha) * vectorScore + alpha * fullTextScore;
}

// You should implement your combine vector search and full-text search.
function combineSearchResults(
  vectorResults: RankedResult[],
  fullTextResults: RankedResult[],
): RankedResult[] {
  return combinedResults;
}

async function hybridSearch(
  botId: string,
  topK: number,
  similarityThreshold: number,
  msg: string,
  alpha: number,
) {
  const vectorResults = await vectorSearch(
    botId,
    topK * 2,
    similarityThreshold,
    msg,
  );

  const fullTextResults = await fullTextSearch(botId, topK * 2, msg);

  const combinedResults = combineSearchResults(vectorResults, fullTextResults);

  combinedResults.forEach((result) => {
    result.rrfScore = calculateRRFScore(
      result.vectorRank,
      result.textRank,
      alpha,
    );
  });

  // Sort by RRF score (desc) and return topK results
  return combinedResults.sort((a, b) => b.rrfScore - a.rrfScore).slice(0, topK);
}
```

We have the properties from type RetrievalModel, and we can pass them into a reusable component. Then we have three card components representing each method. 

![Retrieval settings for search method](assets/hybrid-search-retrieval-settings-chatbot-builder.webp)

We have the result after the search, the return will be formatted by the interface RankedResult. We have the properties content, sourceType, vectorRank, textRank, rrfScore will be rendered to card components.

![Retrieval page for testing](assets/hybrid-search-retrieval-page-chatbot-builder.webp)

## Evaluating and metric in information retrieval

Understanding how well your search system works is importance. Metrics in information retrieval help us gauge the performance of these systems. Let’s break down some key metrics like recall, precision, F1-score, and Normalized discounted cumulative gain (NDCG) to see how they work in evaluating search systems.

![Metric in information retrieval flow](assets/hybrid-search-metric-evaluation.webp)

### Recall, precision, and F1 score

- **Recall:** Recall measures how many relevant documents your search system retrieves out of the total relevant documents available. Example recall is 75% (6 out of 8 relevant documents), this suggests the system is quite good at retrieving most of the relevant documents.
- **Precision:** Precision measures how many of the retrieved documents are actually relevant. Example \*\*\*\*precision is 80% (4 out of 5 documents are relevant). This shows the system is very efficient at retrieving relevant documents with minimal irrelevant ones.
- **F1 score:** The F1 score is the harmonic mean of recall and precision, providing a single metric to evaluate overall performance. Example F1 score is 0.705, with a recall of 75% and a precision of 66.6%. This shows the system is fairly balanced and effective overall, combining the strengths of different search methods.

![Recall, precision and F1 score](assets/hybrid-search-metric-unware.webp)

### Normalized discounted cumulative gain

Normаlizeԁ Disсounteԁ Cumulаtive Gаin (NDCG) is а metriс useԁ in informаtion retrievаl to meаsure the effeсtiveness of seаrсh engines, reсommenԁаtion systems, аnԁ other rаnking аlgorithms. This metriс evаluаtes rаnking quаlity by tаking into ассount both а relevаnt item’s рosition аnԁ its imрortаnсe or relevаnсe. Pаrtiсulаrly vаluаble in sсenаrios vаluing higher-rаnkeԁ results over lower ones – where ԁifferent query outсomes сoulԁ holԁ vаrying levels of relevаnсe.

**Example:** Azure AI has identified various user query categories, this images provides definitions and examples for various types of queries used in our evaluation dataset.

![Normalized discounted cumulative gain example](assets/hybrid-search-metric-ware-example.webp)

NDCG@3 comparison across query types and search configurations. All vector retrieval modes used the same document chunks (512 token chunks w/25% overlap with Ada-002 embedding model over customer query/document benchmark). Sentence boundaries were preserved in all cases.

![Normalized discounted cumulative gain point](assets/hybrid-search-metric-ware-point.webp)

## Conclusion

To summarize, we will choose a hybrid search in cases where we need to combine meaning and precision. Using the re-rank model will increase accuracy, but the cost and context window length will be challenges. After that, we make decisions based on metrics when testing the datasets. Recall lets you know the percentage of missing chunks or get all chunks relevant, and precision makes sure the chunk retrieval is correct with the chunk you want. This metric will tell you what kind of search method will fit your datasets.

## Reference

- https://www.pinecone.io/learn/offline-evaluation
- https://www.evidentlyai.com/ranking-metrics/ndcg-metric
- https://deepchecks.com/glossary/normalized-discounted-cumulative-gain
- https://superlinked.com/vectorhub/articles/optimizing-rag-with-hybrid-search-reranking
- https://www.assembled.com/blog/better-rag-results-with-reciprocal-rank-fusion-and-hybrid-search


Hybrid search comes to solving problems in vector search and full-text search. Let's see how it works. In this article, we will know how hybrid search approach and how to evaluate each search method through metrics in information retrieval.

Evaluating search engine in RAG systems


Collecting user feedback is importance for improving the accuracy and relevance of responses. One simple yet powerful feedback mechanism is the thumbs up and thumbs down system. This article explores how labeling feedback in this manner can enhance the training data for LLMs and provides a case study for obtaining human feedback.

## Pearson correlation coefficient

The Pearson correlation coefficient is a statistical measure that quantifies the strength and direction of the relationship between two variables. It ranges from -1 to +1, where +1 indicates a perfect **positive** linear relationship, -1 indicates a perfect **negative** linear relationship, and 0 indicates no linear relationship. This coefficient is widely used in various fields to understand how variables are related and to make predictions based on these relationships.

![](assets/pearson-correlation-coefficient.webp)

## Thumbs up and thumbs down feedback

- **Thumbs Up (Positive Feedback)**: When a user interacts with an LLM and receives a helpful, accurate, or satisfying response, they provide a "thumbs up" or a positive rating. This indicates that the LLM's output was valuable and aligned with the user's intent.
- **Thumbs Down (Negative Feedback)**: Conversely, if a user receives an unhelpful, incorrect, offensive, or otherwise unsatisfactory response, they give a "thumbs down" or a negative rating. This feedback signals that the LLM's output needs improvement.

![](assets/google-feedback-form.webp)

## Simplifying user interaction and increasing engagement

**User-Friendly Feedback Mechanism:** Using thumbs up and down is a simple way to get user feedback. When it's quick and easy, more people are likely to join in. This means more feedback, which gives the model better data to learn from.

**Encouraging Consistent Feedback:** Since giving a thumbs up or down is so easy, people can keep giving feedback without feeling it's a hassle. This steady feedback is really important for models that need constant input to get better over time. The more feedback the model gets, the better it can understand and guess what users like.

## Case studies

**Sentiment Analysis in Review Systems:** In sentiment analysis, using thumbs up and down can really help in figuring out if reviews are positive or negative. For example, an algorithm that sorts product reviews can use this simple feedback to get better at understanding how people feel. This makes it easier for the model to catch the subtle ways people express their opinions, leading to more accurate results.

![](assets/tiki-collect.webp)

**Content Moderation and Filtering:** In content moderation, using thumbs up and down feedback can help spot inappropriate or harmful stuff. When users give a thumbs down to something they don't like, the model learns to filter out similar content in the future. This feedback loop is key to keeping online spaces safe and positive. Platforms like YouTube and Twitter use likes and dislikes to manage and filter content. This feedback helps algorithms find trends, popular posts, and catch possible misinformation or harmful content.

![](assets/youtube-collect-form.webp)

### Reference

- [Pearson correlation coefficient (r) | Guide & Examples (scribbr.com)](https://www.scribbr.com/statistics/pearson-correlation-coefficient/)
- [What’s missing to evaluate foundation models at scale - TruEra](https://truera.com/ai-quality-education/generative-ai-observability/whats-missing-to-evaluate-foundation-models-at-scale/)
- [How content filtering makes it possible to do moderation at scale – Besedo](https://besedo.com/blog/content-filtering-vs-moderation/)
- [(PDF) Thumbs up or thumbs down? Semantic orientation applied to unsupervised classification of reviews (researchgate.net)](https://www.researchgate.net/publication/248832100_Thumbs_Up_or_Thumbs_Down_Semantic_Orientation_Applied_to_Unsupervised_Classification_of_Reviews)
- [How to make the most out of LLM production data: simulated user feedback | by Pasquale Antonante, Ph.D. | Towards Data Science](https://towardsdatascience.com/how-to-make-the-most-out-of-llm-production-data-simulated-user-feedback-843c444febc7)


The article talks about how important human feedback is for making large language models better, especially using thumbs up and down. It explains the Pearson correlation coefficient, which measures how variables are related. The thumbs system is an easy way for users to show if responses are helpful or not, which encourages more feedback and keeps users engaged. This feedback helps improve training data and makes the models more accurate. The article includes examples in sentiment analysis for reviews and content moderation, showing how this feedback helps understand user opinions and filter out bad content on sites like YouTube and Twitter.

Thumbs up and thumbs down pattern


Getting consistent and accurate answers from LLMs is a major challenge. One effective strategy to address this challenge is incorporating a robust feedback mechanism within the app to collect user feedback. The memo will focus on how we collect logs of this feedback, showing how important it is to have a system for collecting input from users. It will also dive into how we design a database to safely store all this valuable feedback, making sure it's organized and useful for improving our models.

## Overview of feedback mechanism

While we will dig into the details of feedback collection and its impact on LLMs, it’s better to understand the whole picture first. Here’s a step-by-step look at how we use feedback to make our models better:

1. **Collect logs**: All interactions, implicit and explicit feedback, along with additional context like prompt history and model parameters, are logged.
2. **Train**: With this data, we can choose between fine-tuning and pre-training. Most human feedback will be trained with the RLHF method. 
3. **Validate**: The updated model is then validated through A/B testing and with validation agents to ensure the changes lead to improved performance.
4. **Deploy and monitor**: The new model is deployed, and its performance is continuously monitored using the ongoing collection of user feedback.

![Feedback Diagram](assets/feedback-mechanism.webp)

## How user feedback improves AI: implicit and explicit methods

### Implicit feedback

Implicit feedback comes from how users behave when they interact with an AI model, without them directly telling us what they think. There are a few main ways we notice this:

1. **Stop generate**: If users stop the AI from generating a response quickly, it usually means they don't like what it's saying or find it irrelevant. For example, if they read a few lines and feel it's not helpful, they might stop it.
2. **Double check response**: When users go to search engines to check information from the AI, it shows they might not trust what it said. This often happens if the AI gives answers that seem made up, especially with numbers or dates.
3. **Regenerate**: If users ask for a new answer, it means they didn't like the first one. They might use a button to get a different response when the first one doesn't meet their needs. For instance, they might want a better answer to the same question.

![Implicit feedback](assets/implicit_feedback.webp)

### Explicit feedback

Explicit feedback involves users directly communicating their satisfaction or dissatisfaction with the AI’s responses. The methods include:

1. **Like/Dislike**: Users can give a thumbs up if they like the response or thumbs down if they don’t. This quick feedback helps us know how well the AI’s answers are working. For example, a thumbs up means they found it useful, while a thumbs down means it wasn’t helpful.
2. **Scoring**: Users can give a score to show how good they think the response is. This lets them give detailed feedback on whether the answer was great or just okay. For instance, a 5-star rating helps us see where the AI did well, and a 4-star rating shows areas for improvement.
3. **Surveys and question**: These ask users specific questions about how well the AI is doing. They can give detailed feedback on what they like and what could be better. For example, a survey might ask about accuracy, how easy it was to use, and if the tone was right.
4. **Textual feedback**: Users can write their thoughts on the AI’s responses. This lets them explain in detail what worked or didn’t work for them. For instance, they might say how a response helped with their work or where it missed the mark.

![Explicit feedback](assets/explicit_feedback.webp)

## Organizing user feedback for LLM application

After collecting user feedback, we'll organize the data using the schema below. This structured approach makes it easier to analyze and understand the feedback. By categorizing it properly, we can see exactly where improvements are needed. This organized data helps create datasets for fine-tuning or pretraining our large language models.

![Database table for storing human feedback](assets/database_schema_feedback.webp)

**chat_feedback table:**

- `id` A unique identifier for each feedback entry.
- `thread_id` Links the feedback to a specific conversation thread.
- `chat_id` Identifies the particular message within the thread that the feedback refers to.
- `type_id` Categorizes the type of feedback provided. We can define enum type_id for storing specify facts.
- `notes` Allows users to add specific comments or details about their feedback.

**Relationships:**

- `chat_feedback.thread_id` references `threads.id` This links feedback to the appropriate conversation thread, ensuring we can trace feedback back to the entire conversation context. When we need to load history chat, we can count total thumbs-up and thumbs-down user reactions there are.
- `chat_feedback.chat_id` references [`chats.id`](http://chats.id) This connects the feedback to the specific message. We can render history chat again, improving the user experience of specific chats.
- `chat_feedback.type_id` references `feedback_type.id` This categorizes the feedback, making it easier to analyze patterns and specific issues.

**feedback_type table:**

- `id` A unique identifier for each type of feedback.
- `name` A descriptive name for the feedback type.

![History chat UI](assets/history_chat_feedback.webp)

![Feedback type](assets/type_feedback.webp)

We need to track user feedback by `chat_id` and `thread_id`, as well as `feedback_type` and `notes`. We will provide a selection to describe how users interact with the LLM response, which will be stored in `type_id`. We define the types of feedback in the feedback_type table, and we can define common facts for the user to easily choose.

![Feedback form UI](assets/ui_feedback_form.webp)

Imagine you're chatting with a chatbot. If you receive a response that's too long, you can click the dislike button then select "Too long" from the feedback options and add a note explaining that the information could be more detailed. This feedback is saved in the system, linked to the specific `chat_id` and `thread_id`, and categorized under the "Too long" type.

Let's say you're chatting with a bot to get some tips on baking a cake. You ask, "How can I make my cake fluffy?" Instead of giving you baking advice, the bot starts talking about the history of cakes, which isn't what you wanted. So, you hit the "stop" button because the response isn't relevant to your question. This action tells us the answer wasn't helpful or on-topic.

## What’s next

Collecting feedback creates a valuable dataset for later training step, using method like Reinforcement Learning from Human Feedback (RLHF). To understanding step by step using human feedback for fine-tuning large language model, you can found articles about [RLHF method](https://dwarvesf.hashnode.dev/challenges-faced-when-researching-rlhf-with-openassistant) then we go next step training human feedback at [here]().

## Conclusion

By tracking interactions and analyzing feedback, we can keep improving the outputs to be more accurate and user-friendly. Understanding how to design a database to log the feedback will help us prepare good datasets for later training step.

## Reference

- [Evaluating LLM outputs - cohere.com](https://cohere.com/blog/evaluating-llm-outputs)
- [Human in the loop feedback - orq.ai](https://docs.orq.ai/docs/human-in-the-loop-feedback)
- [Human in the loop - machine learning - definition & examples | Encord](https://encord.com/blog/human-in-the-loop-ai)
- [How to test LLMs in production? (leewayhertz.com)](https://www.leewayhertz.com/how-to-test-llms-in-production/#A/B-testing)


Improving AI models requires collecting accurate user feedback,which can be difficult. The article explores more into the importance of collecting human feedback and discusses the need for creating an organized database to effectively store and use this valuable input. We hope to continuously improve our large language models, ensuring they deliver greater performance and match user expectations more effectively.

Design feedback mechanism for LLM applications


## Overview

When Large Language Models (LLMs) becomes popular and an essential tool for growing businesses, it signals a transition toward more complex and efficient data processing. Instead of outputting raw text, the models may now generate structured data in formats such as JSON or XML. This allowed the information to be directly integrated into company databases, removing the need for human processing. Businesses can use structured outputs to streamline their data workflows, decrease processing time, and improve the accuracy and reliability of their analytics.

## Why structured output is needed

[A survey of 51 industry professionals](https://arxiv.org/pdf/2404.07362v1) investigated the contexts and motivations behind limitations placed on Large Language Models. The findings identified two key constraint categories: low-level and high-level. Low-level constraints focus on technical aspects, guaranteeing the generated content adheres to a specific format and length. High-level constraints, on the other hand, address semantic and stylistic aspects, ensuring the outputs are meaningful, avoid factual errors (hallucination), and maintain a desired style. By implementing these constraints, developers can streamline the development process, improve the user experience by ensuring consistent and clear outputs, and ultimately guarantee the quality and usability of what LLMs produce.

- This consistency fosters user trust and satisfaction. Users know what to expect from the LLM, leading to a more positive experience. For example, when an LLM summarizes news articles, structured outputs guarantee all summaries follow the same format (e.g., headline, key points, source), making it easy for users to understand the information without encountering unexpected variations in layout.
- This allows for seamless integration with existing development tools. For instance, when an LLM generates product descriptions for an online store, structured outputs ensure the descriptions fit perfectly into the product database, saving developers time on reformatting.
- Structured outputs provide pre-defined formats (e.g., JSON, XML). This allows developers to leverage LLMs for automated tasks. For instance, an LLM can generate financial reports in a structured format like JSON. Developers can then directly integrate this data into existing financial dashboards.

## The first signals in the emergence of structured output

**Pre-2022:**

- **Focus on text generation:** LLMs primarily focused on generating creative text formats like poems, code, scripts etc. Structured output wasn't a major area of research at this point.

**During 2022:**

- **User workarounds:** Developers started resorting to manual parsing techniques (regular expressions, custom parsers) to extract structured information from LLM outputs. This was inefficient and limited scalability.
- **Prompt crafting:** Users experimented with crafting prompts that subtly nudged LLMs towards generating outputs with a desired structure, although success was limited.

**Late 2022/Early 2023:**

- **LangChain:** emerged as a Python library specifically designed to streamline information extraction from various sources, including LLM outputs. It provided key functionalities that laid the groundwork for structured output:
  - **Document loaders:** These loaders allowed LangChain to handle data from different sources, including LLM outputs. This was crucial for treating LLM text as a data source for structured information extraction.
  - **Parsers:** Parsers within LangChain enabled users to define how to extract the desired structured information from the unstructured LLM text. This offered a more systematic approach compared to manual parsing techniques prevalent before LangChain.

**During 2023:**

- **JSONFormer (research phase):** This approach explores a novel "structured decoding" technique for a subset of JSON schemas. While under development, it holds promise for even finer control over LLM output structure.
- **OpenAI's JSON mode:** This feature allowed users to provide a JSON schema within the prompt, essentially creating a template for LLM generation, leading to increased accuracy and consistency.
- **Kor:** This library streamlined structured output extraction by allowing users to provide both a schema and example data for the LLM. This improved the understanding of desired format and content for better-structured outputs.

**During 2024:**

- **LlamaIndex**: introduced core "structured output" features. This signified a move towards integrating structured output capabilities directly within LLMs:
  - **Function calling APIs:** Users could specify desired output formats (e.g., JSON) directly within the LLM prompt, making guidance more intuitive.
  - **Output parsers:** Parsers could be used before and after LLM calls to ensure the output adhered to the specified structure, adding control.
- **LLM-structured-output:**
  - Ensuring structured outputs adhere to specific formats. This repository provides tools and examples specifically focused on JSON schema validation.
  - By implementing an "acceptor" system, the repository verifies if the LLM's generated text conforms to a predefined JSON schema. This functionality promotes data accuracy and reliability in structured output generation, a key aspect for integrating LLMs into various applications

![Timeline of structured output library](assets/history-of-structured-output-for-llms_timelinecycle.webp)

There's been a clear progression from user workarounds and external frameworks to functionalities embedded directly within LLMs. We've seen increased control over output format, with advancements like JSON schemas and schema-example combinations. Research continues to address accuracy, flexibility, and seamless integration of structured output functionalities across LLM platforms.

## Challenge and future of structured output

Structured output strives for a balance between two seemingly opposed forces:

- **Structured data requirements:** Businesses and applications often require data in specific formats for analysis, reporting, and integration. Imagine needing financial data from an LLM report, but it comes back as a free-flowing narrative. Structured output aims to bridge this gap.
- **The creativity of LLMs:** LLMs excel at generating creative text formats. Confining them to rigid structures can stifle their potential. The ideal solution allows LLMs to adhere to a format while still retaining some flexibility in their output.

**Challenges to Overcome**

- **Accuracy:** One of the main challenges facing LLMs is ensuring consistency and without errors creation inside complex formats. These models may exhibit difficulties when dealing with complex structures or subtle data, which could result in incorrect output. For instance, early structured outputs might have produced financial reports with factual errors due to the LLM's difficulty handling specific formats.
- **Flexibility:** Balancing structure and creativity in LLM outputs is crucial. While structured data is necessary, it’s important that LLMs also creatively express information within a specified framework. Otherwise, they risk becoming rigid and robotic in their responses.
- **Integration:** Structured output functionalities currently may require external libraries or custom coding, depending on the LLM platform. Seamless integration into various platforms and applications is essential for wider adoption. Historically, structured outputs were often limited to specific LLM platforms, necessitating additional coding for use in other environments.

**The Future of Structured Output**

By addressing these challenges and continuing research, structured output has the potential to unlock exciting possibilities:

- **More accurate and flexible outputs:** LLMs will be able to generate data that adheres to complex structures while retaining some creative freedom, offering the best of both worlds.
- **Wider range of applications:** Structured output will become more accessible and integrated into various platforms, enabling applications in data analysis, report generation, form completion, and more.
- **Enhanced human-AI collaboration:** Humans and LLMs will work together more effectively to produce high-quality structured outputs. Human feedback can guide the LLM, leading to a more efficient and productive interaction.

## Reference

- https://arxiv.org/pdf/2404.07362v1
- https://github.com/langchain-ai/langchain
- https://github.com/outlines-dev/outlines
- https://github.com/1rgs/jsonformer
- https://github.com/otriscon/llm-structured-output
- https://github.com/jxnl/instructor
- https://www.youtube.com/watch?v=yj-wSRJwrrc
- https://www.timlrx.com/blog/generating-structured-output-from-llms
- https://medium.com/@kyeg/unlocking-structured-outputs-with-agents-8b5a564b5d44


When Large Language Models (LLMs) becomes popular and an essential tool for growing businesses, it signals a transition toward more complex and efficient data processing. Instead of outputting raw text, the models may now generate structured data in formats such as JSON or XML. This allowed the information to be directly integrated into company databases, removing the need for human processing. Businesses can use structured outputs to streamline their data workflows, decrease processing time, and improve the accuracy and reliability of their analytics.