## When

Building software with Large Language Models (LLMs) involves several steps, from planning to deployment. LLM tracing emerges as a final step in this process, providing ongoing insights and enabling continuous improvement of LLM-powered applications.

![](assets/llm-tracing-build-steps.webp)

## Why

Before diving into tracing, it's important to understand the fundamental difference between traditional software and LLM-powered applications:

- **Traditional software**: Deterministic, based on explicit instructions written by programmers.
- **With LLMs**: Probabilistic, based on neural networks with weights determined through training.

![](assets/llm-tracing-architecture.webp)

Why LLM Tracing is Necessary:

1. **Unpredictable outputs**: LLMs can produce different outputs for the same input due to their probabilistic nature.
2. **Black box nature**: The decision-making process of an LLM is opaque.
3. **Complex interactions**: LLMs often interact with multiple components (e.g., retrieval systems, filters, classifiers, external APIs) in ways that aren't immediately obvious.
4. **Performance variability**: Performance can vary significantly based on input complexity, model size, and hardware.
5. **Evolving behavior**: LLMs can exhibit evolving behavior through fine-tuning or in response to different prompts.
6. **Error diagnosis**: "Errors" in LLMs might be subtle, like hallucinations or biased responses.
7. **Continuous improvement**: LLMs can be improved through better prompts, fine-tuning, or model updates.

## Key metrics

| **Basic** | **Evaluating** |
| ------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------ |
| Latency Throughput Error Rate Resource Utilization Execution Time ... | Factual Accuracy Relevance Bias Detection and Fairness Hallucination Rate Coherence ... |

While basic metrics like latency and throughput measure operational performance, evaluative metrics dig deeper into the actual output and behavior of the LLM.

## Tools

Some popular tools that support various aspects of LLM tracing:

- [LangSmith](https://docs.smith.langchain.com/)
- [Phoenix (by Arize)](https://arize.com/)
- [TraceLoop](https://arize.com/)
- [OpenTelemetry](https://opentelemetry.io/blog/2024/llm-observability/)
- [Portkey](https://portkey.ai/)

## References

- https://colab.research.google.com/github/Arize-ai/phoenix/blob/main/tutorials/llm_ops_overview.ipynb
- https://arize.com/blog-course/llm-evaluation-the-definitive-guide/
- https://docs.smith.langchain.com/how_to_guides/tracing
- https://karpathy.medium.com/software-2-0-a64152b37c35
- Demo: https://colab.research.google.com/gist/tienan92it/490dd65748518a9abc73cdf4bd84583d/welcome-to-colab.ipynb

Understanding LLM Tracing - Principles, Techniques, and Applications in building LLM-powered AI systems.

LLM tracing in AI system


## Generative AI

![](assets/developing-rapidly-with-generative-ai_ai-eco.webp)

Generative AI is a subset of artificial intelligence that focuses on creating new content, such as images, text, or audio, based on patterns learned from existing data.

## Stages for building LLM-powered features

![](assets/developing-rapidly-with-generative-ai_llm-building-stages.webp)

### 1. Identify use cases

The first stage is to identifying where generative AI can make an impact. The common challenges can be:

- Involve analysis, interpretation, or review of unstructured content (e.g. text) at scale
- Require massive scaling that may be otherwise prohibitive due to limited resources
- Would be challenging for rules-based or traditional ML approaches

### 2. Define requirements

This phase requires a thoughtful analysis to select the best-suited LLM and to frame the problem as a prompt to an LLM. Several factors of defining product requirements:

- **Latency**: How fast does the system need to respond to user input?
- **Task complexity**: What level of understanding is required from the LLM? Is the input context and prompt super domain-specific?
- **Prompt length**: How much context needs to be provided for the LLM to do its task?
- **Safety**: How important is it to sanitize user input or prevent the generation of harmful content and prompt hacking?
- **Language support**: Which languages does the application need to support?
- **Estimated QPS**: What throughput does our system eventually need to handle?

### 3. Prototype

Selecting off-the-shelf LLM which use for the prototype. The general idea is that if problems can't be adequately solved with state-of-the-art foundational models like GPT-4, then more often than not, those problems may not be addressable using current generative AI tech.

The key step at this stage is to create the right prompt. To do this, a technique known as [AI-assisted evaluation](https://arize.com/blog-course/llm-evaluation-the-definitive-guide/) can help to pick the prompts that lead to better quality outputs by using metrics for measuring performance.

![](assets/developing-rapidly-with-generative-ai_evaluating-prompts.webp)

### 4. Deploying at scale

![A high-level architecture for an LLM application](assets/developing-rapidly-with-generative-ai_llm-arch.webp) This involves setting up the infrastructure to handle the expected load, monitoring the system's performance, and ensuring that the feature continues to meet the requirements set in the previous stages. There are 2 ways to consider for deploying:

- **Using commercial LLMs**: this is greate to accessing to top-notch models, don't have to worry about setting up the tech, but the expenses can add up quickly.

- **Self-hosted LLMs**: can reduce costs dramatically - but with additional development time, maintenance overhead, and possible performance implications.

## References

- [Developing rapidly with generative AI](https://discord.com/blog/developing-rapidly-with-generative-ai)
- [Artificial intelligence, machine learning , deep learning, GenAI and more](https://medium.com/womenintechnology/ai-c3412c5aa0ac)
- [LLM evaluation: everything you need to run, benchmark LLM evals](https://arize.com/blog-course/llm-evaluation-the-definitive-guide/)


Generative AI overview and the different stages of building an LLM-powered feature

Developing rapidly with generative AI


Data-fetching libraries are software tools or frameworks that can help improve the performance and scalability of your application by handling network requests and data processing more efficiently. In React, one of the key challenges these libraries address is optimizing rendering to avoid unnecessary re-rendering of components when the underlying data remains unchanged. This helps to prevent situations where components re-render multiple times despite no changes in the data, resulting in improved efficiency and smoother user experiences.

Let's dive in how it works.

## Deduplication

Data-fetching libraries typically use caching strategies to reduce the number of network requests that need to be made, and improve the performance of the application. By using a unique key as the identifier and stale-while-revalidate as cache validation, the library will return responses from cached data or make a network request.

![](assets/render-optimization-in-data-fetching-libraries_render-optimization-in-data-fetching-1.webp)

## Data selection

Everytime a request is sent, the corresponding components will be updated by 3 very common stateful values:

```json
{ data, error, fetching }
```

Look at `fetching`, this flag is quite useful if you want to display a loading indicator. But it's also kinda unnecessary if you don't do that. Then your component will render twice even though nothing changed in data because this flag is always true when a request is in-flight and otherwise. Unexpected render happens when a hook request is used like:

```js
const data = useRequest(...)
```

For this use-case, most libraries is designed to notify on what properties are using in component only. So it's recommended to use destructuring instead:

```js
const { data, error } = useRequest(...)
```

## Structural sharing

Structural sharing is the most advance feature that can optimize rendering over data selection. One of the data-fetching libraries has turned on out of the box is [React Query](https://tanstack.com/query/latest/). To use this feature, makes sure your data is kept referential identity on every level. As an example, suppose we have the following data structure:

```json
[
  { "id": 1, "name": "Learn React", "status": "active" },
  { "id": 2, "name": "Learn React Query", "status": "todo" }
]
```

Now suppose first todo is changed into the *done* state:

```diff
[
- { "id": 1, "name": "Learn React", "status": "active" },
+ { "id": 1, "name": "Learn React", "status": "done" },
 { "id": 2, "name": "Learn React Query", "status": "todo" }
]
```

Structural sharing will attempt to compare the old state and the new and keep as much of the previous state as possible.

This comes in very handy when using selectors for partial subscriptions:

```js
// ✅ will only re-render if _something_ within todo with id:2 changes
// thanks to structural sharing
const { data } = useTodo(2);
```

In some instances, especially when having very large datasets, structural sharing *can* be a bottleneck. Let's use it carefully.

## References

- https://tkdodo.eu/blog/react-query-render-optimizations
- https://nextjs.org/docs/app/building-your-application/data-fetching
- https://swr.vercel.app/docs/advanced/performance
- https://tanstack.com/query/latest/docs/react/guides/caching


Data-fetching libraries are software tools or frameworks that can help improve the performance and scalability of your application by handling network requests and data processing more efficiently.

Render optimization in data-fetching libraries

![](assets/continuous-translation_eaf982ca480d0677ec9d8fd34553b51a_md5.webp)

Continuous translation (CT) is a modern approach to translation management that involves synchronizing software development and translation workflows. This means that developers, translators, and product owners work together in a continuous cycle to ensure that all translations are up-to-date and aligned with the latest software developments.

Unlike traditional translation workflows, Continuous translation eliminates the need for file and space synchronization between different stakeholders, streamlining the translation process and promoting better collaboration. By implementing Continuous translation, companies can achieve faster development cycles, higher-quality translations, and a better user experience for their customers.

## Working with localization

### How localization generally works

![](assets/continuous-translation_0ca3440b9dbd840d349d587dd7fd6a1d_md5.webp)

The classical or typical approach in any software project will have translated data be coupled with the repository (plus its software release) and its access to the data. This data could exist in a database or directly as a file. Changes to the data would be dependent on the coupling of the git repository and the translation management system, which becomes a blocker for translation teams.

### Alternative solutions to localization

In order to reduce the friction between the developing team and the translation team, we can remove data and process coupling between systems of both teams by centralizing the data to allow for continuous translation. In this article we’ll discuss 2 possible solutions to integrate continuous translation into our current system:

- **Solution 1**: implement an ad-hoc solution ourselves with help from Google Sheets (using Google Translator under the hood).
- **Solution 2**: translation as a service - we use a third party service for manage our translations.

### Solution 1: Google Sheets

Google Sheets is a free and easy-to-use tool that allows us to manage translations with all stakeholders. Before deploying the app, developers can use Git actions to fetch the latest version of the Google Sheets and generate the locale translation files.

Here are what the steps to use Google Sheets would look like for managing translations:

**Step 1:** Create a Google Sheet that contains all supported translation items. Here is an example **[template](https://docs.google.com/spreadsheets/d/1jjVDCMAmS6WySmB7L25yNCZX-X3jEwrrqLhqOBzVEs0/edit?usp=sharing)** you can use. **Step 2:** Fetch data from Google Sheets using the following code:

_Make sure to replace the _`***sheetId***`_ and _`***sheetName***`_ with your own values_

```javascript
// Save on './public/spreadsheet.ts' file
const sheetId = "<Sheet-ID>";
const sheetName = "<sheet-name>";
const baseUrl = `https://opensheet.elk.sh/${sheetId}/${sheetName}`;

// Path to store all translation data
const translateDataPath = "./public/locales/translate-data.json";

export const getJsonData = async () => {
 const res = await fetch(baseUrl, {
 method: "GET",
 headers: {
 "Content-Type": "application/json",
 accept: "*/*",
 authority: "opensheet.elk.sh",
 },
 mode: "cors",
 credentials: "omit",
 });

 if (res.ok) {
 return await res.json();
 }
};

getJsonData().then((data) => {
 const fs = require("fs");
 let myObject = data;

 // Writing to our JSON file
 var newData = JSON.stringify(myObject, null, 2);
 fs.writeFile(translateDataPath, newData, (err) => {
 // Error checking
 if (err) throw err;
 console.log("New data added");
 });
});
```

**Step 3: **After having obtaining the translation data, generate locale files for supported languages using the following code:

```javascript
// Save on './public/manage-translations.ts' file
import path from "path";
import fs from "node:fs/promises";
import fsExtra from "fs-extra";
import _ from "lodash";

import dotenv from "dotenv";
dotenv.config({ path: ".env.local" });
dotenv.config({ path: `.env.${process.env.NODE_ENV}` });
dotenv.config();

const validLang = ["en-US", "de-DE", "fr-FR"];
const defaultLanguage =
	process.env.LOCALE == null ? "en-US" : validLang.includes(process.env.LOCALE) ? process.env.LOCALE : "en-US";

const configs = {
	defaultLanguage,
	otherLanguages: validLang.filter((lang) => lang !== defaultLanguage),
	rootExportPath: "./public/locales",
};

const allLocales = [];

async function generateJSONFiles() {
	const data = await fs.readFile("./public/locales/translate-data.json", "utf8");
	const jsonData = JSON.parse(data);
	const locales = Object.keys(jsonData[0]).filter((key) => key !== "elementId");
	const result = locales.map((locale) => {
		const data = {};
		jsonData.forEach((item) => {
			data[item.elementId] =
```

### Solution 2: Use a translation management platform

For this approach, we can use [Locize](https://locize.com/) as our continuous localization management platform. This approach isn’t limited to Locize, but the idea is to have a platform to decouple software release from the translation work and minimize work friction for translation.

![](assets/continuous-translation_5d971ebb8af780ed2ad9e7626daf0d8c_md5.webp)

Locize has integration support for a variety of frontend systems. You can integrate Locize by following steps:

- Follow [https://docs.locize.com/integration/getting-started](https://docs.locize.com/integration/getting-started) to create a Locize account and project.
- After the project is created, we will have `project id` and `api key`.
- Use [https://github.com/locize/locize-cli](https://github.com/locize/locize-cli) to synchronize the existing translations with Locize

There are three ways to use `Locize` in your app:

- **Approach 1**: Use Locize live download on the client-side only. This option involves bundling translations in your app to prevent an elevated amount of downloads on the server-side. Before deploying your app, synchronize your translations with Locize so that they are bundled in your app. This way, your server-side will not generate any downloads to the Locize CDN during runtime, but only on the client-side.
- **Approach 2:** Configure Locize to download translations live on both client (browser) and server (node.js).

> Do not use this option if you have a `serverless environment` as it can generate too many download requests and run up your bill.

- **Approach 3:** Bundle translations with your app. This option involves bundling translations in your app at build time. It's recommended to use this option if you have a small number of translations or if your translations don't change frequently.

### Comparisons between solutions

**Solution 1** recommends using a custom translation file and providing translations for each language in a separate JSON file. The translation file is then loaded on the server side and used to render the content in the appropriate language. This solution is relatively simple and straightforward to implement, but it can become cumbersome to manage as the number of languages and translations grows.

**Solution 2** proposes using a translation management platform, with one example using Locize, which allows for continuous localization management. This solution involves integrating Locize into the application, synchronizing the existing translations with Locize, and then bundling the translations in the application using one of three different possibilities, depending on the specific use case. This solution requires more setup and configuration but can provide a more scalable and streamlined approach to managing translations.

In summary, `Solution 1` is a simpler and cheaper approach to manage translations, while `Solution 2` is a more advanced solution that provides more flexibility and scalability in managing translations.

### Pros and cons when integrating Continuous translation into your app

**Pros**

- **Faster development cycles**: With Continuous translation, developers and translators work together in a continuous cycle, ensuring that translations are updated in real-time as new features are developed. This leads to faster development cycles and quicker time-to-market.
- **Improved translation quality**: Continuous translation promotes better collaboration between developers and translators, which can lead to higher-quality translations that accurately reflect the intended meaning of the original content.
- **Better alignment between teams**: By eliminating file and space synchronization issues, Continuous translation helps to align developers, translators, and product owners more closely, reducing communication errors and promoting better collaboration.
- **Reduced costs**: Traditional translation workflows can be time-consuming and costly. By streamlining the translation process, Continuous translation can help to reduce translation costs and improve return on investment.

**Cons**

- **Requires significant coordination**: Continuous translation requires a high level of coordination between different teams, including developers, translators, and product owners. This can be challenging to manage, particularly for larger projects.
- **Potential for errors**: Continuous translation requires real-time updates to translations, which can increase the risk of errors and miscommunication. This requires careful management and quality control.
- **Requires specialized tools**: Implementing Continuous translation requires specialized tools and technologies, which can add to the overall cost of the project.
- **Not suitable for all projects**: Continuous translation may not be suitable for all projects, particularly those with limited budgets or resources. Traditional translation workflows may be more appropriate for smaller projects or those with less frequent updates.

## Conclusion

Multilingual support is a very important function for web or mobile applications nowadays. Users will come from all over the world and always ask for support for their language. The two options above have different advantages and disadvantages, so you need to consider the exact scope of the product to have the best choice for your team. Both methods can meet the needs of constantly translating products to support new features or new products, but it will cost production as well as quality assurance.

Continuous translation (CT) is a modern approach to translation management that involves synchronizing software development and translation workflows. This means that developers, translators, and product owners work together in a continuous cycle to ensure that all translations are up-to-date and aligned with the latest software developments.

Continuous translation


### What

WebAssembly (abbreviated *Wasm*) was launched in 2017 as a low-level assembly-like language with a compact binary format, so it’s fast to load, execute, and run with near-native performance. It is designed as a portable target for the compilation of high-level languages like C/C++/Rust, enabling deployment on the web for client and server applications. That means you do not write WebAssembly, you compile other high-level languages to it.

![](assets/webassembly_wasm-architecture.webp)

### How

With JavaScript, the code is included in the website and is interpreted as it runs because JavaScript variables are dynamic. The only way to know what the types are for sure is to monitor the code as it executes, which is what the JavaScript engine does. Once the engine is satisfied that it knows the variable’s types, it can convert that section of code into machine code.

![](assets/webassembly_wasm-js-how-it-work.webp)

WebAssembly isn’t interpreted but, rather, is compiled into the WebAssembly binary format by a developer ahead of time. Because the variable types are all known ahead of time, when the browser loads the WebAssembly file, the JavaScript engine doesn’t need to monitor the code. It can simply compile the code’s binary format into machine code.

![](assets/webassembly_wasm-how-it-works.webp)

### High-level goals

- Be fast, efficient, and portable -- WebAssembly code can run at near-native speed regardless of platform.
- Be readable and easily debuggable -- although WebAssembly is a low-level assembler-like language, it has a human-readable text format. This makes it possible to write, read and debug code yourself.
- Be secure -- Actually, WebAssembly is specified to be run during a safe and sandboxed execution environment. Like other web code, it'll enforce the browser's same-origin and permissions policies.
- Don’t break the web -- WebAssembly is designed so that it plays nicely with other web technologies and maintains backwards compatibility.

### Use cases

- Image / video editing.
- Peer-to-peer applications (games, collaborative editing, decentralized and centralized).
- Music applications (streaming, caching).
- Live video augmentation.
- VR and augmented reality (very low latency).
- Scientific visualization and simulation.
- Developer tooling (editors, compilers, debuggers, …).
- Fat client for enterprise applications (e.g. databases).

### Reference

https://developer.mozilla.org/en-US/docs/WebAssembly https://webassembly.org/ https://livebook.manning.com/book/webassembly-in-action/chapter-1/ https://www.xenonstack.com/insights/a-beginners-guide-to-webassembly


WebAssembly (abbreviated *Wasm*) was launched in 2017 as a low-level assembly-like language with a compact binary format, so it’s fast to load, execute, and run with near-native performance.

Webassembly


Published on August 30, 2022, Chrome 105 included Container Queries, one of the most highly requested features in CSS.

## The problem with media queries

When we want to create a responsive layout, we can use media queries to adjust styles based on the screen size of the device viewing our site. However, media queries have some limitations. For example, we cannot use them to style individual components based on their parent's width; we can only adjust the entire page. This is problematic because we cannot create responsive components; only responsive pages.

## What is Container Queries?

The CSS3 property "container queries" allows us to style elements based on the size of a container. It is similar to a Media Query, except it evaluates against the size of a container instead of the size of the viewport.

## How to use Container Queries?

To query a component based on its parent width, we need to use the `container-type` property with possible values: `size`, `inline-size`, `block-size`, `style`, `state`.

```css
.sidebar {
  container-type: inline-size;
}
```

Now we can start to query a container using `@container`. This will query the nearest containment context.

```css
@container (min-width: 300px) {
  .content {
    display: none;
  }
}
```

Additionally, Containers can be named with `container-name` property. This allows us to query a specific container.

```css
.sidebar {
  container-type: inline-size;
  container-name: my-sidebar;
}
```

or shorthand syntax

```css
.sidebar {
  container: my-sidebar / inline-size;
}
```

Then, to query a specific container, we can use `@container` with `container-name`.

```css
@container my-sidebar (min-width: 300px) {
  .content {
    display: none;
  }
}
```

## Browser support

Currently, Container Queries is only available in [modern browsers](https://caniuse.com/?search=Container%20Queries):

- Chrome 105+
- Safari 15+
- Edge 105+

Universal support can be achieved by using [Polyfill](https://github.com/GoogleChromeLabs/container-query-polyfill).

## Reference

- https://developer.mozilla.org/en-US/docs/Web/CSS/CSS_Container_Queries
- https://javascript.plainenglish.io/css-container-queries-3393fbeb6ea8
- https://ishadeed.com/article/container-queries-are-finally-here/
- https://developer.chrome.com/docs/devtools/css/container-queries/


Container Queries is a CSS feature that allows us to style elements based on the size of a container.