
## Introduction

In the rapidly evolving landscape of artificial intelligence, Retrieval-Augmented Generation (RAG) systems have emerged as a powerful paradigm for combining the strengths of retrieval-based and generative models. Caching is a vital technique that boosts performance by storing frequently accessed information. This allows the system to quickly retrieve data without having to repeatedly access large databases, reducing latency and computational costs

## Cache

### KV cache

the KV Cache is a technique used in LLMs to store and reuse intermediate computations (keys and values) to speed up text generation, making the model more efficient, especially when generating longer sequences. In this post, the KV is out of scope. We will focus on the other types of cache.

### Prompt cache

At the level of a large language model (LLM) system, a typical AI workflow often involves repeatedly passing the same input tokens to the model. Without a prompt cache, the model must process the system prompt for each query, leading to redundant computations. However, with a prompt cache, the model processes the system prompt only once for the initial query, improving efficiency by eliminating repetitive processing.

For applications with long system prompts, a prompt cache can significantly reduce both latency and cost. It is also beneficial for queries involving long documents. For instance, if many user queries pertain to the same extensive document, like a book or codebase, caching this document allows for efficient reuse across multiple queries.

Example of prompt cache implementation:

- Gemini context caching: <https://ai.google.dev/gemini-api/docs/caching>
- LLAMA: <https://github.com/ggerganov/llama.cpp/blob/master/examples/main/README.md#prompt-caching>

This caching technique is typically implemented by inference APIs that we use. We could consider this when select the ai provider / inference API for our RAG system.

### Exact cache

If prompt cache and KV cache are unique to foundation models, exact cache is more general and straightforward. Your system stores processed items for reuse later when the exact items are requested. For example, if a user asks a model to summarize a product, the system checks the cache to see if a summary of this product is cached. If yes, fetch this summary. If not, summarize the product and cache the summary.

Exact cache is also used for embedding-based retrieval to avoid redundant vector search. If an incoming query is already in the vector search cache, fetch the cached search result. If not, perform a vector search for this query and cache the result.

### Semantic cache

A semantic caching system aims to identify similar or identical user requests. When a matching request is found, the system retrieves the corresponding information from the cache, reducing the need to fetch it from the original source.

For instance, queries like **What is the capital of Vietnam?**, and **What the capital of Vietnam is?** all convey the same intent and should be identified as the same question.

There are 2 place we could implement semantic cache:

- Between the user and the vector database: This would help us keep the advantage of generative model, the answer is different every time with the same context.
- Between the user and the generative model: This would make the answer to be the same every time with the same context. This is useful when we want to keep the answer consistent and reduce the cost of generating the same answer multiple times.

Placing it at the model’s response point may lead to a loss of influence over the obtained response. Our cache system could consider **"Explain the French Revolution in 10 words"** and **"Explain the French Revolution in a hundred words"** as the same query. If our cache system stores model responses, users might think that their instructions are not being followed accurately.

Based on the use case, we could choose the right place to implement the semantic cache. But most of the time, we will choose the first option to keep the advantage of generative model.

![](assets/rag-caching-semantic-cache.png)

## Semantic cache implementation

### Cache between user and vector database

We would need some tools/libraries for this:

- Embedding tool: to transform the sentences into fixed-length vectors, also know as embeddings. Like: [sentence transformers](https://github.com/UKPLab/sentence-transformers) or could use api of openai if you have budget/access to it.
- Vector database: to store the embeddings. Like: [chromadb](https://github.com/chroma-core/chroma), [pgvector](https://github.com/pgvector/pgvector).

![](assets/rag-caching-query-vectordb.png)

To be able to determine the similarity between the user query and the cached queries, we need to store user query embeddings in the vector database. When a user query comes in, we transform it into an embedding and search for the most similar embeddings in the vector database. If the similarity score is above a certain threshold, we consider the user query to be the same as the cached query and return the cached response.

Example of transforming the user query into embeddings using openai api:

```typescript
type EmbeddingData = {
  object: string;
  index: number;
  embedding: number[];
};
let data: EmbeddingData[] = [];

const response = await fetch("https://api.openai.com/v1/embeddings", {
  method: "POST",
  body: JSON.stringify({
    input: contents,
    model: "text-embedding-3-small",
    dimensions: 1024,
  }),
  headers: {
    "Content-type": "application/json",
    Authorization: "Bearer " + openAICred,
  },
});

type JSONResponse = {
  data?: EmbeddingData[];
};
const resBody = (await response.json()) as JSONResponse;
const data = resBody?.data ?? [];
```

After that we stored the embeddings in the vector database. This is example code using drizzle-orm with postgresql (with pgvector extension):

```typescript
// Schema definition in schema.ts
const semanticCache = createTable(
  "semantic_cache",
  {
    id: uuid("id").notNull().primaryKey(),
    typeId: integer("type_id"),
    key: text("key").notNull(),
    value: jsonb("value"),
    vector: vector("vector", { dimensions: 1024 }),
    createdAt: timestamp("created_at").notNull().defaultNow(),
    expiredAt: timestamp("expired_at"),
  },
  (self) => ({
    vectorHnswIndex: sql`CREATE INDEX kv_cache_vector_hnsw_idx ON kv_cache USING hnsw (vector vector_cosine_ops) WITH (m = 16, ef_construction = 64)`,
  }),

  // Example store the embeddings
  await db.insert(schema.semanticCache).values({
    id: uuidv7(),
    typeId: input.type,
    key: input.key,
    value: JSON.stringify(input.value),
    vector: rs?.embeddings,
    expiredAt: dayjs().add(input.durationSecs, "seconds").toDate(),
  }),
);
```

How we search for the similar embeddings in the vector database:

```typescript
const rows = await db
  .select({
    contexts: schema.semanticCache.value,
    similarity: sql<number>`1 - (${schema.semanticCache.vector} <=> ${sql.raw(`'[${msgEmbeddings.join(",")}]'::vector`)})`,
  })
  .from(schema.semanticCache)
  .where(
    and(
      eq(schema.semanticCache.typeId, CacheTypeEnum.UserQueryEmbedding),
      gt(schema.semanticCache.expiredAt, new Date()),
      sql<boolean>`(1 - (${schema.semanticCache.vector} <=> ${sql.raw(`'[${msgEmbeddings.join(",")}]'::vector`)})) > ${CACHE_EMBEDDING_SIMILARITY_THRESHOLD}`,
    ),
  );
```

After that, we could return the cached response if the similarity score is above a certain threshold.

### Cache between user and generative model

![](assets/rag-caching-user-query.png)

This is similar to the previous implementation, the difference is that instead caching the embeddings, we cache the response of the generative model. As mentioned above, this is useful when we want to keep the answer consistent and reduce the cost of generating the same answer multiple times. But it reduces the flexibility of the generative model.

## Conclusion

Caching is a vital technique that boosts performance by storing frequently accessed information. This allows the system to quickly retrieve data without having to repeatedly access large databases, reducing latency and computational costs. In RAG systems, caching can be implemented at various levels, including KV cache, prompt cache, exact cache, and semantic cache. The choice of cache type depends on the use case and the desired trade-offs between performance, cost, and flexibility.

## Reference

- <https://huyenchip.com/2024/07/25/genai-platform.html#step_4_reduce_latency_with_cache>
- <https://arxiv.org/pdf/2311.04934>
- <https://github.com/UKPLab/sentence-transformers>
- <https://huggingface.co/learn/cookbook/en/semantic_cache_chroma_vector_database>


Caching is a vital technique that boosts performance by storing frequently accessed information. Let's see how it works with RAG. In this article, we will know how cache could be implement in RAG system.

Evaluating caching in RAG systems


One common issue we often tackle in backend engineering is writing test cases. In this article, we will explore the techniques for crafting effective tests in Go, discussing best practices for writing unit tests and utilizing mocks to achieve better isolation. Although our primary focus lies in unit testing-related practices, it is important to note that Golang also supports integration testing. We will also tackle the subject of integration testing in a future article, where we will examine the details and best practices for integration testing in Golang.

## Introduction

### Importance of testing in software development

Testing is crucial in software development to catch bugs and errors, ensure maintainability and modularity, and improve security, and overall software quality. With the rise of cybersecurity threats, testing is becoming increasingly important to ensure software systems are secure and reliable.

What follows is a non-comprehensive list of the benefits you get from adopting unit testing:

- **Unit tests enable earlier bug detection and resolution**
- **Your suite of unit tests becomes a safety net for developers**

A comprehensive suite of unit tests can act as a safety net for developers. By frequently running the tests, they can assure their recent modifications to the code haven’t broken anything

- Unit tests can contribute to higher code quality

This item is a natural consequence of the previous one. Since unit tests act as a safety net, developers become more confident when changing the code. They can refactor the code without fear of breaking things, driving the general quality of the codebase up.

- **Detect code smells in your codebase**

If the ease of adding unit tests to a codebase is a good sign, the opposite is also true. Having a hard time creating unit tests for a given piece of code might be a sign of code smells in the code—e.g. functions that are too complex.

### Overview of Golang testing framework

The Golang testing package offers a user-friendly framework to create unit tests, benchmarks, and examples, streamlining the development process in Golang by enabling execution from the command line. Package testing allows for a variety of test types, including performance, parallel, and functional testing, as well as any combination these.

**Steps for writing test suite in Golang:**

- Create a file whose name ends with \_test.go
- Import package testing by import “testing” command
- Write the test function of form\ <u>func TestXxx(_testing.T)_</u> which uses any of Error, Fail, or related methods to signal failure.
- Put the file in any package.
- Run command go test

Example of a test file:

```go
package main

import (
	"testing"
)

// test function
func TestYourFunc(t *testing.T) {
	actualString := YourFunc()
	expectedString := "dwarvesv"
	if actualString != expectedString{
		t.Errorf("Expected String(%s) is not same as"+
		" actual string (%s)", expectedString,actualString)
	}
}
```

## Strategies for writing effective tests

### Make your code testable and easy to test

When working on code projects, developers often devote a large portion of their time to choosing the right frameworks, libraries, databases, and other third-party components, while the importance of testing is sometimes overlooked.

Proper testing actually makes your project better because it encourages you to:

- Apply clean code: write short functions, handle a single task per function, etc.
- Write extendable and agnostic code through the use of abstractions, interfaces and mocks.
- Understand the business logic better by testing regular/edge cases and high coverage of these.
- Avoid legacy, long-untouched and unmaintainable code — tests will ease the process of maintaining and verifying changes to code so it doesn’t rot.

### Writing clear and concise test cases

One of the most important of a good test is easy to read and maintain, it should be taken in mind as important as implementing:

**Naming test case** The name of your test should consist of three parts:

- The name of the method being tested.
- The scenario under which it's being tested.
- The expected behavior when the scenario is invoked.

Examples:

- Bad naming: `Error 1`, `invalid input 1`, `test 1`
- Good naming: `Should returns same number WHEN input single number`, `Should returns 0 WHEN emtpy string`

**Table driven testing** A test can quickly become unreadable, repetitive, and overall annoying when the function you want to test is handling too many tasks, especially when there are many different cases you want to test, for example:

```go
package main

import (
   "github.com/stretchr/testify/assert"
   "testing"
)

func TestHadAGoodGame(t *testing.T) {
   tests := []struct {
      name     string
      stats   Stats
      goodGame bool
      wantErr  string
   }{
      {"sad path: invalid stats", Stats{Name: "Sam Cassell",
         Minutes: 34.1,
         Points: -19,
         Assists: 8,
         Turnovers: -4,
         Rebounds: 11,
         }, false, "stat lines cannot be negative",
      },
      {"happy path: good game", Stats{Name: "Dejounte Murray",
         Minutes: 34.1,
         Points: 19,
         Assists: 8,
         Turnovers: 4,
         Rebounds: 11,
      }, true, ""},
   }
   for _, tt := range tests {
      isAGoodGame, err := hadAGoodGame(tt.stats)
      if tt.wantErr != "" {
         assert.Contains(t, err.Error(), tt.wantErr)
      } else {
         assert.Equal(t, tt.goodGame, isAGoodGame)
      }
   }
}
```

### Use interfaces and avoid file I/O, API call

When writing tests, it's important to use interfaces and avoid file I/O and API calls wherever possible. You want your tests to be fast, independent, isolated, consistent, and not flaky. Here are some best practices to keep in mind:

**Use interfaces** By using interfaces, you can decouple your code from its dependencies, making it easier to test in isolation. Instead of calling concrete implementations, you can call interfaces that define the behavior you need, for example:

```go
type repository interface {
  GetRecipe(recipeID string) (domain.Recipe, error)
  CreateRecipe(recipe domain.Recipe) error
  UpdateRecipe(recipe domain.Recipe) error
}
```

**Mock dependencies** To test code that relies on external dependencies, use mocks to simulate the behavior of those dependencies. This approach allows you to test your code in isolation, without relying on external resources.

```go
import (
    "testing"

    "github.com/stretchr/testify/assert"
)

func Test_getFromDB(t *testing.T) {
    mockDB := NewMockDB(t)
		mockDB.On("GetFlavor").Return("Chocolate", nil)
    flavor := getFromDB(mockDB)
    assert.Equal(t, "chocolate", flavor)
}
```

**Avoid file I/O**

File I/O can be slow and unreliable, making it difficult to test code that relies on it. Instead, consider using interfaces to abstract away file I/O and using mocks to simulate file operations during testing.

```go
func Test_getFromDB(t *testing.T) {
		mockReader := reader.NewMock()
		mockReader.On("Read", mock.Anything).Return(100, nil)
    scanSvc := scan.NewInstance(mockReader)

		expectedSize := 100
    assert.Equal(t, scan.size(), expectedSize)
}
```

**Avoid API calls** Like file I/O, API calls can be slow and unreliable, making it difficult to test code that relies on them. Instead, consider using interfaces to abstract away API calls and using mocks to simulate API responses during testing.

```go
func Test_AcceptJobRequest(t *testing.T) {
		mockEmailGwy := new(email.MockGateway)
		mockEmailGwy.On("SendEmailWithTemplate", mock.Anything, mock.Anything).Return(nil)

		workerCtrl := worker.New(mockEmailGwy)
		err := workerCtrl.acceptAndNotify()
		// Some asserts here
}
```

### Covering edge cases and boundary conditions

As we all know, this is a basic test strategy, but this reveals most of the potential bugs. Because humans usually break the rule, and that would break the happy flow. It's important to cover edge cases and boundary conditions to ensure that your code can handle extreme or unexpected values. Here are some tips for covering edge cases and boundary conditions in your tests:

- **Test extreme values**: Be sure to test extreme values, such as the maximum and minimum values that your code can handle.
- **Test unexpected input**: Be sure to test any weird input values or characters that might look like it would affect the test.
- **Test corner cases**: Be sure to test corner cases, such as scenarios where multiple inputs or conditions intersect. This approach can help you catch issues with complex logic or interactions between different parts of your code.

## Test coverage

Test coverage is defined as a metric in Software Testing that measures the amount of testing performed by a set of tests. It will include gathering information about which parts of a program are executed when running the test suite to determine which branches of conditional statements have been taken.

In simple terms, it is a technique to ensure that your tests are testing your code or how much of your code you exercised by running the test.

- **Very poor: 0-20% coverage**. This means that very few or no unit tests have been written to test the code, which can result in bugs and errors going unnoticed.
- **Poor: 21-40% coverage**. This means that some unit tests have been written, but a significant amount of code remains untested.
- **Acceptable: 41-60% coverage**. This means that a reasonable number of unit tests have been written to test the code, but there is still room for improvement.
- **Good: 61-80% coverage**. This means that a large percentage of the code has been covered by unit tests, and most potential bugs and errors have been caught.
- **Very good: 81-100% coverage**. This means that almost all code has been covered by unit tests, and the likelihood of bugs and errors slipping through is very low. However, achieving 100% coverage may not always be practical or necessary, depending on the nature and complexity of the code.

Although it depends on the project's status, ideally, we recommend aiming for a test coverage between **61-80%**. However, don't become obsessed with the number; the primary goal is to write tests that help us catch bugs effectively.

## Tooling and library

Golang possesses a robust testing framework; however, employing supplementary tools can enhance the development experience and reduce code creation efforts.

- **Mocking**: Rather than creating mock code manually, consider utilizing a mocking library, such as gomock or mockery, which supports mocking and generates mocks from interfaces, thereby reducing time expenditure.
- **Assert**: The default Golang testing framework has limited assertion capabilities. Alternatively, tools like testify provide improved support and more user-friendly assertions.

## Conclusion

In this article, we've covered the basics of testing in Golang and explored some strategies for writing effective and maintainable tests. We've seen best practices for using interfaces, avoiding file I/O and API calls, automating unit tests, and covering edge cases and boundary conditions.

By following these strategies and best practices, you can write tests that are easier to run, understand, and maintain. By investing time in testing, you can catch bugs and errors earlier in the development process, which can save time and improve the overall quality of your code.

Remember, testing is not a one-time task but an ongoing process that should be integrated into your development workflow. With the right tools, frameworks, and mindset, testing can become a natural and valuable part of your development process, helping you build more reliable and maintainable software.

## References

[Overview of testing package in Golang](https://www.geeksforgeeks.org/overview-of-testing-package-in-golang/)

[5 tips for better unit testing in golang](https://blog.devgenius.io/5-tips-for-better-unit-testing-in-golang-b25f9e79885a)

[https://learn.microsoft.com/en-us/dotnet/core/testing/unit-testing-best-practices](https://learn.microsoft.com/en-us/dotnet/core/testing/unit-testing-best-practices)

[https://www.testim.io/blog/unit-testing-best-practices/](https://www.testim.io/blog/unit-testing-best-practices/)

[https://www.freecodecamp.org/news/a-beginners-guide-to-testing-implement-these-quick-checks-to-test-your-code-d50027ad5eed/](https://www.freecodecamp.org/news/a-beginners-guide-to-testing-implement-these-quick-checks-to-test-your-code-d50027ad5eed/)

[https://testing.googleblog.com/2020/08/code-coverage-best-practices.html](https://testing.googleblog.com/2020/08/code-coverage-best-practices.html)


An in-depth exploration of arrays and slices in Go, covering their differences, internal implementations, and key operations like append. Learn about fixed-length arrays, flexible slices, and how Go manages memory allocation for growing slices.