
In most software, monitoring is an accessory. For a system that moves money, like a crypto swap service, it's a core part of the engine. If your gauges are wrong, the engine is broken. When building the observability for the ICY Backend, our problem wasn't just to see if the server was 'up.' It was to build a nervous system for it, one that could feel its own state without revealing secrets that would be financially fatal.

## What not to measure

This led us to the central tension: we needed total observability but also near-total secrecy. Most monitoring thrives on detailed labels like user IDs. In crypto, a wallet address isn't just a label; it's a key. Exposing it in a dashboard would be like engraving your bank password on the outside of your house.

So our first principle was ruthless selectivity. Metric cardinality became a security feature, not just a technical one. The rule was absolute: no transaction hashes, no addresses, no amounts. Our metrics could show what operation failed, but never for whom or for how much.

## The shape of a request

So if we can’t use the most revealing labels, what is left to measure at the system’s front door, its HTTP API? The question becomes finding the most expressive yet safe dimensions of a request.

We found the answer in the three primary colors of web service observability: rate, errors, and duration. These tell you almost everything you need to know about the load on the system and its ability to cope. We captured them with a few fundamental metrics. A counter for total requests, a histogram for request duration, and a gauge for active requests.

The power, as always, was in the labels. We settled on three: the HTTP method, the endpoint template, and the resulting status code. This combination is powerful. It lets you ask questions like, "What is the 95th percentile latency for POST requests to /swaps that result in a 200 status?" without ever touching sensitive data.

![alt text](assets/icy-swap-http.png)

The key insight here was in the endpoint label. We couldn't use the raw request path, like `/api/v1/user/123/transactions`, because that would create a new metric series for every user, defeating our security goal. Instead, we instrumented the router to provide the normalized path template: `/api/v1/user/:id/transactions`. This small distinction is what makes high-utility HTTP metrics possible in a secure environment.

And the gauge for active requests turned out to be surprisingly insightful. While rate and duration tell you what has already happened, the number of active requests tells you about pressure in the system right now. If it starts climbing while the request rate stays flat, you know something is slowing down. It’s an early warning sign of saturation, a leading indicator of trouble.

## What is health?

The next question was, how do you know if the system is truly healthy? A simple /healthz endpoint is trivial; it's like checking for a pulse. It confirms the system is alive, but not that it can do any real work.

So we built a richer set of probes, a form of synthetic monitoring designed for an external service like Uptime Robot to watch. Instead of one status, our dashboard shows several vital signs. The first is the simple pulse check (/healthz). We then added another, `/api/v1/health/db`, to ask a more meaningful question: "Can you talk to your database?"

The trickiest part is handling unreliable external APIs. Treating a failure from the Bitcoin network like a local one would cause unnecessary downtime.

This is where the circuit breaker pattern is critical. It gracefully isolates external failures, preventing them from taking down our whole system. Our health check for these services, `/api/v1/health/external`, uses this logic. If a circuit is open, it reports a "degraded" status, not "unhealthy."

![alt text](assets/icy-swap-healthz.png)

This gives our Uptime Robot dashboard a richer vocabulary. It’s no longer just green or red, but also has a yellow light for when the system is "wounded, but alive", giving a much more accurate picture of its state.

## Gauges on the outside world

But these health checks, these green and yellow lights, are just a summary. They tell you if something is wrong, but not how wrong. A service being "degraded" is useful information, but is it slow? Is it erroring out? Is the circuit breaker about to trip again?

To answer these questions, you need quantitative data. This is where we go beyond the simple status check and measure the performance of every single call to an external service. We created a standard set of Prometheus metrics for this purpose. A histogram, `icy_backend_external_api_duration_seconds`, to track latency. A counter, `icy_backend_external_api_calls_total`, to track the rate of calls and their success or failure status. And most importantly, a gauge, `icy_backend_circuit_breaker_state`, that explicitly reports whether each circuit is closed (1), open (0), or half-open (0.5).

![alt text](assets/icy-swap-metrics-external.png)

This is what we plot in Grafana. It gives us a high-fidelity view of our dependencies. We can see the latency to the Bitcoin API begin to creep up minutes before our circuit breaker trips. We can overlay our application's error rate with the external API's error rate and see a direct correlation. These graphs tell a story. They don't just tell us that a service is degraded; they show us the precise shape of its degradation. This is the difference between knowing a storm is coming and having a weather radar to track its every move.

## The silent workers

The most subtle layer of health, however, was in the background. A great deal of the work in a crypto system, such as indexing new transactions and processing swaps, happens in cron jobs. These are silent workers. They can fail, or worse, become stuck in an infinite loop, consuming resources without anyone noticing until it’s too late. How do you monitor something that has no user-facing request?

![alt text](assets/icy-swap-job-metrics.png)

Our solution was a thread-safe manager that every job had to check in with. When a job started, it registered itself. When it finished, it reported its status as either success or failure. We built a watchdog to detect jobs running for an unusually long time, for example more than 15 minutes for a swap process, and flag them as “stalled.” This brought our background processes, the most hidden part of the machine, into the light.

## The cost of watching

Of course, all this watching comes at a cost. Every check, every metric, every log adds a tiny bit of overhead. In a high-frequency financial system, nanoseconds matter. We set ourselves an almost absurdly low budget for the overhead of our main HTTP monitoring middleware: less than 1 millisecond per request.

Achieving this felt like tuning a race car engine. We pre-computed metric labels at startup so we weren't doing string manipulation on every request. We were careful about memory allocations. The final result was an overhead of around 493 nanoseconds per request. This number wasn't just a performance metric; it was proof that observability didn't have to come at the expense of speed. We could have our microscope without slowing down the patient.

## What we learned

What we built feels less like a collection of tools and more like a coherent system. We learned that security must be designed in from the start, not sanitized later. That "health" is not one question, but a series of layered ones. And that you must assume the world will fail, and build mechanisms like circuit breakers to survive.

The work isn't finished. The next frontier is moving from passive observation to active response, like distributed tracing or automated recovery. But what we've built is a solid foundation: an engine where the gauges are part of the design, not just bolted on.


We built a monitoring system for a cryptocurrency backend that provides deep observability while protecting sensitive financial data through layered health checks and resilient, security-first architecture.

Monitoring the ICY Swap backend


## Thinking in teams

I recently got a request to "track user visits to more pages," which sounded simple. But it unraveled into questions about scope, performance, and privacy that most AI coding assistants can't handle. They're great for writing a single function, but ask for a full feature, and they start losing context.

My own attempts felt like this. Using a single AI was like having a distractible intern doing everything from architecture to QA. It was clear I was on a path to building the wrong solution. The problem wasn't the AI's capability, but my approach. I was asking a soloist to perform a symphony.

It finally clicked for me that you can’t expect one person to do the work of an entire engineering department. You have to build a team. That sparked a thought: what if AI was organized as a group of specialists? One agent could handle research, another could focus on planning, and another on testing. It reflects how we approach building software, and it seemed like a promising direction to take.

## The Five-phase flow

This led me to design a workflow with five distinct agent roles, organized into a five-phase process. The key idea is that the agents don't all talk to each other at once. That would be chaos. Instead, they work sequentially, each producing documentation that becomes the input for the next. The system is managed by a master orchestrator that handles the handoffs, much like a project manager ensuring each person has what they need to start their work.

It looks something like this:

1. **Analyze & Research:** A Researcher agent digs into the problem space.
2. **Planning:** A Project Manager agent creates architectural plans and specifications.
3. **Test Case Design:** A Test Case Designer defines how we'll know if the solution works, before a line of code is written.
4. **Implementation:** A Feature Implementer writes the code to pass those tests.
5. **Quality Assurance:** A QA Engineer validates the whole thing against the original requirements.

Before any of this happens, though, there's a critical pre-phase. The master orchestrator has a detailed conversation with the user to clarify the requirements. This isn't just about getting a yes or no; it's an exploration that often uncovers hidden assumptions. The goal is to turn a vague request into a concrete, documented plan.

## How it works in practice

Let's return to that "track more page visits" request.

My initial prompt was, "I want to track user visits to more pages. Currently, we only track when users go to the home page."

A simple AI might have just started spitting out code. Instead, the master orchestrator analyzed the existing codebase and came back with questions. It had already figured out I only tracked one visit per session and laid out four possible interpretations of my request, from simply tracking more pages to adding deep engagement metrics. It recommended Option 1—tracking individual page navigations—and asked clarifying questions about API calls, privacy, and the admin dashboard.

![alt text](assets/coding-agent-team-0.png)

This initial back-and-forth was transformative. My vague idea became a concrete plan. I decided to track every page navigation, make a single API call per page to avoid spamming the server, and enhance the admin dashboard. The orchestrator documented this in a file, `final-requirements.md`, complete with timestamps and unique IDs for each requirement. This document became the source of truth for the entire project.

With the requirements locked in, the team got to work.

The **Researcher** spent four minutes and used 14 tool calls to analyze my routing architecture and research best practices for the specific stack.

The **Project Manager** then created Architecture Decision Records (ADRs) and detailed specifications. It broke the work down into route-level tracking, session management, and dashboard components, all tracing back to the IDs in the requirements file.

Next, the **Test Case Designer** spent fourteen minutes and 35 tool calls creating a comprehensive suite of tests. There were unit tests for the tracking logic, integration tests for the analytics service, and end-to-end tests for user journeys. This felt like a lot of work upfront, but it was really just diligence.

Only then did the **Feature Implementer** start writing code. It built the TypeScript interfaces, the tracking middleware, and the new dashboard components, with the explicit goal of making all the tests pass.

![alt text](assets/coding-agent-team-1.png)

Finally, the **QA Engineer** validated the whole implementation. It checked not only that the code worked but that it correctly fulfilled the original requirements from `final-requirements.md`. Did it track *all* page visits? Did it avoid duplicate API calls? Did the dashboard show meaningful insights? This phase wasn't just about finding bugs; it was about ensuring I had built what I'd set out to build.

![alt text](assets/coding-agent-team-2.png)

The result was a production-ready analytics system, complete with tests and documentation that explained not just what was built, but why.

## What I learned

The difference was noticeable. I saw fewer post-deployment bugs, especially for complex features. The documentation became radically better because every decision was captured as it was made. And onboarding new engineers became easier because they could read the session logs and understand the thinking behind a feature. The "works on my machine" problem for complex setups virtually disappeared.

But the biggest change wasn't a number on a chart. It was a feeling of predictability. Complex features no longer felt like a gamble.

If you wanted to build something like this, the lessons I learned are straightforward.

**First, think in terms of roles, not just prompts.** Start with three: a Researcher, a Planner, and an Implementer. You can add more specialized roles as you find you need them.

**Second, make documentation the centerpiece of the workflow.** Every task should start with a timestamped directory. The structure I use looks like this:

```text
docs/sessions/YYYY-MM-DD-HHMM/
├── requirements/
├── research/
├── planning/
├── test-cases/
└── implementation/
```

The output of one agent in its directory becomes the input for the next. This creates an audit trail that is invaluable. Each phase ends by creating a `STATUS.md` file, which is a narrative of what was done, what decisions were made, and why, all linked back to the original requirements. It’s a quality gate, not just a checkbox.

The most surprising thing is that this system feels less like operating a machine and more like managing a very efficient team. The documentation reads like a series of well-documented conversations and handoffs.

## Realistic expectations

This workflow isn’t magic. It doesn't produce a perfect, finished feature in a single pass. What it does produce is a very strong first draft—maybe 50-70% of the way to the final solution. From there, a human developer needs to step in, assess what's missing, and perhaps run the process again on smaller, more targeted tasks.

And maybe that's the right model. The goal isn't to replace human developers, but to give them a much better starting point. I've spent so much of my time on the scaffolding of software development—understanding requirements, setting up tests, writing boilerplate. The agent team automates much of that, leaving me to focus on the hard parts that require real judgment and creativity.

It turns out that a simple request to track page visits led me to a new way of thinking about building software with AI. The answer wasn't a better AI, but a better process. By structuring the work like a human team, I didn't just get better code; I got a system that documents its own thinking. And in the long run, that might be the most valuable thing it builds.


I've been experimenting with making AI assistants work not as a single tool, but as a specialized team. It seems to work surprisingly well.

The Coding Agent Team


To ensure the operational reliability of our **memo.d.foundation**, we have developed a comprehensive, dual-mode monitoring system that combines proactive **synthetic monitoring** with reactive **instrumental alerts**. This integrated strategy provides complete visibility into platform health, from business metrics to real-time system events.

## Key benefits of our dual-mode approach

Our hybrid monitoring strategy delivers significant operational advantages by addressing the limitations of traditional single-method systems:

- **Proactive and reactive coverage**: We combine scheduled "scout" checks for landscape-level health with event-driven "messenger" alerts for immediate issue notification, eliminating visibility gaps.
- **Early issue detection**: Synthetic monitoring identifies potential data quality or pipeline bottlenecks before they impact user experience or downstream processes.
- **Real-Time event awareness**: Instrumental monitoring provides immediate feedback on critical events, such as NFT mints and CI/CD pipeline status, enabling rapid engineering response.
- **End-to-end data integrity**: Our checks span the entire data lifecycle, from Markdown content ingestion and **Parquet** file health to database consistency and AI embedding generation.

![memo-monitoring](assets/memo-monitoring.webp)

### Proactive health audits: Synthetic monitoring

Our synthetic monitoring platform operates on a fixed schedule, executing a series of checks against our core data assets and business logic. It serves as our early warning system, providing a regular pulse on the health of the entire **memo.d.foundation** system and catching issues before they escalate.

#### NFT system reporting (`memo-nft-report.ts`)

This daily report provides a comprehensive overview of our minting pipeline and NFT collection activity. It leverages **DuckDB** to execute federated queries across both our canonical **Parquet** vault and our production **PostgreSQL** database, unifying disparate data sources into a single, coherent view.

**Federated data integration:**

The system joins data from a remote Parquet file and a PostgreSQL table to generate a holistic report.

```typescript
// Dual-source data integration with DuckDB
const vaultMetrics = await connection.runAndReadAll(`
  SELECT
    COUNT(CASE WHEN should_mint = true THEN 1 END) as mintable_total,
    COUNT(CASE WHEN should_mint = true AND minted_at IS NOT NULL THEN 1 END) as minted_count
  FROM read_parquet('https://memo.d.foundation/db/vault.parquet')
`);

const collectionMetrics = await connection.runAndReadAll(`
  SELECT
    COUNT(*) as total_events,
    SUM(amount) as total_collected
  FROM memo_nft_db.memo_nft.memo_minted_events
`);
```

This daily audit, running via a `cron` schedule in **GitHub Actions**, tracks key performance indicators, including:

- **Minting pipeline health**: Monitors success rates, queue depth, and pending content.
- **Collection activity**: Analyzes total collection events, revenue, and unique collectors.
- **Author and content Trends**: Identifies top contributors and popular content tags.

#### Data vault integrity (`monitor-vault-parquet.ts`)

This check is designed to verify the completeness and quality of our core knowledge vault. It provides a quick, color-coded health status that allows us to assess data integrity at a glance.

**Data quality metrics:**

Our script assesses the vault based on a predefined set of quality metrics.

```typescript
interface VaultMetrics {
  totalRecords: number;
  missingDates: number;
  missingAuthors: number;
  missingEmbeddings: number;
  pendingMint: number;
  pendingArweave: number;
}
```

![memo-monitoring-1](assets/memo-monitoring-1.png)

**Health status logic:**

The system uses tiered thresholds to classify the vault's health, ensuring we prioritize critical issues effectively.

```typescript
const hasWarnings = metrics.missingDatesPercent > 25 || metrics.missingAuthorsPercent > 40;
const hasCritical = metrics.pendingMint > 20 || metrics.missingEmbeddings > 100;
const healthStatus = hasCritical ? '🔴 Critical' : hasWarnings ? '🟡 Warning' : '🟢 Healthy';
```

This provides **actionable insights** by immediately flagging which specific metrics have crossed their warning or critical thresholds, enabling focused remediation.

### Real-time event notifications: Instrumental monitoring

Where synthetic monitoring is proactive, our instrumental monitoring is reactive. It operates across several integration layers to provide immediate notifications based on specific system events.

#### NFT minting alerts

When a new article is successfully minted as an NFT, our `notify-discord-minted-articles.ts` script is triggered. This process handles community communication by sending a rich, formatted Discord notification. We leverage the **Model Context Protocol (MCP)** to generate context-aware messages that are more informative and engaging than standard text.

**MCP Integration for rich notifications:**

```typescript
await mcpDiscord.callTool({
  name: "discord-send-embed",
  arguments: {
    username: "Memo NFT",
    webhookUrl: process.env.DISCORD_WEBHOOK_URL,
    content: generatedMessage,
    title: "📢 New notes minted in the Memo",
    autoFormat: true,
  }
});
```

This system ensures both optimal performance and high reliability through:

- **Connection resilience**: Implements a retry mechanism for establishing the initial connection.
- **Exponential backoff**: If a notification fails, the system waits progressively longer before retrying, preventing API rate-limiting issues.
- **Graceful error handling**: Failures are logged without crashing the parent workflow.

![memo-monitoring-2](assets/memo-monitoring-2.png)

#### CI/CD workflow monitoring

We have integrated instrumental monitoring directly into our **GitHub Actions** workflows. This provides real-time status updates on deployments and other critical automated processes, sending success or failure notifications to a dedicated Discord channel.

**Standardized notification steps:**

```yaml
- name: Notify Discord on Success
  if: success()
  uses: sarisia/actions-status-discord@v1
  with:
    webhook: ${{ secrets.DISCORD_WEBHOOK_URL }}
    title: '✅ Deployment Completed Successfully'
    color: 0x00ff00

- name: Notify Discord on Failure
  if: failure()
  uses: sarisia/actions-status-discord@v1
  with:
    webhook: ${{ secrets.DISCORD_WEBHOOK_URL }}
    title: '❌ Deployment Failed'
    description: 'Please check workflow logs for details.'
    color: 0xff0000
```

### System architecture and implementation

Our monitoring platform is built on a modern, flexible technology stack chosen for its performance, ease of integration, and alignment with our team's expertise.

#### The DuckDB analytics engine

We selected **DuckDB** as our core data processing engine due to its unique capabilities for in-process analytics. It allows us to perform complex analytical queries on local and remote data sources without the overhead of a traditional data warehouse.

**Key advantages:**

- **Federated queries**: Natively queries remote Parquet files and connects to our live **PostgreSQL** database in a single session.
- **High performance**: Columnar-vectorized query execution is highly optimized for analytical workloads.
- **Zero-dependency**: Runs entirely in-memory, simplifying our CI/CD environment setup.

**PostgreSQL integration:**
The connection process involves loading the `postgres` extension and attaching our remote database as a read-only source.

```typescript
async function setupDuckDBConnections(): Promise<DuckDBConnection> {
  const instance = await DuckDBInstance.create(':memory:');
  const connection = await instance.connect();

  // Load extensions and attach remote database
  await connection.runAndReadAll('LOAD postgres;');
  await connection.runAndReadAll(`
    ATTACH '${process.env.DB_CONNECTION_STRING}' AS memo_nft_db (TYPE postgres, READ_ONLY);
  `);

  return connection;
}
```

### Security and data handling

We built our monitoring system with security and responsible data handling as core principles:

- **Secure credentials management**: All credentials, such as API keys and database strings, are managed as **GitHub Secrets**. Access is limited by the **principle of least privilege**, and our services connect using a dedicated **read-only** database user to prevent accidental data modification.
- **Privacy by design**: We practice **data minimization** in all public-facing reports to protect community privacy. For instance, collector wallet addresses are truncated to show activity trends without revealing full identities.

  ```typescript
  // Address truncation for privacy
  const truncatedAddress = String(row.address || '').substring(0, 8) + '...';
  ```

### Success metrics

We track a set of key performance indicators to measure the effectiveness of our monitoring infrastructure:

- **Monitoring with Uptime**: Targeting 99.9% uptime for all monitoring services, ensuring continuous visibility.
- **Alert latency**: Average delivery time for critical alerts is under 2 minutes.
- **Data freshness**: Reports leverage data that is never more than 6 hours old.
- **Query performance**: Complex analytical reports complete in under 30 seconds.

### What's next?

While the current work provides a solid foundation, we have several enhancements planned to further improve our monitoring capabilities:

- **Automated remediation**: For common, well-understood issues, we will explore self-healing capabilities to reduce manual intervention.
- **Executive dashboards**: We will create higher-level dashboards that distill technical metrics into clear business insights for leadership.


Learn about our dual approach to monitoring at memo.d.foundation, combining synthetic and instrumental monitoring to ensure system health and community engagement.

Monitoring


Think of a single **Model Context Protocol (MCP)** server, like the Anthropic memory example or potentially our own knowledge base interface, as a specialized tool provider. It's essentially a defined protocol endpoint that exposes a set of functions – **tools** – that an **LLM** can call.

![](assets/multi-mcp-data-integration-0.png)

Now, **Multi-MCP data integration** is simply about connecting a primary system – like our evolving knowledge base or a central coordinating **LLM** – to _multiple_ of these specialized MCP servers simultaneously.

What's the result? You dramatically expand the **repertoire** of tools available to the central **LLM**. Instead of just the tools provided by one server, it gains access to the combined capabilities of all connected servers. This works cleanly because the tools are inherently **namespaced**, either by the protocol itself or by the integration layer.

The **LLM** wouldn't just see multiple `search_nodes` tools; it would see `knowledgeBaseServer.search_nodes`, `realtimeSensorServer.search_nodes`, etc. This behind-the-scenes namespacing that exists for MCP servers prevents collisions and provides clarity on which capability belongs to which subsystem.

![](assets/multi-mcp-data-integration-1.png)

It's analogous to microservices or API aggregation. You don't build one monolithic program to do everything; you build specialized services and orchestrate them. Multi-MCP integration allows an LLM to orchestrate across diverse, specialized data sources and functional capabilities exposed via this common protocol.

## How does this feed our knowledge base "brain"?

Simple. When the central system uses a tool from an integrated MCP server – say, fetching sensor data using realtimeSensorServer.get_temperature or retrieving discussion points using meetingNotesServer.get_summary – the retrieved data can be formatted and ingested as a new observation into our observation_log. The payload's source field naturally accommodates this, indicating which MCP server originated the data.

```json
// Example: Observation ingested from an external 'SensorMCP'
{
  "timestamp": "2025-04-17T08:30:00Z",
  "payload": {
    "context_id": "factory-floor:sensor-1a:reading-5987",
    "content": "Temperature reading obtained from Sensor MCP.",
    "entities": [
      { "name": "Sensor-1A", "type": "device" },
      { "name": "Temperature", "type": "measurement" }
    ],
    "relations": [
      {
        "from": "Sensor-1A",
        "to": "Temperature",
        "type": "measures",
        "value": 35.5,
        "unit": "C"
      }
    ],
    "coined_terms": [],
    "source": {
      "source_type": "mcp_server",
      "source_identifier": "SensorMCP/get_temperature", // Namespace indication
      "ingestion_timestamp": "2025-04-17T08:30:05Z"
    },
    "tags": ["iot", "sensor", "temperature", "factory"]
  },
  "operation": "insert",
  "confidence": 0.95
  // ... processed_timestamp ...
}
```

So, multi-MCP integration turns our knowledge base into a central nexus. It doesn't just learn from one direct feed; it learns from the aggregated capabilities and data streams provided by a whole network of specialized MCP servers, allowing it to build a much broader and more interconnected understanding of its environment. It's about leveraging distributed function calling to fuel centralized knowledge synthesis.

This theoretical framework of Multi-MCP integration isn't just an abstract concept; it's the direct architectural solution we implemented to tackle our persistent challenge of fragmented data. Recognizing that our valuable knowledge was siloed across various applications, we applied the Multi-MCP strategy to build a central nervous system for our information.

## The Brain DB: centralizing data for actionable intelligence

Our important data was often spread out, some even locked away in application databases like ConsoleLabs, making it difficult to see the whole picture. To fix this, we built a central place to gather everything, called a Knowledge Hub (or Brain DB). We use specialized services that follow a shared set of rules (the Model Context Protocol or MCP), ensuring they can all talk to each other and share data consistently. For example, one service, MCP ConsoleLabs, follows these rules to safely retrieve data from the ConsoleLabs application. All the information gathered by these services flows into our main Knowledge Hub (using TimescaleDB), giving us one place to see, analyze, and learn from our combined data over time.

![](assets/multi-mcp-data-integration-2.png)

Let's walk through a practical example: analyzing ICY token activity. First, the need arose for a comprehensive report covering a specific period. Using MCP ConsoleLabs, we directly queried the relevant application database to retrieve the raw transaction details. This raw data, potentially enriched with other context, was then processed — by MCP Client (Claude or agents) —transforming basic logs into a structured summary highlighting key entities, relationships, volumes, and patterns.

![](assets/multi-mcp-data-integration-3.png)

Finally, this structured analysis, formatted as JSON, was inserted into our Knowledge Hub's observation_log table using the duckdb_insert tool. During insertion, the MCP client will self-assess its confidence (e.g., 0.92), reflecting its evaluation of the data’s completeness and clarity, using a query similar to the following:

![](assets/multi-mcp-data-integration-4.png)

As a result, the Knowledge Hub now holds a verifiable, structured record of the ICY token analysis for that period, tagged with crucial metadata like the processing timestamp and our confidence level.

Simply collecting knowledge is just the first step; the real value unfolds as we watch this information over time. This helps us move beyond just knowing *what* happened to understanding *why* it happened and deciding *what to do next*. Because our Knowledge Hub uses TimescaleDB to record everything chronologically, we can easily spot trends, like changes in ICY token activity month over month. We can also connect the dots, linking spikes in transactions to specific events like marketing campaigns, or noticing unusual patterns that might signal opportunities or risks.


Multi-MCP data integration - receiving data from other MCP servers

## Introduction

System prompts are the hidden architects of AI interactions. They silently guide AI assistants like Claude to produce consistent, useful responses tailored to specific use cases. But crafting an effective system prompt is both art and science - especially for complex technical scenarios.

In this article, I'll walk through a real process of developing a system prompt for a multi-database analytics use case, drawing from an actual conversation where we identified requirements, tested approaches, and refined instructions to create a specialized database assistant.

To access multiple databases, we needed an MCP server to provide tools that will be using `duckdb` to connect databases. This setup allowed us to merge tables from different databases and perform complex queries.

![struggle-exploring](assets/build-a-system-prompt-0.png)

## Understanding the use case: multi-database analysis

Our journey began with a specific need: creating a system prompt that would enable Claude to effectively explore and query information across multiple databases. The challenge was significant as the assistant needed to add the MCP server to Claude's configuration, discover available databases, identify and understand tables within those databases, connect information across different data sources, and present findings in a useful format.

This represents a common pattern in enterprise environments where data exists in silos, but insights require connecting information across those boundaries.

## Discovery

### Initial exploration

Before diving into the system prompt, we needed to understand Claude's capabilities and limitations. This involved a series of exploratory queries to see how well Claude could interact with the databases.

We started with a simple question: "How many ICY token are transferred in this week?" This query was intended to test Claude's ability to navigate the databases and provide relevant information.

In the beginning, Claude struggled to find the right information. The initial attempts resulted in errors and confusion, Claude will attempt to use different queries and approaches, or in worst cases, it would need more guidance on how to access and query the databases effectively.

![struggle-exploring](assets/build-a-system-prompt-1.png)

### Testing basic database interactions

Claude's first attempt to understand its database interaction capabilities:

```sql
SELECT datname FROM pg_catalog.pg_database;
```

This initial query revealed the available databases in the system, which included specialized databases like `pay_db`, `profile_db`, and `central_db`.

### Exploring database structures

The next step involved exploring the structure of the databases. Claude understood that it needed to identify the tables within each database to find relevant information:

```sql
SELECT tablename FROM pg_catalog.pg_tables WHERE schemaname = 'public';
```

This revealed dozens of tables across different databases. We discovered that similar entities (like "tokens" and "profiles") existed in multiple databases, suggesting potential relationship points.

So it then examined specific table structures:

```sql
SELECT column_name, data_type 
FROM information_schema.columns 
WHERE table_schema = 'public' AND table_name = 'tokens' 
ORDER BY ordinal_position;
```

### Testing cross-database queries

A critical moment came when Claude tested cross-database queries. It encountered an error trying to directly access the `public.tokens` table without specifying the database:

```sh
Error: Table with name tokens does not exist!
Did you mean "pay_db.public.tokens or central_db.public.tokens"?
```

This error revealed that Claude needed to use fully qualified table names when accessing tables across databases, a crucial insight for the system prompt.

### Practical use case testing

So we asked Claude to move from exploration to practical application by testing a real-world query:

```sql
SELECT gct.*, t.symbol, t.name 
FROM central_db.public.guild_config_community_tokens gct
JOIN pay_db.public.tokens t ON gct.mochi_token_id = t.id
WHERE t.symbol = 'ICY';
```

This successful cross-database query demonstrated that with proper guidance, Claude could indeed connect information across databases - in this case, linking community tokens from one database with token definitions from another.

## Crafting the system prompt

What made this system prompt development process unique was that it emerged from actual problem-solving rather than theoretical planning. Here's how the process unfolded:

### Self-Exploration and learning through errors

When initially asked about ICY token emissions, Claude didn't have a predefined approach. Instead, it began by:

1. **Exploring blindly**: The first query to find ICY tokens led to an error because Claude didn't specify the database:

   ```sh
   Error: Catalog Error: Table with name tokens does not exist!
   Did you mean "pay_db.public.tokens or central_db.public.tokens"?
   ```

2. **Learning from failures**: This error actually provided valuable information - it revealed that "tokens" existed in multiple databases and required qualification.

3. **Iterative correction**: So Claude adjusted its approach:

   ```sql
   SELECT * FROM pay_db.public.tokens WHERE symbol = 'ICY' OR name LIKE '%ICY%' LIMIT 1;
   ```

   This worked and revealed the token information.

4. **Building complexity gradually**: After finding the token, Claude needed to locate transactions. It discovered through trial and error that the token was part of a community token system with relations spanning multiple databases.

5. **Cross-database discovery**: Through incremental exploration, Claude discovered how the `community_token_transactions` table in the `central_db` database related to tokens in the `pay_db` database.

6. **Solving real problems**: Eventually, Claude constructed a cross-database query that successfully pulled ICY token emission data:

   ```sql
   SELECT category, SUM(CAST(amount AS DECIMAL)) as total_amount
   FROM central_db.public.community_token_transactions
   WHERE community_token_id = '9232d25e-b7c6-ad4b-d3d8-14becc6deb58'
   AND created_at >= CURRENT_DATE - INTERVAL 7 DAY
   AND source = 'automation'
   GROUP BY category
   ORDER BY total_amount DESC;
   ```

### From experience to instruction

After achieving correct results through iterative exploration, I asked Claude to generate a system prompt based on the previous steps and information gathered. This request marked a critical transition from practical problem-solving to creating reusable guidance.

"After confirming that you could successfully query and connect data across multiple databases, please generate a system prompt that would capture this process systematically," I explained. "The goal was to transform the ad-hoc exploration we had just witnessed into a repeatable methodology that could be applied to similar multi-database scenarios."

This request initiated the crystallization of practical experience into formal instruction. Starting with the database discovery techniques that had worked in practice, Claude developed a comprehensive framework that covered the sequence of queries that successfully identified available databases and their contents, techniques for preventing the catalog errors encountered during our session, templates for the successful joins across different databases, and structured methods for organizing complex multi-database results.

![generate-system-prompt](assets/build-a-system-prompt-2.gif)

The resulting system prompt wasn't theoretical - it was anchored in the actual steps that had successfully answered complex queries spanning multiple data sources. Each component was directly informed by our experience working through real database challenges.

By asking for this formalization, I effectively transformed a single successful interaction into a reusable template that captured both the technical approach and the strategic thinking behind multi-database exploration.

## Testing the system prompt

After creating our system prompt, we tested it with a practical query: "Which guild using vault?", Claude followed our exploration protocol to discover relationships between guilds and vaults, then provided comprehensive information about the guild, its vault configuration, and transaction patterns.

![test-system-prompt](assets/build-a-system-prompt-3.gif)

## Key takeaways for effective system prompts

From our process, several principles emerged for creating effective system prompts:

1. **Let the AI learn by doing**: Allow the AI to explore and solve real problems before creating the system prompt
2. **Build from real experiences**: Base your prompt on actual successful problem-solving patterns
3. **Document both successes and failures**: Include guidance based on errors encountered and how they were overcome
4. **Provide structured protocols**: Formalize the successful exploration paths into step-by-step instructions
5. **Crystallize best practices**: Extract patterns from successful interactions that can be reused
6. **Test in real scenarios**: Validate the system prompt against actual use cases


Building the use case: how to use Claude to build the system prompt

How to use Claude to build the system prompt


Ensuring services are up and running is crucial. But how do you monitor _everything_, including internal tools and sensitive APIs, without exposing them to the world? This is the story of how we adopted Upptime, leveraging the power of GitHub Actions and Secrets to achieve comprehensive and secure uptime monitoring.

## Monitor public and private services

We needed a reliable way to monitor the uptime and performance of all our services, which included both public-facing services and internal services with sensitive endpoints. For public-facing services, such as our website and public APIs, transparency was crucial. Users rely on knowing if there's a disruption. On the other hand, internal services and sensitive endpoints, which include tools used by our team or APIs that should remain inaccessible to the public, had to be handled more discreetly. Directly exposing their status endpoints could lead to security vulnerabilities or invite unwanted attention.

**Why hide certain endpoints?**

- **Security:** Many internal endpoints are not designed for public exposure. Hiding them reduces the potential attack surface.
- **Privacy:** Some endpoints might reveal internal infrastructure details.
- **Preventing noise:** Keeping internal endpoints out of public configuration prevents automated scanners and bots from hitting them unnecessarily.
- **Complexity:** Some internal checks might require specific headers or authentication tokens that are best kept secret.

We needed a solution that could handle both scenarios: transparent monitoring for public services and secure, hidden monitoring for private ones.

## Using Upptime as a monitoring tool

We found our answer in [Upptime](https://upptime.js.org). It's an open-source uptime monitor and status page powered entirely by GitHub Actions, Issues, and Pages. The GitOps approach allows configuration to live in a Git repository, making changes trackable and collaborative. It's cost-effective, as it runs primarily on free GitHub Actions tiers, although we use self-hosted runners for more control. Automation is another benefit, with checks running automatically on a schedule. Additionally, transparency is enhanced as it generates a static status page easily deployable via GitHub Pages. Lastly, it excels in secret management by integrating seamlessly with GitHub Secrets.

## Configure `.upptimerc.yml` file

The heart of our Upptime setup is the `.upptimerc.yml` file in our `dwarvesf/upptime` repository:

```yaml
# Change these first
owner: dwarvesf # Our GitHub organization
repo: upptime # The repository hosting Upptime
user-agent: lmquang # A custom user agent for checks
runner: self-hosted # We use our own runners for reliability

# Add your sites here
sites:
  # Publicly visible services - URL is directly in the config
  - name: Public API
    url: https://public-api.domain./healthz

  # Sensitive internal services - URL is stored securely
  - name: My Secret API
    url: ${{ secrets.SECRET_API_URL }} # Magic! Reads from GitHub Secrets
  - name: Another Internal Tool
    url: ${{ secrets.INTERNAL_TOOL_HEALTH }}

assignees: # Assign issues to these folks on downtime
  - lmquang

status-website:
  publish: true # Yes, publish the status page
  # Custom domain pointing to the GitHub Pages site
  cname: status.d.foundation
  # Branding and messaging
  favicon: https:/storage.host/uploads/-/system/appearance/favicon/1/LogoD_1024.png
  logoUrl: https://storage.host/company-logo/32c5b772aec460924dbe0d60ce73f1c6.png
  name: Dwarves Foundation Status
  introMessage: This is the status page which uses **real-time** data from [Dwarves Foundation](https://dwarves.foundation) services. Internal services are monitored but not listed here.
  # navbar: ... (optional custom links)

i18n:
  footer: Powered by [Upptime](https://upptime.js.org)
# See https://upptime.js.org/docs/configuration for more options
```

**The Key:** Notice how `My Secret API` uses `url: ${{ secrets.SECRET_API_URL }}`. When the GitHub Actions workflow runs, it securely injects the actual URL from the repository's secrets settings. The sensitive URL _never_ appears in the public configuration file.

## How GitHub Actions automation actually works

Upptime relies on a set of workflows defined in `.github/workflows/`:

1. **`uptime.yml` (Runs every 5 mins):** This is the core checker. It fetches the site list from `.upptimerc.yml`, securely resolving any `${{ secrets.* }}` variables. It pings each URL, records the status (up/down) and response time, and commits this data to the `history/` directory. If a site is down, it automatically creates a GitHub Issue and assigns it.
2. **`response-time.yml` & `summary.yml` (Run daily):** These workflows process the raw data in `history/`, calculating historical performance metrics and generating summary files (like `history/summary.json`). They also update the status badges in the `README.md`.
3. **`site.yml` (Runs daily):** This workflow takes the processed data and builds the static HTML/CSS/JS status website. It then deploys this website to the `gh-pages` branch, making it live on `status.d.foundation`.

![service_monitoring_with_upptime](assets/service_monitoring_with_upptime.png)

## A transparent (and secure) status page

Upon visiting `status.d.foundation`, your browser retrieves the static website constructed using `site.yml` from the `gh-pages` branch. The JavaScript code on the page directly fetches public status data, including `summary.json` and recent history, from the `dwarvesf/upptime` repository via GitHub’s raw file access or API. Subsequently, the page dynamically displays the status of our publicly available services.

Crucially, the status page _only_ displays information about the services configured with public URLs in `.upptimerc.yml`. The sensitive endpoints, while monitored constantly by the `uptime.yml` workflow using secrets, are never exposed on the public status page or in the repository's version history.

This setup gives us the best of both worlds: transparent, real-time status updates for our public-facing services, and secure, automated monitoring for our internal infrastructure, all managed through a simple, code-based system.


Discover how Dwarves Foundation uses Upptime and GitHub Actions for transparent public uptime monitoring while securely keeping tabs on internal services.

Secure and transparent uptime monitoring with Upptime and GitHub secrets


Swapping ICY tokens for Bitcoin means exchanging one type of digital currency for another across different blockchain systems. Since ICY tokens (on the Base chain) and Bitcoin (on its own blockchain) operate on incompatible networks, specific tools are needed to make this process work. Below, I’ll explain the tools, why a direct swap isn’t possible, how the swap happens, and how the price is determined.

## Tools used in the swap

Swap Contracts: Automated programs on the Base chain that securely manage the swap process.
Treasury Wallets: Digital wallets that hold ICY tokens and Bitcoin during the exchange.
Icy-Backend: A system that receives your swap request, tracks it, and triggers the Bitcoin transfer.

## Why it’s not a direct swap

A direct swap isn’t possible because ICY tokens and Bitcoin use different blockchains. The Base chain is a modern system with advanced features, while Bitcoin’s blockchain is older and more limited. These differences prevent direct transfers, so tools like swap contracts and oracles are used to bridge the gap.

## How the swap works

Swapping ICY tokens for Bitcoin is a straightforward process that combines user actions, system automation, and secure on-chain technology. Here’s how it works in a concise, step-by-step breakdown:

**Initiate the Swap**

You start by clicking "Swap" on the website. Enter the amount of ICY you want to trade and your Bitcoin address. The system saves your request by listen emitted events on the swap contract in a database to ensure it’s tracked.

**ICY Tokens Are Burned**

The Swap Contract processes your request, permanently removing (or "burning") your ICY tokens from circulation. It then signals that the swap is underway.

**Bitcoin Is Delivered**

The backend regularly checks events on the Swap Contract to detect your swap request, use request's information such as BTC amount, BTC address, and sends it to your address from the treasury wallet. If any issues arise, the system automatically retries using cronjobs to ensure delivery.

![alt text](assets/cross-chain-transfers-implementing-a-token-swap-from-base-chain-to-bitcoin-1.png)

## How the price is set

For more details on how the price is set, please refer to the [How much is your ICY worth](https://memo.d.foundation/playbook/community/how-to-swap-icy-to-btc-copy/) guide.

## Conclusion

This process uses specialized tools and steps to securely swap ICY tokens for Bitcoin, overcoming the challenges of their different blockchain systems while maintaining fairness in pricing.


This guide shows how to implement a token swap from the Base Chain to Bitcoin.

Implement a token swap from the Base chain to Bitcoin for cross-chain transactions


Ready to convert your ICY to Bitcoin? We've made the process simple and secure. Let's walk through it together.

## Getting started

Before you begin, you'll need two things: a Bitcoin (BTC) wallet to receive your funds and some ETH on the Base network for gas fees. Don't have a BTC wallet? No problem! You can set one up using trusted providers like Electrum, Trust Wallet, or UniSat. If you need help with ETH for gas fees, just reach out to our team on the Dwarves Foundation Discord, we're here to help!

## Step 1: Connect to icy.so

First, visit [icy.so](https://icy.so) and click "Connect Wallet" to select your preferred wallet (MetaMask, Coinbase Wallet, etc.). Follow the connection prompts, and make sure you're on the **Base network**.

![Connect wallet interface showing the wallet connection button and network selection](assets/icy-swap-connect-wallet.webp)

## Step 2: Make the swap

Once connected, you'll need to select **ICY** as your source token and **BTC** as your destination token. Enter the amount you want to swap (remember, the minimum is `20 ICY`), input your **BTC wallet address**, and click "Swap". You'll need to confirm the transaction in your wallet.

![Animated demonstration of the swap process on icy.so](assets/icy-swap-process.gif)

## Step 3: Track your transaction

After confirming, you can watch your transaction in the "Recent Transactions" section. Hover over the transaction to see details, including the service fee. Wait for the "Pending" status to complete.

![Transaction tracking interface showing pending status and service fee details](assets/icy-swap-transaction-status.webp)

Processing time varies based on network conditions, gas fees paid, and system load. Don't worry if it takes some time, we ensure 100% of transactions are processed.

## Important details to remember

Keep these key points in mind when swapping your ICY:

- The minimum swap amount is `20 ICY`
- Each transaction has a service fee of `3,000 units`
- Gas fees are paid in ETH on the Base network
- Processing may be delayed during high network activity

Need help? Our team is always ready to assist on the Dwarves Foundation Discord. We're here to make your ICY swap experience smooth and successful!


A friendly guide to converting your ICY tokens to Bitcoin (BTC) using our platform. We'll walk you through the process step by step, from wallet setup to transaction monitoring.

How to swap ICY to BTC


At Dwarves, we faced the challenge of efficiently transcribing and summarizing our weekly OGIF event recordings for our Brainery knowledge hub. This required developing a scalable data pipeline capable of processing YouTube videos, extracting audio, and leveraging AI models for transcription.

Our solution needed to handle diverse video formats, support concurrent processing, and integrate seamlessly with existing infrastructure. The goal: democratize access to OGIF content while enabling powerful search and analytics capabilities.

## Data pipeline design

The transcription data pipeline is part of a larger system that includes a REST API for job management, a task queue for asynchronous processing, and a web interface for user interaction. The pipeline architecture can be divided into three main stages: Data Extraction, Data Transformation, and Data Loading and Analysis.

```mermaid
graph LR

    %% Extract Stage
    subgraph "Data Extraction"
        A[YouTube API] -->  B[Video Metadata]
        A --> C[Audio Data]
    end

    %% Transform Stage
    subgraph "Data Transformation"
        B --> D[Process Audio]
        C --> D
        D --> E[Transcription Text]
        E --> F[Cleaned Text]
    end

    %% Load Stage
    subgraph "Data Loading and Analysis"
        F --> |Store Transcription| G[(Database)]
        G --> I[Analyze Transcription Data]
        I --> J[Insights]
    end

    %% Style Definitions for Consistency
    style A fill:#f9f,stroke:#333,color:#000
    style D fill:#bfb,stroke:#333,color:#000
    style I fill:#bbf,stroke:#333,color:#000
    style J fill:#fbf,stroke:#333,color:#000

```

### Data extraction

The data extraction stage involves fetching video metadata and audio data from the YouTube API. The Video Downloader component interacts with the YouTube API to retrieve the necessary information.

1. **Video metadata**: The Video Downloader fetches the video from YouTube using the provided URL. It downloads the video file and extracts the video duration. The downloaded video is then passed to the next stage of the pipeline for audio extraction.
2. **Audio data**: The Video Downloader extracts the audio data from the YouTube video. It retrieves the audio stream in a suitable format for further processing.

### Data transformation

The data transformation stage takes the extracted audio data and performs various processing steps to prepare it for transcription and analysis.

1. **Audio processing**: The Audio Preprocessor component compresses the extracted audio data to minimize its size. This compression step optimizes the audio for efficient transmission to Groq's transcription service while maintaining the necessary audio quality for accurate transcription.
2. **Transcription**: The Transcription Engine takes the preprocessed audio data and converts it into text using AI-based transcription models. It may utilize services like [Groq AI](https://groq.com/) to perform accurate speech-to-text conversion.
3. **Text cleaning**: The transcribed text undergoes a cleaning process to remove any artifacts, formatting issues, or inconsistencies. The cleaned text is then ready for storage and analysis.

### Data loading and analysis

The data loading and analysis stage involves storing the transcribed text in a database and performing various analyses to derive insights from the transcription data.

1. **Database storage**: The cleaned transcription text is stored in a PostgreSQL database along with relevant metadata. This allows for efficient retrieval and querying of the transcription data.
2. **Text analysis platform**: The stored transcription data is made available to a text analysis platform, which performs various analytics tasks to extract insights and generate meaningful results.
3. **Insights generation**: The text analysis platform applies techniques such as sentiment analysis, topic modeling, and keyword extraction to derive valuable insights from the transcription data. These insights can be used for content summarization, trend analysis, and data-driven decision-making.

## Core workflow

The transcription workflow involves the User, API, Database, Downloader, S3Storage, Transcriber, GroqAI, and OpenAI.

```mermaid
sequenceDiagram
    participant User
    participant API
    participant Database
    participant Downloader
    participant S3Storage
    participant Transcriber
    participant GroqAI
    participant OpenAI

    User->>API: Submit YouTube URL
    API->>Database: Create Job (Pending Status)
    Database-->>API: Job ID Returned
    API-->>User: Job Submission Confirmation

    loop Job Processing
        Downloader->>Database: Fetch Pending Download Jobs
        Downloader->>Database: Lock Job
        Downloader->>YouTube: Download Video
        Downloader->>Downloader: Convert to Audio
        Downloader->>S3Storage: Upload Compressed Audio
        Downloader->>Database: Update Job Status (Uploaded)
    end

    loop Transcription
        Transcriber->>Database: Fetch Uploaded Jobs
        Transcriber->>Database: Lock Job
        Transcriber->>S3Storage: Download Audio
        Transcriber->>GroqAI: Transcribe Audio
        GroqAI-->>Transcriber: Transcription Result

        alt Optional Typo Correction
            Transcriber->>OpenAI: Refine Transcription
            OpenAI-->>Transcriber: Cleaned Transcription
        end

        Transcriber->>Database: Store Transcription
        Transcriber->>Database: Update Job Status (Completed)
        Transcriber->>S3Storage: Delete Temporary Audio
    end

    User->>API: Query Job Status
    API->>Database: Retrieve Job Details
    Database-->>API: Job Status/Transcription
    API-->>User: Return Results

```

The main steps are:

**Job submission:**

- User submits YouTube URL to API
- API creates a new "Pending" job in Database
- API confirms job submission to User with Job ID

**Job processing - Downloader:**

- Downloader fetches pending jobs from Database
- Downloader locks job, downloads video, converts to audio
- Compressed audio uploaded to S3Storage
- Job status updated to "Uploaded"

**Transcription - Transcriber:**

- Transcriber fetches uploaded jobs from Database
- Transcriber locks job, downloads audio from S3 Storage
- Audio sent to GroqAI for transcription
- Transcription refined by OpenAI
- Final transcription stored in Database
- Job status updated to "Completed"
- Temporary audio file deleted from S3 Storage

**Result retrieval**:

- User queries job status through API
- API retrieves job details and transcription from Database
- API sends transcription results to User

The API acts as the intermediary between User and backend components. The Database stores job information and transcriptions. The Downloader handles video downloading and audio conversion, while the Transcriber manages transcription, interacting with GroqAI and OpenAI. S3Storage is used for temporary audio storage.

The workflow ensures efficient job processing through asynchronous processing and job locking. The separation of Downloader and Transcriber allows for parallel processing and scalability. The optional typo correction step with OpenAI enhances transcription quality.

## Performance benchmarks

The system is designed to handle the following benchmarks:

- Process 100 simultaneous transcription jobs
- Handle videos from 5 minutes to 2 hours
- Complete processing within 5 minutes per video
- Maintain 90%+ transcription accuracy
- Ensure sub-500ms API response times
- Complete jobs within 15 minutes

To ensure the pipeline could handle the expected scale and provide timely results, several optimizations were implemented:

1. **Parallel processing**: The Video Downloader, Audio Preprocessor, and Transcription Engine were designed to process multiple jobs concurrently. The number of parallel workers can be dynamically adjusted based on load.
2. **Asynchronous API calls**: The interactions with external services (YouTube, Groq, OpenAI) were made asynchronous to avoid blocking the main pipeline flow.
3. **Intelligent chunking**: Dynamic chunking of audio files allowed optimizing for the input constraints of the AI transcription services while minimizing total API calls.
4. **Temporary file management**: Audio files were stored in S3 only for the duration of processing and deleted afterwards to minimize storage costs.
5. **Database connection pooling**: A pool of reusable database connections was used to avoid the overhead of establishing new connections for each operation.

Robust error handling and monitoring were critical to ensure pipeline reliability and maintainability:

1. **Retry policies**: Each stage of the pipeline was configured with appropriate retry policies to handle transient failures from external services or temporary resource constraints.
2. **Dead-letter queues**: Jobs that repeatedly failed even after retries were moved to a dead-letter queue for manual inspection and intervention.

## Technology stack

The transcription service leverages the following technologies and tools:

**Core platform**

- Python 3.9+
- Flask/FastAPI for RESTful APIs
- Celery + Redis for task queue
- Gunicorn for WSGI server

**AI/ML services**

- Groq AI for transcription
- OpenAI GPT-4 for text refinement
- Custom rate limiting and retry logic

**Data & storage**

- PostgreSQL for persistent storage
- Redis for caching/queues
- AWS S3 for file storage
- `Boto3` for AWS operations

**Media processing**

- `yt-dlp` for video downloading
- `FFmpeg` for video manipulation
- `pydub` for audio processing

**Infrastructure & devOps**

- Docker + Docker Compose
- GitHub Actions for CI/CD
- Prometheus + Grafana monitoring
- Nginx reverse proxy

**Security & documentation**

- JWT authentication
- SSL/TLS encryption
- OpenAPI/Swagger documentation

## Lessons learned

Key successes of the project include:

1. Modular decoupling of downloader, transcriber and API logic improved scalability and maintainability. Issues could be identified and fixed quickly in each module.
2. Optimized resource allocation by isolating components and catering to their specific needs. This led to efficient performance without over or under-provisioning.
3. Asynchronous architecture allowed non-blocking processing of long-running jobs. Users receive immediate job ID and can poll for status without holding up resources.

An area for improvement is the current video chunking implementation, which needs refinement to ensure perfectly synced transcription. The team is exploring alternatives to split videos more intelligently based on breaks and allow finer-grained processing.

## Conclusion

Building the YouTube transcription data pipeline required carefully orchestrating the interaction between several system components and external services. The ETL pattern provided a clear structure to reason about the data flow and transformation steps.

The asynchronous, event-driven architecture allowed each stage of the pipeline to scale independently and handle failures gracefully. Techniques like parallel processing, intelligent chunking, and connection pooling helped achieve the required performance and throughput.

The end result was a robust and efficient data pipeline that could reliably transcribe a high volume of YouTube videos and make the content searchable and accessible across the organization. The pipeline unlocked new ways to distill insights from previously opaque video content.


A technical case study of creating an automated system that downloads videos, processes audio, and generates transcripts using AI services like Groq and OpenAI.

Building data pipeline for OGIF transcriber


Hedge Foundtion, a private trading platform serving select traders, required a robust centralized monitoring system to ensure platform reliability and prevent financial losses. Given the high-stakes nature of trading operations, the system needed to provide real-time alerts, maintain data integrity, and optimize resource allocation to protect traders from potential monetary losses.

## Understanding the unique challenges

As a privately-owned platform with a limited user base, Nghenhan faces unique challenges:

1. **High-stakes trading**: Each user represents a significant portion of the platform's trading volume. Any system failure or data loss can lead to substantial financial losses for these traders.
2. **Reputational risk**: With a smaller user base, any issues with the platform can quickly erode trust and lead to user attrition. Maintaining a stellar reputation is essential for Nghenhan's long-term success.
3. **Resource allocation**: While the user base is limited, the platform must still be equipped to handle peak usage times and spikes. Efficient resource allocation is critical to ensure reliable performance without overspending on infrastructure.

## Mitigating financial losses through proactive monitoring

To address these challenges, Nghenhan must implement a proactive monitoring strategy that focuses on:

1. **Real-time alerts**: The monitoring system must provide instant notifications for any anomalies or threshold breaches. This allows the team to react swiftly and minimize the duration and impact of any issues.
2. **Data integrity**: Ensuring the accuracy and synchronization of trading data is paramount. Any data loss or discrepancies can trigger false alarms or missed opportunities, leading to financial losses for traders.
3. **Resource optimization**: Monitoring resource utilization helps Nghenhan allocate resources effectively during peak times while avoiding over-provisioning during normal usage.

## Implementing Grafana and Prometheus for robust monitoring

Integrating Grafana and Prometheus provides Nghenhan with a powerful centralized monitoring solution. Let's dive deeper into how these tools work together and examine the system diagram:

![](assets/nghenhan-monitoring-system-diagram.webp)

- **Backend services**: The backend services of Nghenhan's trading platform expose metrics through an HTTP endpoint, which Prometheus scrapes at regular intervals.
- **Prometheus server**: The Prometheus server scrapes the metrics from the backend services and stores them as time series data. It also handles the querying and alerting functionality.
- **Alert manager**: The Alert Manager is a component of Prometheus that handles the routing and management of alerts. It receives alerts from Prometheus and sends notifications to the configured receivers, such as the admin or notification channels.
- **Grafana**: Grafana fetches data from Prometheus to create visualizations and dashboards. It also allows users to set up alerts and explore historical data.
- **Notification receivers**: The notification receivers are the endpoints or channels where alerts are sent, such as email, Discord, or custom webhooks. The admin can also receive notifications and take appropriate actions based on the alerts.

### Prometheus as data collector

Prometheus serves as the primary data collection and monitoring tool, scraping metrics from various services and recording health and performance information. The setup involves configuring Prometheus to gather data on key metrics, including:

**CPU and Memory usage**

Making sure that system resources are not reaching critical thresholds.

![](assets/nghenhan-cpu-usage.webp)

**Error rates**

Tracking the number of errors in real time to quickly detect discrepancies.

![](assets/nghenhan-error-rate.webp)

**Data synchronization status**

Monitoring the synchronization of data from Binance to ensure its latest version without any data loss.

![](assets/data-sync-status.webp)

**Binance rate limit monitoring**

Implementing a rate limit monitoring system to ensure that requests to Binance are still compliant with the rate limits. This will prevent data loss during periods of high network traffic.

![](assets/nghenhan-binance-rate-limit.webp)

**Service back-off restarting**

due to multiple issues, such as resource limits, configuration errors, or dependency failures.

![](assets/nghenhan-service-back-off-restarting.webp)

### Grafana as data visualizer for insightful observations

Grafana complements Prometheus by providing robust data visualization capabilities. With Grafana, Nghenhan can create dynamic dashboards that display real-time data on service performance. These dashboards include:

**Real-time alerts**

Configured alerts notify our team of any anomalies, such as sudden increases in CPU usage or error rates, etc. that exceed established thresholds.

Here’s an example of how we configured conditions on Grafana to trigger an alert using the Alert Manager

![](assets/nghenhan-real-time-alert.webp)

The setup above will trigger an alert if data exceeds the threshold, and the Alert Manager will send it to Discord by webhook.

![](assets/nghenhan-discord-alert.webp)

**Interactive graphs**

We utilize visual representations of data that help us easily identify trends during peak trading times and spikes.

![](assets/nghenhan-interactive-graph.webp)

**Historical data analysis**

Grafana’s capabilities allow us to analyze historical data to understand system behavior and improve resource allocation strategies.

![](assets/nghenhan-historical-analytics.webp)

## Conclusion

To sum up, Nghenhan's decision to adopt a centralized monitoring system powered by Grafana and Prometheus is a testament to its dedication to providing a reliable and efficient trading platform. By focusing on real-time monitoring and ensuring data synchronization, Nghenhan can proactively identify and resolve potential issues, minimizing downtime and financial losses for its users. This monitoring system not only bolsters Nghenhan's operational capabilities but also serves as a foundation for future growth.


A technical case study for implementing centralized monitoring for a trading platform using Grafana and Prometheus, focusing on real-time alerts, data integrity, and resource optimization to prevent financial losses.

Setup centralized monitoring system for Hedge Foundation trading platform


Organizations are always looking for ways to improve efficiency and productivity. Large Language Models (LLMs) are a powerful technology that can help create smart internal tools. However, using LLMs in existing workflows can be complicated and resource-heavy. This is where managed LLMOps comes into play, providing a smoother way to develop and deploy LLM-powered tools. In this post, we'll see how platforms like Dify enhance the workflow for building internal AI tools.

## The challenge of building LLM-powered internal tools

LLMs have great potential to enhance business processes, but using them for internal tools comes with challenges:

1. **Complexity**: Using LLMs requires deep technical knowledge of AI and machine learning.
2. **Resource intensity**: Training and fine-tuning LLMs can be expensive and time-consuming.
3. **Maintenance**: Keeping LLM-based tools updated and running well needs ongoing attention.

These factors make it hard for many organizations to fully use LLM technology, especially those without dedicated AI teams.

## Managed LLMOps: A solution for efficient development

Managed LLMOps platforms solve these challenges by offering a complete environment for developing, deploying, and managing LLM-powered applications. These platforms simplify the complexity of LLMs, allowing developers and business users to focus on creating valuable tools instead of dealing with AI infrastructure.

## Dify: An example of managed LLMOps in action

Dify is a great example of a managed LLMOps platform that makes creating LLM-powered internal tools easier. Here are some key features:

1. **User-friendly interface**: Dify has an easy-to-use interface that lets both technical and non-technical users create AI applications.
2. **Pre-built templates**: Users can start quickly with templates for common use cases.
3. **Customization options**: Dify allows for deep customization when needed.
4. **Integrated workflow management**: The platform includes tools for managing the entire lifecycle of AI applications.

To make the deployment process even smoother and manage resources better, we use [elest.io](http://elest.io/) to deploy Dify. [Elest.io](http://elest.io/) is a fully managed DevOps platform, similar to DigitalOcean's marketplace but with more open-source applications. This approach offers several benefits:

- **Simplified setup**: [Elest.io](http://elest.io/) automates much of the deployment process, reducing the time and expertise needed to get Dify up and running.
- **Cost control**: By using [elest.io](http://elest.io/)'s infrastructure, we can easily manage and optimize costs across different cloud providers.
- **Flexibility**: [Elest.io](http://elest.io/) supports various cloud providers, allowing us to choose the best and most cost-effective option for our needs.
- **Scalability**: As our LLM tool usage grows, [elest.io](http://elest.io/) makes it easier to scale our Dify deployment to meet increasing demands.

This combination of Dify's powerful LLMOps capabilities and [elest.io](http://elest.io/)'s streamlined deployment process creates an efficient, cost-effective solution for organizations looking to use LLMs in their internal tools. This makes it possible for an average developer to build and deploy an internal tool in minutes.

## Case studies: Internal tools built with Dify

To show the advantage of managed LLMOps, let's look at some example tools built using Dify:

### 1. SQL sorcerer

This tool turns everyday language into SQL queries, allowing non-technical team members to extract insights from databases without learning complex query languages. We use it to make queries to our DuckDB database, integrating it with our memo website and knowledge base.

![assets/building-llm-powered-tools-with-dify-sql-sorcerer.webp](assets/building-llm-powered-tools-with-dify-sql-sorcerer.webp)

![assets/building-llm-powered-tools-with-dify-sql-sorcerer-test-duckdb.webp](assets/building-llm-powered-tools-with-dify-sql-sorcerer-test-duckdb.webp)

### 2. OGIF memo summarizer

Specialized in extracting information from YouTube transcripts, this tool quickly generates time-stamped summaries with key points, saving hours of manual video analysis. We recently wrote an article about this tool, which you can read [here](../ai/how-we-crafted-the-ogif-summarizer-bot-to-streamline-weekly-knowledge-sharing.md).

### 3. Discord summarizer assistant

Using large language models, this workflow helps translate and summarize conversations across different languages, making global team communication easier.

![assets/building-llm-powered-tools-with-dify-discord-summarizer.webp](assets/building-llm-powered-tools-with-dify-discord-summarizer.webp)

These tools help us significantly increase our productivity. We extensively use Claude 3.5 Sonnet, and occasionally GPT-4 for more detailed instructions. Without the hard work of @innno\_ and her efforts on our social media platforms, we wouldn't be able to build these tools.

## Best practices for developing internal LLM tools

When using managed LLMOps platforms like Dify to create internal tools, consider these best practices:

1. **Start with a clear use case**: Identify specific pain points or inefficiencies in your workflows that LLMs can address.
2. **Iterate based on feedback**: Regularly collect and incorporate user feedback to improve your tools.
3. **Ensure data privacy**: When dealing with sensitive internal data, make sure your LLM applications follow your organization's security policies.
4. **Monitor performance**: Use the analytics provided by your LLMOps platform to track usage and optimize performance.

## Conclusion

Managed LLMOps platforms like Dify are making AI technology accessible to organizations of all sizes, allowing them to create powerful internal tools without needing extensive AI expertise. By simplifying the development and deployment process, these platforms are paving the way for a new era of AI-augmented productivity tools. As LLM technology continues to advance, we can expect to see even more innovative applications that transform the way we work.


Discover how managed LLMOps platforms like Dify streamline the development of AI-powered internal tools. Learn about the challenges of integrating LLMs, the benefits of managed solutions, and real-world examples of efficient AI tool creation. Perfect for businesses looking to enhance productivity with AI without extensive technical expertise.