
## My workflow with Overleaf and ChatGPT

A few weeks ago, I was asked to make a slide deck for a team meeting. Normally, I usually use [Google Slides](https://workspace.google.com/products/slides/) or [Markdown via Marp](https://marp.app/) to make a simple slide for presentation. But this meeting is more serious, so I needed to make a professional, high-standard slide deck. This requirement _made_ me think of using [Overleaf](https://www.overleaf.com/), a tool that helps create slides in a professional format. It worked so well that I want to share my experience. This is my story, walking through the problems I faced, the solution I found, and some tips that might help you too.

### Consistent and polished slides

While tools like [Google Slides](https://workspace.google.com/products/slides/) are great for quick presentations, ensuring a consistently professional and polished look for a more serious meeting can present its own set of challenges. Even seemingly simple tasks, like maintaining uniform fonts, precise spacing, and a cohesive design across all slides, can become surprisingly time-consuming and require meticulous attention to detail. This can detract from the core task of crafting compelling content.

I experimented with markdown via [Marp](https://marp.app/), hoping for a more efficient way to create slides. I found the writing process faster, but I struggled to achieve the level of visual refinement needed for a professional presentation. The output, while functional, lacked the polished aesthetic that would convey the seriousness of the meeting. This experience underscored the need for a tool that could not only streamline the creation process but also inherently produce a high-quality, professional visual output. That's why I decided to explore Overleaf. I knew it was designed to create professional documents and slides with built-in themes that ensure consistency and a polished appearance with minimal effort. Furthermore, its features like online collaboration, and debugging tools made it an even more attractive option for ensuring a smooth and efficient workflow.

![Overleaf](assets/overleaf.png)

But to make it work, normally, we need to know how to use LaTeX. Not everyone is familiar with it, and it can be a barrier to entry for some. But this is the age of AI, and we can use it to make it easier. Let's see how.

### A two-step solution

I came up with a simple way to make things easier, after some trial and error late at night. I decided to use ChatGPT to write the content and Overleaf to handle the formatting. It felt like having one helper for ideas and another for design. Here’s how it worked. First, I asked ChatGPT to help me. I’d give it a request like: "Write a 3-slide presentation in a format I can use, about implementing a data snapshot pattern to persist historical data, with 3 bullet points per slide." It quickly gave me a draft with titles, points, and a structure I could use. It wasn’t perfect, but it was a great starting point. Then, I’d take that draft and put it into Overleaf. I’d pick one of its predefined themes, hit “Recompile,” and get clean slides fast. Overleaf’s live preview let me make small changes as I went, and its online setup made it easy for my team to join in. No more confusion over file versions. It all came together smoothly.

This method worked well for a few reasons. ChatGPT saved me time on writing, turning hours into minutes. It gave me a clear structure, so I didn’t have to plan everything myself. Overleaf made the slides look good with its exporter and themes, without me needing to do much. And it made teamwork simple, keeping us all on the same page. It turned a slow task into something quick and manageable. I was really happy with how it turned out.

### A real example

Let me share one time I used this method, preparing slides on persisting historical data for a tech talk. I’d been working on a project about using the data snapshot pattern to store historical data, like in a cryptocurrency trading system, and I wanted to explain it. I asked ChatGPT: "Write a 3-slide presentation in a format I can use, on implementing a data snapshot pattern to persist historical data, 3-4 points per slide." It gave me something I could work with, like this:

```latex
\begin{frame}{Slide 2: Why Use Snapshots?}
    \begin{itemize}
        \item Captures data at a specific time
        \item Prevents recalculation errors
        \item Speeds up report generation
    \end{itemize}
\end{frame}
```

I copied it into Overleaf, picked a nice predefined theme, and watched it turn into proper slides. I made a few small changes and added a point about how snapshots help with long-term trend analysis. In about 15 minutes, I had a finished deck. I exported it as a PDF, presented it at the tech talk, and the audience found it clear and useful. It didn’t feel like a chore. It felt like a small win, and I left the talk feeling good.

### Turning this article into slides

Here’s where it gets interesting. I used this very article to test my workflow again, turning it into a slide deck with Overleaf. I wanted to see if it could handle something I’d already written, and it did. I asked ChatGPT: "Take this article and make a 4-slide presentation in a format I can use, with 3 points per slide, summarizing my workflow." It gave me a good starting point, like this:

```latex
\begin{frame}{Slide 1: The Struggle with Slides}
    \begin{itemize}
        \item Writing content takes too long
        \item Organizing ideas is hard
        \item Teamwork gets messy
    \end{itemize}
\end{frame}
```

I pasted it into Overleaf, chose a clean predefined theme, and added a slide for each section: the problem, the solution, an example, and tips. I adjusted the wording a bit and recompiled it. In under 20 minutes, I had a deck ready to share with you. It shows that this method works, even on itself. You could do the same with this article. Try it, and you’ll see how fast it comes together.

### Automating with Dify

![Workflow](assets/workflow.png)

If you want to take this even further, I’ve streamlined the whole process using a tool called Dify. It automates the workflow, making it even easier for anyone to follow. The process starts with your content idea, which gets analyzed and optimized by a content tool. Then, it’s turned into a format you can use, styled, and finalized. After that, it’s uploaded to a gist for easy access, and you get the code as a result. If something goes wrong, it retries up to three times to ensure it works. This setup saves so much time, especially if you’re not comfortable with the manual steps. If you’re feeling lucky, you can try our Dify workflow directly [here](https://prompt.d.foundation/app/eb483740-3915-4aea-9fc4-5c50eb4700f5/workflow). It’s a great way to see the process in action without doing all the steps yourself. You can also check the example result [here](https://www.overleaf.com/read/jhywvqsdvwxk#8a280e), this is the result of the workflow when I typed "introduce latex for presentation generation".

### Tips that helped

After using this a few times, I found some ways to make it better. Be specific with ChatGPT. Tell it to keep things short or use lists, and it’ll save you cleanup time. Play with Overleaf’s predefined themes to make slides look nicer without extra work. I’ve found ones like “Copenhagen” or “Berlin” work well. For my tech talk slides, I asked ChatGPT to summarize key points, which saved me from digging through details. And I kept ChatGPT’s raw text in a separate file, just in case something went wrong in Overleaf. These small steps made the process smoother and more reliable.

### Making it useful for you

To help you get the most out of this, I’ve added a few things. You’ll see real examples, like the snapshot slide or the article summary above. Try them out yourself and see how they work. Picture a simple flow: prompt goes to ChatGPT, then to a format you can use, then to Overleaf, and finally to slides. That’s the process in a nutshell. If you’re working on a topic like data persistence, change the prompt to fit your needs, like “snapshot patterns for e-commerce.” One time, this saved me when I made a full deck in two hours instead of two days. Just check ChatGPT’s work. It’s good, but it can miss details if you don’t guide it. You’ll catch those quickly with a little practice.

#### Wrapping it up

That’s my story. It’s how I turned slide-making from a hassle into something simple with Overleaf and ChatGPT. It’s not complicated, just a practical fix that gets the job done. Next time you’ve got a presentation coming up, maybe about data patterns or your own project, try this out. It’s helped me more than once, and I think it could help you too. What do you think? Give it a go, adjust it to fit you, and tell me how it works. See you at the next meeting. Your slides will be done, and the stress won’t be there.


This article shares a workflow for making slide decks with Overleaf and ChatGPT. It solves issues like slow content creation using ChatGPT and formats with Overleaf’s themes. It includes examples, tips, and a Dify automation for engineers.

Create slides with Overleaf and ChatGPT


## Data safeguarding strategies

Data is an important part of software development and one of the most valuable assets for any organization, especially in economics and finance. Along with the growth of business models, a large amount of data is generated diversely. Keeping data safe is critical for business. Data is lost or becomes wrong, which can cause irreversible loss. For example, in the banking system or stock marketplace, if one transaction record is missed, it can lead to a chain of consecutive wrong behaviors. It can even cause money losses for individual users as well as organizations.

In software development, we have many strategies to safeguard the data. Depending on each use case, some strategies can be listed here:

- **Data encryption** marks the original data by ciphertext, making it harder to access.
- **Data security policy** defines rules to manage data access and role-based permissions.
- **Data lifecycle management** defines the framework to manage data from creation to destruction.
- **Data backup and recovery** regularly backs up key data to restore them once any critical issue happens.

Each strategy aligns with specific stages of the data lifecycle and can be combined to maximize protective capabilities.

This overview provides some strategies to protect the data safely. We will continue delving into a more specific problem in the rest of this post, and explore the way we deal with it.

## Problem with storing large amounts of transactional data that are not accessed frequently

### What problem are we solving?

Imagine you are developing a financial application that produces tens of thousands of transactions per day by users because of their cryptocurrency trades. These transactions are firstly stored in the data lake as raw records. Once a trade round (normally 3 months of trading) is over, the raw transactions in this period will be used to produce the final reports and persist to the database. After this time, these records will not be used anymore in both the trading process and summary calculation except for data auditing or report recovery in the future.

Once the project continues running, the amount of data becomes bigger. It requires us to spend more money to expand the database. The amount of data grows quickly also leading to decreased performance of any operation that needs to interact with the database.

This situation lets us think about the data archive which is a strategy helping to offload unused data into long-term storage at minimal cost.

### Data archive, why do we need it?

We first take a look at **data backup** which is the cyclic process of duplicating the entire or a part of data, wrapping it in a stable format then storing it in a secure place. This process is scheduled periodically to make sure we always have at least a copy of production data readies to restore at any time one issue happens.

![alt text](assets/data-backup-and-restore.png) _Figure 1: Simple example in SQL to represent the data backup and recovery process_

In my point of view when writing this post, backup leans toward the action that captures the state of the database for rolling back to the specific point in the past. By using the backup data, we can do the "disaster recovery" in time when a critical problem needs to hotfix. It is often complex and expensive. Backup and recovery in this context can also impact the ongoing work on the production. Data can be lost or wrong if the strategy is not executed carefully.

The **data archive** may or may not be similar to **data backup**, depending on your definition for each of them. For me, they are similar but with some differences.

While backup comes from production data hotfix problems, data archive focuses on long-term data-keeping. With the growth of production data, especially in transactional applications, a lot of data is not needed for normal execution. However, they are required to reproduce important metrics and auditing in the future. **The data archive is the progress of shelving data that has reached the end-of-life in an organized manner to be easy to use later.**

![alt text](assets/data-backup-and-archive.png) _Figure 2: Visualization the differences between data backup and data archiving_

By implementing the **data archive** strategy, we can decrease the live database's pressure significantly and optimize storage costs while still ensuring long-term data availability.

### Recovery using archived data

Archived data is normally not used for production data hotfix or rollback application state. Instead, it is used to recover critical data such as data snapshots, market reports, or even legal matters like audits.

This progress is often executed manually. This means that the data is archived automatically each time it persists after an operational phase is completed. This period is determined depending on your application. It can be monthly, yearly, or each trading round in the trading application. However, once the recovery is required to execute, it should be run manually by the administrator.

This data recovery strategy mainly focuses on calculating instead of restoring. The calculated result usage depends on your use case. However, once it is used to recover the database directly, it must be ensured that it does not interfere with the normal operation of the application or alter any online information as when rolling back the system state using backup data. This process has another name that is called **forward recovery**.

This approach has some advantages:

- Can regenerate data even without having direct backups
- Provides data validation through reprocessing
- Often results in cleaner data since it goes through current business rules
- Can be useful for audit purposes

## Implementing archive-based recovery strategy for trading application

Back to the first example that was used at the second part to raise the problem. Assume we have a high-frequency cryptocurrency trading platforms that produce 50,000 transactions per day, which accumulates to approximately 4.5 million transactions in a single trading cycle of three months. At an average size of 2KB per transaction, this translates to nearly 9GB of raw data every cycle.

To deal with this situation, we can design a simple archive and recovery strategy as following:

```mermaid
flowchart TD
    subgraph "Production Environment"
        A[Trading System] -->|Real-time Transactions| B[Data Lake]
        B -->|Raw Records| C[PostgresQL]
    end

    subgraph "Archival Process"
        C -->|3-month cycle| D[Dedicated Compute Environment]
        D -->|Encrypted Archives| E[Archive Storage]
        D -->|Metadata| F[Lightweight Database]
    end

    subgraph "Recovery Process"
        G[Admin Request] -->|Search| F
        F -->|Archive Location| E
        E -->|Retrieve Archive| H[Dedicated Compute Environment]
        H -->|Processed Data| I[Analysis Instance]
    end

    style A fill:#f9f,stroke:#333,stroke-width:2px
    style E fill:#blue,stroke:#333,stroke-width:2px
    style H fill:#green,stroke:#333,stroke-width:2px
```

_Figure 3: diagram to visualize the workflow of a archive and recovery strategy implementation to resolve the problem with the data of high-frequency cryptocurrency trading platforms_

**Archiving Workflow**:

- After each trading cycle (e.g., 3 months), transactional records are processed and moved to cloud-based storage. These records are compressed and encrypted for security and cost optimization.
- Metadata for these archived transactions is maintained in a lightweight database for quick lookup.

**Recovery Workflow**:

- When data is required, administrators search the metadata for the relevant archive.
- The archived records are retrieved and reprocessed using a dedicated compute environment to generate the required reports or validate metrics.
- If needed, the processed data can be restored to a separate database instance for further analysis without affecting the production environment.

## Conclusion

From this discussion, we have seen how archive and recovery strategy can address specific challenges such as efficiently handling large volumes of rarely accessed data. Implementing a robust archive and recovery system provides several benefits, including long-term data availability, cost-effective storage, and support for audits or legal requirements. This strategy is particularly valuable for industries like finance, healthcare, and e-commerce, where data integrity and accessibility are critical.

This knowledge is essential for system architects, database administrators, and developers who manage large-scale applications with growing data needs. Understanding and implementing this strategy equips teams to handle data growth effectively, ensuring their systems remain reliable, secure, and future-ready.


A guide to implementing data archival and recovery strategies for high-volume transactional application.

Building a data archive and recovery strategy for high-volume trading system


Crypto trading is not just gambling. It has strategies. Once traders can take advantage of it effectively, they can earn. One of the most popular strategies in cryptocurrency is **Hedge**, which is often mentioned with the name **Hedge Bitcoin**.

In simple words, assuming that we are living in a world where we can only just buy/sell salt and pepper. As a merchant, you buy one for speculation. But the risk happens when the price of your holding asset decreases; you lose. With the **Hedge** strategy, you buy both of them with the belief that if one of them decreases, the other will increase. For example, if the price of salt increases while the price of pepper decreases, you buy more pepper and sell a part of your salt.

The real strategy is more complex than my example. But we can understand that **Hedge Bitcoin** means trading entirely other assets on the exchange in the inverse direction to decrease the risk of Bitcoin trading. **To execute this strategy effectively, traders must monitor how liquidity flows between Bitcoin and Altcoins across the exchange, revealing their relative performance patterns.**

### How can we extract market-wide insights from trading data

Performance, in our context, represents the token's profit and loss. For example, if I say, _Performance of BTC is 10%,_ it means BTC has profited 10% since we placed the order. This number closely relates to the asset's liquidity in the market. If you can't imagine the relationship between them, try thinking about the price first. The price of an asset only increases when its total liquidity in the market rises. The change can come from scenarios such as the following:

- Liquidity is moved from other assets to this asset
- Liquidity from new buyers entering the market

![alt text](assets/liq-price-pnl-relationship.png) _Figure 1: Relationship between asset liquidity, price, and trading PnL_

Price movements serve as indicators of liquidity flows between assets. When capital moves between Bitcoin and Altcoins, it triggers price changes that result in trading profits or losses. This chain reaction works in reverse too. Trading PnL patterns reveal price movements, which in turn expose the underlying liquidity shifts in the market.

Finally, market performance can be calculated using the following formula:

```math
market_{perf} = BTC_{perf} - Alt_{perf} = \frac{BTC_{pnl}}{BTC_{init}} - \frac{\sum {Alt_{pnl}}}{\sum{Alt_{init}}}
```

_Formula 1: Formula to calculate market performance_

This formula serves as a powerful indicator, revealing liquidity movements across the market whether capital is flowing from Bitcoin to Altcoins, from Altcoins to Bitcoin, or entering/exiting the market entirely. Unlike evaluating price, which provides a general view of trends, performance allows us to quantitatively assess the market. For example:

- When market performance is 5% and BTC performance is 2.5%, this means 2.5% of the funds are moved from Altcoins to BTC, and no new funds have entered the market.
- When market performance is -10% and BTC performance remains unchanged, this means 10% of funds are poured into the market from outside.

Now we have our framework to hedge. How can we obtain this data? Based on _Formula 1_, we need the trading PnL of BTC and all Altcoins. So our business is to prepare a Binance account; depositing money into it; opening a BTC long position with 50% of our funds, and splitting the rest 50% among short positions on all Altcoins. Finally, we wait for PnL changes at each interval, usually a minute; calculate the performances from the PnLs, and record them in our database for later use.

### From data to dashboard: implementing a Go web interface to render performance charts

At the end of the previous part, we mentioned the data gathering. Let me show you how was it constructed.

![alt text](assets/market-perf.png) _Figure 2: Market performance data_

In _Figure 2_, you can easily see that a large amount of liquidity has been pumped into the market. So, how easy is it to get an overview of the market when these figures are visualized? This is the reason why we are trying to do it here, until now.

Market performance alone is not enough. In addition to estimating future market trends for trading, we also need to evaluate the performance of past trades. To achieve this, I have integrated our trading history into the charts. These include:

- **Trade round**: The time period from the beginning to the end of each trade.
- **Trading account PnL history**: The PnL changes of a specific Binance account over time.

By visualizing both market trends and historical trading data, we gain a more comprehensive understanding of our trading efficiency and decision-making process. The first step is selecting the most appropriate type of chart. It must ensure that when multiple data sources are combined, the visualization retains clarity, readability, and meaning. A mixed chart is ideal for this purpose, combining lines to represent market performance and trading PnL changes over time with a double bar chart that juxtaposes BTC and Altcoin performance to highlight their variations. Finally, scoping each trade within a separate window allows us to analyze individual trading periods in detail.

![alt text](assets/perf-chart.png) _Figure 3: The mixed chart that represent the relationship between market performance and trading effective_

#### Aggregating lines from multiple sources

We will begin the implementation with aggregating and aligning data from different source. The performance data, round period and PnL records each come with their own structure and time frames. To ensure everything aligns properly on the time axis, the data is mapped using time-based Golang map as following.

```go
performanceMap := make(map[string]Performance)
for _, perf := range performances {
    performanceMap[perf.Time.Format("2006-01-02 15:04:05")] = perf
}

pnlMap := make(map[string]Pnl)
for _, pnl := range pnls {
    pnlMap[pnl.Time.Format("2006-01-02 15:04:05")] = pnl
}
```

_Code 1: Simple mapping value of performance and pnl to the time axis_

#### Construct the chart

```go
line := charts.NewLine()
line.SetGlobalOptions(
    charts.WithTitleOpts(opts.Title{Title: "Performance and PnL"}),
    charts.WithXAxisOpts(opts.XAxis{Name: "Time"}),
    charts.WithYAxisOpts(opts.YAxis{
        Name: "Performance (%)",
        Min: -maxPerf,
        Max: maxPerf,
    }),
)

line.ExtendYAxis(opts.YAxis{
 Name: "PnL ($)",
 AxisLabel: &opts.AxisLabel{
  Formatter: "{value} $",
 },
 Min: -maxPnL,
 Max: maxPnL,
 AxisLine: &opts.AxisLine{
  Show:            &trueval,
  OnZeroAxisIndex: 1,
 },
})

line.AddSeries("Performance", yAxisPerf)
line.AddSeries("Unrealized PnL", yAxisPnl, charts.WithLineChartOpts(opts.LineChart{YAxisIndex: 1}))

bar := charts.NewBar()
bar.SetXAxis(xAxis).
    AddSeries("AltCoin Perf", yAxisShortPerf).
    AddSeries("BTC Perf", yAxisLongPerf)

line.Overlap(bar)
```

_Code 2: Code snippet demonstrates how to create a mixed chart using the **go-echarts** library_

To combine line and bar charts to visualize market performance, PnL, and BTC/Altcoin performance. The process begins with defining a line chart using the `charts.NewLine()` function. This line chart is configured with global options such as the title, X-axis for time, and a Y-axis labeled "Performance (%)", which ranges from `-maxPerf` to `maxPerf`. This setup ensures that the performance data is plotted on a dedicated axis, making it easy to interpret trends over time.

Next, a secondary Y-axis is added to represent PnL, labeled "PnL ($)". This axis is configured with its own range (`-maxPnL` to `maxPnL`) and includes an axis line centered on zero for better visual balance. By extending the Y-axis with this configuration, the chart supports two distinct datasets on different scales, ensuring both performance and PnL are displayed clearly without visual clutter.

The `line.AddSeries` method is used to add the performance data and unrealized PnL data to the chart. Each dataset is represented as a separate line, with the PnL data assigned to the secondary Y-axis using `charts.WithLineChartOpts(opts.LineChart{YAxisIndex: 1})`. This approach ensures that performance and PnL are plotted on their respective axes, maintaining the clarity of the visualization.

To include BTC and Altcoin performance data, a bar chart is created using `charts.NewBar()`. This bar chart shares the same X-axis as the line chart and is populated with series for "AltCoin Perf" and "BTC Perf" using the `AddSeries` method. The bar chart highlights how the performance of these two asset classes changes over time, complementing the overall visualization.

The `line.Overlap(bar)` method combines the line and bar charts into a single cohesive visualization. This allows the user to analyze market trends, PnL changes, and asset performance simultaneously within one chart.

After all, we may still missing something. Yes it is the area to represent the boundaries of trade rounds. It is a bit easy, **go-echart** provides us the option `charts.WithMarkAreaNameCoordItemOpts` to integrate the mark areas to the line chart. Our business is construct each area boundaries by specify its coordinates using trade round start, end time.

```go
opts.MarkAreaNameCoordItem{
    Coordinate0: []interface{}{trade.OpenedTime.Format("2006-01-02 15:04:05"), -maxYAxis},
    Coordinate1: []interface{}{trade.ClosedTime.Format("2006-01-02 15:04:05"), maxYAxis},
    ItemStyle: &opts.ItemStyle{
     Color: "rgba(255, 255, 255, 0.3)", // White color with blur effect
    },
   },
```

_Code 3: Code snippet to construct MarkAreaNameCoordItem depend on trade period_

### Conclusions

This project demonstrates how we can combine Golang and go-echarts to build powerful visualizations that provide deep insights into crypto trading performance. By integrating market trends, historical PnL data, and trading rounds into a single chart, we create a tool that allows traders to make informed decisions with clarity and precision.

The challenges of aligning multiple datasets, ensuring readability, and maintaining meaningfulness were overcome with careful design and thoughtful implementation. This visualization not only simplifies the analysis of complex trading data but also empowers users to refine their strategies and improve efficiency. As the crypto market continues to evolve, tools like these will be indispensable for staying ahead of the curve.


Implementing a Golang-based visualization for crypto market performance indicators, focusing on Bitcoin vs Altcoin dynamics and trading strategy effectiveness through interactive charts and data analysis

Visualizing crypto market performance: BTC-Alt dynamic indicators in Golang


## Executive summary

Recovering historical trading profit and loss (PnL) data is a critical challenge for finance and cryptocurrency platforms. When historical records are unavailable, users cannot validate past trading strategies, assess long-term performance, or reconcile discrepancies. This blog details how I tackled this problem by transforming a technically daunting challenge into a robust, maintainable data pipeline solution.

## Background and context

### What is trading PnL?

In trading, **Profit and loss (PnL)** represents financial outcomes:

- **Realized PnL**: The actual profit or loss from completed trades.
- **Unrealized PnL**: The potential gain or loss from open positions.

For instance, when you close a Bitcoin position at a higher price than you entered, your realized PnL reflects the profit after fees. If the position is still open, unrealized PnL tracks potential outcomes as prices fluctuate.

### Why does historical PnL matter?

Historical PnL data provides traders with:

1. **Performance insights**: Understanding which strategies worked and which didn’t.
2. **Compliance and reporting**: Regulatory or internal needs often require accurate historical data.
3. **Strategy validation**: Testing new algorithms against past market conditions relies on accurate PnL records.

### The problem at hand

While designing a trading PnL chart for my platform, a significant gap emerged: historical PnL data for certain periods was missing. The existing system calculated PnL in real-time but didn’t store intermediary data, making reconstruction impossible without extensive changes to the codebase.

## The challenge

> “How can we reconstruct historical trading PnL data efficiently when the original records no longer exist?”

This question encapsulated two core issues:

1. **Data loss**: Real-time calculations discarded intermediary steps, leaving gaps in historical records.
2. **System complexity**: The platform’s codebase, written in Elixir, was intricate, tightly coupled, and unfamiliar to me.

Moreover, the system’s reliance on multiple data sources (trades, market prices, fees) and the sheer volume of transactions compounded the problem.

## Technical requirements

To reconstruct PnL, the following were essential:

- **Historical trade data**: Information about each executed trade, including sizes, directions, and timestamps.
- **Market prices**: Historical price data to calculate unrealized PnL.
- **Fee details**: Trading commissions, funding rates, and other costs affecting PnL.
- **Efficient processing**: Handling massive datasets without overloading system resources.

## System analysis

### From complex code to data flows

Instead of delving into intricate application logic, I reimagined the system as a series of **data flows**, where data is ingested, transformed, and stored across multiple layers. Below is the existing flow:

```mermaid
flowchart LR
    subgraph Sources
        B[Binance]
        R[REST API]
    end

    subgraph DataLake
        ETS[Elixir ETS]
    end

    subgraph Processing
        MV[Materialized Views]
        DB[(PostgreSQL)]
    end

    subgraph Output
        A[Analytics]
        Rep[Reports]
    end

    B --> ETS
    R --> ETS
    ETS --> DB
    DB --> MV
    MV --> A
    MV --> Rep
```

- **Data sources**: Trading data originates from Binance (market data, trades) and a REST API.
- **Temporary storage**: Elixir ETS stores raw data temporarily before processing.
- **Transformation**: Postgres stores normalized data, which is further refined using materialized views for specific use cases.
- **Outputs**: Processed data powers analytics and reporting tools.

### Reconstructing the flow

From the above flow of data, we can easily determine which parts of the flow we should reproduce to find the old PnLs.

- Firstly, data comes from Binance
- Second, data passes through ETS before processing
- Finally, data is transformed to missing PnL and stored in Postgresql DB

One more important thing is the formula to calculate PnL when transforming data in the 3rd step.

- For the realized PnL that represents confirmed gains or losses from closed trades and affects your actual cash balance. The formula is:

  ```go
  Realized PNL = Σ(closed trade realized PNL)
  			   - Σ(commission, funding, insurance)
  ```

- For the unrealized PnL, it is the potential profit or loss from open trades. In the simple way, we can calculate it via the following formula
  ```go
  Unrealized PNL = Position Size * Direction of Order * (Mark Price - Entry Price)
  ```

There are many things that must be reproduced. But we will not implement all of them completely. Some useful data stored in the Postgresql DB can be reused. Let’s check!

- Trading positions information that contain:
  - Open trade history as `user_trades`
    - Realized PNL of closed trades
    - Commission fee for trades
    - Historical data contains the price, and quantity of assets when open, and close trades. These data can be used to calculate the average entry price at the time proper trade is opened or closed.
  - Transfer, profit, and fee as `future_incomes`
    - Funding fee
    - Locked positions commission fee
    - Insurance fee (for the future)

Comparing to the available data to the above formulas, we can see that everything is enough to calculate the unrealized PnL without Binance. But Binance is needed in retrieving the old marking prices to calculate historically unrealized. So we can illustrate the new flow as follows.

```mermaid
flowchart LR
    subgraph Input
        B[Binance API]
        DB[(Trading Info DB)]
    end

    subgraph Processing
        P[Data Mapping]
        C[Parallel Processing]
    end

    subgraph Storage
        PNL[(PnL Storage)]
    end

    B -->|Price Data| P
    DB -->|Trading History| P
    P --> C
    C -->|Results| PNL
```

## Implementation

The reconstruction process involves five major steps:

1. **Data collection** Fetch necessary data from two sources:

   - **Database**: Historical trading data, fees, commissions.
   - **Binance API**: Historical Kline data for price points.

2. **Mapping data** Group the data by trading pairs (tokens) for efficient processing.

3. **Token-level calculation** For each token:

   - Use minute-level Kline data to calculate fees, realized PnL, unrealized PnL, and entry prices.
   - Apply cumulative calculations to ensure accuracy.

4. **Aggregate results** Sum PnL across all positions for the user’s account.

5. **Storage and visualization** Save results back into the database and visualize them in the PnL chart.

```mermaid
flowchart TD
    subgraph DataCollection
        F[Fees Data]
        T[Trade Data]
        K[Kline Data]
    end

    subgraph Processing
        M[Map by Token]
        C[Calculate per Token]
        A[Aggregate Results]
    end

    F --> M
    T --> M
    K --> M
    M --> C
    C --> A
```

## Outstanding challenges

**Volume of data** Minute-level Kline data is essential for accuracy, but retrieving and processing it is resource-intensive:

- One month of data requires **43,200 points per token**.
- Accounts with **300+ open positions** significantly amplify the workload.
- Binance API limits Kline data to **1,500 points per request**, introducing additional complexity.

**PnL accuracy** PnL, specifically realized PnL, is stuck to the trade set to help us know the total PnL of this trade set by accumulating the closed trade PnL and fee over time. So if we retrieve the list of user trades randomly, it may produce the wrong PnL and let our report make nonsense.

## Optimization strategies

- **Time-series state reconstruction**: Our trading events naturally fall into their proper timeline. Each trade, fee, and price change finds its proper place in the chronological sequence. So our system can reconstruct a trading position's PnL at any moment.
- **Map-reduce**: As mentioned above, an account needs a long time to process. So forcing all our data through a single filter is impossible. The real benchmark test takes me about 5 hours to recover 1 trade set. By creating mapping by trading pairs, the data can be processed in parallel and done in minutes.
- **Parallel processing**: Awakening that each token has its own PnL let us think about trying to process them in parallel to reduce the time cost.
- **Running total**: Basically, each trade set is considered done once all positions of the account are closed completely. It means from a specific time, we can track the cumulative sum of positions (both long and short) for each account. A trade cycle is complete when the cumulative quantity equals zero. So we can update the flow a bit to resolve the missing trade set problem.

```mermaid
flowchart TD
    TS[Find Trade Sets]
    P[Process Each Set]
    C[Calculate PnL]

    TS --> P
    P --> C
```

---

## Quality assurance

To validate the reconstruction process:

- **Unit testing**: Test with small data samples, validating each step with detailed logs.
- **Cross-validation**: Compare reconstructed PnL with existing records in the database.
- **Visual analysis**: Render reconstructed data onto charts to ensure trends align with expected strategies.

## Conclusion

This case study highlights the power of a **data-centric approach** in solving financial system problems. By treating the challenge as a structured data pipeline problem, we avoided risky codebase modifications and developed a robust, scalable solution. Techniques such as parallel processing, time-series reconstruction, and efficient data retrieval were key to solving the problem within system constraints.

This approach demonstrates that **data-driven solutions** can effectively address complex challenges while maintaining flexibility and performance for future needs.


A detailed look at how we rebuilt historical trading PnL data through an efficient data pipeline approach, transforming a complex problem into a maintainable solution.

Reconstructing historical trading PnL: a data pipeline approach

Binance is one of the most popular Decentralized Exchanges worldwide, so the demand for building Binance-integrated applications is growing daily. My team is also onboarding. We have a deal that requires us to build a Binance trading application with the ability to trade on multiple accounts simultaneously. In this way, our clients can optimize their trading progress as much as possible.

Everything worked well at the beginning, motivating the clients to increase the amount of trading accounts and assets. The nightmare came at this moment. The funds began transferring between accounts to balance the strategies, making the client hard to control the fund and its flow. They must log in to each Binance account to track the transfer history manually. This behavior looks bad.

This emergency lets us begin record every transfers between accounts in the system, then notify to the clients continuously.

## Limitations of Binance income history

To record every transfers, we need the help of Binance APIs, specifically is [Get Income History (USER_DATA)](https://developers.binance.com/docs/derivatives/usds-margined-futures/account/rest-api/Get-Income-History). Once calling to this endpoint with proper parameters, we can retrieve the following `JSON` response.

```JSON
[
	{
 	"symbol": "",					// trade symbol, if existing
 	"incomeType": "TRANSFER",	// income type
 	"income": "-0.37500000", // income amount
 	"asset": "USDT",				// income asset
 	"info":"TRANSFER",			// extra information
 	"time": 1570608000000,
 	"tranId":9689322392,		// transaction id
 	"tradeId":""					// trade id, if existing
	},
	{
 		"symbol": "BTCUSDT",
 	"incomeType": "COMMISSION",
 	"income": "-0.01000000",
 	"asset": "USDT",
 	"info":"COMMISSION",
 	"time": 1570636800000,
 	"tranId":9689322392,
 	"tradeId":"2059192"
	}
]
```

_Code 1: JSON response of Binance API Get Income History (USER_DATA)_

Our job is just passing `TRANSFER` as `incomeType` to filter out other types of Binance transactions. Then we can store these records for use later. But when looking at this response, can you imagine the limitations that I mentioned in the title of this part? Yes! you actually can't know where the fund comes from or move to? Just only can detect whether it is a deposit or withdrawal by using the sign, which is not enough in our system where every account is under our control. If it is hard for you to understand, the result of the transfer notification is look sus as below screenshot.

![Sporadic and confusing transfer logs](assets/nghenhan-bad-logging.png) _Figure 1: Sporadic and confusing transfer logs that lack clear relationships between transactions_

To me, it looks bad. Ignore the wrong destination balance because of another issue with the data, this logging is too sporadic, hard to understand, and confusing. We can't understand how the fund is transferred. In my expectation, at least, it should like following.

![Clear and connected transfer logs showing fund flow between accounts](assets/nghenhan-better-logging.png) _Figure 2: Clear and connected transfer logs that show the complete flow of funds between accounts_

If you pay attention to the `JSON` response of Binance API, an idea can be raised in your mind that "_Hmm, it looks easy to get the better version of logging by just only matching the transaction ID aka tranId field value_". Yes, it is the first thing that popped into my mind. Unfortunately, once the transfer happens between two accounts, different transaction IDs are produced on each account side.

## Our approach to transfer history mapping

### Current implementation

It can make you a bit of your time at the beginning when looking at the response of Binance API and ask yourself "Why does Binance give us a bad API response?". Bit it is not a dilemma. And Binance API is not as bad as when I mentioned it. This API serves things enough for its demand in the Biance. And more general means can serve more use cases at all.

Enough to explain, now, we get to the important part: matching transfers to make the transfer history logging becomes more robust. I think we have more than two ways to do it. But because this issue comes from a data aspect, we will use a database solution to make it better.

Of course, we need to know the current query first. But it is inconvenient when sharing the source code here. So I will use a flow chart to replace it. This chart can also help us easy to imagine what's happening. It is easy, but the real query is not just to get from transfer history and show everything directly. To know the balance change, one needs to do some additional steps.

```mermaid
flowchart LR
 subgraph Input
 FI[Future Incomes]
 ACBS[Balance Snapshots]
 end

 subgraph Processing
 TD[Transfer Data]
 TT[Transfer Time]
 BB[Before Balance]
 AB[After Balance]
 end

 subgraph Output
 FR[Final Record]
 end

 %% Data collection
 FI --> TD
 TD --> TT

 %% Balance processing
 TT --> ACBS
 ACBS --> BB
 ACBS --> AB

 %% Final calculations
 BB --> FR
 AB --> FR
 TD --> FR
```

_Figure 3: Current flow to build transfer history_

The flow chart above shows how the current system produced transfer tracking logging.

- From `Future Incomes`, we simply query transfer information such as amount, time, and its sign.
- Using the time of transfer, query `Balance snapshots` to detect balance before and after it is changed by the transfer.

### How to make it better?

To do it better, we need to match the transfers together to know the source and destination of the fund. To match the transfers together, we need to specify what is the transfer before and after it (**with the assumption that transfers of the same fund on the send and receive side happen in a small gap of time, and two transfers can't happen in the same time**). We are lucky that Postgresql provides us with two convenient window functions, LEAD and LAG. LEAD is used to access a row following the current row at a specific physical offset. On the other hand, LAG helps with previous row access. With simple syntax and better performance, it is our choice to do transfer paring.

```sql
WITH matched_transfers AS (
 SELECT
 ...,
 LEAD(...) OVER (ORDER BY fi.time) AS next_...,
 LAG(...) OVER (ORDER BY fi.time) AS prev_...,
```

_Code 2: SQL query to match transfer by using LEAD and LAG_

Once we match each transfer with its previous and follows, we can easily detect type of each transfer by following script.

```sql
CASE
 WHEN amount < 0
 AND next_amount > 0
 AND (amount + next_amount = 0)
 AND (next_time - time < interval '5 seconds')
 THEN 'INTERNAL_TRANSFER'
```

_Code 3: SQL query to detect type of each transfer depend on it transaction before and after it_

It is not enough, we can list the following types, and each type has a separate way of detecting:

- Internal transfers (between accounts)
- External transfers out (withdrawals)
- External transfers in (deposits)

Everything is fine, from the two above queries, we can produce the record of the transfer with sender and receiver information. But don't miss the balance change. To do it, we need to select proper before and after balances depending on the time of transfer. Imagine we have 100 transfers, and the total amount of records of balance snapshot reaches million or more, it is a real nightmare.

There is a more subtle way. We can group close transactions of the same account together into a group, then just only need to query the balance of the account at the beginning of the group and calculate other balances by accumulating the amount.

```sql
SUM(CASE
 WHEN sender_time_gap > interval '20 seconds' THEN 1
 ELSE 0
END) OVER (
 PARTITION BY from_account
 ORDER BY time
) AS sender_group
```

_Code 4: SQL to group transfers of the same account and order by time_

```sql
FIRST_VALUE(
 COALESCE(
 (SELECT current_balance
 FROM account_current_balance_snapshots bs
 WHERE bs.account_id = from_account
 AND bs.created_at <= time
 ORDER BY bs.created_at DESC
 LIMIT 1),
 0
 )
) OVER (...)
```

_Code 5: SQL to find balance for the first record of each transfer group that is the result of Code 4_

Now, we have transfer history, in this, each record has its type, and information of the records before and after it. These records are also grouped together, and the leader of each group has its balanced information. Everything readies for querying the final result. Before going to the result, we may be missing a important step that is calculate balance for each transfer in the transfer group. To do it, Postgresql provides us some other interesting window functions. Tale a look following code.

```sql
GREATEST(0, (
 sender_initial_balance +
 SUM(signed_amount) OVER (
 PARTITION BY from_account, sender_group
 ORDER BY time
 ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW
 )
))
```

_Code 6: SQL to calculate balance for each transfer in the transfer group by using window functions ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW_

Let's break down the window frame `ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW`:

- `UNBOUNDED PRECEDING` means "start from the very first row in the partition". In our case, it starts from the first transfer in the group
- `CURRENT ROW` specifies "up to the transfer we're currently calculating"
- Together, they create a sliding window that grows as we move through the transfers, always starting from the first transfer and including all transfers up to the current one

This black magic save us from the danger from self join and `RECURSIVE` when calculating the accumulated total.

After all, every we are building can be wrapped in the below chart.

```mermaid
flowchart TD
 subgraph Input["Data Sources"]
 FI[Future Incomes]
 ACBS[Balance Snapshots]
 end

 subgraph Processing["Enhanced Processing"]
 RT["Transfers Pairing (LEAD/LAG Analysis)"]
 TWT["Type Detection (Internal/External Classification)"]
 TWG["Transfer Grouping (Time-Based Clustering)"]
 GFB["Find Initial Balance (Starting States)"]
 TWB["Calculate Balances (Running Totals)"]
 end

 subgraph Output["Enhanced Output"]
 FR["Final Record: - Paired Transfers - Balance Changes - Transfer Types - Time Relationships"]
 end

 %% Data flow
 FI --> RT
 RT --> TWT
 TWT --> TWG
 TWG --> GFB
 ACBS --> GFB
 GFB --> TWB
 TWB --> FR
```

_Figure 4: Upgraded process to build transfer history_

## Conclusions

From the problem to the idea and finally is the implementation, nothing is too difficult. Every normal software developer can do it even better. But to do the huge thing, we first should begin from the smaller and make it done subtly and carefully. From this small problem, I learned some things:

- **The answer may lie in the question itself.** Instead of blaming Binance API for being so bad, we can take a sympathetic look at it, and see if there is anything we can get out of it.
- **One small change can make everything better.** When comparing the original transfer tracking log, and the version after upgrading with some small changes in the DB query, there is a huge difference when seeing the new one. This reminds uss that impactful solutions don't always require complex architectures – sometimes they just need careful refinement of existing approaches.
- **Data challenges are often best addressed through data-driven solutions**. Rather than seeking fixes elsewhere, the key is to leverage the inherent patterns and structure within the data itself.

A deep dive into building a robust transfer tracking syste m for Binance accounts, transforming disconnected transaction logs into meaningful fund flow narratives through SQL and data analysis

Building better Binance transfer tracking


## Why Devbox? The big picture

- [The game-changing reasons to choose Devbox](introduction/the-reason-for-being.md)

## The building blocks

- [Devbox: Supercharge your development workflow](introduction/devbox.md)
- [Devbox services: Tame complex setups with ease](introduction/devbox-services.md)
- [Devbox plugins: Extend and customize like a pro](introduction/devbox-plugins.md)

## From zero to hero

- [Craft your perfect shell with Devbox](guide/run-your-own-shell.md)
- [Devbox.json: The blueprint of your dev environment](guide/devbox-json.md)
- [Beyond containers: Devbox's containerless revolution](guide/containerless.md)

## What's Next?

- [Reinventing Docker builds with Nix and Devbox](research/nix-in-building-docker-images.md)

## Our story

- [Life before Docker: The old ways](story/devbox-a-world-before-docker.md)
- [Docker: The promise and the pain](story/devbox-docker-adoption-and-challenges.md)
- [Enter Nix and Devbox: Our eureka moment](story/devbox-nix-and-our-devbox-adoption.md)
- [Devbox in action: Transforming our local dev setup](story/devbox-local-development-env.md)
- [Devbox in production: Our success story](story/devbox-production-success-story.md)

## Deep dives: For the curious

- [Nix shell: The foundation of rock-solid environments](introduction/nix-shell.md)
- [Nix flakes: Package management reimagined](introduction/nix-flakes.md)
- [Devbox vs Nix: Simplicity meets power](introduction/why-devbox-but-not-nix.md)


A comprehensive guide to Devbox: The tool that creates isolated, reproducible development environments anywhere, without Docker or VMs, powered by Nix.

§ Devbox


Remember when setting up a new project felt like assembling IKEA furniture blindfolded? Yeah, those days are over. Here's how Devbox revolutionized our workflow, from dev to production.

## The big picture: Devbox across our projects

We didn't just dip our toes into Devbox - we dove in headfirst. **Two main projects** and a handful of side projects later, we're swimming in efficiency.

## The spotlight

### Dev environment: from hours to minutes

Before Devbox:

1. Clone repo
2. Install dependencies (pray for version compatibility)
3. Set up databases
4. Configure environment variables
5. Sacrifice a goat to the dev gods
6. Maybe start coding

With Devbox:

1. Clone repo
2. `devbox shell`
3. Start coding

Yeah, it's that simple. Our `devbox.json` looks something like this:

```json
{
  "packages": ["nodejs@14", "postgresql@13", "redis@6"],
  "shell": {
    "init_hook": ["npm install", "npm run db:setup"]
  }
}
```

New team member? "Here's the repo, run `devbox shell`." Boom, they're productive on day one.

### Deployment: Dockerfile? More like DockerSMILE

Remember the days of handcrafting Dockerfiles? Neither do we. Now we just:

```bash
devbox generate dockerfile
```

And we get a Dockerfile that's perfectly synced with our dev environment. No more "works on my machine" syndrome. If it works in dev, it works in prod.

## Leveling up: custom flakes for the win

We didn't stop at basic Devbox usage. We dove into the world of Nix Flakes to create custom, reusable configurations. Here's one of what we cooked up:

```nix
{
  description = "A flake that adds the timescaledb extension to Postgresql";

  inputs = {
    nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable";
    flake-utils.url = "github:numtide/flake-utils";
  };

  outputs = { self, nixpkgs, flake-utils }:
    flake-utils.lib.eachDefaultSystem (system:
      let
        pkgs = import nixpkgs {
          inherit system;
          config.allowUnfree = true;
        };
        psqlExtensions = [
          "timescaledb"
        ];
      in {
        packages = {
          postgresql = pkgs.postgresql_15.withPackages (ps:
              (map (ext: ps."${ext}") psqlExtensions));
        };

        defaultPackage = self.packages.${system}.postgresql;
      });
}
```

This flake is our secret weapon for projects needing PostgreSQL with TimescaleDB. It's like ordering a custom pizza - PostgreSQL is the base, and TimescaleDB is the topping we always want.

Why is this cool? Let me count the ways:

- **Consistency**: Every developer gets the exact same PostgreSQL setup, TimescaleDB included.
- **Flexibility**: Need to add another extension? Just add it to psqlExtensions. It's that easy.
- **Portability**: This flake works on any system Nix supports. Linux, Mac, doesn't matter.
- **Versioning**: We're using PostgreSQL 15 here. Want to test with 14? Just change one number.

But the real magic happens when we use this in a Devbox project. We just add this to our devbox.json:

```json
{
  "packages": ["path:/path/to/our/postgresql-flake"]
}
```

## The secret sauce: our internal service repo

We created a central repository of Devbox configurations for popular services. This isn't your grandma's config file - it's a Swiss Army knife for dev environments. Take a look at this beauty:

```json
// devbox.json
{
  "$schema": "https://raw.githubusercontent.com/jetpack-io/devbox/0.10.1/.schema/devbox.schema.json",
  "packages": {
    "github:NixOS/nixpkgs#darwin.apple_sdk.frameworks.CoreText": "",
    "nodejs": "18",
    "pnpm_8": "8.15.9",
    "pkg-config": "0.29.2",
    "pango": {
      "version": "1.52.2",
      "outputs": ["dev"]
    },
    "libpng": "1.6.43",
    "giflib": "5.2.2",
    "librsvg": {
      "version": "2.58.2",
      "outputs": ["dev"]
    },
    "python3": "3.11.9",
    "pixman": "0.43.4",
    "cairo": {
      "version": "1.18.0",
      "outputs": ["dev"]
    },
    "libjpeg": {
      "version": "3.0.3",
      "outputs": ["dev"]
    },
    "elixir": "1.15.7",
    "github:baenv/timescalepg-fake#postgresql": "",
    "redis": "7.2.5",
    "redis-plus-plus": "1.3.12",
    "apacheKafka": {
      "version": "2.13-3.8.0",
      "outputs": ["out"]
    },
    "kafkactl": "5.0.6",
    "zookeeper": "3.9.2"
  },
  "env": {
    "PKG_CONFIG_PATH": "$DEVBOX_PACKAGES_DIR/lib/pkgconfig",
    "KAFKA_CONFIG": "$DEVBOX_PACKAGES_DIR/config"
  },
  "shell": {
    "init_hook": [
      "echo 'Welcome to devbox!' > /dev/null",
      ". $VENV_DIR/bin/activate",
      "sudo cp $KAFKA_CONFIG/zookeeper.properties $KAFKA_CONFIG/zoo.cfg"
    ],
    "scripts": {
      "zookeeper": "sudo $DEVBOX_PACKAGES_DIR/bin/zkServer.sh --config $KAFKA_CONFIG start-foreground",
      "zookeeper-daemon": "sudo $DEVBOX_PACKAGES_DIR/bin/zkServer.sh --config $KAFKA_CONFIG start",
      "zookeeper-stop": "sudo $DEVBOX_PACKAGES_DIR/bin/zkServer.sh --config $KAFKA_CONFIG stop",
      "zookeeper-status": "sudo $DEVBOX_PACKAGES_DIR/bin/zkServer.sh --config $KAFKA_CONFIG status",
      "zookeeper-kill": "sudo kill $(ps aux | grep zookeeper | awk '{print $2}')",
      "kafka": "sudo $DEVBOX_PACKAGES_DIR/bin/kafka-server-start.sh $KAFKA_CONFIG/server.properties",
      "kafka-daemon": "sudo $DEVBOX_PACKAGES_DIR/bin/kafka-server-start.sh -daemon $KAFKA_CONFIG/server.properties",
      "kafka-stop": "sudo $DEVBOX_PACKAGES_DIR/bin/kafka-server-stop.sh $KAFKA_CONFIG/server.properties"
    }
  },
  "include": [
    "plugin:postgresql"
  ]
}

// process-compose.yaml
version: "0.5"

processes:
  postgresql:
    command: |
      rm -rf ${PGDATA}
      initdb --username=postgres
      if ! grep -q "shared_preload_libraries = 'timescaledb'" $PGDATA/postgresql.conf; then
        echo "shared_preload_libraries = 'timescaledb'" >> $PGDATA/postgresql.conf
      fi
      pg_ctl start -o "-k $PGHOST"
    is_daemon: true
    shutdown:
      command: "pg_ctl stop -m fast"
    availability:
      restart: "always"
    readiness_probe:
      exec:
        command: "pg_isready"

  redis:
    command: "redis-server $REDIS_CONF --port $REDIS_PORT"
    availability:
      restart: on_failure
      max_restarts: 5

  zookeeper:
    command: "devbox run zookeeper-daemon"
    is_daemon: true
    shutdown:
      command: "devbox run zookeeper-stop"
    availability:
      restart: "always"
    readiness_probe:
      exec:
        command: "sudo devbox run zookeeper-status | grep -q 'Mode: standalone'"
        interval: 20s
        timeout: 20s
        retries: 5

  kafka:
    command: "devbox run kafka-daemon"
    is_daemon: true
    shutdown:
      command: "devbox run kafka-stop"
    availability:
      restart: "always"
    depends_on:
      zookeeper:
        condition: process_healthy
    readiness_probe:
      exec:
        command: "kafkactl get brokers | grep -v 'connection refused'"
        interval: 10s
        timeout: 10s
        retries: 5
```

These are not just config files; they look like love letters to productivity. Let's break it down:

1. **Kitchen sink included**: Node.js, Python, Elixir, Redis, Kafka - it's like an all-you-can-eat buffet for developers.

2. **Version control**: Every package is pinned to a specific version. No more "it worked yesterday" syndrome.

3. **Cross-platform magic**: See that `darwin.apple_sdk` package? That's making sure our Mac users don't feel left out.

4. **Custom sauce**: We've got our own TimescaleDB-enabled PostgreSQL package in there. Because why settle for off-the-shelf when you can have gourmet?

5. **Environment setup**: PKG_CONFIG_PATH and KAFKA_CONFIG are preset. It's like having your IDE set up before you even open it.

6. **Script city**: Look at those Zookeeper and Kafka scripts. Starting a distributed system is now as easy as `devbox run kafka`.

7. **Plugin power**: We're using the PostgreSQL plugin, because why reinvent the wheel when you can turbocharge it?

Here's the kicker: Need a Kafka cluster for local testing? Just copy this file, run `devbox shell`, and boom - you're running a mini data center on your laptop.

Want to spin up a multi-service setup with Node.js, Python, and a sprinkle of Elixir? It's all there, ready to rock.

This isn't just configuration; it's constellation. All these services, perfectly aligned, ready to shine in your development universe.

Is it overkill for a simple React app? Maybe. But for our complex, multi-service architectures, this is the difference between a day of setup and a minute of magic.

## The bottom line

Devbox didn't just change our tools; it changed our culture. "It's too complex to set up locally" is no longer an excuse. "It works on my machine" is no longer a problem.

Is it perfect? No. Sometimes we still need to dive into Nix for complex setups. But for 90% of our needs, Devbox is our go-to.

Life's too short for bad dev environments. Make yours awesome with Devbox.


How we transformed our development and deployment process with Devbox.

Devbox in production: our success story


Containers are great, but sometimes they're overkill. With Devbox Services and Plugins, you can create a sleek, containerless environment for local development. Let's build a Golang backend to show you how it's done.

## Setting the stage

First things first, let's init a Devbox shell:

```bash
devbox init
```

This gives you a bare-bones `devbox.json`:

```json
{
  "$schema": "https://raw.githubusercontent.com/jetify-com/devbox/0.12.0/.schema/devbox.schema.json",
  "packages": [],
  "shell": {
    "init_hook": ["echo 'Welcome to devbox!' > /dev/null"],
    "scripts": {
      "test": ["echo \"Error: no test specified\" && exit 1"]
    }
  }
}
```

## Adding Golang to the mix

Fire up your Devbox shell and add Go:

```bash
devbox shell
devbox add go
```

Boom! Go is now installed in your project root. Let's check:

```bash
which go
# /Users/you/your-project/.devbox/nix/profile/default/bin/go
```

## The database dilemma

Let's say you've got a killer Go API for managing books. You try to run it:

```bash
go run main.go
# 2024/07/28 17:00:10 Error connecting to database: "dial tcp 125.235.4.59:5432: connect: operation timed out"
# exit status 1
```

Oops! No database. But don't worry, Devbox has your back.

## PostgreSQL to the rescue

Add PostgreSQL to your Devbox:

```bash
devbox add postgresql
```

Devbox doesn't just install PostgreSQL. It sets up a whole environment:

- Creates a `process-compose.yaml` for you
- Sets up `PGHOST` and `PGDATA` environment variables
- Gives you commands to manage your database

Initialize your database:

```bash
initdb --username=yourusername
```

## Fire it up

Now you've got options to start your services:

- For background mode: `devbox services start`
- For monitoring mode: `devbox services up`

Create your database:

```bash
createdb bookstore --username=yourusername --password
# Enter your password when prompted
```

## The moment of truth

Update your Go code with the new database details:

```go
const (
    host     = "localhost"
    port     = 5432
    user     = "yourusername"
    password = "yourpassword"
    dbname   = "bookstore"
)
```

Run it:

```bash
go run main.go
# Successfully connected to the database!
# 2024/07/28 20:43:46 Starting server on :8080
```

## More than just PostgreSQL

Remember that `devbox.json` we started with? Forget it. Let's look at a real-world example that'll knock your socks off:

```json
{
  "$schema": "https://raw.githubusercontent.com/jetpack-io/devbox/0.10.1/.schema/devbox.schema.json",
  "packages": {
    "github:NixOS/nixpkgs#darwin.apple_sdk.frameworks.CoreText": "",
    "nodejs": "18",
    "pnpm_8": "8.15.9",
    "pkg-config": "0.29.2",
    "pango": {
      "version": "1.52.2",
      "outputs": ["dev"]
    },
    "libpng": "1.6.43",
    "giflib": "5.2.2",
    "librsvg": {
      "version": "2.58.2",
      "outputs": ["dev"]
    },
    "python3": "3.11.9",
    "pixman": "0.43.4",
    "cairo": {
      "version": "1.18.0",
      "outputs": ["dev"]
    },
    "libjpeg": {
      "version": "3.0.3",
      "outputs": ["dev"]
    },
    "elixir": "1.15.7",
    "github:baenv/timescalepg-fake#postgresql": "",
    "redis": "7.2.5",
    "redis-plus-plus": "1.3.12",
    "apacheKafka": {
      "version": "2.13-3.8.0",
      "outputs": ["out"]
    },
    "kafkactl": "5.0.6",
    "zookeeper": "3.9.2"
  }
  // ... (env and shell configurations omitted for brevity)
}
```

This isn't just a configuration file. It's a manifesto for containerless development.

## Breaking it down

Let's unpack this beast:

1. **Multiple languages**: Node.js, Python, and Elixir all living in harmony. No "it works on my machine" excuses here.

2. **Precise versioning**: Every package is pinned to a specific version. Reproducibility? Check.

3. **System libraries**: Pango, Cairo, libpng - we're not just installing runtimes, we're building a complete system.

4. **Databases and messaging**: PostgreSQL, Redis, and Kafka. A full backend stack without a single `docker-compose.yml` in sight.

5. **Custom packages**: See that `github:baenv/timescalepg-fake#postgresql`? That's a custom package pulled straight from GitHub. Try doing that easily with Docker.

## The magic of Devbox services

With Devbox Services, you're not just installing these packages - you're orchestrating them. Check out these scripts:

```json
"scripts": {
  "zookeeper": "sudo $DEVBOX_PACKAGES_DIR/bin/zkServer.sh --config $KAFKA_CONFIG start-foreground",
  "kafka": "sudo $DEVBOX_PACKAGES_DIR/bin/kafka-server-start.sh $KAFKA_CONFIG/server.properties",
  // ... more scripts omitted
}
```

Start Zookeeper and Kafka with a simple `devbox run zookeeper` and `devbox run kafka`. No Docker, no fuss.

## Why this matters

1. **Speed**: No container overhead means faster startup times and lower resource usage.
2. **Flexibility**: Need to add a system library? Just add it to your `devbox.json`. No need to rebuild a Docker image.
3. **Transparency**: Everything is defined in one file. No hidden layers, no mysterious base images.
4. **Reproducibility**: Every developer gets the exact same environment, down to the system libraries.

## The bottom line

Containers had their moment. But for local development, Devbox offers a level of control and simplicity that containers can't match. It's not just about running your code - it's about crafting the perfect environment for it to thrive.

Ready to leave containers behind? Give Devbox a shot. Your future self (and your team) will thank you.

## References

- [Devbox services guide](https://www.jetify.com/devbox/docs/guides/services/)
- [Devbox plugins guide](https://www.jetify.com/devbox/docs/guides/plugins/)
- [Creating custom Devbox plugins](https://www.jetify.com/devbox/docs/guides/creating_plugins/)


Build a lean, mean Golang backend without the container bloat. Here's how.

Ditch the containers: go containerless with Devbox

Ever wished you could clone your perfect dev environment? With `devbox.json`, you can. This little file is the beating heart of your Devbox setup. Let's crack it open and see what makes it tick.

![Devbox.json Configuration](assets/config-ref.webp)

## The anatomy of devbox.json

### Packages: Your toolbox

```json
"packages": [
 "rustup@latest",
 "libiconv@latest"
]
```

This is where you list all the packages you need. Think of it as your project's shopping list. Need Rust? Throw it in. Need a specific version? Just add `@<version>` after the package name.

**Pro tip:** Use `devbox search <package>` to find available versions.

### ENV: Set the stage

```json
"env": {
 "PROJECT_DIR": "$PWD"
}
```

Here's where you declare or override environment variables. It's like setting up the perfect lighting before a photoshoot - everything just works better.

### Shell: Your command center

```json
"shell": {
 "init_hook": [
 ". conf/set-env.sh",
 "rustup default stable",
 "cargo fetch"
 ],
 "scripts": {
 "build-docs": "cargo doc",
 "start": "cargo run",
 "run_test": [
 "cargo test -- --show-output"
 ]
 }
}
```

This is where the magic happens:

- `init_hook`: These commands run every time you fire up your Devbox shell. Perfect for setup tasks.
- `scripts`: Define your own commands here. It's like creating shortcuts for complex tasks.

### Include: Extend your powers

```json
"include": [
 "github:org/repo/ref?dir=<path-to-plugin>",
 "path:path/to/plugin.json",
 "plugin:php-config"
]
```

Need more firepower? Use `include` to add extra configurations:

- Pull plugins from GitHub
- Use local plugins
- Activate built-in plugins

## Put it all together

Here's what a fully-loaded `devbox.json` might look like:

```json
{
 "packages": ["rustup@latest", "libiconv@latest"],
 "env": {
 "PROJECT_DIR": "$PWD"
 },
 "shell": {
 "init_hook": [". conf/set-env.sh", "rustup default stable", "cargo fetch"],
 "scripts": {
 "build-docs": "cargo doc",
 "start": "cargo run",
 "run_test": ["cargo test -- --show-output"]
 }
 },
 "include": [
 "github:org/repo/ref?dir=<path-to-plugin>",
 "path:path/to/plugin.json",
 "plugin:php-config"
 ]
}
```

## The bottom line

Your `devbox.json` is more than just a config file - it's a blueprint for the perfect dev environment. Spend some time getting it right, and you'll save countless hours down the road.

Remember, a well-crafted `devbox.json` is like a good pair of shoes: it should fit perfectly and take you anywhere you want to go.

## References

- [Devbox.json configuration reference](https://www.jetify.com/devbox/docs/configuration/)
- [Devbox search command](https://www.jetify.com/devbox/docs/cli_reference/devbox_search/)
- [Devbox plugins guide](https://www.jetify.com/devbox/docs/guides/plugins/)

Master your Devbox environment with this no-nonsense guide to devbox.json

Devbox.json: Your project's DNA


So you've installed Devbox with that nifty one-liner:

```bash
curl -fsSL https://get.jetify.com/devbox | bash
```

Now what? It's time to run your own shell, and Devbox gives you two flavors: local and global. Let's break 'em down.

## Devbox local: isolation is bliss

Want to create a bubble where you can install and run anything without messing up your system? Devbox Local is your new best friend.

Here's how it works:

1. Navigate to your project:

   ```bash
   cd path/to/your/awesome/project
   ```

2. Initialize Devbox:

   ```bash
   devbox init
   ```

   This creates two magical files: `devbox.json` and `devbox.lock`. Think of them as the blueprint and the snapshot of your environment.

3. Add packages:

   ```bash
   devbox search go
   devbox add go
   ```

   Your `devbox.json` now looks something like this:

   ```json
   {
     "$schema": "https://raw.githubusercontent.com/jetify-com/devbox/0.12.0/.schema/devbox.schema.json",
     "packages": ["go@latest"],
     "shell": {
       "init_hook": ["echo 'Welcome to devbox!' > /dev/null"],
       "scripts": {
         "test": ["echo \"Error: no test specified\" && exit 1"]
       }
     }
   }
   ```

4. Fire up your shell:

   ```bash
   devbox shell
   ```

   Want true isolation? Use `devbox shell --pure`. It's like your shell went into witness protection - new identity, no baggage.

## Devbox global: your system, supercharged

Want to use Devbox as your primary package manager? Devbox Global has got your back.

Here's the secret sauce:

1. Add this to your shell's RC file (like `~/.zshrc` or `~/.bashrc`):

   ```bash
   eval "$(devbox global shellenv)"
   ```

2. Restart your terminal or run `source ~/.zshrc` (or whatever your RC file is).

3. Start adding packages globally:

   ```bash
   devbox global add go
   ```

4. If you see a warning about your shell being out of date, just run:

   ```bash
   refresh-global
   ```

Wondering where all this magic happens? Try:

```bash
which go
# or
devbox global path
```

You'll likely see something like `~/.local/share/devbox/global/default/`. That's where Devbox keeps its global configuration.

## The bottom line

Whether you go local or global, Devbox gives you the power to create the perfect development environment. No more "it works on my machine" nightmares. Just pure, predictable, awesome shells.

So what are you waiting for? Fire up Devbox and start building something amazing.

## References

- [Installing devbox](https://www.jetify.com/devbox/docs/installing_devbox/)
- [Create a dev environment with devbox](https://www.jetify.com/devbox/docs/quickstart/)
- [Use devbox as your primary package manager](https://www.jetify.com/devbox/docs/devbox_global/)
- [Nix shell: the foundation](../introduction/nix-shell.md)


Master the art of running your own shell with Devbox - locally or globally. No more environment headaches.

Devbox shell: your dev environment, your rules


Ever spent hours configuring a new package in your dev environment? Yeah, we've all been there. It sucks. That's why we created Devbox Plugins.

## The plugin revolution

Imagine this: You need to add Nginx to your project. Without plugins, you're in for a world of hurt - custom configs, environment variables, file management... ugh.

But with Devbox Plugins? It's a whole new ballgame:

1. Add Nginx to your project.
2. ...
3. That's it. You're done.

No, seriously. The plugin handles everything else:

- Slaps down a rock-solid default config
- Exposes the right env vars for easy tweaking
- Organizes config files so you're not playing hide-and-seek later
- Sets up a service so you can start/stop Nginx with a single command

And the best part? This all happens automagically when you add the package. No extra steps, no headaches.

## Plugin flavors: Built-in or build your own

We've got two types of plugins to suit your needs:

### 1. Built-in plugins: The easy button

These are our pre-baked plugins for popular packages. Nginx, PostgreSQL, Node.js - we've got you covered. Just add the package, and the plugin kicks in automatically.

Want to see what's available? [Check out our Built-in Plugins docs](https://www.jetify.com/devbox/docs/guides/plugins/#using-plugins).

### 2. Custom plugins: For the DIY crowd

Need something special? No problem. You can create your own plugins following our [dead-simple schema](https://www.jetify.com/devbox/docs/guides/creating_plugins/#plugin-design). Host them locally or on GitHub - whatever floats your boat.

## The plugin lifecycle: How the magic happens

Plugins aren't just static config files. They're active participants in your Devbox shell's lifecycle:

![Devbox Shell Lifecycle](assets/devboxshell_lifecycle.webp)

Every time you fire up a shell, run a script, or start a service, your plugins spring into action, making sure everything's set up just right.

## Anatomy of a plugin

Here's what a plugin looks like under the hood:

```
my-awesome-plugin/
├── README.md              # Because documentation matters
├── plugin.json            # The brains of the operation
├── config/
│   ├── my-plugin.conf     # Default configs
│   └── process-compose.yaml  # Service definitions
└── test/
    ├── devbox.json        # For testing your plugin
    └── devbox.lock
```

The star of the show is `plugin.json`. This is where you define what your plugin does, what packages it needs, what environment variables it sets - everything.

## The bottom line

Devbox Plugins aren't just a nice-to-have. They're a game-changer. They take the pain out of package setup, letting you focus on what really matters: building awesome stuff.

So the next time you're adding a package to your Devbox project, remember: there's probably a plugin for that. And if there isn't? Well, maybe it's time to build one.

## References

- [Devbox plugins guide](https://www.jetify.com/devbox/docs/guides/plugins/)
- [Creating custom Devbox plugins](https://www.jetify.com/devbox/docs/guides/creating_plugins/)
- [Nix package manager](https://nixos.org/)
- [Nginx documentation](https://nginx.org/en/docs/)


Discover how Devbox Plugins streamline your development workflow by automating package setup and configuration

Devbox plugins: Turbocharge your dev setup


Ever wished you could manage your development services as easily as with Docker Compose, but without all the Docker baggage? Say hello to Devbox Services, powered by process-compose.

## What's process-compose?

Think of process-compose as Docker Compose's leaner, meaner cousin. It's a dead-simple way to orchestrate your non-containerized apps. No more wrestling with Dockerfiles, volumes, or registries. Just define your services, and you're off to the races.

Here's what a `process-compose.yml` might look like:

```yaml
version: "0.5"

processes:
  postgresql:
    command: |
      if ! grep -q "shared_preload_libraries = 'timescaledb'" $PGDATA/postgresql.conf; then
        echo "shared_preload_libraries = 'timescaledb'" >> $PGDATA/postgresql.conf
      fi
      pg_ctl start -o "-k $PGHOST"
    is_daemon: true
    shutdown:
      command: "pg_ctl stop -m fast"
    availability:
      restart: "always"
    readiness_probe:
      exec:
        command: "pg_isready"

  telegram-bot:
    command: |
      cd telegram-bot
      make install
      make dev
    availability:
      restart: "always"
    depends_on:
      postgresql:
        condition: process_healthy
    readiness_probe:
      http_get:
        host: 127.0.0.1
        scheme: http
        path: "/healthz"
        port: 5001
      initial_delay_seconds: 5
      period_seconds: 10
      timeout_seconds: 5
      success_threshold: 1
      failure_threshold: 3
```

## Breaking it down

Let's dissect this beast:

### PostgreSQL process

```yaml
postgresql:
  command: |
    if ! grep -q "shared_preload_libraries = 'timescaledb'" $PGDATA/postgresql.conf; then
      echo "shared_preload_libraries = 'timescaledb'" >> $PGDATA/postgresql.conf
    fi
    pg_ctl start -o "-k $PGHOST"
  is_daemon: true
  shutdown:
    command: "pg_ctl stop -m fast"
  availability:
    restart: "always"
  readiness_probe:
    exec:
      command: "pg_isready"
```

This little chunk of YAML is doing some heavy lifting:

1. It checks if TimescaleDB is enabled. If not, it adds it to the config.
2. Fires up PostgreSQL with a custom socket directory.
3. Tells process-compose this is a long-running daemon.
4. Defines how to gracefully shut down PostgreSQL.
5. Sets it to always restart if it crashes.
6. Uses `pg_isready` to check if PostgreSQL is good to go.

### Telegram bot process

```yaml
telegram-bot:
  command: |
    cd telegram-bot
    make install
    make dev
  availability:
    restart: "always"
  depends_on:
    postgresql:
      condition: process_healthy
  readiness_probe:
    http_get:
      host: 127.0.0.1
      scheme: http
      path: "/healthz"
      port: 5001
    initial_delay_seconds: 5
    period_seconds: 10
    timeout_seconds: 5
    success_threshold: 1
    failure_threshold: 3
```

Here's where it gets interesting:

1. We're setting up a Telegram bot with a simple `make install && make dev`.
2. It'll restart if it crashes.
3. It won't start until PostgreSQL is healthy. No more race conditions!
4. We've got a fancy HTTP health check to make sure the bot is actually working.

## Devbox services: your new best friend

Now, here's the kicker: Devbox wraps all this process-compose goodness into a simple CLI. No need to fumble with process-compose directly. Just use these magic commands:

- `devbox services ls` - See what's cooking
- `devbox services restart` - Give your services a kick
- `devbox services start` - Fire everything up
- `devbox services stop` - Shut it all down

It's that easy. No containers, no fuss, just your services running smoothly in your Devbox environment.

## The bottom line

Devbox Services with process-compose gives you the power of containerized workflows without the overhead. It's perfect for development environments where you want simplicity and speed.

## References

- [process-compose Documentation](https://github.com/F1bonacc1/process-compose)
- [Devbox services guide](https://www.jetify.com/devbox/docs/guides/services)
- [PostgreSQL documentation](https://www.postgresql.org/docs/)
- [TimescaleDB documentation](https://docs.timescale.com/)


Discover how Devbox Services uses process-compose to wrangle your daemon applications without the container overhead

Devbox services: tame your daemons with process-compose


Remember the last time you onboarded a new developer? Or tried a new tool without wrecking your setup? Yeah, it probably sucked. But it doesn't have to.

## The dev environment dream

Imagine this:

- Your whole team works in identical environments
- You test new tools without fear
- Your dev setup is lightning fast
- Version conflicts? Ancient history
- Your perfect environment follows you everywhere

Sounds too good to be true? Meet Devbox.

## Why Devbox rocks

Devbox is like a magic wand for your command line. It conjures perfect dev environments in seconds. Here's why it's a game-changer:

1. **One file rules all:** Everything lives in a single `devbox.json`. Dependencies, env vars, init scripts – it's all there.

2. **Set it and forget it:** Define once, use forever. Devbox handles the rest.

3. **Consistency is king:** "Works on my machine" becomes "Works on every machine."

Here's what a `devbox.json` looks like:

```json
{
  "packages": ["nodejs@14", "python@3.9", "postgresql@13"],
  "env": {
    "DATABASE_URL": "postgresql://localhost/myapp"
  },
  "shell": {
    "init_hook": ["npm install", "flask db upgrade"]
  }
}
```

This gives everyone the same Node.js, Python, and PostgreSQL setup, plus any custom config you need.

## Nix power, Devbox simplicity

Under the hood, Devbox uses Nix. Don't worry – you don't need to learn Nix (but if you do, it opens up even more possibilities).

Want a package? Devbox gives you 80,000+ options from Nixpkgs. Need bleeding-edge stuff? Use Nix flakes directly. It's all seamless.

## Get started in 60 seconds

```bash
# Install Devbox
curl -fsSL https://get.jetify.com/devbox | bash

# Create a project
mkdir my-awesome-project && cd my-awesome-project
devbox init

# Add packages
devbox add nodejs python

# Fire it up
devbox shell
```

Boom. You've got a pristine environment with Node.js and Python, ready to rock.

## Why Not Just Nix?

Devbox is Nix with training wheels. It's for developers who want the power without the learning curve. Think of it as a race car with an automatic transmission.

Want to dive deeper?

- [Nix shell: the foundation](https://nixos.org/manual/nix/stable/command-ref/nix-shell.html)
- [Why choose Devbox over plain Nix](https://www.jetify.com/devbox/docs/devbox_vs_other_tools)

## The bottom line

Devbox isn't just a tool. It's a revolution in managing dev environments. It makes "works on my machine" a universal truth, not an excuse.

Is there a learning curve? Sure. But once you go Devbox, you'll wonder how you ever lived without it.

Ready to supercharge your dev setup? Give Devbox a shot. Your future self (and your team) will thank you.

## References

- [Devbox documentation](https://www.jetify.com/devbox/docs/)
- [Nix package manager](https://nixos.org/)
- [Nixpkgs repository](https://github.com/NixOS/nixpkgs)
- [Nix flakes](https://nixos.wiki/wiki/Flakes)


Forget setup headaches. Devbox delivers instant, rock-solid dev environments powered by Nix.

Devbox: your dev environment on steroids


Ever tried to share your perfect dev setup, only to hear "It doesn't work on my machine"? Nix Flakes is about to make that headache a thing of the past.

## What's a Flake?

Think of a Flake as a supercharged `package.json`. It's a self-contained unit that defines everything your project needs, from dependencies to build instructions. And just like `package-lock.json`, there's a `flake.lock` to keep everything pinned and reproducible.

## Why Should You Care?

1. **Reproducibility**: Same setup, every machine, every time. No more "but it works on my machine" excuses.
2. **Discoverability**: Share your setup as easily as sharing a Git repo. Others can use your work without headaches.
3. **Flexibility**: Customize existing packages without breaking a sweat.

## Flakes in action: the PostgreSQL Timescale saga

Let's say you need PostgreSQL with TimescaleDB. But oh no, there's no pre-built package! No problem. Here's how you'd whip one up with Flakes:

```nix
{
  description = "PostgreSQL on Steroids: Now with TimescaleDB!";

  inputs = {
    nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable";
    flake-utils.url = "github:numtide/flake-utils";
  };

  outputs = { self, nixpkgs, flake-utils }:
    flake-utils.lib.eachDefaultSystem (system:
      let
        pkgs = import nixpkgs {
          inherit system;
          config.allowUnfree = true;
        };
        psqlExtensions = [
          "timescaledb"
        ];
      in {
        packages = {
          postgresql = pkgs.postgresql_15.withPackages (ps:
              (map (ext: ps."${ext}") psqlExtensions));
        };

        defaultPackage = self.packages.${system}.postgresql;
      });
}
```

## Breaking it down

1. **Inputs**: We're grabbing the latest Nixpkgs and some handy utilities.
2. **Outputs**: This is where the magic happens. We're creating a custom PostgreSQL package for every system Nix supports.
3. **The secret sauce**: `postgresql = pkgs.postgresql_15.withPackages (...)` - This line is adding TimescaleDB to PostgreSQL.

## Why this rocks

1. **Customization made easy**: Need more extensions? Just add them to `psqlExtensions`.
2. **Universal**: Works on any system Nix supports. Linux, Mac, doesn't matter.
3. **Shareable**: Anyone can use this Flake to get the exact same PostgreSQL setup.

## The bottom line

Nix Flakes aren't just a feature; they're a revolution in package management. They make "works on my machine" a universal truth, not an excuse.

Ready to make your dev setup bulletproof? Give Nix Flakes a shot. It's time to take control of your dependencies.

## References

- [Nix flakes documentation](https://nix.dev/concepts/flakes.html)
- [Nix flake command reference](https://nix.dev/manual/nix/2.22/command-ref/new-cli/nix3-flake)
- [Reddit: explaining flakes](https://www.reddit.com/r/NixOS/comments/131fvqs/can_someone_explain_to_me_what_a_flake_is_like_im/)


Discover how Nix Flakes revolutionize package management with reproducibility and discoverability

Nix flakes: next-level package management


Ever wished you could clone your perfect dev setup to any machine? Or test a new package without breaking your system? That's where Nix Shell shines. It's not just a tool; it's a game-changer for developers.

## What Makes Nix Shell Special?

Nix Shell drops you into an isolated environment faster than you can say "dependency hell." Here's why it's revolutionary:

1. **Zero system pollution**: Play with packages all you want. Your host system stays pristine.
2. **Reproducible environments**: Same setup, every machine, every time. No more "works on my machine" excuses.
3. **Multiple versions, no conflicts**: Need Python 2 and 3? Node 12 and 16? No problem. Nix Shell handles it like a pro.

## How Nix pulls off this magic

### The Nix store: Fort Knox for your packages

Imagine a fortress where every package lives in its own unbreakable vault. That's the Nix store (usually at `/nix/store`). No package can mess with another. It's like giving each app its own private island.

### Hashed versions: every package gets a fingerprint

Nix doesn't just store packages; it gives each version a unique fingerprint. Look at these beauties:

```bash
/nix/store/1fxz1flmv4a4m5pvjmmzxlaznjzybjcp-go-1.21.3/
/nix/store/i04a1a6qgxhjw6c0ld2b3x1v815sbxjc-go-1.22.3/
```

That gibberish? It's a hash that represents everything about that specific version. Different versions live side by side, no fighting.

### User environments: your personal playground

Nix creates a sandbox just for you. Install packages without begging for sudo. It's your space, your rules.

## Nix shell: two flavors of awesome

1. **Team player mode**: Ditch the global `$PATH` for a pure, reproducible environment. Perfect for team projects where "it works on my machine" isn't good enough.

2. **Personal power-up mode**: Keep your global `$PATH` and supercharge your everyday shell. It's like your regular shell, but with extra muscle.

## The bottom line

Nix Shell isn't just a tool; it's a philosophy. It says, "Hey, your development environment should be rock-solid, portable, and pain-free."

Imagine never again fighting with conflicting versions. Picture spinning up the exact same environment on any machine in seconds. That's the Nix Shell promise.

Ready to give it a shot? Trust us, your dev life is about to get a whole lot smoother.

## References

- [Nix shell manual](https://nix.dev/manual/nix/2.22/command-ref/nix-shell)
- [Why you should try Nix](https://nixos.org/guides/nix-pills/01-why-you-should-give-it-a-try)
- [Nix package manager](https://nixos.org/)


Discover how Nix Shell creates rock-solid dev environments that work everywhere, every time.

Nix shell: bulletproof development environments


## The pursuit of consistency

With the rise of cloud services, we first adapted Docker as a containerization tool to wrap our application into an isolated environment for running on the cloud for production. Then we begin setting up a development environment inside a container or take advantage of docker-compose to run multiple services without installing locally. This helps us share the environment easily, and run the same environment repeatably with minimum changes with just only 1-2 script files.

## Docker's Achilles heel

### Bad build cache

Docker build is actually not very effective because basically, we need to rebuild the entire environment each time our code is changed when developing. This rebuild process not only affects the image layer where the code is placed but also lets other layer below it is rebuilt. It takes more time and resource to do it.

![alt text](assets/docker_layers.png)

### Pulling from internet is not stable

The installed application version can be specified in the `apt-get add` script, but its dependencies are not stable. Repositories can be updated, transitive dependencies are changed without changing the main package version. So once we run `apt-get update`, the produced image may not be the same as the previous build.

### Docker is heavy on non-Linux OS

On Windows and MacOS, Docker containers are run on a Linux VM. So we need to split resources from the host machine to serve this VM. It makes our computer become slow.
![alt text](assets/docker_container_run.png)

### How to resolve problems?

The problems can be summed up as slow, asynchronous, and cumbersome when developing locally. We actually can resolve them by using some Docker hack. But we can also do it by avoiding using Docker (or changing your machine if you want). Among a variety solutions, we consider trying using Devbox. What's it?

## Devbox

### Do not build, do not need VM

Powered by Nix which is known as a cross-platform package manager for Unix-like systems with the ability to build any package, or application running natively on your machine with 100.000+ available Nix packages. Devbox will simply scope an isolated workspace, and then install all dependencies to help you run your project. All these dependencies are native to your OS, so just only need to run without any other middleman.

### Consistent dependencies

All packages in the Devbox workspace are actually linked from your local Nix storage, where every application is installed. They are identified by a unique hash and are linked together by a dependency tree. So it will not change over time. So the Devbox environment can be reproduced perfectly and easily to share between teammates with just a script file.

![alt text](assets/nix_deps_graph.png)

The hash is different between versions of the same application. So we have different versions of a package in our Nix storage. And easily link it to different projects that require different versions of a package.

## Conclusion

Embracing Devbox has not only streamlined our development process but has also fostered a more collaborative and productive team dynamic. As we continue to scale our projects, Devbox's reproducibility and efficiency will undoubtedly remain key factors in our success.


The reason for being

Let's cut to the chase: Devbox is Nix with training wheels, and that's a good thing. Here's why we're all in on Devbox for building killer dev environments.

## The power of Nix, without the pain

Devbox is built on Nix. That means you get all the good stuff:

- Rock-solid isolated environments
- Reproducibility that'll make your ops team weep with joy
- The ability to run your setup on any machine, anywhere

But here's the kicker: you get all of this without having to learn Nix. It's like getting a race car with an automatic transmission.

## Devbox: Nix's cool, user-friendly cousin

Here's where Devbox shines:

1. **Developer-First**: Nix can do a million things. Devbox does one thing: create awesome dev environments. It's laser-focused on making your coding life better.

2. **Simple interface**: Compare these:

 Nix:

 ```nix
 { pkgs ? import <nixpkgs> {} }:
 pkgs.mkShell {
 buildInputs = [ pkgs.nodejs pkgs.yarn ];
 }
 ```

 Devbox:

 ```json
 {
 "packages": ["nodejs", "yarn"]
 }
 ```

 Which one would you rather explain to a new team member?

3. **No new language required**: Nix has its own programming language. It's powerful, sure, but do you really want to learn a new language just to set up your dev environment? With Devbox, if you can write JSON, you're golden.

## But What About...?

You might be thinking, "Sure, but what if I need Nix's advanced features?" Here's the secret: Devbox can tap into those too. Need a custom Nix derivation? Devbox has your back. Want to use Nix Flakes? Go for it.

The difference is that with Devbox, you start simple and add complexity only when you need it. With pure Nix, you're dealing with that complexity from day one.

## The bottom line

Devbox isn't just a tool; it's a philosophy. It says, "Hey, your dev environment should be powerful AND easy to set up."

Is Nix more powerful in the hands of an expert? Maybe. But Devbox makes everyone on your team an expert on day one. And in the real world, that's what matters.

Ready to make your dev setup both bulletproof and brain-dead simple? Give Devbox a shot. Your future self (and your team) will thank you.

## References

- [Devbox: improving on Nix](https://www.reddit.com/r/NixOS/comments/z97cwy/devbox_predictable_development_environments/)
- [Devbox on Hacker News](https://news.ycombinator.com/item?id=32600821)
- [Nix language documentation](https://nix.dev/manual/nix/2.18/language/)
- [Nix shell: the foundation](https://nixos.org/manual/nix/stable/command-ref/nix-shell.html)
- [Nix flakes: next-level package management](https://nixos.wiki/wiki/Flakes)

Discover why Devbox is the smart choice for developers who want Nix's power without the learning curve

Devbox vs Nix: why we chose simplicity

Let's cut to the chase: Devbox is Nix with training wheels, and that's a good thing. Here's why we're all in on Devbox for building killer dev environments.

## The power of Nix, without the pain

Devbox is built on Nix. That means you get all the good stuff:

- Rock-solid isolated environments
- Reproducibility that'll make your ops team weep with joy
- The ability to run your setup on any machine, anywhere

But here's the kicker: you get all of this without having to learn Nix. It's like getting a race car with an automatic transmission.

## Devbox: Nix's cool, user-friendly cousin

Here's where Devbox shines:

1. **Developer-First**: Nix can do a million things. Devbox does one thing: create awesome dev environments. It's laser-focused on making your coding life better.

2. **Simple interface**: Compare these:

 Nix:

 ```nix
 { pkgs ? import <nixpkgs> {} }:
 pkgs.mkShell {
 buildInputs = [ pkgs.nodejs pkgs.yarn ];
 }
 ```

 Devbox:

 ```json
 {
 "packages": ["nodejs", "yarn"]
 }
 ```

 Which one would you rather explain to a new team member?

3. **No new language required**: Nix has its own programming language. It's powerful, sure, but do you really want to learn a new language just to set up your dev environment? With Devbox, if you can write JSON, you're golden.

## But What About...?

You might be thinking, "Sure, but what if I need Nix's advanced features?" Here's the secret: Devbox can tap into those too. Need a custom Nix derivation? Devbox has your back. Want to use Nix Flakes? Go for it.

The difference is that with Devbox, you start simple and add complexity only when you need it. With pure Nix, you're dealing with that complexity from day one.

## The bottom line

Devbox says, "Hey, your dev environment should be powerful AND easy to set up."

Is Nix more powerful in the hands of an expert? Maybe. But Devbox makes everyone on your team an expert on day one. And in the real world, that's what matters.

Ready to make your dev setup both bulletproof and brain-dead simple? Give Devbox a shot.

## References

- [Devbox: improving on Nix](https://www.reddit.com/r/NixOS/comments/z97cwy/devbox_predictable_development_environments/)
- [Devbox on Hacker News](https://news.ycombinator.com/item?id=32600821)
- [Nix language documentation](https://nix.dev/manual/nix/2.18/language/)


Docker build is not deterministic. This means that with the same `Dockerfile`, different build times (or builds on different machines) can result in different outcomes.

This issue stems from the fact that _Docker build has access to the public internet_. While this ability is often necessary for pulling packages from repositories like Ubuntu's, it can also lead to some unexpected consequences. One of these consequences is mentioned above.

Some specific issues related to this problem include:

- [Unstable package installation](unstable-package-installation.md)
- [Shadow copies](shadow-copies.md)


An overview of the non-deterministic nature of Docker builds and related issues

Docker build issues


On different machines with different nixpkgs versions, Nix build will result different packages.

To handle this issue, Fixed-output derivation is raised. It simply means the result of the Nix build called derivation, is represented by a fixed hash. Any change when re-building will let the hash change. Once the current hash and the new hash are different, the build fails.

The question is how can we know this hash in the first build time? Simply, just mock a random value, then run build to get the right hash in the error message as below.

```bash
hash mismatch in fixed-output derivation '/nix/store/...':
  wanted: sha256-AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=
  got:    sha256-ACTUALCORRECTHASHHEREXXXXXXXXXXXXXXXXXXXXX=
```

This mechanism does not just only prevent Nix build from wrong global nixpkgs but also helps on tracking whether something, is
downloaded from the internet, is right.

As you can see, even when Nix has mechanisms to decrease the risk of downloading resources from unreliable sources, it also can
control everything downloaded from the Internet, and can be used to build and result the same package at any time.

---

#### References

_Nix: what are fixed-output derivations and why use them?_ (2023, February 24). Brian McGee. Retrieved August 2, 2024, from
https://bmcgee.ie/posts/2023/02/nix-what-are-fixed-output-derivations-and-why-use-them/

_Advanced Attributes - Nix Reference Manual_. (n.d.). nix.dev. Retrieved August 2, 2024, from https://nix.dev/manual/nix/2.18/
language/advanced-attributes.html?highlight=outputHash


An explanation of fixed-output derivations in Nix and their role in ensuring reproducible builds

Fixed-output derivation in Nix


Nix gives us the ability to build Docker image with minimum deterministic by providing some solid mechanisms.

Firstly, using Nix, we can specify exactly the version of each package and its dependencies, and keep it completely the same each time the package build happens. It is called reproducible build.

To achieve it, Nix originally combined two mechanisms, Pinning nixpkgs and Fixed-output derivation.

- [Pinning nixpkgs](pinning-nixpkgs.md)
- [Fixed-output derivation](fixed-output-derivation.md)

Above is not all. Nix raised a new feature called [Flake](../introduction/nix-flakes.md) that has input and output that are determined exactly. To do it, Flake defines a standard structure for a Nix project where input (dependencies) and output are specified and locked. So your build now can avoid the dependency on the global state of Nix.


An overview of how Nix enables deterministic Docker image builds through reproducible builds, pinning, and Flakes

Nix - build the same thing at any time

Docker's great, but let's face it: building images can be a pain. Enter Nix, the tool that's about to change your Docker game forever.

## The Docker dilemma

Before we dive into Nix, let's talk about why Docker image building can be such a headache:

1. **Inconsistent builds**: Ever had an image work on your machine but fail in CI? Yeah, we've all been there.
2. **Bloated images**: Why are your images so big? Probably because you're carrying around a bunch of stuff you don't need.
3. **Slow builds**: Waiting for builds is like watching paint dry, except less exciting.
4. **Dependency hell**: Managing dependencies in Docker can feel like juggling chainsaws.

## Nix to the rescue

Nix isn't just a package manager; it's a philosophy. Here's how it solves Docker's biggest pain points:

### 1. Build the same thing, every time

With Nix, "works on my machine" becomes "works on every machine." Here's why:

- **Deterministic builds**: Nix locks down every dependency, right down to the system libraries. Same inputs always equal the same outputs.
- **Reproducibility**: Build an image today,
 Claude can make mistakes. Please double-check responses.

Docker's great, but let's face it: building images can be a pain. Enter Nix, the tool that's about to change your Docker game forever.

## The Docker dilemma

Before we dive into Nix, let's talk about why Docker image building can be such a headache:

1. **Inconsistent builds**: Ever had an image work on your machine but fail in CI? Yeah, we've all been there.
2. **Bloated images**: Why are your images so big? Probably because you're carrying around a bunch of stuff you don't need.
3. **Slow builds**: Waiting for builds is like watching paint dry, except less exciting.
4. **Dependency hell**: Managing dependencies in Docker can feel like juggling chainsaws.

## Nix to the rescue

Nix isn't just a package manager; it's a philosophy. Here's how it solves Docker's biggest pain points:

### 1. Build the same thing, every time

With Nix, "works on my machine" becomes "works on every machine." Here's why:

- **Deterministic builds**: Nix locks down every dependency, right down to the system libraries. Same inputs always equal the same outputs.
- **Reproducibility**: Build an image today, next week, or next year - you'll get the same result.
- **Version pinning**: No more "latest" tag roulette. Nix lets you specify exact versions of everything.

### 2. Lean, mean, shipping machines

Nix doesn't just build images; it builds efficient images:

- **Minimal images**: Nix includes only what you need, nothing more. Your images go on a diet without even trying.
- **Deduplication**: Nix is smart about reusing components across images. Build ten images, and you might only store the equivalent of two or three.
- **Layer optimization**: Nix understands your dependency tree, creating Docker layers that make sense.

### 3. Speed demon builds

Waiting for builds is so last year:

- **Caching on steroids**: Nix caches at a much more granular level than Docker. Change one file? Only rebuild what's necessary.
- **Parallel builds**: Nix can build multiple parts of your image simultaneously. More cores = faster builds.
- **Incremental updates**: Updating a single dependency doesn't mean rebuilding the world.

### 4. Dependency management that doesn't suck

Say goodbye to dependency hell:

- **Declarative definitions**: Describe your entire system in a Nix expression. No more imperative Dockerfile gymnastics.
- **Conflict resolution**: Nix can handle multiple versions of the same library without breaking a sweat.
- **Transitive dependencies**: Nix tracks the entire dependency tree, so you don't have to.

## Putting it all together

Here's a taste of what a Nix-based Docker build might look like:

```nix
{ pkgs ? import <nixpkgs> {} }:

pkgs.dockerTools.buildImage {
 name = "my-awesome-app";
 tag = "latest";
 contents = [
 (pkgs.buildEnv {
 name = "app-env";
 paths = [
 pkgs.nodejs-14_x
 pkgs.yarn
 (pkgs.writeScriptBin "start-app" ''
 #!/bin/sh
 yarn start
 '')
 ];
 })
 ];
 config = {
 Cmd = [ "start-app" ];
 WorkingDir = "/app";
 };
}
```

This little snippet creates a Docker image with Node.js, Yarn, and your application, all neatly packaged and ready to run.

## The bottom line

Nix isn't just a tool; it's a superpower for Docker image building. It brings determinism, efficiency, and sanity to a process that often feels like herding cats.

Is there a learning curve? Sure. But once you go Nix, you'll wonder how you ever lived without it. Your Docker images will be leaner, meaner, and more reliable than ever before.

So, are you ready to take your Docker game to the next level? Give Nix a shot. Your future self (and your ops team) will thank you.

## References

- [Nix package manager](https://nixos.org/)
- [Nix pills tutorial](https://nixos.org/guides/nix-pills/)
- [NixOS Docker tools](https://nixos.org/manual/nixpkgs/stable/#sec-pkgs-dockerTools)
- [Docker best practices](https://docs.docker.com/develop/develop-images)

Ditch the Docker headaches. Learn how Nix brings sanity to image building with determinism and efficiency.

Nix: revolutionizing Docker image builds

As I mentioned in the [Build the same thing at any time](nix-build-the-same-thing-at-any-time.md), Nix knows the exact content of packages and dependencies before builds. So it can take advantage of this characteristic to avoid duplicated building different layers with the same content but different instructions.

It is even too great to take advantage of content-addressable storage. Nix can do more than that.

With a deep understanding of Nix on the package that it prepares to build. Nix can build a dependency tree exactly. From this insight, Nix groups all closely related packages and their dependencies in the same layer. Besides, files of large sizes can be placed in a separate layer to maximize the cache. In this way, the needed number of layers is decreased and is more effective.

Nix can evaluate the changeability of each package depending on update history, the place in the dependency tree, and the type of the package. Then it uses some complex techniques to determine the order of layers. So the layers with the least changed layers are placed at the bottom.

Instead of copying the same files to different layers. Nix creates symlinks or hardlinks to share resources between layers. It helps Nix optimize the size of the image but still maintains the logical structure. Thereby increasing the pull/push speed significantly. As a result of this ability, dependencies changing now can just only update the reference links.

One more point, Nix creates Docker images from scratch. It means Nix does not need any available image as a base layer as a normal Docker build. For example, we usually need Ubuntu and Alpine images to build our Docker image.

Finally, creating a Docker image in Nix is just building the program into a package with Nix and making sure it works. Then turn it into a docker image. The build-time dependencies are actually not needed here. So it is easy to remove built-time related tools and redundant libraries.

After all the above things, we have a Nix with the ability to build Docker image more efficiently, faster, and cheaper.

---

#### References

_Using the build cache._ (n.d.). Docker Docs. Retrieved August 2, 2024, from <https://docs.docker.com/guides/docker-concepts/building-images/using-the-build-cache/>

Wang, E. (2022, September 13). _Construction and analysis of the build and runtime dependency graph of nixpkgs_. Tweag. Retrieved August 2, 2024, from <https://www.tweag.io/blog/2022-09-13-nixpkgs-graph/>

_Understanding the image layers_. (n.d.). Docker Docs. Retrieved August 2, 2024, from <https://docs.docker.com/guides/docker-concepts/building-images/understanding-image-layers/>

_What is the difference between a symbolic link and a hard link?_ (2008, October 9). Stack Overflow. Retrieved August 2, 2024, from <https://stackoverflow.com/questions/185899/what-is-the-difference-between-a-symbolic-link-and-a-hard-link>

Rugyt, A. (2024, March 15). _Nix is a better Docker image builder than Docker's image builder_. Xe Iaso. Retrieved August 2, 2024, from <https://xeiaso.net/talks/2024/nix-docker-build/>

An exploration of how Nix outperforms Docker in building images, leveraging its deep understanding of package dependencies and content-addressable storage

Nix is faster than Docker build


The reproducible issue is resolved in the [previous part](nix-build-the-same-thing-at-any-time.md). But we still have another problem when using Docker build, it is shadow copies. This issue comes from the underutilization of Content-addressable storage in Docker of Docker build. It even lets build time slower.

- [Content-addressable storage in Docker](content-addressable-storage-in-docker.md)
- [Nix is faster than Docker build](nix-is-faster-than-docker-build.md)


An overview of how Nix addresses issues like shadow copies in Docker builds, improving build efficiency and speed

Nix - minimum changes, minimum redundancies


Pinning nixpkgs means you can choose the version of nixpkgs, and then all packages required for building your application and its dependencies are fetched from this nixpkgs version where the version of each package is specified and not changed over time.

```nix
let
  nixpkgs = fetchTarball {
    url = "https://github.com/NixOS/nixpkgs/archive/20.09.tar.gz";
    sha256 = "1wg61h4gndm3vcprdcg7rc4s1v3jkm5xd7lw8r2f67w502y94gcy";
  };
  pkgs = import nixpkgs {};
in
```

What happens once you do not specify the pinning version of nixpkgs? In this case, Nix uses your local nixpkgs version by default.

---

#### References

_FAQ/Pinning Nixpkgs_. (n.d.). NixOS Wiki. Retrieved August 2, 2024, from https://nixos.wiki/wiki/FAQ/Pinning_Nixpkgs


An explanation of pinning nixpkgs in Nix and its importance for reproducible builds

Pinning nixpkgs in Nix


One more issue that comes from internet access of Docker build is Shadow copies aka redundant files that are not cleaned after new versions of packages are installed.

For example

```Dockerfile
FROM ubuntu:20.04
RUN apt-get update && apt-get upgrade -y
RUN apt-get install -y some-package
```

Remind that the Docker build creates a separate layer for each RUN statement. Referring to the above Dockerfile, we can simply explain it as follows.

- The base layer is Ubuntu
- The second layer contains every file that is updated by `apt-get upgrade`.
- The third layer contains new packages

Clearly, the upgrade happens at the second layer without affecting the base layer. This means that new versions of packages are installed without removing old versions of files that are no longer used.

This issue makes the size of the image increase unnecessarily. We can resolve this by Multistage builds also.

But why do we not resolve all the above issues from its root cause?

---

#### References

_Understanding the image layers_. (n.d.). Docker Docs. Retrieved August 2, 2024, from https://docs.docker.com/guides/docker-concepts/building-images/understanding-image-layers/


An explanation of shadow copies in Docker builds

Shadow copies in Docker builds


For example, let's suppose that we have the following `Dockerfile`.

```Dockerfile
FROM ubuntu:20.04
RUN apt-get update && apt-get install -y python3
```

Imagine that the first time you build your Docker image, the version of Python is 3.8.1. However, today when the latest version of Python is 3.9.1, you want to build Docker image again. Transparently, your two images are different no matter that you are using the same Dockerfile. It means you can’t reproduce the same environment in which your application is running.

If you are familiar with Docker, more than one solution pops into your mind. The easiest way is to specify a specific version to install like the following.

```Dockerfile
FROM ubuntu:20.04
RUN apt-get update && apt-get install -y python3=3.8.10-0ubuntu1~20.04.1
```

In this way, the instability can be decreased but can not resolve the issue completely. The reasons can be listed as follows.

- Indirect dependencies: even though you specify the version for main packages, these dependencies that you can’t control, can be changed over time.

- Packages are not supported: Ubuntu repositories can be changed. So your build will fail once any required packages/versions are removed from repositories.

- Ubuntu mirrors aka copies of main Ubuntu repositories can be changed also. Once the packages that you need, are not available, the building will fail.

We have some keywords related to better solutions that I only list here for reference.

- Local cache
- Private registry
- Multistage builds


An explanation of the challenges with package versioning that lets Docker builds unstable

Unstable package installation in Docker


In my previous post, [Ton: Blockchain of Blockchains](ton-blockchain-of-blockchains.md), I talked about some core technologies that make TON strong. However, it was just an overview. To begin developing on TON is not enough. Today, I will introduce some core concepts in TON that you will frequently work with as a TON developer.

## Cells and data storage

**Cells are the atomic unit of data storage in TON**. It is a data structure containing:

![](assets/ton_core_concept_ton_cell.webp)

- Up to 1023 bits of data
- Up to 4 references to other cells

Currently, TON has 2 types of cells: ordinary and exotic

- ordinary: basic, standard cell.
- exotic: specialized cells with specific functions.
  - Pruned Branch Cell: represents a pruned branch in a data structure.
  - Library Reference Cell: references external libraries or modules.
  - Merkle Proof Cell: contains proofs for verifying data in a Merkle tree.
  - Merkle Update Cell: contains information to update a Merkle tree.

When serializing cells into byte arrays, it is called Bag of cells. **Every data stored on TON including blocks and the state of the network is a bag of cells.**

![](assets/ton_core_concept_ton_bag_of_cells.webp)

## Smart contract

As mentioned in the previous post, everything in TON is a smart contract. That contains:

- Address: (workchain_id, account_id) pairs present smart contract as a unique identifier.
- Code: code that can be run on TVM, and represents the rule and business logic of this contract.
- Data: the state of the smart contract, includes variables that store information and can be modified by the contract’s code.
- Balance: the amount of TON held by the contract.
- Others: flags requesting tick-tock calls at each block (active only for fundamental contracts), auto-splitting information and published library cells, etc.

## Message

![](assets/ton_core_concept_ton_message_flow.webp)

To communicate between smart contracts aka actors on TON network, we need to send messages. A message is a package of data that contains the following elements:

- Source Address: Sender’s address.
- Destination Address: Receiver’s address.
- Value: Amount of Toncoins being transferred.
- Payload: Data or instructions for the recipient.
- State Init: Optional initialization data for the recipient.

We have 2 types of messages:

- Internal Messages: Facilitate communication and value transfer between smart contracts within the TON network. They are used for contract-to-contract interactions.
- External Messages: Allow users and external entities to interact with smart contracts. They are typically used to invoke functions or services provided by the smart contracts.

## Transaction

On TON, we have the transaction that records the state changes of processing a message. That basically contains:

- Inbound Message: The message that triggered the transaction.
- State Changes: Modifications to the account or smart contract state, such as balance updates or data modifications.
- Outbound Messages: Any new messages generated as a result of processing the inbound message.
- Gas Fees: Computational resources consumed during the transaction.

Not every transaction leads to outgoing messages or updates to the contract’s storage; this depends on the specific actions defined by the contract’s code.

One more important point to note is that unlike Ethereum or most other synchronous blockchains, where each transaction can contain several smart contract calls, in TON, **a transaction is executed on a single smart contract, and smart contracts communicate through messages.**

## Gas

In Solidity, gas concerns are minimal for contract developers. If a user provides insufficient gas, the transaction will be completely reverted (though the gas spent will not be refunded). If sufficient gas is provided, the actual costs will be calculated and deducted from the user’s balance automatically.

In TON, the scenario differs:

- Insufficient gas leads to partial transaction execution.
- Excess gas must be refunded, a responsibility falling on the developer.
- When multiple contracts exchange messages, each message requires individual control and calculation.

TON does not automatically calculate gas. The entire transaction execution, with all its outcomes, can be lengthy, potentially leaving the user with an insufficient toncoin balance by the end. **So the developer must take care of gas costs**. However, calculating gas is not an easy task. So we often need to set a minimum gas limit for each transaction, then refund the excess gas later. For example:

- Today, every transaction costs around ~0.005 TON.
- And NFT marketplaces usually take an extra amount of TON (~1 TON) and return (1 - transaction_fee) later.

> FYI: Fee formula on TON transaction_fee = storage_fees

    + in_fwd_fees
    + computation_fees
    + action_fees
    + out_fwd_fees

## Conclusion

With all the above concepts, we can now begin developing on TON. But these are actually not enough. We will continue diving deep into more complex concepts such as data format, transaction layout, or bounceable addresses when developing something in the next post.


Base concepts to begin with TON development

Ton's base concepts


TON (The Open Network) is a blockchain platform originally developed by the team behind Telegram Messenger. By taking advantage of unique sharding technology, Multi-blockchain architecture and Instant hypercube routing protocol, TON aims to enable fast transactions and smart contracts with a high level of scalability and security.

To understand what actually happens under these shiny names of technologies, this post will dissect one by one in the simplest way.

## Actor model, everything is a smart contract

![](assets/ton_blockchain_of_blockchains_ton-actor-model.webp)

Firstly, TON is a concurrency model that facilitates the efficient execution of smart contracts and other decentralized applications by organizing computational entities (actors) that interact through message passing. So technically, every actor in TON is represented as a smart contract including our wallet which can be considered as a simple actor.

Each actor has its own storage and behavior. We can imagine that storage is the location where we will store the state of the actor or some other data. So we can temporarily avoid it here, and go to the detail in another post where we will prepare for writing our first smart contract on TON. The main point that I want to bring to you in this post is "What does the actor do?"

## Actor behavior

Take a look at the following image. It is actually the sequence of jobs that an actor actually does.

![](assets/ton_blockchain_of_blockchains_ton-actor-behavior.webp)

1. Event Trigger: An actor is typically activated upon receiving a message.
2. Event Handling: The actor routes the event to the appropriate handler in its `contract code`, utilizing its properties to process the event in the TVM (TON Virtual Machine).
3. State Modification: During event execution, the actor can modify its own properties such as its `contract code` or `data`.
4. Message Generation: Optionally, the actor can generate outgoing messages.
5. Standby Mode: After processing the event, the actor returns to standby mode to wait for the next event.

Finally, when we combine all the above steps together, it will result in a transaction.

## Chain and shard

![](assets/ton_blockchain_of_blockchains_ton_chain_of_txs.webp)

We basically have a transaction after the behavior of any actor is done. Then when there is more than one transaction in a sequence, it may be called a **chain**. In case it is a chain of transactions that is owned by a single account, it is called `AccountChain`.

![](assets/ton_blockchain_of_blockchains_ton_account_chain.webp)

Then a group of account chains will be stored in the same location called shard called **AccountShard**. In the same way build an **AccountChain**, **ShardChain** which is a chain of **AccountShard**, and **BlockChain** which is the chain of all shards.

In TON, we can consider that have 2 types of **BlockChain**

- Workchain: Blockchain with your own rules - This is the Blockchain that will run normal transactions such as swaps, transfers, etc.
- Masterchain: Blockchain of Blockchains - This is the Blockchain that manages other Workchains for the synchronization of message routing and transaction execution. Under the hood, it is also considered as a Workchain.

Currently, TON has 2 chains, Masterchain and Basechain.

![](assets/ton_blockchain_of_blockchains_ton_blockchain.webp)

## Splitting and Merging, What makes TON more scalable?

Because a **ShardChain** is composed of distinct **AccountChains**, it can be easily divided.

For instance, if a **ShardChain** manages events for one million accounts but encounters a transaction volume too high for a single node to handle, we can split this chain.

By dividing it into two smaller **ShardChains**, each responsible for half a million accounts, we ensure each new chain is processed by a different subset of nodes.

Similarly, if certain shards become underutilized, they can be combined into a larger shard.​⬤

> Side note from TON document: To make splitting and merging deterministic, an aggregation of AccountChains into shards is based on the bit-representation of account addresses. For example, address looks like (shard prefix, address). That way, all accounts in the shardchain will have exactly the same binary prefix (for instance all addresses will start with 0b00101).

#### Instant hypercube routing

In the infinite sharding approach, every account aka smart contract is treated as if it were itself in a separate **ShardChain**. Accounts interact only by sending messages to one another, adhering to the actor model where each account operates as an independent actor.

So TON needs an efficient way to deliver and process messages between **ShardChain**. It is Instant hypercube routing with the following characteristics

- Hypercube Structure: Network as a hypercube with shards as vertices and communication paths as edges.
- Multi-Dimensional Routing: Shards are addressed by binary strings; routing involves flipping bits in the address.
- Scalability: Efficiently scales with more shards; each shard knows a logarithmic number of neighbors.
- Instant Routing: Rapid message propagation through direct communication paths.
- Fault Tolerance: Multiple alternative paths ensure robustness despite shard or path failures.

If it is hard to imagine, in some aspects, you can take a look at **E-Cube Routing** to get the idea.

![](assets/ton_blockchain_of_blockchains_e_cute_routing.webp)

## Conclusion

In summary, TON stands out as a “blockchain of blockchains” due to its advanced architecture and design principles. Its use of unique technologies like sharding, the Actor model, and Instant hypercube routing allows it to achieve high scalability, fast transactions, and robust security. By seamlessly integrating multiple blockchains into a unified network, TON not only enhances interoperability but also paves the way for a more connected and efficient decentralized ecosystem. Whether you’re a developer or a blockchain enthusiast, understanding the inner workings of TON offers valuable insights into the future of blockchain technology.


Talk about TON and its core technologies. Why do we call TON the 'Blockchain of blockchains'?

Ton: blockchain of blockchains

Welcome back to our Devbox series! Previously, we've explored the journey from the early days of virtualization to the widespread adoption of Docker. Finally, we try using Nix and Devbox to enhance software development process. If you need to catch up, you can check out the earlier posts here:

- [Devbox #1: the world before Docker](devbox-a-world-before-docker.md)
- [Devbox #2: our Docker adoption and its challenges](devbox-docker-adoption-and-challenges.md)
- [Devbox #3: the overview into Nix & how we use Devbox @ Dwarves](devbox-nix-and-our-devbox-adoption.md)

Beyond the theories, in this expanded part, we'll show how we use Devbox to create an isolated, reproducible, and portable local development environment.

![](assets/devbox-local-development-env_devbox.gif)

## Engineering = programming + time + people

Programming itself is challenging enough—solving problems by writing code that works. However, engineering extends this challenge: it involves ensuring that code continues to function over time and can be effectively managed and improved by a team of programmers that is changed continuously. This introduces complexities that go beyond initial development. As mentioned in the [What is Software Engineering?](https://research.swtch.com/vgo-eng)

Once the project scales and team grows, the complexity of managing dependencies, tools, and environments also increases. It is an inherent outcome of the collaborative essence of software engineering.

## We can’t control everything manually

Dwarves experienced this complexity firsthand as we welcomed a lot of new talents and projects with varying sizes and tech stacks. Team members are rotated to another project once the current project is done. Because each project has different scales and requirements, each time onboarding happens, team members need to install different tools to begin developing. This situation causes their machine to be filled with a lot of redundant applications and dependencies over time.

Besides, we have no way to track what dependencies are installed or not currently. So we must install them one by one until the project can be executed. You can have your checklist in the _README_. But it just only helps you manually checking it is installed or not, you can’t make sure the installed stuff is compatible with your available applications and you also don’t know the exact version of them. The challenge of managing different tools and dependencies became evident. It forces our members to spend a lot of time setting up new projects under anxiety.

## The road to Devbox

To address this challenge, we adopted several strategies that other successful teams have used. Using containerization tools like Docker can ensure a consistent environment across different projects by packaging applications and their dependencies into isolated containers. Virtual environments and package managers can help manage dependencies specific to each project, reducing conflicts and redundancies. However, Docker images become quite large and unwieldy, making them difficult to manage and distribute efficiently. On the other side, virtual environment and package manager often lack the robustness needed for truly reproducible builds and environments. Finally, we found Nix as a savior.

We took a look at Nix with the expectation of creating a configurable, reproducible, and portable development environment that helps us quickly onboard anybody to any new project with a few simple commands. It also brings an easy way to manage all installed applications in an isolated or semi-isolated environment.

But Nix needs a huge effort to be applied such as separated syntax and mechanisms. It is also too big to serve our purpose. So we need something more simple, and lightweight but can also take advantage of Nix. This is the reason why finally we chose Devbox for a few first experiments. You can read about Devbox [here](devbox-nix-and-our-devbox-adoption.md).

## Devbox simple setup

Our purpose is creating a configurable, isolated, reproducible and portable development environment. So we use Devbox to create an isolated shell in our project root.

```shell
cd way/to/the/project
devbox init
devbox shell --pure
```

In the above commands, when `--pure` is specified, Devbox creates an isolated shell inheriting almost no variables from the current environment. A few variables, in particular `$HOME`, `$USER`, and `$DISPLAY`, are retained.

Once Devbox shell is shown, we can install everything for running our the project including database, git, programming language, code editor, etc. Powered by Nix, Devbox has more than 80,000 packages containing everything you want. You can also specify what is the version that you want when installing it.

```shell
devbox add go@1.21.3
devbox add docker
devbox add docker-compose
devbox add vim
devbox add git
devbox add colima
...
```

After running above commands, the file `devbox.json` in your project root should look like following:

```JSON
{
 "$schema": "<https://raw.githubusercontent.com/jetify-com/devbox/0.10.7/.schema/devbox.schema.json>",
 "packages": [
 "docker@latest",
 "go@1.21.3",
 "docker-compose@latest",
 "git@latest",
 "qemu@latest",
 "vim@latest",
 "vscode@latest",
 "colima@latest"
 ],
 "shell": {
 "init_hook": [
 "echo 'Welcome to devbox!' > /dev/null"
 ],
 "scripts": {
 "test": [
 "echo \\"Error: no test specified\\" && exit 1"
 ]
 }
 }
}
```

You can bring this file to anywhere you want. Once you type `devbox shell`, the exact shell with the same state will be initiated with full installation for all dependencies.

## Bring Devbox to Makefile

To quickly onboarding newbie without any Devbox knowledge, we also trying to turn it to our Makefile as following.

```makefile
shell:
 @if ! command -v devbox >/dev/null 2>&1; then curl -fsSL <https://get.jetpack.io/devbox> | bash; fi
 @devbox install
 @devbox shell

```

With the above simple script, the system firstly checks if Devbox is installed or not. If it is not installed, we try to fetch and install it. Then `devbox install` helps us install all packages mentioned in `devbox.json`. Finally, `devbox shell` starts the shell with ability to inherit your host machine installed things.

Basically, it’s all things that we must do to reach our purpose. But sometimes, edges case still happens. We need some outstanding steps in the first to have a thorough preparation for other people coming later.

## Container runtime configuration

Without container-less purpose, we also need to install container runtime to use `docker-compose` and `docker` in the project. Colima is a nice thing to do it.

There are no problems in the n-time running the Devbox shell. But if it is the first time, we must run the shell with `--pure`. In this situation, we need some small cheats to run Colima.

First is mounting `/usr/bin` to shell `$PATH`, by this way the system can use `sw_vers` to get the environment version. Colima needs this to be started.

```shell
@export PATH=${PATH}:/usr/bin && colima start
```

One more issue comes from Docker config, which is used when running `docker-compose`. We need to cheat by removing `credsStore` in the `~/.docker/config.json`, or installing `osxkeychain` to run Colima normally. You can check [this thread](https://stackoverflow.com/questions/67642620/docker-credential-desktop-not-installed-or-not-available-in-path/72888813#72888813) for more information if you reach this issue.

## Real life usage

You can find a practical example of using Devbox in [our memo repository](https://github.com/dwarvesf/memo.d.foundation). This repository uses Devbox to create an isolated, reproducible, and portable development environment, showcasing how to manage dependencies and streamline development workflows.

## Conclusion

In this installment, we've moved from theory to practice, demonstrating how Devbox creates an isolated, reproducible, and portable development environment. We've shown how to set up Devbox, install dependencies, and integrate it with a Makefile for ease of use. By addressing container runtime challenges, we've ensured a smooth development experience. We hope these insights help streamline your own workflows and enhance your development practices. Thank you for following along, and happy coding!

Expanded section that focuses on Devbox and its practices on setup a local development environment.

Using Devbox to setup local development environment


This is the 3rd post of Devbox series includes

- [Devbox #1: The world before Docker](devbox-a-world-before-docker.md)
- [Devbox #2: Our Docker adoption and its challenges](devbox-docker-adoption-and-challenges.md)
- [Devbox #3: The overview into Nix & how we use Devbox @ Dwarves](devbox-nix-and-our-devbox-adoption.md)

In the previous session, that is [Devbox #2: Our Docker adoption and its challenges](devbox-docker-adoption-and-challenges.md) , we talked about “How Docker is adopted in my development team and its challenges encountered along the way?” And I also talked about a solution to separate us from Docker and containers in setup a local software development environment. And now, It is coming!

As known as a command-line tool, Devbox lets you easily create isolated shells for development. But it doesn't work alone. Because of powering by Nix under the hood, whether you're a newbie or an experienced Nix user, we we need to dissect Devbox beginning by taking about Nix and it power that is superior than Docker.

## Nix is more than a cross-platform package manager

From the [official Nix website](https://nixos.org/), we learn that Nix is a unique cross-platform package management tool that takes a reproducible, declarative and reliable approach to system configuration. However, there's more to Nix than just being a package manager. In fact, the holy trinity of Nix includes Nixpkgs, NixDSL, and NixOS. While they are all called Nix, you can break them down into their respective components for a clearer understanding. With Nix, you use the NixDLS to make Nixpkgs build packages that can be anything from software to entire NixOS images.

Nix builds packages in isolation from each other. This means that if a package works on one machine, it will also work on another. Nix is also declarative and reliable, making it easy to share development and build environments for your projects regardless of the programming languages and tools you're using. Additionally, Nix ensures that installing or upgrading one package doesn't break other packages, allowing you to roll back to previous versions if needed and preventing any inconsistent state during upgrades.

So, what sets Nix apart from Docker?

## Docker build is linear history

In Docker, each line in the Dockerfile is built into a separate Docker layer when it's executed. It's important to understand that even a minor change in any layer will prompt Docker to rebuild all subsequent layers, regardless of whether changes were made to those lower layers.

Let's take a look at the following Dockerfile.

```BASH
FROM ubuntu
# ...
RUN apt-get install -yq gcc
RUN apt-get install -yq vim mc
RUN apt-get install -yq big package
RUN some-expensive-operation
# ...
```

When you modify any line in a Dockerfile, the underlying layers are rebuilt from scratch, regardless of whether those lower layers have changed or not as in the following diagram. Once `RUN apt-get install -yq vim` is modified to `RUN apt-get install -yq vim mc`, the big package is also rebuilt after that.

![Image1](assets/devbox-nix-and-our-devbox-adoption_1.webp)

## Nix build is dependency graph

In contrast to Docker, Nix build works differently. It uses symlinks to achieve atomic deployment of new versions of the system configuration. This makes rollbacks easy to perform and reduces the complexity of managing dependencies. Under the hood, Nix creates a "map" of all the packages and their dependencies, providing a high-level view of the entire system.

![Image2](assets/devbox-nix-and-our-devbox-adoption_2.webp)

Imagine you have a big box full of toys. Inside the box, there are lots of different toys that are connected to each other in different ways. For example, one toy might be connected to another toy by a string. If you want to play with just one toy, you need to find it in the box and take it out. But if you want to play with all the toys at once, you need to open the box and take them all out.

Nix works the same way. It creates a "map" of all the packages on your computer, and how they depend on each other. When you want to use a package, Nix finds it in the map and uses it. If you want to update a package, Nix changes the map to reflect the new version.

The magic of Nix is that it makes sure that everything works together correctly. It's like having a special box that knows how all the toys are connected, so it can help you play with them without getting confused or losing any of them.

## Docker can access internet while building image

As we discussed in the [Devbox #2: Our Docker adoption and its challenges](devbox-docker-adoption-and-challenges.md), Docker builds can access the public internet, so we can't guarantee that the same image will be built every time, as everything on the internet can change in minutes. Additionally, there are no checks to ensure that the files being fetched are actually the ones you intended for your Docker image. Even if we can push the image to a registry and pull it for running identical containers on different computers. In some edge cases when we can’t access to the online repository so need to build another one, we may not be able to build the required image again if some dependencies have changed on the internet.

## Nix ensures external sources are immutable

Take a look at the following diagram, we can see with the same build expression, Nix can result in the same images. In contrast, we can just only build a Docker Image one time and bring this result everywhere if don’t want any change to happen. How can Nix make it happen?

![Image3](assets/devbox-nix-and-our-devbox-adoption_3.webp)

![Image4](assets/devbox-nix-and-our-devbox-adoption_4.webp)

Nix restricts external sources from being changed without being detected. As a result, Nix build always produces functionally the same output each time they are run. This also makes making changes safer, as a change requires a rebuild. The reason for this is that Nix provides the ability to configure the system state using Nix expressions in the Nix language. Essentially, it's an instruction to Nix on how should it build the package and what the final result should look like.

To further illustrate the point that Nix builds produce the same output each time, we can understand that Nix is implemented as a pure function that always returns the same result for the same input parameters. This means that if you run the same Nix expression multiple times, you will always get the same output.

## Nix also can build Docker image better than Docker build

Nix can build Docker images better than Docker build, here's why: Nix takes advantage of a dirty secret deep within Docker - a content-aware store. However, since Docker build isn't designed to utilize this feature. Nix, on the other hand, can create layered images that only upload the changed layers, making updates more efficient.

A layered image puts every dependency into its own image layer so you only upload the parts of your image that have actually changed. For example, making an update to the webp library to fix a trivial bounds checking vulnerability because nobody writes those libraries in memory-safe languages? The only thing that'd need to be uploaded is that single webp library layer.

![Image5](assets/devbox-nix-and-our-devbox-adoption_5.webp)

Additionally, if you have multiple services in the same repository, they'll share Docker layers with each other without any extra configuration. It's not possible to achieve this level of efficiency with Docker without creating multiple common base images, each containing a bunch of tools and unnecessary bloat that some of your services may never use.

## But Nix needs effort, Devbox is more easy

After all, Nix looks superior to Docker. But not everything is perfect. Nix is more complex, If you and your team do not want to destroy everything and start again, migrating to Nix needs a lot of effort. But we have another simple way to apply Nix in the local development environment, that is using Devbox.

Powered by Nix, Devbox is a command-line tool that makes it easy to create isolated shells for development. By defining the list of packages needed for your development environment, Devbox creates a dedicated space for your application to run without anything related to container or VM.

![Image6](assets/devbox-nix-and-our-devbox-adoption_6.webp)

In practice, Devbox works similarly to a package manager like Yarn - except the packages it manages are at the operating system level (something you would normally install with brew or apt-get). With Devbox, you can access over 400,000 package versions from the Nix Package Registry.

Devbox offers an intuitive interface for creating development environments using the Nix Package Manager, without requiring any knowledge of the Nix language. So, how can we incorporate Devbox into our team's workflow?

## Case study: memo.d.foundation

Looking at [memo.d.foundation](https://github.com/dwarvesf/memo.d.foundation), we use Devbox as a tool to create reproducible development environments, making it easy for new team members to get started with minimal effort. By setting up a Devbox shell config with needed dependencies, new joiners can quickly get started without installing anything beyond Devbox. They can then focus on making the application run locally, avoiding the issue of "It works on my machine." Non-tech team members working on content for this repository can also easily run the project without any concerns.

All above ideas are proved by below transparent configuration file.

![Image7](assets/devbox-nix-and-our-devbox-adoption_7.webp)

Jut only need to bring it to anywhere then using `devbox shell` to setup an reproduced development environment without any outstanding steps likes following.

![Image8](assets/devbox-nix-and-our-devbox-adoption_8.webp)

Finally, easy to run our project without installing any other stuffs

![Image9](assets/devbox-nix-and-our-devbox-adoption_9.webp)

## Case study: docker-less development with Devbox services

In addition, we use Devbox to create a containerless development environment by leveraging Devbox services. Under the hood, it takes advantage of Process-Compose, a simple and flexible scheduler and orchestrator written in Nix, to manage non-containerized applications. With Devbox, we can create and manage servers in the shell using a Docker-compose-like approach without any actual Docker containers needed.

When we install a package using Devbox, it creates a file `process-compose.yaml` in the `./.devbox/virtualenv/` directory. This allows Devbox to manage every package installed inside the shell.

For example, when installing PostgreSQL using Devbox add Postgresql, the resulting `process-compose.yaml` file contains instructions for executing the DB, as shown in the following code.

```BASH
version: "0.5"

processes:
  postgresql:
    command: "pg_ctl start -o \"-k $PGHOST\""
    is_daemon: true
    shutdown:
      command: "pg_ctl stop -m fast"
    availability:
      restart: "always"%

```

If you familiar with docker-compose, I think it is easy to understand process-compose as well. By standing all `process-compose.yaml` both from the root of project and in the `.devbox/`, you can manage how many services are ready to serve by using `devbox services ls`. Then when you enter `devbox services up`, all services in this project will be instructed to run. The special point here is that you can custom the `process-compose.yaml` at the root of project to get the same result as docker-compose without using container.

You also can run different Devbox shells parallel, but can’t run different services from different shells using the same port because we have no separate network here. Take a look at the following example when I try running 2 Postgresql services in different shells with the same port.

![Image10](assets/devbox-nix-and-our-devbox-adoption_10.webp)

So depending on what applications you are running, you will have the proper way to config different ports. For example, can change add one more argument to change the default port of Postgresql in the `process-compose.yaml` to get parallel processes.

```BASH
version: "0.5"

processes:
  postgresql:
    command: "pg_ctl start -o \"-k $PGHOST -p 5433\""
    is_daemon: true
    shutdown:
      command: "pg_ctl stop -m fast"
    availability:
      restart: "always"

```

![Image11](assets/devbox-nix-and-our-devbox-adoption_11.webp)

For now, this is the list of usecases I can provide. As I continue to learn and adapt, I will introduce more in the future.

## Conclusion

In summary, Nix and Devbox offer exciting new possibilities for streamlining software development processes. By providing a more efficient way to create reproducible development environments, Nix and Devbox can help teams overcome challenges and improve overall workflow. While they may not completely replace Docker, they can effectively address different problems and provide additional support where Docker may be struggling.


The overview into Nix & how we use Devbox @ Dwarves


This is the 2nd post of Devbox series includes

- [Devbox #1: The world before Docker](devbox-a-world-before-docker.md)
- [Devbox #2: Our docker adoption and its challenges](devbox-docker-adoption-and-challenges.md)
- [Devbox #3: The overview into Nix & how we use Devbox @ Dwarves](devbox-nix-and-our-devbox-adoption.md)

In the previous session that is [Devbox #1: The world before Docker](devbox-a-world-before-docker.md), we discussed the world from the beginning before container and Docker concepts emerged as groundbreaking advancements in software development. But nothing is perfect! Container and Docker also have their own concerns. So in this session, we will dive deep into Docker in practice with the real-life adoption of my team. Then, we can discover the challenges of containers and Docker and find the right way to go to get more performance.

From the beginning, the desire to produce an isolated environment is raised to provide ability for multiple users accessing a computer concurrently with full resource utilization via a singular application. Over time, various other purposes have been explored and implemented. With the introduction of Docker, container technology became extensively utilized in software development. Two of the most popular uses are creating reproducible development environments and enabling continuous deployment.

## Container deployment era

Compare to other traditional solutions for creating isolated environments, Docker container offers a lot of superior advantages. It requires a significantly smaller resource footprint with the ability to spin up and down faster by using the same OS kernel instead of encapsulating independent OS. This approach allows for leveraging numerous benefits of containerization in the software deployment process.

![](assets/devbox-docker-adoption-and-challenges_01.webp)

Currently, creating and deploying entire applications becomes simple through the use of Docker. With the available immutability characteristics of the Docker image, we can also decouple applications from infrastructure by generating application container images at release time rather than at deployment time.

Furthermore, owing to the preeminence of Docker containers, a lot of services supporting them have experienced robust growth. With millions of pre-packaged applications available on DockerHub, initiating a project becomes swift with just a few commands. Numerous public and private cloud providers offer managed container services, enabling immediate deployment of our applications online. Moreover, an extensive ecosystem has emerged around containers and Docker, providing support for monitoring, security, networking, storage, and integration, letting us have the ability to create a smooth process from development, integration to deployment and testing.

## Docker in development

Sounds massive! But I know some of us will never touch on the entire benefits of Docker containers except if we are a DevOps professional. So, to be more developer-friendly, why don't we take a closer look at the development environment aspect?

When beginning my software development career, I had the opportunity to work in a team with almost all previous-generation developers. In this environment, durability and certainty are prioritized above all else, so everyone is fine with installing a bunch of stuff on their computers to get the project off the ground. I actually needed over 3 days to get the project running, despite having quite detailed documents. After that, I knew others also needed more time. It is acceptable when this is the onboarding phase. The nightmare actually begins when your laptop encounters issues and the deadline is looming. Regardless of how familiar you are with the project, you still need at least 1 day to reinstall everything the risk of not remembering the dependency version. But that's not even the end of it. After I left the project, my laptop became polluted, and I actually didn’t know how to deal with it except by reinstalling my OS.

In another Golang project, a similar situation occurred. When we attempted to install all dependencies locally for development, one of us unwittingly upgraded the version of the Protobuf generator. Consequently, when the code is committed, thousands of changes are generated, even if only one line of code was updated. After this issue arose, we adopted Docker as a lifesaver.

Docker provides an easy way to define and start a local development environment with just a few instructions using a Dockerfile. Through this transparent approach, you can precisely control any library, version, and related resources required to run your application and lock down configurations as needed. So, once you identify the specific library update that caused issues, resolving it becomes straightforward.

![](assets/devbox-docker-adoption-and-challenges_02.webp)

Utilizing Docker simplifies the creation of a reproducible development environment. By initializing dependencies as containers and connecting our application to them, we ensure consistency. However, in this way, the code is actually running in the local environment. For maximum container isolation, we can encapsulate our application in its own container and integrate it with others. Sharing the development environment then becomes as simple as sharing Docker configuration and scripts, enabling seamless collaboration among team members, regardless of location.

It is not just for the backend side; on the frontend, containerization is also widely applied. Docker can be used to create containers that contain the frontend code, dependencies, and any related configurations to run the application, such as images for building specific APIs integrated with this website for use in local development with proper datasets.

However, nothing is perfect, Docker container has its own concerns.

## Nothing is perfect

Yeah, Docker is really fast. It only takes anywhere from a few milliseconds to a few seconds to start a Docker container from a Docker image. But how do you feel when every time you change the code, you have to rebuild the Docker image and restart the container again for debugging? That would be a real nightmare. To avoid it, you can only run the application locally with Docker container dependencies, or rack your brain to find a way to optimize the Dockerfile. Most of the time, it's fine, but the real problem occurs in edge cases.

The same issue arises when our team tries to pack all related development tools into a Docker image. While it successfully avoids the problem of different versions of dependencies, this approach encounters a bottleneck as the time to start the application is longer than usual.

So what is actually happening? In the Docker, each modification to the codebase necessitates rebuilding the image and restarting the container. Despite leveraging build caching, this process can be time-consuming if not managed carefully. It's crucial to recognize that even a minor change in any layer prompts Docker to rebuild all subsequent layers, irrespective of whether alterations were made to those lower layers.

![](assets/devbox-docker-adoption-and-challenges_03.webp)

Furthermore, incorporating packages into a Docker image without proper consideration can lead to inefficiencies. Executing `apt-get upgrade` at the onset of your Docker build might replace files within the container image. Consequently, these surplus files generate redundant shadow copies, gradually consuming additional storage space over time.

One significant issue that is often overlooked is that Docker builds have access to the public internet. If dependencies are pulled directly from the internet during builds, it can make it difficult to ensure reproducibility of builds over time. Different versions of dependencies may be pulled, leading to inconsistencies between builds.

For example, we often include something like `RUN apt-get install ...` in the Dockerfile. This command handles everything necessary for your container to successfully execute your application. However, as mentioned above, this approach doesn't ensure complete reproducibility of the Docker image over time. Each time this command is run, the version of dependencies installed may vary. To mitigate this, we can specify the version of dependencies. However, if that exact version is no longer available, Docker will throw an error.

## What’s new gate?

So, with all the challenges mentioned above, do we have any way to avoid them in a peaceful manner? Certainly, there are various ways to address these problems, but none of them are perfect or bad.

Most of them involve optimizing your approach to using Docker. However, I would like to introduce another approach that keeps us away from Docker during development but still allows us to leverage Docker for deployment. We'll explore that next time in the [Devbox #3: The overview into Nix & how we use Devbox @ Dwarves](devbox-nix-and-our-devbox-adoption.md).


Our second Devbox session, where we discussed our Docker adoption and challenges.

Our docker adoption and its challenges


This is the 1st post of Devbox series includes

- [Devbox #1: The world before Docker](devbox-a-world-before-docker.md)
- [Devbox #2: Our Docker adoption and its challenges](devbox-docker-adoption-and-challenges.md)
- [Devbox #3: The overview into Nix & how we use Devbox @ Dwarves](devbox-nix-and-our-devbox-adoption.md)

## The it-works-on-my-machine time

Sometimes, in my software development team, a strange issue is raised. Nobody can evaluate the issue because 'It works on their machine’.One of Docker's most powerful abilities is cross-platform compatibility, which enables seamless deployment across Linux, Windows, and macOS environments. In this way, it broadens adoption and facilitates interoperability in heterogeneous computing environments.

![](assets/devbox-a-world-before-docker_untitled.webp)

Positively, It is fine because the software actually works somewhere. So you can scope the area of issue and try troubleshooting. But, it is also a red alert on our development process.

I believe it's not just my problem. Throughout the ongoing evolution of software development, others have likely encountered similar problem. This leads me to wonder, 'Have they implemented strategies to address these issues over time?'

Right, as software technology develops, we always have ways to improve the output. It encompasses more than just a solution, it represents a rich evolutionary history within the realm of software development.

## First bullet: Virtual machine

During the 1960s, researchers at IBM's Cambridge Scientific Center delved into the development of the first virtual machine operating system. They aimed to address the burgeoning demand for time-sharing capabilities within the computing industry of that era. Despite initial setbacks encountered with the System/360 series, which lacked adequate time-sharing functionalities, persistent efforts eventually led to the creation of CP/CMS as the first virtual machine.

As VM evolved and became more widely known, its benefits for software development processes would have become apparent. Developers and engineers outside of IBM, upon learning about CP/CMS and its virtual machine capabilities, began to explore how similar concepts could be applied to their development workflows.

By the late 1990s, products like VMware Workstation emerged, enabling the running of multiple virtual machines on a single PC. This facilitated software testing across diverse environments.

Not only just for development testing, VM has been widely adopted for creating reproducible development environments that can be shared among team members by taking advantage of its characteristics such as **reproducibility, isolation, portability, and snapshotting.**

However, challenges arise when it comes to sharing the environment among team members. These bottlenecks include ensuring consistent setups, managing resource constraints, handling dependencies, and coordinating collaboration due to the complexity and resource requirements of VMs. Additionally, VM configurations may depend on the underlying hypervisor and guest operating system.

## Containerization, a breath of fresh air

![](assets/devbox-a-world-before-docker_untitled-2.webp)

While everyone is struggling with VM, the modern containerization movement of the early 2010s was kickstarted by Linux Containers, followed by Docker's public unveiling at PyCon in Santa Clara in 2013, with its subsequent open-source release in March of that year. Since then, there has been a significant transformation in the software development process.

Have a lot of things to discuss, but we can distill the variances between VMs and Containers into the following summary

|                   | Container                                                         | Virtual Machine                                     | Conclusion    |
| ----------------- | ----------------------------------------------------------------- | --------------------------------------------------- | ------------- |
| Isolation         | Share host kernel, lightweight                                    | Run full OS, heavier                                | VM is heavier |
| Resource Overhead | Minimal, efficient use of resources                               | Higher, each VM requires its own OS                 | VM is heavier |
| Startup Time      | Almost instant                                                    | Slower due to booting entire OS                     | VM is heavier |
| Security          | Good isolation, but potential risks if host kernel is compromised | Strong isolation between VMs, larger attack surface | VM is heavier |
| Deployment        | Highly portable                                                   | Flexible but heavier                                | VM is heavier |

## The opening act with Linux containers

As I mentioned above, Linux Containers (LXC) was perhaps the first implementation of a complete container manager. It provide a lightweight and efficient approach to deploying and isolating applications

Unlike VM, which require the overhead of running multiple operating system instances, Linux containers share the host operating system's kernel while maintaining separate namespaces for processes, networking, and file systems. This results in faster startup times, lower resource consumption, and greater scalability compared to VMs.

However, one disadvantage of using Linux Containers directly is the potential lack of comprehensive tooling and ecosystem support, which can lead to integration challenges and a less streamlined development and deployment experience.

In that situation, Docker was born with more innovations.

## Docker - innovation of innovation

One of Docker's most powerful abilities is cross-platform compatibility, which enables seamless deployment across Linux, Windows, and macOS environments. In this way, it broadens adoption and facilitates interoperability in heterogeneous computing environments.

Moreover, Docker has a variety system of extensive tooling, and ecosystem that make a big improvement in container management. such as Docker CLI and Docker Hub. These resources streamline every facet of container deployment, from image creation to orchestration, significantly simplifying the development workflow.

Docker's robust ecosystem also garners widespread support from major cloud providers, ensuring effortless integration and scalability in cloud environments. Its user-friendly interface and intuitive workflow further contribute to its appeal, empowering developers to expedite deployment processes without compromising efficiency or reliability.

Consequently, Docker emerges as the quintessential choice for modern containerization needs, offering unparalleled flexibility, convenience, and efficiency to developers and organizations worldwide.

## Final thoughts

In summary, the transition from traditional Virtual Machines (VMs) to containerization, epitomized by Docker, represents a pivotal advancement in software development. Docker's lightweight, portable containers have revolutionized software development and deployment. However, this transformation is not merely about adopting a new technology; it's a journey marked by adaptation and discovery. In the [Devbox #2: Our Docker adoption and its challenges](devbox-docker-adoption-and-challenges.md), we will uncover the story: how Docker is adopted in my development team and its challenges encountered along the way.


A brief history about the world before Docker and why we need devbox for local development

The world before Docker


## What is this?

- **ZK-SNARKs** stands for **Zero-Knowledge Succinct Non-Interactive Argument of Knowledges**.

- Nowadays, we often hear this word when mentioning [zk-rollups](). But it is actually a **Privacy-enhancing technology**, and has a lot of applications that we will delve into in another post.

## Decomposition

- **ZK** aka **Zero-Knowledge** mean:
  - Prove possession of certain information.
  - Without revealing that information.
  - **For example:**
    - Given the hash of a random number.
    - The prover could convince the verifier that a number with this hash value exists without revealing what it is.
- **Succinct**:
  - Can be verified within a few milliseconds.
  - no matter how long the statement is.
- **Non-interactive**:
  - In the first version of ZK, the prover and verifier had to communicate repeatedly for multiple rounds.
  - Now, by implementing this characteristic, the proof consists of a single message sent from the prover to the verifier.

## Implementation by example

### Example 1: Function C

- You have a program denoted `C`.
- C have 2 input `C(x,w)`:
  - `x` is the public stuff, that can be shared with anyone.
  - `w` is the secret witness.
- The condition `C(x,w)===true`, means "Prover actually knows a secret witness `w` satisfied a statement related to `x`".
- As a prover, how can we prove that we know `w`, without sending `w` to the verifier to check with the statements?
- **Can map this example to the example of the hash function of the previous part. Let's do it, and go to the next example.**

### Example 2: Bob, Alice, and hash

- Bob is given a hash `H`.
- Alice is given the original string `S` satisfied the condition when hashing `S` by a hash function such as `SHA`, `H` is issued (aka `SHA(S) === H`).
- How can Alice prove to Bob know that she knows the `S`?
- Normally
  - Alice needs to send `S` to Bob.
  - Bob needs to hash again to check `SHA(S) === H`.
- But in this case, `S` is a secret witness, and must not send to any locations, how can Alice prove this statement to Bob?
  - Solution:
    - Alice need a **proof** to send to Bob.
    - This **proof** can prove `SHA(S)===H` is true. Mapping to program `C`, with public `x` as `H` and private `w` as `S`, in other words, when this **proof** proves `C(x,w)===true`, it means Bob can confirm that Alice knows this `w` aka `S` satisfied `SHA(S)===H` without knowledge about `S`.
    - Example of `C`:
      - `function C(x, w) { return ( sha256(w) == x );}`
    - **Note that `C(x,w)===true` is proved by the `proof`, not by itself. How can?**

### Example 3: Implementation of zk-SNARKs in simple words

- A **ZK-SNARK** consists of three functions `G, P, V` defined as follows:

  - **Key generator** aka `G`

    - `G(lambda, C) = pk,vk`
      - `lambda`: secret parameter. **Noted this**.
      - `C`: a program that proves that `prover` knows `w` by `C(x,w)=true`. Aka above `C` in **Example 1** and **Example 2**.
      - `pk`: proving key
      - `vk`: verification key
    - `pk` and `vk` are public parameters that only need to be generated once for a given program `C`.
    - Can assume that:
      - **From `pk`, we can create a `proof` that can be verified by using `vk` in a pre-defined way.**

  - **Prover function** aka `P`

    - `P(pk,x,w) = prf`
      - `pk`: proving key issued from `G`
      - `x`: public input
      - `w`: private witness
      - `prf`: proof proves that prover knows `w` satisfy program `C`.

  - **Verifier function** aka `V`
    - `V(vk,x,prf)=true/false`
      - `vk`: verification key generated from `G`
      - `x`: public input
      - `prf`: proof generated from `P`

- By above functions, `V(vk,x,prf)=true` means:

  - With the proof `prf` generated depends on a logic that includes `x`, `w` and `pk`.
  - And `pk` is a pair with `vk` so they are closely related to each other.
  - So when having `vk`, `x`, `prf`, by a pre-defined way, we can confirm that `C(x,w)===true`.

- **Note that `lambda` can cause security issues when used in real work**

  - Reason: anyone who knows `lambda` can generate fake proofs
  - Example:
    - From any `C`, and known `lambda`, can find a pair `f_pk` and `f_vk`
    - From `f_pk`, malicious actor can generate a `fake_prf` that represents `C(x,w)==true` when checking with `f_vk`.
  - **Solution**: [multi-party-ceremonies]() for **Trusted Setup** -> To build `lambda`.

- Resolve the **Example 2**:

  - Bob uses `G` to generate key, send `pk` to Alice
  - Alice gen `prf` -> Sent to Bob
  - Bob verifies using `vk`

  - **Issue**: Bob can't be a prover because he holds the `Lambda`.
    - A trusted independent group separate from Alice and Bob could run the generator and create the proving key pk and verification key vk in such a way that no one learns about lambda.

### Example 4: In Ethereum [zk-rollups]()

- Can add the building blocks of the verification algorithm to Ethereum in the form of precompiled contracts.
  - [layer-2](), run `G` to generate `pk` and `vk`
  - [layer-2](), the operator use `pk` to generate `proof`
  - [layer-1](), the verifier contract use `vk`, `proof` and public `input x` (can be state changes/Merkle root hash, bla bla ble ble)
  - [layer-1](), if valid -> trigger transaction / append ZK blocks / etc.

## References

- https://consensys.net/blog/developers/introduction-to-zk-snarks/
- https://ethereum.org/en/developers/docs/scaling/zk-rollups/
- https://vitalik.eth.limo/general/2022/06/15/using_snarks.html


ZK-SNARKs is a type of zero-knowledge proof that allows one party to prove to another that a given statement is true, without revealing any additional information. This article provides an overview of zk-SNARKs, how they work, and their applications in blockchain technology.

zk-SNARKs


## What?

- Ethereum scaling solutions
- Separate blockchain
  - Extends Ethereum
  - Inherits the security guarantees of Ethereum
- All user transactions on the Layer 2 can ultimately settle back to Layer 1
- Ethereum also functions as a data availability layer for Layer 2s
  - Layer 2 will post their transactions data onto Ethereum
  - Rely on Ethereum for data availability
    - Used to get the state of Layer 2
    - Dispute transaction of Layer 2

## Why?

Blockchain has 3 desirable properties

- Decentralized
- Secure
- Scalable

"can only achieve 2 out of 3" - Blockchain trilemma

High demand -> Need to scale without sacrificing decentralization and security => Need Layer 2 to scale Blockchain that takes advantage of robust decentralized security of Layer 1

## How?

- Communicate with Layer 1 by submitting bundles of transactions
- Layer 1 handles security, data availability, and decentralization
- Layer 2 handles scaling by computing and sending finalized proofs to Layer 1 -> Remove transaction loading.

### Rollup

- Preferred layer 2 scaling solution in Ethereum
- Reduce gas fees by up to 100x compared to Layer 1
- Rollup bundle ("roll up") hundreds of transactions into a Layer 1 transaction => Fee will be dived/distributed to all users (owners of these hundreds of transactions) -> Cheaper
  - For example:
    - 1 Layer 1 transaction is paid for 1eth as fees
    - 100 Layer 2 transactions rolled up in 1 Layer 1 transaction are also paid for 1eth. So 1 Layer 2 transaction is just only needed 0.01 eth to execute.
- Rollup is executed outside Layer 1 (in Layer 2), but finalized result (proof) is submitted to Layer 1 => and can be secured by Layer 1 security mechanisms.
- Have 2 approaches (different on posting transaction data to L1):
  - Optimistic
  - ZK Rollups aka Zero-knowledge Rollups

## Example

- Arbitrum One
- Optimism
- Boba Network

## References

- https://ethereum.org/en/layer-2


Explore Layer 2 solutions for Ethereum scaling, including rollups and their benefits. Learn how Layer 2 extends Ethereum's capabilities, reduces gas fees, and maintains security while improving transaction throughput and efficiency.