As an engineer at Dwarves, you will be working closely with a team of talented, kind people and working directly with our clients. There is a lot of freedom to contribute to the quality of the project and improve, or prove yourself.

Energy - data engineering

Tom Nguyen, Dwarves' Data Lead, shares how remote working fits his lifestyle perfectly, allowing him to stay productive without distractions while maintaining effective team collaboration

Tom Nguyen

#3 Tom Nguyen on remote working

data-erd

data-lifecycle

database-sharding

Multi-MCP data integration - receiving data from other MCP servers

sql-practices

Latest study and application of Data Engineering

§ Data Engineering

Materialized view is the cache of views. It pre-computes, stores, and optimizes data access when created, and automatically refreshes to ensure real-time data availability.

Materialized view pattern

Multi-column index in DB

Learn how to transition from application development to data engineering by building a modern project reporting system. Discover essential mindset shifts, best practices, and hands-on look using tools like DuckDB, and Modal. Master system-level architecture and data pipeline design for scalable enterprise solutions.

A Lens to Modern Data Engineering

Building a data-driven project reporting system: A lens into modern data engineering

A Bloom filter is a probabilistic data structure used for testing whether an element is a member of a set or not. It's space-efficient compared to other data structures like hash tables, but it may give false positives (indicating that an element is in the set when it's not) and never gives false negatives (indicating that an element is not in the set when it actually is not)...

Bloom filter

To improve and strongly go-live the data pipeline, besides apply best practices and pillar for Data Pipeline Native Solution, a design framework and pattern are robustly help us in...

Data pipeline design framework

A comprehensive guide to understanding circular references in database design, including types, challenges, real-world applications, and solutions for managing them effectively. Covers self-references, circular dependencies, and strategies for maintaining data integrity while working with circular relationships.

Database design circular

When we started working on this project aimed at the creative community, we faced an interesting challenge. We needed to build a model that was similar to a task but with a key difference: it had to support custom, dynamic, and extensible properties. If you’ve ever used Notion and appreciated how flexible its objects are, you’ll know exactly what we were trying to achieve.

Designing a model with dynamic properties

As systems scale to meet growing demands, databases must evolve alongside them to maintain performance and integrity. This document explores best practices for managing database changes, maintaining knowledge, and ensuring smooth integration. Topics covered include knowledge sharing, repository structuring, continuous integration, and database refactoring, with real-world examples illustrating their application.

Evolutionary database design: managing change and scaling with the system

Conflict-free Replicated Data Types (CRDTs) (aka convergent replicated data type or commutative replicated data type) are data structures that can be replicated across multiple computers in a network, where the replicas can be updated independently and concurrently without coordination between them, and enable operations to always converge to a final state consistent among all replicas.

Introduction to CRDT

Local-first software is an approach to software development that emphasizes data ownership, offline functionality, and real-time collaboration. This model ensures data is primarily stored on the user's device, enhancing privacy and control while enabling seamless synchronization and collaboration without relying on continuous internet connectivity

¶ Local-first software

Learn how explicit locking in PostgreSQL helps manage concurrency control in high-workload databases by preventing data conflicts with table-level, row-level, and advisory locks.

Managing dataflow and SQL database with concurrency control

We are the head of census bureau for the state of California and tasked with finding the finding the population of all cities in California. All the resources we want are ready but we have only four months to finish the task. Calculating the population of all cities for a big state like california is not an easy task. The sensible thing to do is to divide the state by city and make individuals in charge of each city to calculate the population of each city where he is in charge of...

¶ MapReduce

The motivation of this thoughts that trying to discover the LLMs and Generative AI...

Quick learning vector database

Sequential read/write operations involve accessing data in a continuous, linear manner. This typically occurs when transferring large files or accessing a large file on the drive. In sequential read/write operations, the drive can read or write data from a series of blocks, allowing for faster performance...

Sql and how it relates to disk reads and writes

Learn how to choose between raw SQL, ORM, and SQL generators, optimize database performance with smart indexing, avoid rounding errors, and write efficient queries for better security and speed.

Sql practices orm vs plain sql

Structured Query Language (SQL) is widely used for managing data in relational database management systems (RDBMS). In the context of SQL, the optimization of database queries forms the foundation for efficient data retrieval, providing quicker response times and increasing overall database performance. One key strategy to optimize SQL queries involves the concept of "Sargable" queries...

SQL saragable queries and their impact on database performance

Data-fetching libraries are software tools or frameworks that can help improve the performance and scalability of your application by handling network requests and data processing more efficiently.

Render optimization in data-fetching libraries

An in-depth exploration of arrays and slices in Go, covering their differences, internal implementations, and key operations like append. Learn about fixed-length arrays, flexible slices, and how Go manages memory allocation for growing slices.

Slice and array in Golang

collecting-analytics-data-is-not-the-same-as-doing-research

This article explores chunking strategies to handle context limitations in Large Language Models (LLMs) like GPT. It covers preprocessing data, selecting chunk sizes, and how to maintain coherence in various content types such as unstructured, Markdown, and LaTeX.

Chunking strategies to overcome context limitation in LLM

An overview of how to implement long-term memory in a ChatGPT-based chatbot using a Vector Database (VectorDB) to store conversation history and retrieve the most relevant past interactions.

Storing long-term memory in ChatGPT using VectorDB

An exploration of query caching in Large Language Models (LLMs), focusing on how semantic vector databases can drastically improve efficiency and reduce computational costs by reusing cached answers for semantically similar queries.

Query caching for large language models

In spite of having taken the world by storm, Large Language Models(LLM) still has some limitations such as limited context window and a knowledge cutoff date. Retrieval-Augmented Generation(RAG) steps in to bridge this gap by allowing LLMs to access and utilize external knowledge sources beyond their training data. However, data is not text based only, it also can be image, audio, table in docs,...

Multimodal: in rag

An overview of vector databases and their application in LLM systems. This article covers the history and core concepts of vector databases, their use cases, implementation considerations with LLMs, and a comparison of popular vector database options.

Select vector database for LLM

An exploration of natural language to database query systems using MongoDB, examining system prompts and implications for data engineering and agentic workflows.

Natural language to database queries: Text-to-MongoDB

An in-depth look at Dwarves' monthly Project Reports system - a lean, efficient system that transforms communication data into actionable intelligence for Operations teams. This case study explores how we orchestrate multiple data streams into comprehensive project insights while maintaining enterprise-grade security and cost efficiency.

Project reports system: a case study

A deep dive into building a robust transfer tracking syste m for Binance accounts, transforming disconnected transaction logs into meaningful fund flow narratives through SQL and data analysis

Building better Binance transfer tracking

This article provides an overview of the importance of tracking Bitcoin-Altcoin performance indicators in a trading strategy known as Hedge, and explains how to visualize this data effectively. It also demonstrates how to render a chart for this strategy using Matplotlib and Seaborn

Tracking Bitcoin-altcoin performance indicators in BTC hedging strategy

The official Brainery build-log. We're detailing our process for creating a 'second brain' designed for continuous learning and insight generation. Covers motivation, system design, and practical data-to-wisdom strategies.

§ Brainery 🧠

Brainery build log

How data flows through Brainery's second brain system, from Discord, Memo Blog, and GitHub sources to TimescaleDB via MCP, with an LLM-powered interface for natural language queries.

Data flow in Brainery

A migration-less, append-only database design using TimescaleDB's hypertables and continuous aggregates to capture observational data and detect emerging patterns through LLM analysis.

Database design and philosophy

This post explains the multi-stage process of transforming raw data into structured information, and then synthesizing that information into persistent insight and knowledge within a true second brain, leveraging tools like TimescaleDB and LLMs.

Promoting raw data to insight

A technical case study of creating an automated system that downloads videos, processes audio, and generates transcripts using AI services like Groq and OpenAI.

Building data pipeline for OGIF transcriber

Implementing a Golang-based visualization for crypto market performance indicators, focusing on Bitcoin vs Altcoin dynamics and trading strategy effectiveness through interactive charts and data analysis

Visualizing crypto market performance: BTC-Alt dynamic indicators in Golang

A guide to implementing data archival and recovery strategies for high-volume transactional application.

Building a data archive and recovery strategy for high-volume trading system

Discover how a trading platform mitigated database access risks, enhanced security, and ensured data integrity through role-based access control, network isolation, MFA, and robust logging. Learn about the strategies and tools, like Teleport, that transformed operational efficiency and reinforced client trust.

Database hardening for a trading platform

A comprehensive guide on improving cryptocurrency transfer logging systems to provide better transparency and traceability for users and developers.

Transfer mapping: enhancing loggers for better transparency

How do we migrate normal table to timescale table to optimized data storage

Migrate regular tables into TimescaleDB hypertables to improve query performance

A technical exploration of implementing the data snapshot pattern for efficient historical data persistence

Implementing data snapshot pattern to persist historical data

A detailed look into the mcp-playbook's internal code flow, data handling for chat logs and prompts, and interactions with GitHub repositories like prompt-db and prompt-log.

MCP playbook code and data flow

How data flows through the MCP Playbook server, from MCP client requests to tool execution and interactions with the file system and GitHub.

Data flow in MCP Playbook

A detailed look at how we rebuilt historical trading PnL data through an efficient data pipeline approach, transforming a complex problem into a maintainable solution.

#data

B

C

D

E

I

L

M

N

P

Q

R

S

T

V