As systems scale to meet growing demands, databases must evolve alongside them to maintain performance and integrity. This document explores best practices for managing database changes, maintaining knowledge, and ensuring smooth integration. Topics covered include knowledge sharing, repository structuring, continuous integration, and database refactoring, with real-world examples illustrating their application.

Evolutionary Database Design: Managing Change and Scaling with the System

A comprehensive guide to understanding circular references in database design, including types, challenges, real-world applications, and solutions for managing them effectively. Covers self-references, circular dependencies, and strategies for maintaining data integrity while working with circular relationships.

Database design Circular

Learn how to transition from application development to data engineering by building a modern project reporting system. Discover essential mindset shifts, best practices, and hands-on look using tools like DuckDB, and Modal. Master system-level architecture and data pipeline design for scalable enterprise solutions.

A Lens to Modern Data Engineering

Building a Data-Driven Project Reporting System: A Lens into Modern Data Engineering

When we started working on this project aimed at the creative community, we faced an interesting challenge. We needed to build a model that was similar to a task but with a key difference: it had to support custom, dynamic, and extensible properties. If you’ve ever used Notion and appreciated how flexible its objects are, you’ll know exactly what we were trying to achieve.

Designing a model with dynamic properties

Local-first software is an approach to software development that emphasizes data ownership, offline functionality, and real-time collaboration. This model ensures data is primarily stored on the user's device, enhancing privacy and control while enabling seamless synchronization and collaboration without relying on continuous internet connectivity

Local-first Software

A Bloom Filter is a probabilistic data structure used for testing whether an element is a member of a set or not. It's space-efficient compared to other data structures like hash tables, but it may give false positives (indicating that an element is in the set when it's not) and never gives false negatives (indicating that an element is not in the set when it actually is not)...

Bloom Filter

To improve and strongly go-live the data pipeline, besides apply best practices and pillar for Data Pipeline Native Solution, a design framework and pattern are robustly help us in...

Data Pipeline Design Framework

The motivation of this thoughts that trying to discover the LLMs and Generative AI...

Quick Learning Vector Database

Google Data Fusion is a fully managed, cloud-native data integration service that enables users to efficiently build and manage ETL/ELT data pipelines. It is designed to streamline data engineering tasks for enterprise users and is built on top of the open-source project CDAP.

Google Data Fusion

Google Dataproc is a fast and easy-to-use, fully managed cloud service for running Apache Spark and Hadoop workloads. It allows users to process and analyze large-scale data efficiently and at scale.

Google Dataproc

Conflict-free Replicated Data Types (CRDTs) (aka convergent replicated data type or commutative replicated data type) are data structures that can be replicated across multiple computers in a network, where the replicas can be updated independently and concurrently without coordination between them, and enable operations to always converge to a final state consistent among all replicas.

Introduction to CRDT

Structured Query Language (SQL) is widely used for managing data in relational database management systems (RDBMS). In the context of SQL, the optimization of database queries forms the foundation for efficient data retrieval, providing quicker response times and increasing overall database performance. One key strategy to optimize SQL queries involves the concept of "Sargable" queries...

SQL Saragable Queries and Their Impact on Database Performance

Sequential read/write operations involve accessing data in a continuous, linear manner. This typically occurs when transferring large files or accessing a large file on the drive. In sequential read/write operations, the drive can read or write data from a series of blocks, allowing for faster performance...

SQL and how it relates to Disk Reads and Writes

Managing Dataflow And Sql Database With Concurrency Control

As an engineer at Dwarves, you will be working closely with a team of talented, kind people and working directly with our clients. There is a lot of freedom to contribute to the quality of the project and improve, or prove yourself.

Energy - Data Engineering

Multi-column index in DB

Materialized view is the cache of views. It pre-computes, stores, and optimizes data access when created, and automatically refreshes to ensure real-time data availability.

Materialized View Pattern

We are the head of census bureau for the state of California and tasked with finding the finding the population of all cities in California. All the resources we want are ready but we have only four months to finish the task. Calculating the population of all cities for a big state like california is not an easy task. The sensible thing to do is to divide the state by city and make individuals in charge of each city to calculate the population of each city where he is in charge of...

MapReduce

Tom Nguyen, Dwarves' Data Lead, shares how remote working fits his lifestyle perfectly, allowing him to stay productive without distractions while maintaining effective team collaboration

Tom Nguyen

#3 Tom Nguyen on remote working

Sql Practices Orm Vs Plain Sql

How do we migrate normal table to timescale table to optimized data storage

Migrate regular tables into TimescaleDB hypertables to improve query performance

This article provides an overview of the importance of tracking Bitcoin-Altcoin performance indicators in a trading strategy known as Hedge, and explains how to visualize this data effectively. It also demonstrates how to render a chart for this strategy using Matplotlib and Seaborn

Tracking Bitcoin-Altcoin Performance Indicators in BTC Hedging Strategy

Discover how a trading platform mitigated database access risks, enhanced security, and ensured data integrity through role-based access control, network isolation, MFA, and robust logging. Learn about the strategies and tools, like Teleport, that transformed operational efficiency and reinforced client trust.

Database hardening for a trading platform

A guide to implementing data archival and recovery strategies for high-volume transactional application.

Building a data archive and recovery strategy for high-volume trading system

A technical exploration of implementing the data snapshot pattern for efficient historical data persistence

Implementing data snapshot pattern to persist historical data

A technical case study of creating an automated system that downloads videos, processes audio, and generates transcripts using AI services like Groq and OpenAI.

Building data pipeline for OGIF transcriber

A deep dive into building a robust transfer tracking syste m for Binance accounts, transforming disconnected transaction logs into meaningful fund flow narratives through SQL and data analysis

Building better Binance transfer tracking

Implementing a Golang-based visualization for crypto market performance indicators, focusing on Bitcoin vs Altcoin dynamics and trading strategy effectiveness through interactive charts and data analysis

Visualizing crypto market performance: BTC-Alt dynamic indicators in Golang

A comprehensive guide on improving cryptocurrency transfer logging systems to provide better transparency and traceability for users and developers.

Transfer mapping: enhancing loggers for better transparency

A detailed look at how we rebuilt historical trading PnL data through an efficient data pipeline approach, transforming a complex problem into a maintainable solution.

Reconstructing historical trading PnL: a data pipeline approach

An in-depth look at Dwarves' monthly Project Reports system - a lean, efficient system that transforms communication data into actionable intelligence for Operations teams. This case study explores how we orchestrate multiple data streams into comprehensive project insights while maintaining enterprise-grade security and cost efficiency.

Project reports system: a case study

An exploration of natural language to database query systems using MongoDB, examining system prompts and implications for data engineering and agentic workflows.

Natural Language to Database Queries: Text-to-MongoDB

In spite of having taken the world by storm, Large Language Models(LLM) still has some limitations such as limited context window and a knowledge cutoff date. Retrieval-Augmented Generation(RAG) steps in to bridge this gap by allowing LLMs to access and utilize external knowledge sources beyond their training data. However, data is not text based only, it also can be image, audio, table in docs,...

Multimodal in rag

This article explores chunking strategies to handle context limitations in Large Language Models (LLMs) like GPT. It covers preprocessing data, selecting chunk sizes, and how to maintain coherence in various content types such as unstructured, Markdown, and LaTeX.

Chunking strategies to overcome context limitation in LLM

An exploration of query caching in Large Language Models (LLMs), focusing on how semantic vector databases can drastically improve efficiency and reduce computational costs by reusing cached answers for semantically similar queries.

Query Caching for Large Language Models

Data-fetching libraries are software tools or frameworks that can help improve the performance and scalability of your application by handling network requests and data processing more efficiently.

Render optimization in data-fetching libraries

React Server Components (RSCs) offer the flexibility to decide where to render components based on their purpose, diverging from the client-side rendering approach of Single-Page Applications.

React Server Components, NextJs Route and Data Fetching

An overview of vector databases and their application in LLM systems. This article covers the history and core concepts of vector databases, their use cases, implementation considerations with LLMs, and a comparison of popular vector database options.

Select Vector Database for LLM

An in-depth exploration of arrays and slices in Go, covering their differences, internal implementations, and key operations like append. Learn about fixed-length arrays, flexible slices, and how Go manages memory allocation for growing slices.

#data

#

B

C

D

E

G

I

L

M

N

P

Q

R

S

T

V