Data
Data engineering: pipelines, warehouses, time series and storage choices.
Reports
- Migrate regular tables into TimescaleDB hypertables to improve query performance
How do we migrate normal table to timescale table to optimized data storage
- Database hardening for a trading platform
Discover how a trading platform mitigated database access risks, enhanced security, and ensured data integrity through role-based access control, network isolation, MFA,…
- Building a data archive and recovery strategy for high-volume trading system
A guide to implementing data archival and recovery strategies for high-volume transactional application.
- Building better Binance transfer tracking
A deep dive into building a robust transfer tracking syste m for Binance accounts, transforming disconnected transaction logs into meaningful fund flow narratives…
- Reconstructing historical trading PnL: a data pipeline approach
A detailed look at how we rebuilt historical trading PnL data through an efficient data pipeline approach, transforming a complex problem into a maintainable solution.
- Visualizing crypto market performance: BTC-Alt dynamic indicators in Golang
Implementing a Golang-based visualization for crypto market performance indicators, focusing on Bitcoin vs Altcoin dynamics and trading strategy effectiveness through…
Writing
- Data flow in Brainery
How data flows through Brainery's second brain system, from Discord, Memo Blog, and GitHub sources to TimescaleDB via MCP, with an LLM-powered interface for natural…
- Database design and philosophy
A migration-less, append-only database design using TimescaleDB's hypertables and continuous aggregates to capture observational data and detect emerging patterns…
- Promoting raw data to insight
This post explains the multi-stage process of transforming raw data into structured information, and then synthesizing that information into persistent insight and…
Research notes
- Evolutionary database design: managing change and scaling with the system
As systems scale to meet growing demands, databases must evolve alongside them to maintain performance and integrity. This document explores best practices for managing…
- Database design circular
A comprehensive guide to understanding circular references in database design, including types, challenges, real-world applications, and solutions for managing them…
- Building a data-driven project reporting system: A lens into modern data engineering
Learn how to transition from application development to data engineering by building a modern project reporting system. Discover essential mindset shifts, best…
- Designing a model with dynamic properties
When we started working on this project aimed at the creative community, we faced an interesting challenge. We needed to build a model that was similar to a task but…
- Building a local search engine for our Memo website
Explore how we developed a fully local search engine for our memo website using DuckDB-wasm, Transformers.js, and Alpine.js. Learn about hybrid search techniques,…
- Data pipeline design framework
To improve and strongly go-live the data pipeline, besides apply best practices and pillar for Data Pipeline Native Solution, a design framework and pattern are robustly…
- SQL saragable queries and their impact on database performance
Structured Query Language (SQL) is widely used for managing data in relational database management systems (RDBMS). In the context of SQL, the optimization of database…
- A cached table for fast Binance kline API data
Learn how to speed up Binance Kline API data retrieval by using a cached table to efficiently get highest and lowest cryptocurrency prices and timestamps within large…
Show all 41 notes
- dbt for faster data transformation in the warehouse
Discover how DBT (data build tool) simplifies data transformation in warehouses with modular SQL, automation, and software engineering practices for faster, reliable…
- ¶ Local-first software
Local-first software is an approach to software development that emphasizes data ownership, offline functionality, and real-time collaboration. This model ensures data…
- Bloom filter
A Bloom filter is a probabilistic data structure used for testing whether an element is a member of a set or not. It's space-efficient compared to other data structures…
- Introduction to CRDT
Conflict-free Replicated Data Types (CRDTs) (aka convergent replicated data type or commutative replicated data type) are data structures that can be replicated across…
- Sql and how it relates to disk reads and writes
Sequential read/write operations involve accessing data in a continuous, linear manner. This typically occurs when transferring large files or accessing a large file on…
- Building a Redis leaderboard with sorted sets and hashes
Learn how to build a fast user leaderboard system using Redis sorted sets and hashes to track scores and store user profiles like usernames and avatars.
- How Discord stores messages: from MongoDB to Cassandra
Learn how Discord migrated from MongoDB to Cassandra for scalable, low-maintenance data storage, handling billions of messages with predictable performance and eventual…
- Database locking in PostgreSQL: explicit locks for concurrency control
Learn how explicit locking techniques in PostgreSQL help manage concurrency control and prevent data conflicts in high workload databases by using table-level,…
- Self-balancing binary search trees, part 1: trees and binary trees
What a tree is, the vocabulary for talking about one, and the properties and kinds of binary trees, as groundwork for balanced search trees.
- Self-balancing binary search trees, part 2: why a BST needs to be balanced
How a binary search tree orders its keys, why that helps a runway-reservation system, why tree height matters, and what balancing has to guarantee.
- Self-balancing binary search trees, part 3: AVL trees
The AVL invariant, the proof that it keeps tree height at O(log n), and how insertions and deletions restore the invariant with rotations.
- Data Vault modelling: hubs, links and satellites for flexible data lakes
Learn how Data Vault architecture builds flexible, scalable data lakes using hubs, links, and satellites to manage business keys and historical data with pros and cons…
- Hive window and analytic functions: a 10-day moving average example
Learn how to use Hive window and analytical functions to calculate complex metrics like a 10-day moving average on stock data with easy-to-understand SQL examples and…
- Buckets in Apache Hive: fewer tiny files and efficient sampling
Learn how Apache Hive buckets solve issues with dynamic partitions by reducing tiny files, enabling efficient sampling, and improving query performance through bucketing…
- Apache Hive partitions: static and dynamic partitioning for faster queries
Learn how Hive partitions and dynamic partitions optimize query performance by targeting specific data subsets, reducing scan times, and efficiently managing large…
- Hive ORDER BY, SORT BY, DISTRIBUTE BY and CLUSTER BY compared
Learn how to optimize Hive queries by using ORDER BY, SORT BY, DISTRIBUTE BY, and CLUSTER BY to efficiently order large datasets with multiple reducers and improve…
- Hive managed tables versus external tables
Learn the key differences between Hive managed tables and external tables, including when to use each type and how dropping tables affects their underlying data in HDFS.
- Materialized view pattern
Materialized view is the cache of views. It pre-computes, stores, and optimizes data access when created, and automatically refreshes to ensure real-time data…
- Behind a Hive table: structure, storage formats and SerDe
Learn how to create and understand Hive tables, including table structure, storage formats, SerDe functions, HDFS data location, and metadata management in the Hive…
- An introduction to Apache Hive: SQL queries on Hadoop data
Learn how Apache Hive simplifies big data analysis by enabling SQL queries on Hadoop datasets using tables, making MapReduce easier for developers and data analysts.
- ¶ MapReduce
We are the head of census bureau for the state of California and tasked with finding the finding the population of all cities in California. All the resources we want…
- MapReduce components, part 1: input split, map and reduce
The MapReduce problem statement, then how input splits, the map phase and the reduce phase work.
- MapReduce components, part 2: the shuffle phase and combiner
How the shuffle phase moves map output to reducers and how a combiner cuts the data it moves.
- Full-text search in PostgreSQL with GIN indexes
Learn how to boost PostgreSQL search performance using full-text search with GIN indexes, stop words, and custom configurations for faster and flexible text queries on…
- Hadoop distributed file system, part 1: why HDFS and how blocks work
Why Hadoop needs its own file system, how HDFS compares with local file systems, and how block storage and replication work.
- Hadoop distributed file system, part 2: working with HDFS, reads and writes
The HDFS shell commands, then how the name node and data nodes handle a read, a write and replica placement.
- Apache Hadoop and big data, part 1: what big data is, why Hadoop, and the problem
The three Vs of big data, its use cases and challenges, why traditional solutions fall short, how Hadoop answers them, and a 1 TB stock-price problem to test the answer.
- Apache Hadoop and big data, part 2: execution time on one machine and the history of Hadoop
How long a single machine takes to read and process 1 TB, which shows why distributed processing is needed, followed by how Hadoop grew out of the Nutch web crawler.
- The ACID model for reliable database transactions
Learn how the ACID model ensures database transaction reliability with atomicity, consistency, isolation, and durability for accurate, secure, and concurrent data…
- The CAP theorem: consistency, availability and partition tolerance
Learn about the CAP theorem, which explains how distributed databases balance consistency, availability, and partition tolerance, and how different database types…
Playbooks
- Database migration
Checklist and tooling pointers for database migrations on mobile and backend projects, with a safe rollout sequence.
Series
Tags in this topic
Writers
dudaka14, monotykamary9, baenv4, mirageruler3, Bachpx29112, fuatto2, ngolapnguyen2, nnhuyhoang2