§ Data Engineering

Fundamental data system concepts

• Cap theorem - Core theoretical limits of distributed data systems
• ¶ Local-first software - Paradigm enabling collaboration without centralized servers
• Introduction to CRDT
• Bloom filter
• Data pipeline design framework

Data storage technologies

• Hadoop distributed file system (HDFS)
• Vector database fundamentals
  • Quick learning vector database
• Relational database Concepts
  • Database locking
  • Multi-column index in DB
  • SQL and how it Relates to disk reads and writes
  • Evolutionary database design

Big data processing frameworks

• ¶ MapReduce - Programming model for distributed computing
  • MapReduce components
• Apache Hive ecosystem
  • Introduction to Apache Hive
  • Behind a Hive table
  • Partitions on Apache Hive
  • Buckets on Apache Hive
  • Hive window and analytic functions
  • Managed table vs external table
  • Order By vs. Sort By vs. Distribute By vs. Cluster By
• Introduction to Apache Pig
• Google cloud data solutions
  • Google Dataproc
  • Google Data Fusion

Data transformation & analysis

• DBT - the good solution to accelerate data transformation
• Statistics in data analysis
• Data vault modelling
• Overview of BI tools
• Data analysis techniques
  • SQL practices: ORM vs Plain SQL
  • SQL sargable queries and their impact on database performance
  • DuckDB demo and showcase

Specialized data applications

• Full-text search with PostgreSQL
• Creating a fully local search engine on Memo
• Data analyst in retail trading
• Redis leaderboard
• Utilizing cached table for Binance Kline API data processing
• How Discord stores messages - part 1: from MongoDB to Cassandra
• Self-balanced BSTs - AVL trees