Learn how to transition from application development to data engineering by building a modern project reporting system. Discover essential mindset shifts, best practices, and hands-on look using tools like DuckDB, and Modal. Master system-level architecture and data pipeline design for scalable enterprise solutions.

A Lens to Modern Data Engineering

Building a Data-Driven Project Reporting System: A Lens into Modern Data Engineering

How we support, contribute to, and reward open source development in our community

☀️ Open source

A guide on how to check in at the office and claim ICY token rewards.

Office check-in process for earning ICY

Discover how the Dwarves community fosters a culture of continuous learning and knowledge sharing. With a monthly reward pool of up to 2500 ICY, contributors are recognized for sharing valuable insights, especially in areas like AI/LLM, Golang, and more. Get involved and grow with us.

Record and reward sharing at Dwarves

Explore how we developed a fully local search engine for our memo website using DuckDB-wasm, Transformers.js, and Alpine.js. Learn about hybrid search techniques, real-time embeddings, and performance optimizations that deliver fast and accurate results without compromising on privacy or requiring server-side processing.

Building a Local Search Engine for Our Memo Website

Introducing the OGIF Memo Summarizer, a chatbot we developed using Dify in collaboration with @innno_. This tool transcribes YouTube videos and extracts key points from our Oh God It’s Friday (OGIF) sessions. By providing both short and detailed summaries in English and Vietnamese, it significantly enhances our ability to review and reference the diverse knowledge shared every Friday.

How we crafted the OGIF summarizer bot to streamline weekly knowledge-sharing

This is where we share our journey in the ever-evolving landscape of technology. It’s a space where we unveil our latest breakthroughs, discuss our research, and delve into our innovative projects. Just as an artist expresses their vision on a canvas, we use this platform to paint a picture of our technological advancements and how they’re shaping the future.

Tech Canvas

Our Discord isn't just another company server; it's a place for both our team and the wider developer community. This guide explains how it works and where you fit in.

Discord

Dwarves Network Discord

The Dwarves bounty program is the means through which both company peeps and the community can contribute to our daily activities. This includes tasks like building internal tools, engaging in new technology research and development, or sharing knowledge session

👾 Open bounties

Our fifth weekly catchup to kick off the new year! We organize an agenda the day beforehand and consolidate our findings, notable research progress, possibly new tech, insight or technique everyone should know.

Labs Weekly Catchup #5

Our fourth weekly catchup. We organize an agenda the day beforehand and consolidate our findings, notable research progress, possibly new tech, insight or technique everyone should know.

Labs Weekly Catchup #4

We're a team combining tech skills, problem-solving, and clear communication. We help businesses overcome challenges by finding root causes, creating practical solutions, and working closely with clients to implement them effectively.

💼 Consulting team

Our third weekly catchup. We organize an agenda the day beforehand and consolidate our findings, notable research progress, possibly new tech, insight or technique everyone should know.

Labs Weekly Catchup #3

Our second weekly catchup. We organize an agenda the day beforehand and consolidate our findings, notable research progress, possibly new tech, insight or technique everyone should know.

Labs Weekly Catchup #2

Looking for a well-rounded backend engineer with experience in shipping web applications to production, CI/CD with docker centric workflow, unit testing, performance and scaling, etc. with Go/Elixir/Rust.

Backend Engineer, Go/Elixir/Rust

Home

When we conduct an estimation, it is recommended to abandon the transitional “exact hours” assessment method, instead, use the story point based on the Fibonacci number (1, 2, 3, 5, 8, 13, 21…). The number expresses an estimation of the overall effort required to fully implement a backlog item or any piece of work.

Estimation Guidelines

Our first weekly catchup. We organize an agenda the day beforehand and consolidate our findings, notable research progress, possibly new tech, insight or technique everyone should know.

Labs Weekly Catchup #1

DuckDB is a great starting point for modern stacks that don't need to worry about scalability, but want to derive hard analytics from the data that they have. This meeting will do a 3-part demo of DuckDB (15 minutes total):
• Consolidate NYC taxi data in parallel (through map-reduce) over HTTPS
• Aggregate IBM AML transactions (Anti-Money Laundering)
• Some examples of how it is used to consolidate and do basic analytics on note.d and log.console.so

DuckDB demo and showcase

Explores using Golang and the Pion WebRTC library to build high-performance, scalable, low-latency video streaming infrastructure. Covers WebRTC basics (ICE, STUN, TURN), Go's concurrency benefits, Pion's native Go implementation, and a real-world multi-stream security monitoring use case.

Leveraging Golang and WebRTC for High-Performance Video Streaming

Structured Query Language (SQL) is widely used for managing data in relational database management systems (RDBMS). In the context of SQL, the optimization of database queries forms the foundation for efficient data retrieval, providing quicker response times and increasing overall database performance. One key strategy to optimize SQL queries involves the concept of "Sargable" queries...

SQL Saragable Queries and Their Impact on Database Performance

Sequential read/write operations involve accessing data in a continuous, linear manner. This typically occurs when transferring large files or accessing a large file on the drive. In sequential read/write operations, the drive can read or write data from a series of blocks, allowing for faster performance...

SQL and how it relates to Disk Reads and Writes

State, explain, link is a basic style for organizing and explaining ideas in an understandable way. This technique may have many names, but in essence it is a way to organize paragraphs in roughly 3 sentences or more.

State, Explain, Link - An all-purpose writing technique

This guide explores implementing robust authorization for Model Context Protocol (MCP) over Server-Sent Events (SSE) transport, providing a standardized framework for secure AI-to-tool communication while maintaining vendor independence.

Securing your remote MCP servers

A comprehensive guide to implementing granular access control for Model Context Protocol (MCP) servers, allowing organizations to securely expose tool capabilities based on user roles and permissions while maintaining data privacy.

Tool-Level Security for Remote MCP Servers

AI agents, talent shifts, & market dynamics: Explore tech trends in AI, blockchain, & software engineering. Discover key insights & analysis.

Forward Engineering Feb 2025

A case study exploring how we built an AI-powered travel assistant using Ruby and AWS Bedrock, demonstrating how choosing the right tools over popular choices led to a more robust and maintainable solution. This study examines our approach to integrating AI capabilities within existing Ruby infrastructure while maintaining enterprise security standards.

AI-powered Ruby travel assistant

Project Operations Copilots is a suite of AI assistants designed to streamline project management and reporting processes. These copilots, built using Dify and custom implementations, help teams generate insightful project reports, create comprehensive project charters, and facilitate smooth project handovers. This approach represents a shift towards AI-assisted project management, enhancing team efficiency and project success rates.

Project Operations Copilots

Team Copilots is a suite of AI assistants designed to enhance team collaboration and productivity. These copilots, built using Dify and custom implementations, help teams manage tasks, facilitate communication, and improve overall efficiency. This approach represents a shift towards AI-assisted team management, enhancing team performance and project success rates.

Team Copilots

An in-depth look at Dwarves' monthly Project Reports system - a lean, efficient system that transforms communication data into actionable intelligence for Operations teams. This case study explores how we orchestrate multiple data streams into comprehensive project insights while maintaining enterprise-grade security and cost efficiency.

Project reports system: a case study

An exploration of natural language to database query systems using MongoDB, examining system prompts and implications for data engineering and agentic workflows.

Natural Language to Database Queries: Text-to-MongoDB

Explore the October 2024 market report: chatbots as productivity essentials with future potential, AI's role despite investment bubble concerns, and Python's rise in modern tech and AI. Learn why coding tools require strong practices, why companies seek versatile engineers, and how Big Tech's job market favors senior talent, posing challenges for juniors.

October 2024

Market report October 2024

Our visit to DZS Media revealed surprising parallels between film production and software development. Their meticulous planning, focus on getting things right the first time, and expertise-driven approach offers valuable insights for our engineering practices.

DZS Media

#2 Film production lessons for software engineering

In this month's market report, we've covered the hype and skepticism around OpenAI's reasoning models, the quiet rise of lightweight AI, the misuse of serverless, and the shakeup in web frameworks.

September 2024

Market report September 2024

In Q3/2024 Forward Engineering, we explore new tech stacks, engineering excellence, and key trends from the tech market over the past three months, with a focus on AI. Learn about tools like Dify, LangGraph, RAG, and more as we share our experiments and insights. Whether you’re interested in AI-driven workflows or tech market trends like the surge in AI startups and hiring demand.

Forward Engineering September 2024

Journey of Thought Prompting is an innovative technique that leverages AI to craft more effective prompts for large language models. This approach uses the analytical capabilities of AI assistants to help users create detailed system prompts, fill in missing details, and iteratively refine their prompt engineering skills. It represents a shift towards collaborative AI interaction, enhancing problem-solving capabilities for businesses and engineers.

Journey of Thought Prompting: Harnessing AI to Craft Better Prompts

Explore the August 2024 market report to discover how AI and LLM tools like Cursor, Claude 3.5 Sonnet, and Amazon Q are transforming software development. Learn about the shift from generative UI to generative apps, the impact of prompt caching by Anthropic, and the rise of structured outputs in AI.

August 2024

Market report August 2024

In July's market report, we explore Meta's Llama 3.1 and its competitive edge in AI, OpenAI's fine-tuning for GPT-4o-mini, and Claude Sonnet 3.5's rapid development improvements. We also highlight advancements in sports analytics, multilingual text-to-speech, cybersecurity challenges, and trends in software engineering, including "boring" technology and evolving programming language salaries

July 2024

Market report July 2024

Discover how managed LLMOps platforms like Dify streamline the development of AI-powered internal tools. Learn about the challenges of integrating LLMs, the benefits of managed solutions, and real-world examples of efficient AI tool creation. Perfect for businesses looking to enhance productivity with AI without extensive technical expertise."

Streamlining Internal Tool Development with Managed LLMOps: A Dify Case Study

An overview of RAPTOR, a novel approach for improving retrieval-augmented language models for long documents using hierarchical tree summaries.

RAPTOR: Tree-based Retrieval for Language Models

In May's market report, we explore GPT-4o's advancements, the rising demand for veteran programmers, the transformative power of prescriptive AI, React 19's impact on network gaps, the growing complexity of GraphQL codebases, and Bend, a new parallel programming language.

May 2024

Market report may 2024

In april’s market report, we explore the evolving landscape of Meta's Llama 3, LLM, the shift to edge computing, the new Bun 1.1, the trend of copy-pastes UI.

April 2024

Market report April 2024

Our third office hours community discussion, where we'll cover diverse topics ranging from the current state of the dwarf community and their experiences with devbox, insights on and continuation of liquidity markets, advancements in generative AI technology, to fun events like a lucky draw for icy. The goal is to foster learning by sharing weekly topics suggested through tags, encouraging collaborative growth among our members.

#3 Generative AI, Tokenomics, and Finance Talks

OGIF Office Hours #3 - Generative AI, Tokenomics, and Finance Talks

Our second Office Hours. Join the community to exchange knowledge and insights on diverse topics, including Docker alternatives with Nix, security practices and origin stories of our standards, financial discussions on liquidity, company updates, and icy draws.

#2 Devbox as the new Docker, Security Standards, and Understanding Liquidity

OGIF Office Hours #2: Devbox as the new Docker, Security Standards, and Understanding Liquidity

Our first ever Office Hours in our series of OGIFs. Our first day to exchange knowledge and insights on topics and projects we're working on and tools we're using for our internal work and clients.

#1 Markdown Presentations, Research Pipeline, Screenshots How-to

OGIF Office Hours #1: Markdown Presentations, Research Content Pipeline, and Professional Screenshots

In this month’s market report, we explore the evolving landscape of artificial intelligence, API techniques, toolings, programming languages, and the intersection of design and engineering. Our insights shed light on the dynamic shifts and emerging trends across these domains, highlighting the advancements that are shaping the future of technology and software development.

March 2024

Market report March 2024

This month has been a continuation of AI and an upwards trend on FinOps and CyberSecurity. On top of those domains, there has been smaller, but very notable trends on the growth of Software as a Service (SaaS) or MicroSaaS' have rebound and created significant growth for serverless services, as a means to avoid startup and initial running costs. Likewise, AI has been changing how we work and how we design systems to enrich workflows and create more naturally interactive apps.

February 2024

Market report February 2024

Here are some of the few trends we are seeing across the industry, in our projects, as well as our community. This market report isn't exhaustive, at least just yet, with certain trends not yet listed such as Blockchain, data management, DevEx, etc. We see a lot of promising trends and we hope that understanding what engineers and firms are becoming more passionate about that it will help us create a foundation of ideas and knowledge.

January 2024

Market report January 2024

December 2023

Market report December 2023

Sign up for our newsletter to stay up-to-date on our latest news, tips, and updates. We'll deliver valuable content straight to your inbox, keeping you informed and engaged with stuff happening at Dwarves.

Changelog

A collection of both our internal and external events, including the things we do with the Labs team, Consulting team, Operations, team, and the community.

Digest

OGIF (Oh God It's Friday) is our weekly casual office hours meeting, where team members unwind, share updates, and connect in a relaxed environment at the end of each work week.

OGIF - Oh God It's Friday

We have focused on revamping how we learn, familiarize, and work with tech. We are excited to announce that we have updated our Forward Engineering to better reflect the feedback and insights from our Labs team, Operations Team, and Consulting Team.

November 2023

Forward Engineering November 2023

As we step into the exciting opportunities of October, we are at a crucial point in the ever-changing world of technology. Our adventure until now has been for the most part very exciting, with a strong focus on 'Full-stack engineering’ Now, we shift our attention to a topic that is prolific and always true: Open-source.

October 2023

Forward Engineering October 2023

Our adventure until now has been incredibly thrilling, with a strong focus on 'LLM in Production.' Now, we shift our attention to a topic that is always important and never outdated: **Fullstack Engineering**.

August 2023

Forward Engineering August 2023

A comprehensive guide outlining the journey for engineers developing applications on top of Large Language Models (LLMs), covering key stages from understanding AI basics to fine-tuning models.

Story map for LLMs

As we usher in the month of June, we embrace the ever-changing landscape of technological innovation, and focus on a central theme:LLM in Production.

June 2023

Forward Engineering June 2023

As we look back on the past month, we're excited to share our latest engineering report, which highlights our recent achievements, projects, and insights in May 2023.

May 2023

Forward Engineering May 2023

This article explores strategies to overcome OpenAI's token limit using Langchain, including stuffing, map-reduce, refining, and map-rerank methods. It discusses the advantages and drawbacks of each approach in handling large documents or prompts while working within the constraints of OpenAI's API.

Workaround with OpenAI's token limit with Langchain

This article explores Langchain document loaders, explaining their role in overcoming token limits, integrating with vector databases, and demonstrating how to build a question-answering system using CSV data and an AI agent.

Working with langchain document loaders

As we look back on the past month, we're excited to share our latest engineering report, which highlights our recent achievements, projects, and insight in March 2023.

March 2023

Forward Engineering March 2023

Forward Engineering December 2023

We want to demonstrate and encourage individual and organizational learning, where both gaining and sharing knowledge is prioritized, valued, and rewarded. Learning is an ongoing process that never ends in our fast-moving industry.

monotykamary

Contributions