
## What is Bloom Filter?
A Bloom Filter is a probabilistic data structure used for testing whether an element is a member of a set or not. It's space-efficient compared to other data structures like hash tables, but it may give false positives (indicating that an element is in the set when it's not) and never gives false negatives (indicating that an element is not in the set when it actually is not).

![](assets/bloom-filter_bloom_filter.webp)

## How Bloom Filter work?

`Initialization`: You start with a bit array of size `m` (usually a large prime number) initially set to all zeros, and `k` different hash functions.

`Adding Elements`: When you want to add an element to the Bloom Filter, you run it through each of the `k` hash functions, each of which maps the element to one of the `m` bits in the array. You then set those bits to `1`.

`Checking Membership`: To check if an element is in the Bloom Filter, you again run it through each of the `k` hash functions. If all `k` bits are set to `1` in the bit array, the element is probably in the set. If any of the bits are `0`, then the element is definitely not in the set. However, due to potential hash collisions, `false positives are possible`.

## When to use?
Imagine that you have a system which millions user, and then you have a user table which had millions of records. So everytime a new user is registered, you have to query millions records table to check username is existed or not which cost `Time complexity = O(n)` (`n` will be vary upto millions) in case not aggregate further. Instead of that, you can `Checking membership` in Bloom Filter which just have `Time complexity = O(k)` (k is constant number of hash function) and then response to user that username is alreay existed. The result has a small probability of false positive(username not existed but response existed) due to potential hash collion but it acceptable by notify user can register with other username.

Besides above scenario, Bloom Filter can apply in other case like:

- `Caching Systems`: Bloom Filters are commonly used in caching systems to quickly check if a requested item is likely to be in the cache before performing a more expensive lookup. This helps in reducing cache misses and improving overall system performance.

- `Data Deduplication`: In systems where duplicate data needs to be identified and eliminated efficiently, Bloom Filters can help quickly determine whether a new data item is a duplicate of an existing item in the dataset.

- `Web Crawlers and Search Engines`: Bloom Filters can be used in web crawlers and search engines to quickly check whether a URL has been visited or indexed before, reducing redundant crawling or indexing efforts.

- `Spell Checkers`: Bloom Filters can be employed in spell checkers to quickly determine if a word is misspelled by checking against a large dictionary of correctly spelled words.

- `Network Routing`: In networking applications, Bloom Filters can be used to quickly filter out packets or messages that are not likely to match certain criteria, reducing the computational load on routers and improving network performance.

- `Blockchain and Distributed Systems`: In distributed systems like blockchain networks, Bloom Filters can be utilized for lightweight and efficient synchronization of data among nodes, reducing the amount of data that needs to be transmitted.

Overall, Bloom Filters are suitable for applications where memory usage needs to be minimized, and a small probability of false positives is acceptable. However, they are not suitable for scenarios where false positives are unacceptable or where exact membership information is required.

## Pros and Cons?
### Pros:

- `Space Efficiency`: Bloom Filters are very space-efficient compared to other data structures like hash tables or trees. They require only a fraction of the space that would be needed to store the actual set of elements.

- `Fast Membership Testing`: Bloom Filters provide constant-time (O(k)) complexity for membership testing, regardless of the size of the set. This makes them suitable for large datasets where quick membership checks are required.

- `No False Negatives`: Bloom Filters never produce false negatives. If the filter says an element is not in the set, it's guaranteed to be absent. This property is useful in applications where false negatives are unacceptable.

- `Versatility`: Bloom Filters are versatile and can be used in various applications such as caching, spell checking, network routing, and more. They are particularly useful in scenarios where approximate membership testing is acceptable.

### Cons:

- `False Positives`: Bloom Filters can produce false positives, indicating that an element is in the set when it's not. The probability of false positives increases with the number of elements in the set and the parameters chosen for the Bloom Filter (e.g., size of the bit array and number of hash functions).

- `Cannot Remove Elements`: Bloom Filters do not support element removal. Once an element is added to the filter, it cannot be removed without resorting to complex techniques or resetting the filter entirely.

- `Parameter Sensitivity`: The performance and effectiveness of a Bloom Filter are highly sensitive to the parameters chosen, such as the size of the bit array (m) and the number of hash functions (k). Selecting inappropriate parameters can lead to higher false positive rates or increased memory usage.

- `Limited Operations`: Bloom Filters support only two primary operations: adding elements and checking for membership. They do not support other common operations like element retrieval or iteration over the elements in the set.

- `Hash Function Dependency`: Bloom Filters rely heavily on hash functions for their operation. The quality of the hash functions used can significantly impact the performance and effectiveness of the Bloom Filter.

## Simple Implementation:

Nowadays, many cache system integrated bloom filter as a feature. Can use it without implementation, but if you want try to hand-on. the simple implementation is a example:

```go
package main

import (
	"fmt"
	"hash/crc32"
	"hash/fnv"
)

type BloomFilter struct {
	bitArray []bool
	size     uint
	hashFunc []func(data string) uint
}

func NewBloomFilter(size uint) *BloomFilter {
	bf := &BloomFilter{
		bitArray: make([]bool, size),
		size:     size,
		hashFunc: []func(data string) uint{
			func(data string) uint { return uint(fnv.New32().Sum32()) },
			func(data string) uint { return uint(crc32.ChecksumIEEE([]byte(data))) },
			// Add more hash functions here if needed
		},
	}
	return bf
}

func (bf *BloomFilter) Add(data string, numHash int) error {
	if numHash > len(bf.hashFunc) {
		return fmt.Errorf("number of hash functions exceeds the limit")
	}
	for i := 0; i < numHash; i++ {
		index := bf.hashFunc[i](data) % bf.size
		bf.bitArray[index] = true
	}
	return nil
}

func (bf *BloomFilter) Contains(data string) bool {
	for _, hashFunc := range bf.hashFunc {
		index := hashFunc(data) % bf.size
		if !bf.bitArray[index] {
			return false
		}
	}
	return true
}

func main() {
	bloomFilter := NewBloomFilter(100)

	// Add user identities to the Bloom Filter
	userIdentities := []string{"user123", "user456", "user789"}
	for _, identity := range userIdentities {
		bloomFilter.Add(identity, 2)
	}

	// Check for existence of user identities
	fmt.Println(bloomFilter.Contains("user123")) // true
	fmt.Println(bloomFilter.Contains("user789")) // true
	fmt.Println(bloomFilter.Contains("user999")) // false
}
```

A Bloom Filter is a probabilistic data structure used for testing whether an element is a member of a set or not. It's space-efficient compared to other data structures like hash tables, but it may give false positives (indicating that an element is in the set when it's not) and never gives false negatives (indicating that an element is not in the set when it actually is not)...

Bloom Filter


## What is CRDT?
Conflict-free Replicated Data Types (CRDTs) (aka convergent replicated data type or commutative replicated data type) are data structures that can be replicated across multiple computers in a network, where the replicas can be updated independently and concurrently without coordination between them, and enable operations to always converge to a final state consistent among all replicas.

![](assets/introduction-to-crdt_b41416acdcc17fd248cefbd2f333fb6e_md5.webp)

## Why CRDT?
### Addressing Data Modification Dilemma
There are two approaches to handling concurrent modifications in distributed systems:

1. **Strongly Consistent Replication**: In this approach, replicas coordinate for modifications to ensure strong consistency. However, this coordination sacrifices performance and can be limited by the Cap Theorem. The CAP theorem states that in a distributed system, it's impossible to simultaneously guarantee consistency (every read receives the most recent write), availability (every request receives a response), and partition tolerance (the system continues to operate despite network partitions).
2. **Optimistic Replication**: In optimistic replication, users can independently modify data without coordination, which enhances performance. However, this approach may lead to conflicts when multiple replicas receive conflicting modifications. To address conflicts, replicas communicate and resolve conflicts automatically.

### Role of CRDTs
CRDTs (Conflict-free Replicated Data Types) play a crucial role in **optimistic replication systems**. They enable seamless merging of data modifications by performing replication as commutative operations. CRDTs ensure that conflicting modifications from different replicas are resolved automatically, eliminating the need for special conflict resolution code or user intervention. This makes CRDTs a valuable tool for maintaining consistency in distributed systems while maximizing performance.

## How to use CRDT?
CRDTs (Conflict-free Replicated Data Types) are a powerful tool for ensuring data consistency and synchronization in various applications. Here are some examples of how CRDTs can be applied:

**Applications of CRDTs**

- _Mobile Apps_: CRDTs enable seamless synchronization of data across multiple devices used by a single user. This ensures that the user can access and modify their data from any device without conflicts or inconsistencies.
- _Distributed Databases_: CRDTs play a crucial role in maintaining data integrity in distributed databases. They allow replicas of the database to be located in different locations while still ensuring that updates and modifications to the data are propagated correctly across all replicas.
- _Collaboration Software_: CRDTs are particularly useful in collaboration software where multiple users can simultaneously make changes to the same file or data. By using CRDTs, conflicts between concurrent changes can be automatically resolved without the need for centralized coordination or locking mechanisms.
- _Large-scale Data Storage Systems_: CRDTs are instrumental in building large-scale data storage systems that require global scalability. By replicating data using CRDTs, these systems can handle high volumes of data and distribute it across multiple nodes, allowing for efficient and reliable access to the data.

**Decentralized Operation**

- One of the key advantages of CRDTs is that they support decentralized operation. Unlike traditional systems that rely on a single server, CRDTs can thrive in peer-to-peer networks and other decentralized settings. This makes them well-suited for applications and environments where a central authority is not desirable or feasible.

## Conclusion
In short, Conflict-free Replicated Data Types (CRDTs) play a pivotal role in managing conflict resolution. It guarantees the seamless merging of data into a coherent state, irrespective of the modifications made on distinct replicas. Importantly, CRDTs automate this merging process, eliminating the necessity for specialized conflict resolution code or user intervention.

Unlike systems relying on algorithms utilized by platforms such as Google Docs, Trello, and Figma, CRDTs don't presuppose the reliance on a single server. This distinctive quality allows CRDTs to seamlessly integrate into peer-to-peer networks and various decentralized settings, setting them apart in the landscape of distributed data management.

## References
- https://crdt.tech/
- https://redis.com/blog/diving-into-crdts/
- https://jakelazaroff.com/words/an-interactive-intro-to-crdts/#user-content-fn-cvrdt
- https://www.infoq.com/presentations/crdt-production/


Conflict-free Replicated Data Types (CRDTs) (aka convergent replicated data type or commutative replicated data type) are data structures that can be replicated across multiple computers in a network, where the replicas can be updated independently and concurrently without coordination between them, and enable operations to always converge to a final state consistent among all replicas.

Introduction to CRDT


## What are random and sequential reads and writes?
Random and sequential reads and writes refer to the way data is accessed, read, and written on hard disks

![](assets/sql-and-how-it-relates-to-disk-reads-and-writes_disk-sequential-random-access.webp)

Sequential read/write operations involve accessing data in a continuous, linear manner. This typically occurs when transferring large files or accessing a large file on the drive. In sequential read/write operations, the drive can read or write data from a series of blocks, allowing for faster performance.

Random read/write operations, on the other hand, involve accessing small files scattered throughout the drive. This is common when opening multiple files or applications simultaneously, such as a Word document, a spreadsheet, and a web browser. In random read/write operations, the drive needs to access data from random blocks repeatedly, which can result in slower performance. Hard drives have a harder time with random operations compared to SSDs, as the read/write head has to position itself to get the requested data, increasing seek time.

TL;DR: Sequential reads and writes involve accessing data in a continuous, linear manner, while random reads and writes involve accessing small files scattered throughout the drive. Hard drives generally perform better in sequential operations compared to random operations due to the physical limitations of the read/write head.

## What does this mean for database design?
Random and sequential reads and writes have implications for CQRS (Command Query Responsibility Segregation) and database design. CQRS is an architectural pattern that separates read and write operations for a data store, allowing for optimized performance, scalability, and security.

In the context of random and sequential reads and writes, CQRS can benefit from the separation of read and write operations. For example, the read side can use a schema optimized for queries, which may involve more random reads, while the write side can use a schema optimized for updates, which may involve more sequential writes. This separation allows for independent scaling of read and write workloads, potentially reducing lock contentions and improving overall performance.

Moreover, CQRS can take advantage of different storage technologies for read and write operations, such as using SSDs for random reads and traditional hard drives for sequential writes. This flexibility can lead to better performance and resource utilization, depending on the specific use case and requirements of the application.

## SQL Operation on Disk reads/writes
In the context of SQL operations and hard disk actions, different SQL operations can result in either sequential or random read/write actions on the hard disk. Here's a breakdown of some common SQL operations and their corresponding hard disk actions:

1. **SELECT**: Reading data from a table can be either sequential or random, depending on the query and the organization of the data on the disk. If the data is well-organized and the query accesses contiguous blocks, it can result in sequential reads. However, if the data is scattered across the disk, it can result in random reads.

2. **INSERT**: Inserting new data into a table can be either sequential or random, depending on the organization of the data and the table structure. For example, if the data is appended to the end of the table, it can result in sequential writes. However, if the data is inserted into various locations within the table, it can result in random writes.

3. **UPDATE**: Updating existing data in a table can result in random read/write operations, as the data to be updated may be scattered across the disk.

4. **DELETE**: Deleting data from a table can also result in random read/write operations, as the data to be deleted may be scattered across the disk.

5. **INDEX**ing: Creating or updating indexes can result in both sequential and random read/write operations, depending on the organization of the data and the index structure. For example, creating a clustered index can result in sequential writes, while creating a non-clustered index can result in random writes.

## Optimizing SQL for better sequential reads/writes
To optimize PostgreSQL database performance for sequential reads and writes, you can follow these general recommendations:

1. **Table partitioning**: Partition large tables into smaller, more manageable pieces to improve query performance and reduce the amount of data that needs to be scanned.

2. **Indexing**: Create appropriate indexes to speed up query execution and minimize random disk access. Be cautious not to over-index, as it can slow down write operations.

3. **Optimize query execution**: Use the **EXPLAIN** command to analyze query plans and identify potential bottlenecks. Optimize queries by rewriting them or adjusting configuration parameters to improve performance.

4. **Separate tablespaces on different disk drives**: Physically store each tablespace on a different disk drive to prevent the disk from being overloaded with I/O operation requests.

5. **Tune configuration parameters**: Customize PostgreSQL configuration parameters, such as shared_buffers, work_mem, and maintenance_work_mem, to improve read and write performance.

6. **Write-Ahead Log (WAL) configuration**: Adjust WAL configuration parameters, such as wal_buffers, checkpoint_timeout, and checkpoint_completion_target, to optimize write performance.

7. **Filesystem optimization**: Disable `atime` (the timestamp at which the file was last accessed) for the data files to save CPU cycles).

8. **Use CLUSTER or pg_repack**: Reorganize the table data to match the index order, which can improve the performance of sequential scans.

Remember that each PostgreSQL database server's environment is different, so it's essential to test and adjust these recommendations according to your specific needs and use case.

## Conclusion
In summary, SQL operations can result in either sequential or random read/write actions on the hard disk, depending on the organization of the data, the table structure, and the specific operation being performed. The impact of sequential and random reads and writes on SQL database performance can be significant, and it is important to design the database disk storage in such a way that maximum sequential I/O may be performed to optimize performance.

## References
-  https://www.redhat.com/architect/pros-and-cons-cqrs
-  https://www.baeldung.com/cs/sequential-vs-random-write
-  https://superuser.com/questions/1325962/sequential-vs-random-i-o-on-ssds
-  https://medium.com/design-microservices-architecture-with-patterns/cqrs-design-pattern-in-microservices-architectures-5d41e359768c
-  https://learn.microsoft.com/en-us/azure/architecture/patterns/cqrs
-  https://www.howtogeek.com/769286/sequential-vs-random-read-write-operations-for-storage/
-  https://www.usenix.org/legacyurl/impact-sequential-and-random-io
-  http://www.eventstore.com/cqrs-pattern
-  https://medium.com/codex/cqrs-design-pattern-5-things-you-should-know-ecaab3f406cc
-  https://stackoverflow.com/questions/2100584/difference-between-sequential-write-and-random-write
-  https://www.redhat.com/architect/illustrated-cqrs
-  https://kislayverma.medium.com/architecture-pattern-cqrs-7a91e9050b0d
-  https://www.techpowerup.com/forums/threads/sequential-r-w-vs-random-r-w.268820/
-  https://itnext.io/a-practical-guide-to-cqrs-af4e2d797383
-  https://anarsolutions.com/when-to-go-for-cqrs-design-pattern/
-  https://superuser.com/questions/1474993/trying-to-understand-random-access-write-versus-sequential-access-writes
-  https://docs.aws.amazon.com/prescriptive-guidance/latest/modernization-data-persistence/cqrs-pattern.html
-  https://blog.risingstack.com/when-to-use-cqrs/
-  https://news.ycombinator.com/item?id=35878961
-  https://www.upsolver.com/blog/cqrs-event-sourcing-build-database-architecture
-  https://betterprogramming.pub/cqrs-software-architecture-pattern-the-good-the-bad-and-the-ugly-e9d6e7a34daf
-  https://condusiv.com/sequential-io-always-outperforms-random-io-on-hard-disk-drives-or-ssds/
-  https://www.linkedin.com/pulse/how-cqrs-solves-problem-overloading-transactional-database-gontu-1e
-  https://dba.stackexchange.com/questions/285809/do-databases-optimize-random-write-and-read-operations


Sequential read/write operations involve accessing data in a continuous, linear manner. This typically occurs when transferring large files or accessing a large file on the drive. In sequential read/write operations, the drive can read or write data from a series of blocks, allowing for faster performance...

SQL and how it relates to Disk Reads and Writes


Structured Query Language (SQL) is widely used for managing data in relational database management systems (RDBMS). In the context of SQL, the optimization of database queries forms the foundation for efficient data retrieval, providing quicker response times and increasing overall database performance. One key strategy to optimize SQL queries involves the concept of "Sargable" queries.

"Sargable" is derived from "Search ARGument ABLE," signifying that a query can successfully utilize indexes for efficient execution. It is a term that was initially introduced in a 1979 research paper titled "Access Path Selection in a Relational Database Management System" authored by P. Griffiths Selinger et al. (DBA.StackExchange.com). The fundamental idea behind sargable queries is geared towards forming SQL statements that allow the query execution engine to make the best use of indexes whenever available.

## Sargable vs. Non-Sargable Queries
A central feature differentiating sargable from non-sargable queries is the manner in which operations are executed on indexed columns. Non-sargable queries include function calls or operations that use an indexed field in the WHERE clause, a feature that hinders the usage of indexes.

|Non-sargable Query|Sargable Query|
|---|---|
|`SELECT * FROM myTable WHERE SUBSTRING(myColumn, 1, 3) = 'ABC';`|`SELECT * FROM myTable WHERE myColumn LIKE 'ABC%';`|
|`SELECT * FROM myTable WHERE DATEDIFF(day, myDate, GETDATE()) = 7;`|`SELECT * FROM myTable WHERE myDate >= DATEADD(day, -7, GETDATE()) AND myDate < GETDATE();`|
|`SELECT * FROM myTable WHERE ISNULL(myColumn, 'N/A') = 'N/A';`|`SELECT * FROM myTable WHERE myColumn IS NULL;`|

### Wildcards
Attempting to use wildcards at the beginning of a string in a LIKE clause also tends to create non-sargable conditions. For instance, the condition `WHERE name LIKE '%prefix%'` usually results in a table or index scan, which is significantly slower than an index seek.

Let's consider a simple example. Suppose we have a table called "Customers" with columns "CustomerID", "Name", and "Address". We want to find all customers who live in a particular city. A non-sargable query would be:

``` sql
SELECT * FROM Customers WHERE Address LIKE '%New York%';
```

This query is not sargable because the LIKE operator does not allow the query engine to use an index on the Address column. However, we can modify the query to make it sargable:

```sql
SELECT * FROM Customers WHERE Address = 'New York';
```

Now, the query engine can use an index on the Address column to optimize the search process.

### Handling NULL values
Another important aspect of sargable queries is handling NULL values properly. The returned dataset will be empty if we do not handle NULL values correctly. For example, consider the following query:

```sql
SELECT * FROM Customers WHERE Name IS NOT NULL;
```

This query is not sargable because the `IS NOT NULL` predicate does not allow the query engine to use an index. To make this query sargable, we can modify it to:

```sql
SELECT * FROM Customers WHERE Name IS NOT NULL AND Address IS NOT NULL;
```

Now, the query engine can use an index on the Name and Address columns to optimize the search process.

### Calling with functions
A non-sargable query may look like `WHERE YEAR(dateColumn) = 2022`. The problem with such a query is that it requires evaluation of the function `YEAR(dateColumn)` for each row in the table, thus preventing us to use any pre-existing index on `dateColumn`, an operation leading to inefficient table scans.

Sargable queries aim at performing operations responsibly by avoiding function calls on indexed columns whenever possible. For instance, reversing the non-sargable condition from

```sql
SELECT *
FROM myTable
WHERE YEAR(dateColumn) = 2023;
```

to

```sql
SELECT *
FROM myTable
WHERE dateColumn >= '2023-01-01' AND dateColumn < '2024-01-01';
```

allows the query optimizer to use our indexes and not run the function on every row.

## Impact on Performance
The distinction between sargable and non-sargable queries lies primarily in how efficiently they facilitate usage of indexes. Sargable queries allow the database engine to perform index seeks, a process whereby only the matching data in index pages are read, hence reducing the consumption of [input/output (IO) resources and time]().

Several advantages come from employing sargable queries. The main benefit lies in leveraging indexes, thereby improving search speed. Depending on the data type and column values, diverse index types such as clustered indexes, non-clustered indexes, and columnstore indexes can be used.

## Conclusion
In modern databases, where massive data sizes are common, improving query performance can significantly impact the overall system speed and efficiency. Sargable queries provide a crucial optimization strategy that should be taken into account during SQL programming and query design.

By understanding how sargable queries can take advantage and efficiently use indexes, database administrators, and developers can significantly boost search speeds and performance. Furthermore, ways to convert non-sargable queries into sargable queries by avoiding function calls on indexed columns and leveraging efficient use of the LIKE operator and wildcards should be studied for regular practice.

## References
- https://stackoverflow.com/questions/799584/what-makes-a-sql-statement-sargable
- https://www.sqlshack.com/how-to-use-sargable-expressions-in-t-sql-queries-performance-advantages-and-examples/
- https://www.tech-recipes.com/uncategorized/sargable-queries-in-sql-server-with-examples/
- https://www.mssqltips.com/sqlservertip/6795/improve-sql-server-query-performance-searchable-arguments/
- https://dba.stackexchange.com/questions/162263/what-does-the-word-sargable-really-mean


Structured Query Language (SQL) is widely used for managing data in relational database management systems (RDBMS). In the context of SQL, the optimization of database queries forms the foundation for efficient data retrieval, providing quicker response times and increasing overall database performance. One key strategy to optimize SQL queries involves the concept of "Sargable" queries...

SQL Saragable Queries and Their Impact on Database Performance


## Before the removal
### Importance of Zookeeper
Kafka has been using [Zookeeper](https://cwiki.apache.org/confluence/display/ZOOKEEPER/ProjectDescription) for a variety of important functions. It uses Zookeeper to keep track of which brokers are part of the Kafka cluster. This is a critical task, as it enables Kafka to ensure that each broker is working properly and that the entire cluster is functioning as intended.

Additionally, Zookeeper is responsible for electing the leader of a given partition. This is a complex task that requires significant computational power, and Zookeeper is well-suited to handle it.

Another important function that Zookeeper performs for Kafka is storing configurations for topics/permissions and current offsets of each consumer group. Managing configurations is critical for ensuring that Kafka is able to function properly as a distributed system and that each consumer is receiving the correct information.

Finally, Zookeeper sends health-check notifications to Kafka for a variety of events. This includes new topics being created, brokers coming up or dying, and topics being deleted.

### Downsides to Zookeeper
Despite its many advantages, there are also some downsides to using Zookeeper. One challenge is that the Kafka Controller state or metadata often does not match the Zookeeper state. This can lead to confusion and make it difficult to determine the current state of the cluster.

Additionally, there can be performance issues when a broker joins or leaves a cluster, as Zookeeper can be overloaded by a high number of leader elections. Despite these challenges, many organizations continue to find Zookeeper to be a valuable tool in managing their Kafka clusters, and Kafka continues to rely on it for its important functions.

## Why 
The Kafka Controller **state** or **metadata** often does not match the Zookeeper state. Despite this, Kafka uses Zookeeper for several important tasks. For example, it keeps track of which brokers are a part of the Kafka cluster and is responsible for electing the leader of a given partition. Additionally, it stores configurations for topics/permissions and current offsets of each consumer group. Lastly, it sends health-check notifications to Kafka for events such as new topics being created, brokers coming up or dying, and topics being deleted.

One of the advantages of using Zookeeper is simpler deployment and configuration, as it is a *separate Java service*. However, there are also some downsides to using Zookeeper. Performance issues occur when a broker joins or leaves a cluster, as Zookeeper can be overloaded by a high number of leader elections. In addition, scaling Kafka only supports a limited number of partitions, up to **200k partitions**.

Despite the potential challenges, Kafka continues to rely on Zookeeper for its important functions, and many organizations continue to find it a valuable tool in managing their Kafka clusters.

## Architecture
![](assets/the-removal-of-apache-kafka's-dependency-on-zookeeper_kafka_architecture.webp)

### In the old architecture
The controller is a crucial component of the system that manages the state of the Kafka cluster. Upon election, the controller loads its state from a Zookeeper quorum consisting of multiple nodes. This state includes information about topics, their partitions, and the brokers that host these partitions. Once the controller has loaded its state, it begins pushing updates to other nodes in the cluster using `LeaderAndIsr` and `UpdateMetadata` messages. These updates ensure that all nodes in the cluster are aware of the latest state of the system, and that they can take appropriate actions in response to changes in this state. Overall, the controller plays a critical role in ensuring the smooth functioning and reliability of the Kafka cluster.

![](assets/the-removal-of-apache-kafka's-dependency-on-zookeeper_kafka-controller-apis.webp)

### In **KRaft Mode**
Kafka has been using Zookeeper for various critical tasks, such as keeping track of which brokers are part of the Kafka cluster, electing leaders for partitions, storing configurations for topics/permissions, and current offsets of each consumer group, and sending health-check notifications to Kafka.

Despite its advantages, using Zookeeper also poses challenges when it comes to performance and consistency in Kafka's metadata. For example, Zookeeper can be overloaded by a high number of leader elections, especially when a broker joins or leaves a cluster. Additionally, Kafka Controller state or metadata often does not match the Zookeeper state, leading to confusion and difficulty in determining the current state of the cluster.

To address these challenges, Kafka introduced KRaft mode, which replaces the Zookeeper quorum with the Controller quorum. The Controller quorum manages all metadata stored in Zookeeper, including topics, partitions, ISRs, and configs, among others. An active controller is elected using [Raft](https://raft.github.io/), a consensus algorithm that does not rely on any external system. The leader controller handles RPC calls from the brokers, while the follower controllers replicate the data written to the active controller to be ready in case the active controller fails.

With the KRaft mode, the controller quorum does not need to load state from Zookeeper when the leadership changes, which significantly improves performance. Instead, the brokers fetch updates from the active controller via the new `MetadataFetch` API, tracking the offset of the last update fetched and only requesting newer updates.

Kafka's KRaft mode offers various advantages over Zookeeper, such as simpler deployment and configuration since Zookeeper is a separate Java service. Furthermore, it addresses the performance issues associated with Zookeeper, especially when a broker joins or leaves a cluster, by reducing the number of leader elections.

In conclusion, KRaft mode is a crucial improvement to Kafka's architecture, ensuring smooth functioning and reliability of the Kafka cluster by providing a better alternative to Zookeeper for managing Kafka's metadata.

![](assets/the-removal-of-apache-kafka's-dependency-on-zookeeper_time-shutdown-operations-kafka.webp)

For simple local setup, can refer to [Apache Kafka's quickstart guide](https://kafka.apache.org/quickstart) . For more detailed configurations, you can refer to [Kafka's kraft config](https://kafka.apache.org/documentation/#kraft_config) 

## References
- [https://www.conduktor.io/kafka/zookeeper-with-kafka/](https://www.conduktor.io/kafka/zookeeper-with-kafka/ "https://www.conduktor.io/kafka/zookeeper-with-kafka/")
- [https://developer.confluent.io/learn/kraft/](https://developer.confluent.io/learn/kraft/ "https://developer.confluent.io/learn/kraft/")
- [https://raft.github.io/](https://raft.github.io/ "https://raft.github.io/")
- [https://cwiki.apache.org/confluence/display/KAFKA/KIP-500%3A+Replace+ZooKeeper+with+a+Self-Managed+Metadata+Quorum](https://cwiki.apache.org/confluence/display/KAFKA/KIP-500%3A+Replace+ZooKeeper+with+a+Self-Managed+Metadata+Quorum "https://cwiki.apache.org/confluence/display/KAFKA/KIP-500%3A+Replace+ZooKeeper+with+a+Self-Managed+Metadata+Quorum")
- [https://kafka.apache.org/documentation/#zk](https://kafka.apache.org/documentation/#zk "https://kafka.apache.org/documentation/#zk")


Kafka has been using Zookeeper for a variety of important functions. It uses Zookeeper to keep track of which brokers are part of the Kafka cluster. This is a critical task, as it enables Kafka to ensure that each broker is working properly and that the entire cluster is functioning as intended...

Playground > 01_literature > Engineering > Backend