Explore efficient techniques for handling concurrent workloads in Go, including worker pools, fan-out/fan-in patterns, pipelines, and semaphores. This guide provides practical examples, code snippets, and insights into implementing these patterns, discussing their benefits and potential drawbacks. Learn how to improve performance, scalability, and resource management in Go applications while avoiding common pitfalls like deadlocks and excessive resource consumption.

Approaches To Manage Concurrent Workloads Like Worker Pools And Pipelines


## 1. What is Finite Automata? (A Simple Explanation)

Finite Automata (FA), also known as Finite State Machines, are abstract computational models used to recognize patterns or process sequences of symbols. They consist of:

- A finite set of states
- A set of input symbols (alphabet)
- Transitions between states based on input symbols
- An initial state
- A set of final (accepting) states

FAs can be deterministic (DFA) or nondeterministic (NFA). In a DFA, each state has exactly one transition for each input symbol, while in an NFA, a state can have multiple transitions for the same input symbol or even transitions without consuming any input (epsilon transitions).

FAs are widely used in text processing, pattern matching, and lexical analysis in compilers. They efficiently recognize regular languages, making them suitable for tasks like validating email addresses or searching for specific patterns in text.

## 2. What is Computing the Union of Two Finite Automata?

Computing the union of two finite automata means creating a new automaton that accepts all strings accepted by either of the original automata. In other words, if we have two FAs, A and B, their union FA will accept a string if it's accepted by either A or B (or both).

Theoretically, this is done by:

- Creating a new start state
- Adding epsilon transitions from this new start state to the start states of both input automata
- Combining all states and transitions from both automata
- Making all final states from both automata final in the new automaton

This process essentially creates an NFA that represents the union of the languages recognized by the two input automata.

## 3. What is Different Between the Academic Approach and Practical Approach?

The academic approach to computing the union of finite automata, as described above, is straightforward but can be inefficient in practice. The main differences are:

**Academic Approach**:

- Creates a new start state and adds epsilon transitions
- Combines all states from both automata
- Results in an NFA that may need further processing (e.g., determinization) for efficient matching

**Practical Approach**:

- Avoids creating unnecessary states
- Focuses on reachable states only
- Aims to create a more efficient structure for actual pattern matching
- May produce a hybrid automaton that's neither fully deterministic nor fully nondeterministic
- Optimizes for memory usage and matching speed in real-world scenarios

The practical approach, as implemented in Quamina, tries to balance theoretical correctness with performance considerations for large-scale pattern matching.

## 4. How Do We Solve This Using Golang Code?

The Quamina library implements the union of finite automata using a practical approach in the `mergeFAStates` function. Here's a detailed explanation of the implementation:

```go
func mergeFAStates(state1, state2 *faState, keyMemo map[faStepKey]*faState, printer printer) *faState {
    // Memoization to avoid redundant computations
    mKey := faStepKey{state1, state2}
    combined, ok := keyMemo[mKey]
    if ok {
        return combined
    }

    // Combine field transitions
    fieldTransitions := append(state1.fieldTransitions, state2.fieldTransitions...)
    combined = &faState{table: newSmallTable(), fieldTransitions: fieldTransitions}

    // Memoize the result
    keyMemo[mKey] = combined

    // Unpack the transition tables
    u1 := unpackTable(state1.table)
    u2 := unpackTable(state2.table)
    var uComb unpackedTable

    // Merge transitions
    for i, next1 := range u1 {
        next2 := u2[i]
        switch {
        case next1 == next2:
            uComb[i] = next1
        case next2 == nil:
            uComb[i] = next1
        case next1 == nil:
            uComb[i] = next2
        case i > 0 && next1 == u1[i-1] && next2 == u2[i-1]:
            uComb[i] = uComb[i-1]
        default:
            var comboNext []*faState
            for _, nextStep1 := range next1.states {
                for _, nextStep2 := range next2.states {
                    comboNext = append(comboNext, mergeFAStates(nextStep1, nextStep2, keyMemo, printer))
                }
            }
            uComb[i] = &faNext{states: comboNext}
        }
    }

    // Pack the combined table
    combined.table.pack(&uComb)

    // Combine epsilon transitions
    combined.table.epsilon = append(state1.table.epsilon, state2.table.epsilon...)

    return combined
}
```

## Key Aspects of This Implementation:

1. **Memoization**: The function uses a `keyMemo` map to store already computed merged states, avoiding redundant computations and potential infinite recursion.
2. **Combining Field Transitions**: The field transitions from both input states are combined immediately.
3. **Table Unpacking**: The transition tables of both input states are unpacked into a more manageable format for merging.
4. **Merging Transitions**: The function iterates through all possible transitions (0-255 for byte values) and merges them according to several rules:
   - If both transitions are the same, use that transition.
   - If one transition is `nil`, use the non-`nil` transition.
   - If the transition is the same as the previous byte value, reuse the previous merged result.
   - For different non-`nil` transitions, recursively merge the next states.
5. **Packing the Combined Table**: After merging, the combined table is packed back into the efficient `smallTable` format.
6. **Combining Epsilon Transitions**: Epsilon transitions from both input states are combined.

## Advantages:

- Avoids creating unnecessary states by only merging reachable states.
- Handles both deterministic and nondeterministic aspects of the input automata.
- The memoization technique prevents redundant computations, improving efficiency for large automata.
- The use of the `smallTable` structure keeps the memory footprint low.
- By merging transitions byte-by-byte, it maintains compatibility with UTF-8 encoded input.

## Trade-offs:

- The resulting automaton may not be minimal or fully deterministic.
- The recursive nature of the function could lead to stack overflow for extremely large or complex automata.
- Performance depends on the structure of the input automata and can vary significantly based on their complexity.

In practice, this implementation provides a good balance between theoretical correctness and practical efficiency for the pattern matching tasks Quamina is designed to handle. It allows for the combination of multiple patterns into a single automaton structure that can be efficiently traversed during the matching process.


An in-depth guide on implementing the union of finite automata in Golang, focusing on practical efficiency and performance considerations.

Efficient Union of Finite Automata in Golang: A Practical Approach


## An introduction to Vim and how to use Vim with Golang

- Vim is a powerful text-editor, usually used server environments with no graphical interfaces. In this post, I will layer a basic foundation of Vim and show you how to use Vim as an IDE for Go.

## What is Vim

Vim is a highly configurable text editor built to enable efficient text editing, the next version of Vi (Vim = Vi Improved), written by Bram Moolenaar, first released in 1991.

- Vim is often called a "programmer's editor,"
- Vim can be used as IDE for its plugins.
- Vim can support for many platforms.

## Install and Config

- Vim can be downloaded at [vim homepage](https://www.vim.org/download.php) or set up by [brew](https://brew.sh/)

`brew install vim`

- Since some plugins can be in needed of lua, you should set up with this command line`brew install vim --with-lua`
- After finished setting, create a.vimrc file. This file is where you store every config, plugins or bundle, as well as other Vim related function.
- Next, set up vim plugins manager. Currently, I’m using [Vundle](https://github.com/VundleVim/Vundle.vim), you can use [panthogen](https://github.com/tpope/vim-pathogen) if you like. Vundle will help in config those plugins in .vimrc, install or update and other tasks. In order to set up, clone vundle in the folder ~/.vim/bundle/ (folder .vim will appear when you finish installed vim)`git clone https://github.com/VundleVim/Vundle.vim.git ~/.vim/bundle/Vundle.vim`

Copy some configs and basic plugins into .vimrc (please note that the “ in vim is comment)

```javascript
set nocompatible              " be iMproved, required
filetype off                  " required

" set the runtime path to include Vundle and initialize
set rtp+=~/.vim/bundle/Vundle.vim
call vundle#begin()
" alternatively, pass a path where Vundle should install plugins
"call vundle#begin('~/some/path/here')

" let Vundle manage Vundle, required
Plugin 'VundleVim/Vundle.vim'

" The following are examples of different formats supported.
" Keep Plugin commands between vundle#begin/end.
" plugin on GitHub repo
Plugin 'tpope/vim-fugitive'
" plugin from http://vim-scripts.org/vim/scripts.html
Plugin 'L9'
" Git plugin not hosted on GitHub
Plugin 'git://git.wincent.com/command-t.git'
" git repos on your local machine (i.e. when working on your own plugin)
Plugin 'file:///home/gmarik/path/to/plugin'
" The sparkup vim script is in a subdirectory of this repo called vim.
" Pass the path to set the runtimepath properly.
Plugin 'rstacruz/sparkup', {'rtp': 'vim/'}
" Avoid a name conflict with L9
Plugin 'user/L9', {'name': 'newL9'}

" All of your Plugins must be added before the following line
call vundle#end()            " required
filetype plugin indent on    " required
" To ignore plugin indent changes, instead use:
"filetype plugin on
"
" Brief help
" :PluginList       - lists configured plugins
" :PluginInstall    - installs plugins; append `!` to update or just :PluginUpdate
" :PluginSearch foo - searches for foo; append `!` to refresh local cache
" :PluginClean      - confirms removal of unused plugins; append `!` to auto-approve removal
"
" see :h vundle for more details or wiki for FAQ
" Put your non-Plugin stuff after this line
```

Next, to install plugins, open vim and run this command line below (by pressing Esc)

`:PluginInstall`

To up date plugins, use `:PluginUpdate`

That’s enough for you to pick Vim up from the ground.

## Basic Usage Instruction

Vim is a text-editor program first built on UNIX, in order to work in non-user interface environment. At this time, the keyboard hasn’t been upgraded completely, which is why the action on Vim is still different from today.

![](assets/connecting-vim-with-golang_5ca1828b88d41d78ead1d47ac32aeabd_md5.webp)

You can practice using these keys by [vim game](https://vim-adventures.com/)

To open a new file with vim, use the command line: `vim file_name`

There are 3 modes in vim:

- **Normal mode** - appear when you press Ecs, usually has “.” in front of it. Vim will get this as what you type is the command for vim. For example: `:w` is “save document”, `:q` is “exit document”. Vim has its own language, which is called Vim script. Vim script can be executed at normal mode, for example, to print out Hello world, we use

`:echo "hello world"`

- **Insert mode** -appear when you press one of these keys below

a: Insert vào phía sau con trỏ hiện tại A: Insert vào cuối hàng i: Insert vào phía trước con trỏ hiện tại I: Insert vào đầu hàng o: Insert và mở một hàng trống phía dưới O: Insert và mở một hàng trống phía trên

- **Visual mode** – appear when you press v, usually used in selecting a big block text, can be used in copy paste or comment,..

You can read [vim command](http://bullium.com/support/vim.html) to know more about the commands in vim

## Using Vim as Go IDE

In order to use Vim as an IDE for Go, we need to set up some plugins for Go.

The most important thing is [vim go](https://github.com/fatih/vim-go/). Vim Go contains a set of libraries that support Go for Vim, for example `godef`, `gofmt`, `go test`.

To install vim go, copy plugin to file .vimrc

```javascript
Plugin 'fatih/vim-go'
```

Then choosing normal mode by pressing `esc` and type this command

```javascript
:PluginInstall
```

Next, continue typing command to install Go binaries

```javascript
:GoInstallBinaries
```

You can basically code Go with only Vim go, but to make things easier, and also enhance its efficiency, you can research and add up some plugins as below

```javascript
ack.vim
bclose.vim
bufexplorer
nerdtree
nerdcommenter
csapprox
vim - fugitive
gitv
vim - gitgutter
syntastic
neocomplete.vim
neosnippet.vim
auto - pairs
```

In these:

- nerdtree: Help you organize the folders in vim in folder structure, which makes it easier to open file and folder. After finished setting up:

![](assets/connecting-vim-with-golang_ac42721e6ddb17d3204d4596f55f96ce_md5.webp)

Thanks to plugin of git, you can know which file is under editing, which one has just been added up, and which one has not been committed yet.

- neocomplete: Support auto complete vim

![](assets/connecting-vim-with-golang_db43f35dc945a8dde4b51e4b97221c86_md5.webp)

- autopair: Plugin that helps you to quick type using “ ( ‘ [ < by adding “” () “ [] <> as a pair

## Mapping Structure in Vim

Vim has a definition called mapping, which allows its user to mode their keystrokes to support their personal purposes.

```javascript
Khi tôi nhấn key này, tôi muốn bạn làm hành động này thay vì những gì bạn hay thường làm
```

Example: Type some sentences by vim, then run this command

```javascript
:map - x
```

Put your mouse on the text paragraph and press `-`. Vim will instantly delete the characters right under the mouse pointer, as if you’ve just pressed `x`.

At this time, your mapping only works on the text file that you’re editing. In order to mapping your action on all other files, put your mapping on file .vimrc Here are some of the common mappings:

```javascript
nmap    : used in normal mode
imap    : used in insert mode
vmap    : used in visual mode
noremap : can't be overridden by other mappings
```

For example, when coding in Go, we usually use godef to find the definition of that function. In vim-go, we log into normal mode and type `GoDef` But we can mapping like this instead:

```javascript
nnoremap <silent> df :GoDef<cr>
```

That means when we’re at normal mode, the keystrokes df will execute the command: `GoDef`. This Map can’t be overridden by any other normal map.Therefore, if you’re looking for a definition of a function, use the keystroke `df`. This will reduce the time of typing process.

Another example for mapping. I usually use [print](https://github.com/k0kubun/pp) library in print out a better result in terminal. The action of login to debug in Go happens regularly, so I set up a map like this:

```javascript
inoremap <silent> pp pp.Println("")
```

That means when I type in insert mode `pp`, it will automatically switch into `pp.Println("")`, which enables me to code faster.

You can read more at [mapping in vim](http://vimdoc.sourceforge.net/htmldoc/map.html).

## Conclusion

Everything is ready! Your only job is to:

- Experienced in using hjkl in vim, as well as the basic commands in vim.
- Search for more plugins to support for vim. This is a page I find very interesting and useful for [plugins in vim](http://vimawesome.com/).
- The key strong point in vim is customization, you should try to figure out how to create new mappings or function that helps fasten your coding process.

These aspects is what I have learnt about Vim and how to code Go using Vim. There are a lot of things to discover in the world of Vim and I would love to hear all of your comments or sharing.


A comprehensive guide on using Vim as an IDE for Go development. Learn about Vim basics, installation, configuration, essential plugins like vim-go, and customization techniques to enhance your Go coding workflow in Vim.

Connecting Vim With Golang


The extension interface pattern is when an interface embeds another one. The extension pattern helps to add new features to an existing object without changing its original code. 

- Extending third-party packages: When you are working with a third-party package, and you want to add new methods or modify the behavior of existing types without forking or modifying the original package.
- Adding functionality to interfaces: When a package provides a minimal interface and you want to add additional behaviors on top of that without changing the underlying implementation.
- Testing: You can use the extension interface pattern to mock or adapt behaviors of a type for testing purposes, adding features like logging, metrics, or other cross-cutting concerns.

Whether you are working with the standard library (`io`, `http`, `sql`), third-party packages, or your own codebase, this pattern provides a way to add functionality in a flexible, non-intrusive manner.

### 1. **Extending `io.Reader` and `io.Writer`**

The `io.Reader` and `io.Writer` interfaces are simple but versatile interfaces that are widely used in Go. You can extend them to add features like compression, encryption, logging, or even buffering.

**Example: adding logging to an `io.Writer`**

Let’s say you want to add logging functionality to an `io.Writer`. You can use the extension interface pattern to wrap an existing `io.Writer` and log any data written to it.

```go
type LoggingWriter struct {
    io.Writer  // Embed the original io.Writer
}

func (lw LoggingWriter) Write(p []byte) (n int, err error) {
    fmt.Printf("Writing %d bytes: %s\n", len(p), string(p))  // Log the write
    return lw.Writer.Write(p)  // Call the original Write method
}
```

Usage:
```go
func main() {
    var writer io.Writer = LoggingWriter{Writer: os.Stdout}
    
    writer.Write([]byte("Hello, World!"))  
    // Output:
    // Writing 13 bytes: Hello, World!
    // Hello, World!
}
```

This allows you to add logging to any writer without modifying the original `io.Writer` type.

### 2. **Extending HTTP middleware in `http.Handler`**

In web development with Go, the `http.Handler` interface is central to building web servers. It’s common to use the extension interface pattern to create middleware that extends the behavior of `http.Handler`.

**Example: Adding a request logger middleware**

You can create middleware that wraps an `http.Handler` to log HTTP requests.

```go
type LoggingMiddleware struct {
    handler http.Handler
}

func (lm LoggingMiddleware) ServeHTTP(w http.ResponseWriter, r *http.Request) {
    fmt.Printf("Received request: %s %s\n", r.Method, r.URL.Path)  // Log request
    lm.handler.ServeHTTP(w, r)  // Call the original handler
}
```

Usage:

```go
func main() {
    originalHandler := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        w.Write([]byte("Hello, World!"))
    })

    loggingHandler := LoggingMiddleware{handler: originalHandler}
    http.ListenAndServe(":8080", loggingHandler)
}
```

This example extends `http.Handler` to log incoming requests, wrapping the original handler without modifying it.

### 3. **Extending `sql.DB` for database connections**

You can extend the `sql.DB` type from Go’s `database/sql` package to add functionalities like logging, connection retries, or metrics tracking.

**Example: Adding query logging to `sql.DB`**

```go
type LoggingDB struct {
    *sql.DB  // Embed the original sql.DB
}

func (ldb LoggingDB) Query(query string, args ...interface{}) (*sql.Rows, error) {
    fmt.Printf("Executing query: %s\n", query)  // Log the query
    return ldb.DB.Query(query, args...)
}
```

Usage:

```go
func main() {
    db, _ := sql.Open("mysql", "user:password@tcp(127.0.0.1:3306)/dbname")
    
    loggingDB := LoggingDB{DB: db}
    loggingDB.Query("SELECT * FROM users")
}
```

This extension allows you to log SQL queries without altering the behavior of `sql.DB`.

### 4. **Adding caching to HTTP clients**

Go’s `http.Client` is a widely used type for making HTTP requests. You can extend `http.Client` to add caching, retries, or additional logging.

**Example: adding caching to an `http.Client`**

You can wrap an `http.Client` to cache responses based on URLs.

```go
type CachingClient struct {
    client   *http.Client
    cache    map[string]*http.Response
}

func (cc *CachingClient) Do(req *http.Request) (*http.Response, error) {
    if cachedResp, ok := cc.cache[req.URL.String()]; ok {
        fmt.Println("Returning cached response")
        return cachedResp, nil
    }
    
    resp, err := cc.client.Do(req)
    if err == nil {
        cc.cache[req.URL.String()] = resp
    }
    return resp, err
}
```

Usage:

```go
func main() {
    httpClient := &http.Client{}
    cachingClient := &CachingClient{
        client: httpClient,
        cache:  make(map[string]*http.Response),
    }

    req, _ := http.NewRequest("GET", "http://example.com", nil)
    cachingClient.Do(req)  // Fetches from the internet and caches the result
    cachingClient.Do(req)  // Uses the cached result
}
```

This allows you to extend the functionality of the `http.Client` without altering the original type, adding caching behavior.

### 5. **Adding context or timeouts to `http.Request`**

Go's `http.Request` does not have built-in timeout functionality, but you can extend the `http.Request` type to add it.

**Example: timeout extension for `http.Request`**

```go
type TimeoutRequest struct {
    req *http.Request
    timeout time.Duration
}

func (tr *TimeoutRequest) Do(client *http.Client) (*http.Response, error) {
    ctx, cancel := context.WithTimeout(tr.req.Context(), tr.timeout)
    defer cancel()

    reqWithTimeout := tr.req.WithContext(ctx)
    return client.Do(reqWithTimeout)
}
```

Usage:

```go
func main() {
    req, _ := http.NewRequest("GET", "http://example.com", nil)
    timeoutReq := &TimeoutRequest{
        req:     req,
        timeout: 2 * time.Second,
    }

    client := &http.Client{}
    timeoutReq.Do(client)  // The request will timeout after 2 seconds
}
```

This allows you to extend `http.Request` with timeout functionality without modifying the original type.

### 6. **Decorators for `fmt.Stringer`**

Go’s `fmt.Stringer` is a simple but powerful interface used for customizing string representations of types. You can use the extension interface pattern to add additional behaviors when printing.

**Example: Add a prefix to `fmt.Stringer`**

You can wrap a `fmt.Stringer` type to add a prefix to its string representation.

```go
type PrefixedStringer struct {
    prefix string
    fmt.Stringer
}

func (ps PrefixedStringer) String() string {
    return ps.prefix + ps.Stringer.String()
}
```

Usage:

```go
type User struct {
    Name string
}

func (u User) String() string {
    return u.Name
}

func main() {
    user := User{Name: "John"}
    prefixedUser := PrefixedStringer{prefix: "User: ", Stringer: user}

    fmt.Println(prefixedUser.String())  // Output: User: John
}
```
This wraps the original `fmt.Stringer` and adds a prefix to the output.

----

That's good as far as it goes, but I think the key part of extension interfaces is why they're useful and how they're used -- they can add optional functionality to an API which (according to the statically-checked type signature) only takes the "base" interface.

For example, [io.WriteString](https://golang.org/pkg/io/#WriteString) is one of the simplest examples of an extension interface: it takes a plain io.Writer, but if that writer has been extended to support the `io.StringWriter` interface (i.e., has the `WriteString` method), it will use that for efficiency, otherwise fall back to the regular Write method which all io.Writer implementations have.

Sticking with your `File` / `ReadDirFile` example, the `fs.Open` method returns a plain File, but if the "file" is actually a directory, you can convert it to a `ReadDirFile` and use the `ReadDir` extension method. Something like:

```go
f, _ := fs.Open("dir_or_file") // in real life, handle errors
st, _ := f.Stat()
if st.IsDir() {
    // f is a directory
    d := f.(ReadDirFile)
    d.ReadDir(10)
} else {
    // f is a normal file
}
```

As an alternative to calling st.IsDir() (and what would probably be more typical for extension interfaces), you could just check whether the file implements the interface directly, like so:

```go
f, _ := fs.Open("dir_or_file")
if d, ok := f.(ReadDirFile); ok {
    // f is a directory
    d.ReadDir(10)
} else {
    // f is a normal file
}
```

[Source](https://www.reddit.com/r/golang/comments/i6yehu/what_is_the_extension_interface_pattern_in_go/)

Note about using Go extension interface pattern

Go extension interface pattern


## Golang concurrency

When we talk about Golang, its most basic and popular characteristic is concurrency-support. Unlike other languages what is quite complex to build an concurrency system, Go concurrency primitives via Goroutines and channels make concurrent programming easy. So i am going to talk about Goroutines and similar things in other programming languages.

Firstly, we need to know Go is a concurrent language and not a parallel one. So what is the difference from concurrency and parallelism?

### Concurrency vs Parallelism

Concurrency means “out of order” execution. Another concept talk about concurrency as the capability to deal with lots of things at once. It's best explained with an example in real life: During playing Dota 2, let’s say, my mom asked me to buy something. Now i stop playing, go to the grocery and then starts playing again with rebuke of my team. This is a basic example of concurrency. In computing, concurrency is represented by the state when two or more tasks can start, run, and complete in overlapping time periods. It doesn't necessarily mean they'll ever both be running at the same time. Multitasking on a single-core machine is an example.

Parallelism is doing lots of things at the same time like other running tasks on a multicore processor. Let’s come back with the example above. Instead of taking order my mother, I play the game and listen to music at once freely. In this case, playing game and listening to music is "lots of things".

![](assets/go-concurrency_9cb01cacb1edc3373acd7570cc280cd7_md5.webp)

After understanding the way to compare concurrency with parallelism, we can research about Goroutines/channel and other well.

### Goroutines

Goroutines can be understood like a light weight thread. Instead of OS, Goroutines exists only in the virtual space of Go runtime. A Goroutines is started up with initial 2KB of stack size (This is the location where store answer of the question "what is this Go routine's current state?". It is contains local variables as well as pointers to heap allocated variables). Because Go’s stacks are dynamically sized, growing and shrinking with the amount of data stored. So we need only 4KB memory for Goroutines.

![](assets/go-concurrency_d82335b0ffffc63ae92aa7339a5867e8_md5.webp)

Goroutines are cheaper than others thing like it (thread) in other programming language because of ability growing and shrinking depend on programmer's desire. e.g a Java thread initially have 4MB while go-routine is somewhere around 2KB.

While some languages take an "concurrency at OS level" approach. Go implemented its own scheduler in order to keep concurrency concept at language level.

### Channel

Channel is a communication tool for Goroutines. Channel can be understand like pipes. So similar to water flows from one end to another in pipes, a Goroutines write data in one end and we can get it in another by using Channel.

Each channel has a type associated. This type decide data type that channel is allowed to transport and no other types can access into it.

When a data is sent to a channel, the control is blocked in the send statement until some other Go-routine reads from that channel. Similarly when data is read from a channel, the read is blocked until some Go-routine writes data to that channel. It also mean these are **unbuffered**.

In other words, by default Go channel are **unbuffered**.

So how to break this limit?

Like the way God made Eve from a part of Adam, Go provide another channel type that can buffered.

For buffered channel, providing the buffer length as the second argument to make to initialize a buffered channel:

```plain_text
 messages := make(chan string, capacity)
```

Capacity should be greater than 0 for a channel to have a buffer and we can send data to our channel until this is full, similar receives will not block when we read data until empty.

![](assets/go-concurrency_eb379f648c223af56819605adfeb118d_md5.webp)

## Under the hood

### Channel implementation

When we implement a Go channel, a struct will be created, then it looks like following.

To have a comprehensive view, we have a few descriptions about the fields encountered in the channel structure.

![](assets/go-concurrency_23c2cb6d27ce23dcdc500decd9a59398_md5.webp)

- **buf** is the location where our data is actually stored. It is a circular queue.
- **dataqsize** is size of the circular queue. It represent capability of channel. When we declare make(chan int, N), **dataqsize** is N. It also let us know how many Goroutines can write data into this channel until blocked

![](assets/go-concurrency_966768c61d35099632b9e31f581807c6_md5.webp)

- **qcount** represents the number of slots in the buf currently filled up.
- **elemsize** Is the size of a channel corresponding to a single element.
- **elemtype** used when messages are copied over from one Go-routine to the other. It has a bunch of fields which provide type and size information for the type of values the channel can hold.
- **closed** Indicates whether the current channel is in the closed state. After a channel is created, this field is set to 0, that is, the channel is open; by calling close to set it to 1, the channel is closed.

![](assets/go-concurrency_80543471c1dd9c44e63c7fa281d4b6c0_md5.webp)

- **sendx** and **recvx** indicates the current index of buffer — backing array from where it can send data and receive data.
- **recvq** and **sendq** waiting queues, which are used to store the blocked Goroutines while trying to read data on the channel or while trying to send data from the channel.
- **lock** protects all fields in **hchan**, as well as fields in sudogs blocked on this channel.
- **sudog** struct is described in Golang source as a Go Routine in a wait list, such as for sending/receiving on a channel.
-

We mentioned about sudog struct above. What is sudog? See the following image:

![](assets/go-concurrency_2ec27e55f9c531e5aab4f885f5915f30_md5.webp)

We have a full description for sudog struct in source. If something confuse, ignore it by looking at the original concept. Then we have a good description about sudog: sudog represents a Goroutine. recvq and sendq hold the information on the Goroutines which are currently blocking on the channel while receiving and sending respectively. They are pointers to sudog.

An example to know what happen when we declare Goroutines and channel.

![](assets/go-concurrency_026784030ef13c5d006328ea47cfc657_md5.webp)

What will be the structure of the channel before line **No 22?**

![](assets/go-concurrency_28bee5f5381e186532a5ceda7746c017_md5.webp)

In Our Example Code before line 22 there are two Goroutines (goroutineA and goroutineB ) trying to read data from the channel ch.

Since before line 22 on a channel, there is no data we have put on the channel so both the Goroutines blocked for receive operation have been wrapped inside the **sudog** struct and is present on the recvq of the channel. **recvq** and **sendq** are basically linked list, which looks basically as below

![](assets/go-concurrency_2da8e4e08c9af6c09a70149ce0bee0c1_md5.webp)

### Go scheduler

To understand Go scheduler quickly, we will review a few concepts in OS and OS scheduler that support a program is runnable. If you knew about OS thread and OS scheduler, you should skip following section and jump to next part.

Computer Program is just a series of machine instructions that need to be executed one after other sequentially. So thread was born in other to account and sequentially execute the set of instructions it was assigned.

When we run a program, a process is initiated with a thread. A thread have the ability to create more threads. A thread can run parallel or concurrently and keep their own state is safe, local and independent execution of their instructions.

The OS scheduler is an operating system module that is responsible for making sure cores are not idle if there are threads that can be executing. It must also create the illusion that all the threads that can execute are executing at the same time.

If you are running on Linux, Mac or Windows, you are running on an OS that has a preemptive scheduler. This means a few important things. First, it means the scheduler is unpredictable when it comes to what Threads will be chosen to run at any given time. Thread priorities together with events, (like receiving data on the network) make it impossible to determine what the scheduler will choose to do and when.

Existing a piece called Program Counter in a Thread that allows the thread keep track of the next instruction to execute.

![](assets/go-concurrency_bf66c5b925d9c162acc9d0fb055c353e_md5.webp)

There are 3 states in thread

- **Waiting**: This means the Thread is stopped and waiting for something in order to continue. “Something” is waiting for the hardware (disk, network), system calls, synchronization calls (atomic, mutexes).
- **Runnable**: This means the Thread wants time on a core so it can execute its assigned machine instructions.
- **Executing**: This means the Thread has been placed on a core and is executing its machine instructions.

And 2 types of works

- **CPU-Bound**: This is work that never creates a situation where the Thread may be placed in Waiting states.
- **IO-Bound**: This is work that causes Threads to enter into Waiting states. This is work that consists in requesting access to a resource over the network or making system calls into the operating system. **Context switches** is the duration that we swap a thread on and of a core. It take between ~1000 and ~1500 nanoseconds to do its work while the hardware should be able to reasonably execute averagely 12 instructions per nanosecond per core. So a context switch can cost we ~12k to ~18k instructions of latency. In essence, a context switch make lower performance by losing ability to execute a lot of machine instructions. It look like suitable with IO-Bound work but not with CPU-Bound work that requires work constantly.

After surfing over above all things, we have ability to understand following part quickly. It is the section that describe about GO scheduler and how it make Goroutines faster than threads.

### Getting started

When our program start up, it’s given a Logical Processor (P) for every virtual core identified on the host machine. So how many virtual core exist in your computer? Suppose that we have a Macbook with following system report:

![](assets/go-concurrency_8c56d98dd1917a5c479f487977ccfb9f_md5.webp)

We have 1 processor with total 4 cores. As you see the report don’t let us know directly how many virtual core here. But the Intel Core i7 processor has Hyper-Threading, which means there are 2 hardware threads per physical core. This will report to the Go program that 8 virtual cores are available for executing OS Threads in parallel. In another word, a hardware thread serve for a virtual cores.

Every P is assigned an OS Thread (M) that stands for machine. This means when I run a Go program on my machine, I have 8 available threads to execute my work, each individually attached to a P.

Every Go program is also given an initial Go-routine (G). Just as OS Threads are context-switched on and off a core, Goroutines are context-switched on and off an M.

Each P is given a Local Run Queue LRQ that manages the Goroutines assigned to be executed within the context of a P. These Goroutines take turns being context-switched on and off the M assigned to that P. The Global Run Queue(GRQ) is for Goroutines that have not been assigned to a P yet. There is a process to move Goroutines from the GRQ to a LRQ that we will discuss later.

Following figure provides an image of all these components together

![](assets/go-concurrency_ad2ef462e022466f3eab445763ef2bda_md5.webp)

The Go scheduler is part of the Go runtime, which is built into our application. This means the Go scheduler runs in user space. The current implementation of the Go schedule is a cooperating scheduler that needs well-defined user space events.

We can’t predict what the Go scheduler is going to do. This is because decision making for this cooperating scheduler doesn’t rest in the hands of developers, but in the Go runtime. It’s important to think of the Go scheduler as a preemptive scheduler and since the scheduler is non-deterministic, this is not much of a stretch.

There are four classes of events that occur in your Go programs that allow the scheduler to make scheduling decisions. This doesn’t mean it will always happen on one of these events. It means the scheduler gets the opportunity.

- The use of the keyword go
- Garbage collection
- System calls


A comprehensive guide to concurrency in Go, covering goroutines, channels, and the Go scheduler. This article explains the differences between concurrency and parallelism, dives into the implementation details of channels, and explores the inner workings of the Go scheduler. It provides insights into how Go's concurrency model compares to other languages and offers a deep understanding of Go's efficient approach to managing concurrent tasks.

Go Concurrency


## Go As Enterprise Standard Language

### [What is An Enterprise standard language (From Java story)?](why-enterprise-chose-java.md)

### [Why Go?](why-go.md)

### [How to use Go in enterprise?](how-to-use-go-in-enterprise.md)

### [Who is using Go in enterprise?](who-using-golang-in-enterprise.md)

### [When to use Go in enterprise?](when-to-use-golang-in-enterprise.md)

### [Aggregate use cases of successful enterprises using Go]()


There is widespread adoption to take Go as the primary programming language for enterprise-level software development, emphasizing its simplicity, efficiency, and robust standard library. This initiative aims to streamline development processes, improve code maintainability, and leverage Go's strong support for concurrent programming in large-scale enterprise applications.

Go as an Enterprise Standard Language


## How to use Go in enterprise?

- [uber-go/guide]()

- [Golang for enterprise application (Reddit)](https://www.reddit.com/r/golang/comments/h8kfp0/golang_for_enterprise_application/)

- [Golang Benefits and Best Practices for Highly Scalable App Development (Blog post)](https://mobidev.biz/blog/golang-app-development-best-practices-case-studies)

- [A Go library with PID controllers used in robotics (Reddit)](https://www.reddit.com/r/robotics/comments/m7lwmd/a_go_library_with_pid_controllers_used_in/)

- [Reclaiming CPU for free with PGO (Blog post)](https://blog.cloudflare.com/reclaiming-cpu-for-free-with-pgo/)

- [1.5 Million PDFs in 25 Minutes (Blog post)](https://zerodha.tech/blog/1-5-million-pdfs-in-25-minutes/)


This content map explores strategies and best practices for implementing Go as a primary programming language in enterprise environments. It covers Go's strengths in scalability, concurrency, and performance, as well as addressing common challenges and considerations for large-scale adoption in corporate settings.

How to use Go in the Enterprise


## When to use Go in the Enterprise?

- [Golang a programming language for modern enterprise applications (Blog post)](https://faun.pub/golang-a-programming-language-for-modern-enterprise-applications-b117f64d00f6)


This section explores the scenarios and considerations for adopting Go as a programming language in enterprise environments. It discusses the advantages of Go's simplicity, efficiency, and concurrency support, as well as its suitability for building scalable and maintainable enterprise applications.

When to use Go in the Enterprise


## Who is using Go in enterprise?
### Technology Giants

- Google

  Created Go in 2007 Uses Go for many internal tools and services Examples: Kubernetes, Docker (partially) Reference: [Go at Google: Language Design in the Service of Software Engineering](https://go.dev/talks/2012/splash.article)

- Meta (Facebook)

  Ent is a database schema as code tool written in Go Reference: [Eli5: Ent schema as code](https://developers.facebook.com/blog/post/2021/04/26/eli5-ent-schema-as-code-go/)

### Cloud Providers

- Amazon Web Services (AWS)

  Uses Go for several services Example: AWS Lambda Reference: [AWS Lambda Supports Go](https://aws.amazon.com/blogs/compute/announcing-go-support-for-aws-lambda/)

- DigitalOcean

  Command line tool for DigitalOcean API Reference: [doctl](https://github.com/digitalocean/doctl)

### Financial Services

- American Express

  Adopted Go for some of its backend services Reference: [How American Express is using Go](https://go.dev/solutions/americanexpress)

- Zerodha

  Uses Go for its trading platform Reference: [A Lesson in Niche Business DSLs at Scale](https://zerodha.tech/blog/a-lesson-in-niche-business-dsls-at-scale/)

### Media and Entertainment

- Netflix

  Rend is a memcached-compatible server and proxy written in Go Reference: [Rend: Memcached-Compatible Server and Proxy](https://github.com/Netflix/rend)

- Twitch

  Heavily relies on Go for its video streaming infrastructure Reference: [Go's March to Low-Latency, GC-Agnostic](https://blog.twitch.tv/en/2016/07/05/gos-march-to-low-latency-gc-a6fa96f06eb7/)

### Ride-sharing and Delivery

- Uber

  Uber's Go style guide Reference: [uber-go/guide]()

- Einride

  Einride uses Go for its microservices architecture Reference: [Einride's Tech Radar](https://einride.engineering/docs/tech-radar/backend#go)

### Cybersecurity

- Cloudflare

  Extensively uses Go for its edge computing platform and other services Reference: [Cloudflare's Go Projects](https://github.com/cloudflare?language=go)

### Productivity Tools

- Dropbox

  Rewrote some core components in Go for better performance Reference: [Dropbox Go's tools](https://github.com/dropbox?language=go)

---

https://go.dev/talks/2012/splash.article

https://developers.facebook.com/blog/post/2021/04/26/eli5-ent-schema-as-code-go/

https://aws.amazon.com/blogs/compute/announcing-go-support-for-aws-lambda/

https://github.com/digitalocean/doctl

https://go.dev/solutions/americanexpress

https://zerodha.tech/blog/a-lesson-in-niche-business-dsls-at-scale/

https://github.com/Netflix/rend

https://blog.twitch.tv/en/2016/07/05/gos-march-to-low-latency-gc-a6fa96f06eb7/

https://github.com/uber-go/guide/blob/master/style.md

https://einride.engineering/docs/tech-radar/backend#go

https://github.com/cloudflare?language=go

https://github.com/dropbox?language=go


This content map explores the companies and organizations that have embraced Go as their primary programming language for enterprise-level applications.

Who is using Go in enterprise?


## Why Java is chosen by enterprises

### Historical Context

Initially developed by Sun Microsystems, Java was born out of an experimental project. The language, originally named Oak. was designed for use in embedded systems. However, the name Oak was soon found to be unsuitable for trademark protection, as it had already been used by numerous other companies. In a strategic move, Sun Microsystems renamed the language to Java, inspired by the coffee that the developers drank during their coding sessions. Java was then released for free online, and quickly gained popularity among developers.

Sun Microsystems recognized early on the commercial potential of Java. By promoting Java as a versatile and industry-standard language, Sun created a significant revenue stream from licensing fees, services, and software sales. The company also leveraged its hardware business by ensuring that Java was natively supported on its Solaris operating system. This synergy between Java and Solaris meant that Sun could market its hardware as “Java-compatible,” thus driving both software and hardware sales.

Furthermore, Sun's promotion of Java as a language capable of running on any system, thanks to its "write once, run anywhere" capability, addressed a long-standing issue of software incompatibility across different platforms. This feature was especially appealing to enterprises, which often operate in diverse technological environments.

The introduction of Java EE (now Jakarta EE) in December 1999 was a pivotal moment, providing a comprehensive set of technologies and APIs designed specifically for large-scale, distributed, transactional, and mission-critical applications. This suite of tools further cemented Java's place in the enterprise software world, offering a robust framework for developing and deploying complex applications.

### Why Java is an Enterprise Programming Language

- Enterprise applications are rarely standalone; they often comprise multiple interacting systems that must communicate seamlessly. Java excels in **supporting both synchronous and asynchronous messaging**. The language natively supports communication formats such as XML, JSON, and Protocol Buffers (Protobuf), which are integral to facilitating data exchange between applications and services.
- With the growing trend towards web-based interfaces, enterprise applications need to provide **robust web interactions**. Java's ecosystem includes extensive libraries and frameworks for developing web applications and web services. Technologies like JavaServer Pages (JSP), Servlets, and frameworks such as Spring and Jakarta EE (formerly Java EE) enable developers to create scalable and secure web applications, crucial for today’s browser-based enterprise solutions.
- Data is the backbone of enterprise applications, and the ability to interact with various databases is crucial. Java **provides high-quality drivers** for both relational databases (RDBMS) and NoSQL databases.
- Security is a critical concern for enterprises, and Java’s emphasis on safety helps mitigate risks. Unlike languages like C and C++, which are prone to memory management issues such as buffer overflows and dangling pointers, Java **is designed to be memory-safe**.
- Managing user access and authentication efficiently is vital for enterprise systems. Java **integrates well with various authentication and authorization protocols**, including LDAP, Active Directory, SAML, and OAuth. These integrations facilitate centralized management of user identities and permissions, streamlining access control across the enterprise.
- Java **benefits from a large talent pool**, as it is widely taught in academic institutions and has extensive community support. This widespread adoption **makes it easier for enterprises to hire qualified developers** and **leverage the collective knowledge** of a large Java community.
- Enterprises require the flexibility to host applications on various platforms, and Java’s "write once, run anywhere" principle plays a crucial role here. Java applications **can be deployed across different operating systems and cloud environments**, such as AWS, Google Cloud Platform, and Microsoft Azure. This flexibility allows businesses to select the most cost-effective and stable hosting solutions and adapt as their needs evolve.
- Ongoing maintenance and performance monitoring are critical for enterprise applications. Java **supports a wide range of Application Performance Monitoring (APM) tools**, including ElasticAPM and DataDog, which help in tracking application health and performance. Furthermore, Java’s static typing and compilation process make it **easier to refactor code and identify issues before deployment**, enhancing the overall maintainability of applications.
- Java **boasts a rich ecosystem of libraries, frameworks, tools, and Integrated Development Environments (IDEs)**. This extensive support accelerates development processes and enhances productivity by providing developers with pre-built solutions and utilities.

### Why Java and not C/C++

Even though C/C++ is considered faster than Java in almost every benchmark, Java is still more popular because:

- Java is easier to use. This results in shorter development time.
- Java is slower than C/C++, but still fast enough for most use cases. As most applications spend time waiting for I/O and network calls to finish, the run speed of a programming language isn't a primary concern in most cases.
- Java is memory-safe, and doesn't require memory management thanks to its garbage collector.
- Java is platform-independent. Thanks to the JVM, you can run Java software on various platforms.

=> For these reasons, Java gradually becomes a popular language, and is widely used among enterprises.

#### Citations

- BSCAL. _"Sun Launches Promotion Of Java"_, Financial Times Limited 1997, Nov 07 1997. [Link](https://www.business-standard.com/article/specials/sun-launches-promotion-of-java-197110701089_1.html)
- Philip Elmer-Dewitt. _"Why Sun’s Java is hot"_, Time USA, January 22, 1996. [Link](https://time.com/archive/6728450/why-suns-java-is-hot/)
- Paul Pacheco. _"What makes a programming language an enterprise programming language"_, Quora, 2022. [Link](https://www.quora.com/What-makes-a-programming-language-an-enterprise-programming-language)
- Joey DeFrancesco. _"If C++ is faster than Java, how come Java is used in every enterprise application?"_, Quora, 2020. [Link](https://www.quora.com/If-C-is-faster-than-Java-how-come-Java-is-used-in-every-enterprise-application/answer/Joey-DeFrancesco?ch=10&oid=89079315&share=c530e7de&srid=sm2B&target_type=answer)
- Trausti Thor Johannsson. _"If C++ is faster than Java, how come Java is used in every enterprise application?"_, Quora, March 6, 2024. [Link](https://www.quora.com/If-C-is-faster-than-Java-how-come-Java-is-used-in-every-enterprise-application/answer/Trausti-Thor-Johannsson?ch=10&oid=1477743744452601&share=57fb3215&srid=sm2B&target_type=answer)


Java has been widely adopted as the primary programming language for enterprise-level software development, emphasizing its platform independence, robust ecosystem, and extensive libraries. This choice aims to enhance scalability, ensure long-term support, and leverage Java's strong object-oriented programming principles in large-scale enterprise applications.

Why Enterprise Chose Java


## Why Go

### Historical Context

The Go programming language was developed at Google by Rob Pike, Robert Griesemer, and Ken Thompson. Rob Pike’s frustration with slow C++ compilation times led him to discuss the issue with Robert Griesemer, and Ken Thompson, working nearby, joined in. Their collaboration aimed to address the inefficiencies they faced with existing languages in Google’s large-scale environment.

Google’s software infrastructure, involving millions of lines of code and extensive use of C++, Java, and Python, required a language that balanced efficient compilation, fast execution, and ease of use. The existing languages fell short, prompting the team to create Go, which began as a 20% project in September 2007. By January 2008, work on the initial compiler started, and Go was open-sourced in November 2009. It reached its first stable release in March 2012.

Go was designed to improve software development efficiency and scalability at Google. It focused on addressing issues like slow compilation and cumbersome programming practices to enhance productivity and simplify management of large codebases. The language’s design emphasizes practical software engineering solutions over theoretical language research.

### What Go Offers

Go, also known as Golang, stands out as an ideal choice for an enterprise programming language. These are a few of Go’s strong points, which makes it possible for this language to be chosen among enterprises:

- **Simple and Easy to Learn:** Go is designed to be straightforward. Its language features are minimal but effective, allowing developers to grasp and use them quickly. With just a few days of study, developers can become proficient in Go, making it easy to onboard new team members and maintain consistent coding standards across a team.
- **Fast Compilation and Execution:** One of Go's biggest advantages is its fast compilation time. Unlike languages like C++ or Java, which can be slow to compile, Go produces binaries quickly. This means you spend less time waiting for your code to build and more time actually writing and improving it. Additionally, Go binaries run as fully optimized native code, meaning they start up fast and use CPU cores efficiently.
- **Easy Maintenance and Collaboration:** Go promotes code simplicity and clarity. Its standardized formatting and idiomatic ways of doing things make code easy to read and understand. This consistency reduces the cognitive load on developers, making it easier to review, debug, and collaborate on code. Go’s toolchain also supports automated code formatting and refactoring.
- **No Runtime Dependencies:** Go applications are standalone binaries, which means they don't rely on external runtimes or libraries. This results in significantly smaller Docker images compared to languages like Java or Node.js. Deployment is also simplified because you don’t need to worry about installing additional software or managing runtime versions on your servers.
- **Explicit Error Handling:** Go takes a different approach to error handling by requiring developers to handle errors explicitly, making error management clear and predictable. This explicitness helps in identifying and addressing issues more effectively, making your applications more reliable.
- **Cross-Platform Compatibility:** Go’s runtime is lightweight and designed to be portable across various platforms, including macOS, Linux, and Windows.
- **Modern Object-Oriented Features:** Go embraces modern object-oriented principles while avoiding problematic features like inheritance. Instead, it supports composition, which provides better flexibility and scalability. This makes it easier for developers that are already familiar with OOP to switch to Go.
- **Future-Proof:** Go is designed with forward compatibility in mind. Software written in Go will continue to work with newer versions of the language, making it easier to upgrade and benefit from performance and security improvements over time.

### In which scenarios can Go do better than Java

- **When building small to medium-sized services:** Because Go has no virtual machine, huge platform libraries, or a sophisticated garbage collector, and it compiles directly to machine code and supports static linking, Go has a very fast load times and very efficient resource usage. This makes it very easy to quickly spin up a new service with good performance. But when it comes to building large and complex services, Java is a better choice as it has more features and better support, along with a more "concrete" code structure.
- **When building cloud-native, serverless applications:** Because of Go's simplicity and performance, a service written in Go takes minimal time to be spun up compared to those written in Java. This makes it ideal to use Go for building serverless applications, in which the cold start time is usually a problem worth considering.
- **When doing extensive concurrent programming:** Both Golang and Java provide support for concurrent programming. With Java, you can only manage green threads through JVM, which inevitably suffer from context switching and fixed stack size. Go, however, shines in this area with its goroutines and channels. Goroutines are essentially lightweight threads that are managed by the Go runtime, allowing developers to write highly concurrent code easily. This makes Go an excellent choice for applications that require efficient handling of thousands of simultaneous tasks.

### Conclusion

To conclude, when it comes to picking a language for application development, Go is a viable option for enterprises. However, it is not likely that Go will completely replace Java, simply because there is no strong reason and enough benefits for companies to migrate entirely from Java to Go. Java is mostly used for developing Android mobile applications, data processing and transforming, and building applications with complex requirements which are heavily reliant on Java's vast ecosystem of libraries and frameworks. Go is more suitable for building small to medium services, serverless applications, and gives a higher performance when doing concurrent programming. Both languages excel in different areas, and can be used in tandem to build high-performance and reliable applications.

### Citations

- Team Veltris. _"Golang: A Key Programming Language in Future Enterprise Application Development"_, Veltris Blog, December 9, 2019. [Link](https://www.veltris.com/blogs/digital-engineering/why-go-lang-future-enterprise-application-development/)
- Nikhilbhide. _"Golang: A Programming Language for Modern Enterprise Applications"_, Medium, May 25, 2021. [Link](https://faun.pub/golang-a-programming-language-for-modern-enterprise-applications-b117f64d00f6)
- Victor Björklund. _"Will golang replace Java?"_, JawDropping.io, March 1, 2022. [Link](https://jawdropping.io/blog/golang-replace-java/)
- Rob Pike. _"Go at Google: Language Design in the Service of Software Engineering"_, Google Inc, October 25, 2012. [Link](https://go.dev/talks/2012/splash.article)
- Technoidentity. _"Opinion: Go vs Java Microservices"_, Technoidentity. [Link](https://www.technoidentity.com/insights/opinion-go-vs-java-microservices/)
- Russell Cohen. _"Why you can have millions of Goroutines but only thousands of Java Threads"_, RCoh's Blog, April 12, 2018. [Link](https://rcoh.me/posts/why-you-can-have-a-million-go-routines-but-only-1000-java-threads/)


An exploration of the reasons why Go is gaining traction as a preferred programming language for enterprise-level software development, including its simplicity, efficiency, and robust standard library.

Why Go?


Go introduced generics with Go 1.18, which was officially released in March 2022. This update allowed Go developers to write more flexible and reusable code by supporting type parameters, enabling functions, and data structures to work with different types without sacrificing type safety.

Before this, Go was known for its simplicity and type safety but lacked the kind of flexibility that generics bring, which is a feature common in other languages like Java, C#, and C++. Generics were one of the most anticipated features in Go's development, and the 1.18 release was a significant milestone for the language.

Go achieves type safety with generics through **type parameters** and **type constraints**. Here’s how it works:

### 1. Type Parameters

When you define a function, method, or data structure with generics, you specify a type parameter in square brackets `[]`. This type parameter allows the function or type to accept different types without being tied to a specific one. However, the type still needs to conform to certain rules, which leads to the next part—**type constraints**.

Example of a generic function with type parameters:

```go
func Print[T any](items []T) {
    for _, item := range items {
        fmt.Println(item)
    }
}
```

In this case, `T` is a type parameter, and `any` is a built-in constraint that allows any type.

### 2. Type Constraints

Type constraints are used to limit what types the type parameter `T` can represent. Go enforces type safety by ensuring that the types passed to a generic function or type comply with these constraints. A type constraint can either be a specific interface or a built-in constraint like `comparable`, `any`, or custom-defined ones.

Example using a constraint:

```go
// A custom constraint interface that requires a type to implement a method
type Stringer interface {
    String() string
}

func ToString[T Stringer](val T) string {
    return val.String()
}
```

In this example, only types that implement the `Stringer` interface can be used as `T`, ensuring type safety at compile time.

### 3. Compile-Time Checking

Go’s compiler checks the types at compile time. If the provided type doesn’t satisfy the constraint, the program won’t compile, ensuring that incorrect types are not passed to a function or data structure. This is crucial for maintaining Go’s philosophy of simplicity and robustness in type safety.

### 4. Underlying Type Consistency

Go also leverages underlying types in some constraints. For example, the `comparable` constraint ensures that the type parameter can be compared using `==` or `!=`. For this to work, the compiler ensures that any type passed to a function constrained by `comparable` supports these operations, preventing runtime errors.

### 5. Explicit and Simple Type Inference

Go simplifies type safety by inferring types when possible. If the compiler can deduce the type parameter from the context, you don’t need to explicitly specify it, but the compiler still checks that the type is valid according to the constraints.

Example with inferred type:

```go
func Add[T int | float64](a, b T) T {
    return a + b
}

result := Add(3.0, 4.5) // Go infers T as float64
```

### Commutative Diagram


$$
\begin{CD}
\text{GenericCode} @>\text{Parse}>> \text{AST} @>\text{TypeCheck}>> \text{TypedAST} \\
@V\text{Instantiate}VV @V\text{Infer}VV @VV\text{Compile}V \\
\text{ConcreteCode} @>>\text{Verify}> \text{TypeSafeCode} @>>\text{Generate}> \text{ExecutableCode}
\end{CD}
$$


This diagram shows the process of how Go handles generic code:

1. Generic code is parsed into an Abstract Syntax Tree (AST).
2. The AST undergoes type checking, which involves constraint checking and type inference.
3. The resulting TypedAST is then compiled into executable code.
4. Alternatively, generic code can be instantiated with concrete types, verified for type safety, and then generated into executable code.

### Summary

In conclusion, Go achieves type safety with generics through a combination of compile-time type checking, constraint satisfaction, and type inference.

1. **Type parameters** that allow flexibility.
2. **Type constraints** to enforce rules about what types are allowed.
3. **Compile-time type checking** to prevent invalid types from being used.
4. **Simple and explicit type inference** while maintaining safety through constraints.


A quick note on How does Go achieve type safety when it enables generics

How does Go achieve type safety when it enables generics?


Go’s import system, linked directly to git repository paths, was crucial to its early adoption. Unlike most languages, Go’s approach tightly integrates version control with package management, enhancing developer experience and reusability.

Using the git repository URL as the import path gives each package a unique identity, eliminating namespace conflicts and simplifying dependency management.

Example:

```go
import "github.com/username/projectname/package"
```

This approach avoids complex dependency tools, ensuring packages are isolated and traceable—a simple yet effective structure that keeps Go codebases clean and organized.

### Key benefits

**Global uniqueness**  
Git paths give each package a unique identifier (e.g., `github.com/user/repo/package`), avoiding namespace issues, especially in large projects with numerous dependencies.

**Direct version control**  
Go modules let developers pin dependencies to specific git tags or commits, achieving reproducible builds without a central registry. Any git repository, public or private, can serve as a Go package source, reducing reliance on third-party registries.

**Modularity and reusability**  
Git paths encourage modular, self-contained packages, making code easier to reuse and maintain. Direct links to git repositories also simplify code inspection and troubleshooting.

### Comparison to other Languages

Most other languages, like Python, Java, and Ruby, rely on centralized registries (e.g., PyPI, Maven) where packages are fetched by name, often leading to naming conflicts. Go’s git-based identifier, by contrast, provides a direct source link, eliminating centralized naming conventions. Go developers can pull packages directly from repositories, pin versions with `go.mod`, and enjoy simplified, traceable dependency management.

Example `go.mod`:

```go
module myproject

go 1.20

require (
    github.com/user/mathlib v1.3.0
    github.com/otheruser/utils v0.9.2
)
```

Go’s git path-based imports connect package management directly to version control, prioritizing simplicity, clarity, and reusability—key reasons behind Go’s adoption as a preferred language for modular software development.

----

### Go's old $GOPATH story for development and dependencies

[Source](https://utcc.utoronto.ca/~cks/space/blog/programming/GoTheGopathDevelopmentStory)

As people generally tell the story today, [Go](https://golang.org/) was originally developed without support for dependency management. Various community efforts evolved over time and then [were swept away](https://utcc.utoronto.ca/~cks/space/blog/programming/GoIsGooglesLanguage) in 2019 by [Go Modules](https://go.dev/blog/using-go-modules), which finally added core support for dependency management. I happen to feel that this story is a little bit incomplete and sells the original Go developers short, because I think they did originally have a story for how Go development and dependency management was supposed to work. To me, one of the fascinating bits in Go's evolution to modules is how that original story didn't work out. Today I'm going to outline how I see that original story.

In Go 1.0, the idea was that you would have one or more of what are today called [multi-module workspaces](https://go.dev/doc/tutorial/workspaces). Each workspace contained one (or several) of your projects and all of its dependencies, in the form of cloned and checked-out repositories. With separate repositories, each workspace could have different (and independent) versions of the same packages if you needed that, and updating the version of one dependency in one workspace wouldn't update any other workspace. Your current workspace would be chosen by setting and changing `$GOPATH`, and the workspace would contain not just the source code but also precompiled build artifacts, built binaries, and so on, all hermetically confined under its `$GOPATH`.

This story of multiple `$GOPATH` workspaces allows each separate package or package set of yours to be wrapped up in a directory hierarchy that effectively has all of its dependencies 'vendored' into it. If you want to preserve this for posterity or give someone else a copy of it, you can archive or send the whole directory tree, or at least the src/ portion of it. The whole thing is fairly similar to a materialized Python [virtual environment](https://docs.python.org/3/library/venv.html).

(The original version of Go did not default `$GOPATH` to `$HOME/go`, per for example [the Go 1.1 release notes](https://go.dev/doc/go1.1#gocmd). It would take until [Go 1.8 for this default to be added](https://go.dev/doc/go1.8#gopath).)

This story broadly assumes that updates to dependencies will normally be compatible, because otherwise you really want to track the working dependency versions even in a workspace. While you can try to update a dependency and then roll it back (since you normally have its checked out repository with full history), Go won't help you by remembering the identity of the old, working version. It's up to you to dig this out with tools like [the git reflog](https://git-scm.com/docs/git-reflog) or your own memory that you were at version 'x.y.z' of the package before you updated it. And 'go get -u' to update all your dependencies at once only makes sense if their new versions will normally all work.

This story also leaves copying workspaces to give them to someone else (or to preserve them in their current state) as a problem for you, not Go. However, Go did add ['experimental' support for vendoring dependencies](https://go.dev/doc/go1.5) in Go 1.5, which allowed people to create self-contained objects that could be used with 'go get' or other simple repository copying and cloning. A package that had its dependencies fully vendored was effectively a miniature workspace, but this approach had some drawbacks of its own.

I feel this original story, while limited, is broadly not unreasonable. It could have worked, at least in theory, in a world where preserving API compatibility (in a broad sense) is much more common than it clearly is (or isn't) in this one.

Go’s use of git repository paths as package identifiers is a unique and powerful feature. Unlike most languages that rely on centralized package repositories, Go links directly to git paths.

Go import design: using git repo path


Go is a type-safe, cloud-native language designed for simplicity with first-class support for concurrency.

The software community has been hungering for a small, simple, easy-to-learn and pragmatic programming language. Go fits the bill with:

## Simplicity

Go's beauty lies in its powerful simplicity. A list of NO's in Go design contains generics, ternary operation, pointer arithmetic,…. These features are missing because it doesn't fit, and it affects compilation speed, and end up making the language too complicated.

It's not what it included that makes it great; it's what is left out. Even at the expense of writing more code, simplicity acts as a key feature.

### First-class support for Concurrency

Go is not the first language that comes up with concurrency. We had Erlang for that a while ago. But Go is one of the few mainstream languages get it right.

Relying on CSP & "sharing by communicating" model plus the light-weight go-routine implementation makes concurrent programming perform at the best experience. If concurrency plays a vital part in your application, Go should be the first thing in your mind.

### Interface-driven OOP

This guy enables duck typing and dependency injection, two forms that extremely fruitful while avoiding the complexities of inheritance, which proves a notable downside in practice.

## Why Go

Let's be frank, Go is not a language for everything.

- It can't compare with Elixir in term of development speed.
- Rust beats it hard when it comes to performance.
- Simplicity alone? Try Python

Go has its place and shines the brightest with the following application types:

- Cloud application: Go's native feature set made it a naturally fit for the cloud, with concurrency built-in feature, Go is the best choice in the market for the two most common architecture in the cloud: micro-services and distributed system.
- Utilities and stand-alone tools: Go was born to beat the compilation time. It's a savior that the program can be compiled in shorter time and minimal size compare to most of other languages, allow it to be packed and distributed quickly.

## Go in Software Engineering

> Software engineering is what happens to programming when you add time and other programmers.-Russ Cox

Programming is hard. You have a problem, you write a program to solve it. Your program turns out great and works thing out - that's programming. But keeping your program work over time is struggling in many levels. More people coming, your business logic changes or some of your 3rd party library is no more working.

Software Engineering is about keeping your program resilient over time. And Go was born to live by that code, with a firm promise that the program we wrote in day 1 will compile and run perfectly in the future. All of the language technical design and toolset that were made by the concern of Software Engineering aims to simpler the program, make it easier to maintain, coordinate and evolve.


An overview of Go's key features, including simplicity, concurrency support, and interface-driven OOP. Explores Go's strengths in cloud applications and utility development, and its role in software engineering practices for maintaining resilient programs over time.

Go In Software Engineering


Here's another article that I want to reassure everyone to know about it. As Go pushes more type [composition over inheritance](https://go.dev/doc/faq#Is_Go_an_object-oriented_language), the POV on building 'unit' is different compare to other languages.

In Go, packages serve as the basic building blocks for creating **modular, reusable**, and maintainable software. Go's philosophy encourages developers to organize their code in a package-oriented way. 

Treat your packages as base units. This means that, from the outset, you should structure your project into reusable, well-encapsulated packages, each with a clear purpose.

### Key concepts

1. **Encapsulation and exporting**

By default, Go keeps all symbols (functions, variables, constants, types) within a package private unless they are explicitly exported. Exported symbols in Go start with an uppercase letter. This helps enforce encapsulation, exposing only what's necessary for external users while keeping the internal details hidden.
    
For example:

```go
// This function is public and can be used outside the package.
func Add(a, b int) int {
    return a + b
}

// This function is private to the package.
func subtract(a, b int) int {
    return a - b
}
```
    
2. **Separation of concerns**    

Packages should follow the principle of separation of concerns. Each package should serve a single purpose or set of related tasks. This makes the codebase more understandable and easier to maintain.
    
For instance, if you're building a web server, you might separate concerns into different packages like:  
- `http`: Handles HTTP requests and responses.
- `router`: Manages routing of different endpoints.
- `db`: Manages database interactions.

3. **Directory structure**
Go’s tooling is designed to work seamlessly with a package-oriented directory structure. Each directory contains its own package, which can be imported by other parts of your project.

Here's an example directory structure:
    
```go
myproject/
  ├── go.mod
  ├── cmd/            // For command-line tools and executables
  │   └── myapp/
  │       └── main.go
  ├── pkg/            // For libraries and reusable code
  │   └── http/
  │       └── handler.go
  ├── internal/       // For non-public packages
  │   └── config/
  │       └── config.go
  └── vendor/         // Third-party dependencies (if needed)
```

4. **Testing in packages**

Each package should also contain its own unit tests, which are placed in the same directory as the package itself, following Go's testing framework. Test files are named with the `_test.go` suffix and can test both exported and internal functions of a package.
    
Example:
    
```go
// In mathutil/add_test.go
package mathutil

import "testing"

func TestAdd(t *testing.T) {
    result := Add(2, 3)
    if result != 5 {
        t.Errorf("Expected 5, but got %d", result)
    }
}
```
    
5. **Modularity and reusability**

By structuring your code into distinct packages, you create **reusable** building blocks. These packages can be easily shared across different projects or within teams, and since Go’s import system relies on unique paths, there’s no conflict as long as each package’s import path is unique.

### How to apply

To apply package-oriented development with a focus on reusability, follow these steps. Keeping reusability in mind from the start ensures your code is modular, maintainable, and adaptable for future projects.

Please take note that all below examples are to demonstrate the approach

1. **Identify core domains and reusable utility needs**

- **Define key Domains**: Break down the main areas of functionality (or domains) in your project, such as `users`, `orders`, or `products` for an e-commerce app.
- **Identify reusable utilities**: List any generic functionalities, like string manipulation or date formatting, that multiple domains might need. Plan to create specific utility packages for these, separate from domain logic.

Example Structure:

```go
myproject/
  ├── users/
  ├── orders/
  └── util/
       ├── stringutil/
       └── timeutil/
```
2. **Design each package to be self-contained and purpose-driven**

- Each package should have a **single responsibility**, encapsulating everything it needs for its function. This approach makes it easier to reuse entire packages across projects.
- Avoid mixing different concerns. A `users` package, for example, should contain everything about user management (e.g., types, validation, storage) without including unrelated functions.

This single responsibility design ensures that when you need similar functionality in another project, you can reuse the package without modification.

3. **Create generalized, flexible functions**

- When writing functions within a package, think about how they might be used in other contexts. Avoid overly specific parameters or hardcoded values that tie functions to one scenario.
- For instance, instead of a `ValidateUserEmail` function, create a `ValidateEmail` function in a `validation` utility package, making it applicable to emails in any domain.

4. **Use interfaces to decouple dependencies**

- Define interfaces to allow flexible interactions between packages. Instead of directly calling functions from another package, define an interface in the calling package. This way, different implementations can be plugged in as needed.
- For example, if `orders` needs data from `users`, create an interface in `orders` that describes only the needed methods, letting any `User` service that meets this interface be used.

```go
// orders/service.go
package orders

type UserFetcher interface {
    GetUser(userID int) (User, error)
}

type OrderService struct {
    UserService UserFetcher
}
```

Using interfaces like this enhances reusability because each package relies on general contracts rather than specific implementations.

5. **Structure utility packages for broad use**

- Create focused utility packages that are purpose-driven and independent of specific domains. For instance, `stringutil` could contain generic string functions, while `timeutil` could handle time parsing and formatting.
- Organizing utilities in this way makes them truly reusable across any project or domain.

This organization avoids the common “catch-all” `utils` package, promoting well-structured, reusable functions that don’t add unnecessary dependencies.

6. **Document with reusability in mind**

- Write clear documentation for each package, focusing on its purpose, its public API, and how to use it. Document with the mindset that another developer (or future you) may want to reuse it in a different context.
- For utility functions, provide examples in the documentation to clarify their general use.

This documentation makes it easier for others to understand and adopt your package, increasing the likelihood of reuse.

7. **Write independent unit tests for each package**

- Write tests for each package that validate its functionality independently of the rest of the project. This not only ensures correctness but also supports reusability, as each package can be confidently reused without additional modification or testing.
- Use test files (ending with `_test.go`) within each package and focus on testing each function’s behavior as if it were in a standalone environment.

8. **Refactor with reusability in mind**

- As you add features, continually review and refactor to ensure packages remain focused and reusable. If a package is accumulating functions that don’t belong, refactor those into new packages, keeping each package aligned with its single responsibility.

Periodic refactoring keeps packages easy to understand, maintain, and reuse, avoiding monolithic packages that are hard to untangle or apply to new contexts.

----

- **Code organization**: Dividing functionality into small, focused packages keeps the code clean and organized.
- **Reusability**: Once written, a package can be reused in many projects or different parts of a large project.
- **Maintainability**: With a well-structured package layout, code becomes easier to maintain, modify, and extend.
- **Collaboration**: Teams can work on separate packages concurrently, as each package represents an independent unit of functionality.
- **Testing and debugging**: Since each package is modular, testing becomes easier, and debugging issues can be done within the context of specific packages.


In Go, packages serve as the basic building blocks for creating modular, reusable, and maintainable software.

Package first design

An in-depth exploration of message brokers, their use cases, and implementation examples using RabbitMQ in Go. Covers the basics of message queues, their advantages and disadvantages, and common patterns like work queues and publish/subscribe models.

Message Queues And Streaming Platforms Eg Kafka Nats Rabbitmq

Profiling in Go is a powerful tool that can help you identify and fix the subtle issues in your code quickly and efficiently. In this article, we'll explore the world of profiling in Go and show you how you can use it to catch all the issues in your code.

Profiling in Go


### Array

**TL;DR:** The differences between array in Go and C:

- Arrays are values. Assigning one array to another copies all the elements.
- In particular, if you pass an array to a function, it will receive a copy of the array, not a pointer to it.
- The size of an array is part of its type. The types [4]int and [5]int are distinct.

In Go language, the terminology `Array` has a bit different from another language like C, JS, ... In Go, the `array has a fixed length and type `Take a look of array implementation in Go.

![](assets/slice-and-array-in-golang_a650b13e6028a391f8acdc858b08c372_md5.webp)

Its length is part of its type ([4]int and [5]int are distinct, incompatible types). For example, you can compare two arrays have the same type [4]int.

```go
a := [4]int{1,1,1,1}
b := [4]int{0,0,0,0}
c := [4]int{}

fmt.Println(a == b) // false
fmt.Println(b == c) // true
fmt.Println(c == a) // false
```

As you can see, the array does not need to be initialized explicitly, the `array c` in the example above is initial with zero value of an array type (zero value of an integer is 0).

But two array [4]int and [5]int is incompatible with each other

```go
a := [4]int{}
b := [5]int{}

fmt.Println(a == b) // mismatched types [4]int and [5]int
```

You also can let Go's compiler count the array size for you at compile time.

```go
a := [...]int{1,1,1}
b := [3]int{1,1,1}

// in this example, both a and b have a [3]int type and can compare with
// each other

fmt.Println(a == b) // true
```

For representation for an array of [4]int in memory is for integer value laid out `sequentially`

![](assets/slice-and-array-in-golang_8327bf995dd32badeef1e1d0eb4eeda5_md5.webp)

So, in Go, the array is values. An array variable holds the entire array (not a pointer to the first element). Let's say the array is a struct ( but using index instead of named field). Because an array is not a pointer to the first element, so when we assign, pass an array to a function, it will make a copy of its content

### Slice

In Go code, we don't often see array because of its inflexible, slice - on the other hand - is everywhere. Slice is an abstraction built on top the array. Unlike the Array, Slice type has no specified length; you can declare a slice like an array but `without the count element`.

```go
a := []int{1,2,3,4}
```

A slice has three components (will be talking more detail in next section):

- `pointer` : point to underlying array
- `length ` : the number of elements referred to by the slice
- `capacity` : the number of elements in the underlying array

Because slices hold references to an underlying array, so if you assign one slice to another, both refer to the same array.

```go
a := []int{1,2,3,4}
b := a
b[0] = 10

fmt.Println(a) // [10 2 3 4]
```

We can make a slice by using built-in `make` function. When called, `make` allocates an array and returns a slice that refers to that array. Note that the zero value of a slice is `nil.`

```go
// make(type, length[, capacity])
a := make([]int,4,8)

// if we omit capacity, it defaults to the specified length
b := make([]int,4)

// len b = 4
// capacity b = 4
```

or `slicing` an array or another slice using these format

```go
a := []int{1,2,3,4,5,6,7,8,9,10}

b := a[0:5]
fmt.Println(b) // [1 2 3 4 5]

c := a[5:]
fmt.Println(c) // [6 7 8 9 10]

d := a[:5]
fmt.Println(d) // [1 2 3 4 5]

e := a[:]
fmt.Println(e) // [1 2 3 4 5 6 7 8 9 10]
```

The length and capacity of a slice can be inspected using the built-in `len` and `cap` functions.

```go
a := make([]int,4, 8)

fmt.Println(len(a)) // 4
fmt.Println(cap(a)) // 8
```

### Slice internal

Let's take a look at `slice` implementation in Go.

```go
type slice struct{
	array unsafe.Pointer
	len int
	cap int
}
```

For example, if we create a slice by using make([]byte,5), the slice will be structured like this:

![](assets/slice-and-array-in-golang_f18621d9bf057f8c2ea818ca438379c3_md5.webp)

A slice cannot be grown beyond its capacity. Attempting to do so will cause a runtime panic, just as when indexing outside the bounds of a slice or array. Similarly, slices cannot be re-sliced below zero to access earlier elements in the array.

**So the question is, What if we `*append*` an element to a slice which has reached its capacity?**

### Append

Let's dig into a source code (go/src/reflect/value)

![](assets/slice-and-array-in-golang_1f1383e462f5fa432205e471759c4051_md5.webp)

so the `append` built in function do a few things here:

First, it will check an input value, if it's not a `Slice` the program will throw a panic, then the interesting things here is `grow` function , it will take an old slice and length of the new slice we want to append. Let take a look at `grow` function:

![](assets/slice-and-array-in-golang_de7599e21a9ed4cf0e4a9d31169129e2_md5.webp)

So now we understand the mechanism behind the `append` function, it will allocate a `new Slice` which have more capacity than the old one

The interesting thing is how Go decide how much capacity the new slice is. As you can see if the old slice capacity is lesser than 1024, it will double the old slice's capacity, but when it grows bigger than 1024 Go adds `old slice's capacity / 4` to the old capacity

### Appendix

- [https://golang.org/doc/effective_go.html#slices](https://golang.org/doc/effective_go.html#slices)
- [https://blog.golang.org/slices](https://blog.golang.org/slices)


An in-depth exploration of arrays and slices in Go, covering their differences, internal implementations, and key operations like append. Learn about fixed-length arrays, flexible slices, and how Go manages memory allocation for growing slices.

Slice And Array In Golang


One common issue we often tackle in backend engineering is writing test cases. In this article, we will explore the techniques for crafting effective tests in Go, discussing best practices for writing unit tests and utilizing mocks to achieve better isolation. Although our primary focus lies in unit testing-related practices, it is important to note that Golang also supports integration testing. We will also tackle the subject of integration testing in a future article, where we will examine the details and best practices for integration testing in Golang.

## Introduction
### Importance of testing in software development

Testing is crucial in software development to catch bugs and errors, ensure maintainability and modularity, and improve security, and overall software quality. With the rise of cybersecurity threats, testing is becoming increasingly important to ensure software systems are secure and reliable.

What follows is a non-comprehensive list of the benefits you get from adopting unit testing:

- **Unit tests enable earlier bug detection and resolution**
- **Your suite of unit tests becomes a safety net for developers**

A comprehensive suite of unit tests can act as a safety net for developers. By frequently running the tests, they can assure their recent modifications to the code haven’t broken anything

- Unit tests can contribute to higher code quality

This item is a natural consequence of the previous one. Since unit tests act as a safety net, developers become more confident when changing the code. They can refactor the code without fear of breaking things, driving the general quality of the codebase up.

- **Detect code smells in your codebase**

If the ease of adding unit tests to a codebase is a good sign, the opposite is also true. Having a hard time creating unit tests for a given piece of code might be a sign of code smells in the code—e.g. functions that are too complex.

### Overview of Golang Testing Framework

The Golang testing package offers a user-friendly framework to create unit tests, benchmarks, and examples, streamlining the development process in Golang by enabling execution from the command line. Package testing allows for a variety of test types, including performance, parallel, and functional testing, as well as any combination these.

**Steps for writing test suite in Golang:**

- Create a file whose name ends with \_test.go
- Import package testing by import “testing” command
- Write the test function of form\ <u>func TestXxx(_testing.T)_</u> which uses any of Error, Fail, or related methods to signal failure.
- Put the file in any package.
- Run command go test

Example of a test file:

```go
package main

import (
	"testing"
)

// test function
func TestYourFunc(t *testing.T) {
	actualString := YourFunc()
	expectedString := "dwarvesv"
	if actualString != expectedString{
		t.Errorf("Expected String(%s) is not same as"+
		" actual string (%s)", expectedString,actualString)
	}
}
```

## Strategies for Writing Effective Tests

### Make your code testable and easy to test

When working on code projects, developers often devote a large portion of their time to choosing the right frameworks, libraries, databases, and other third-party components, while the importance of testing is sometimes overlooked.

Proper testing actually makes your project better because it encourages you to:

- Apply clean code: write short functions, handle a single task per function, etc.
- Write extendable and agnostic code through the use of abstractions, interfaces and mocks.
- Understand the business logic better by testing regular/edge cases and high coverage of these.
- Avoid legacy, long-untouched and unmaintainable code — tests will ease the process of maintaining and verifying changes to code so it doesn’t rot.

### Writing clear and concise test cases

One of the most important of a good test is easy to read and maintain, it should be taken in mind as important as implementing:

**Naming test case** The name of your test should consist of three parts:

- The name of the method being tested.
- The scenario under which it's being tested.
- The expected behavior when the scenario is invoked.

Examples:

- Bad naming: `Error 1`, `invalid input 1`, `test 1`
- Good naming: `Should returns same number WHEN input single number`, `Should returns 0 WHEN emtpy string`

**Table driven testing** A test can quickly become unreadable, repetitive, and overall annoying when the function you want to test is handling too many tasks, especially when there are many different cases you want to test, for example:

```go
package main

import (
   "github.com/stretchr/testify/assert"
   "testing"
)

func TestHadAGoodGame(t *testing.T) {
   tests := []struct {
      name     string
      stats   Stats
      goodGame bool
      wantErr  string
   }{
      {"sad path: invalid stats", Stats{Name: "Sam Cassell",
         Minutes: 34.1,
         Points: -19,
         Assists: 8,
         Turnovers: -4,
         Rebounds: 11,
         }, false, "stat lines cannot be negative",
      },
      {"happy path: good game", Stats{Name: "Dejounte Murray",
         Minutes: 34.1,
         Points: 19,
         Assists: 8,
         Turnovers: 4,
         Rebounds: 11,
      }, true, ""},
   }
   for _, tt := range tests {
      isAGoodGame, err := hadAGoodGame(tt.stats)
      if tt.wantErr != "" {
         assert.Contains(t, err.Error(), tt.wantErr)
      } else {
         assert.Equal(t, tt.goodGame, isAGoodGame)
      }
   }
}
```

### Use Interfaces and Avoid file I/O, API call

When writing tests, it's important to use interfaces and avoid file I/O and API calls wherever possible. You want your tests to be fast, independent, isolated, consistent, and not flaky. Here are some best practices to keep in mind:

**Use interfaces** By using interfaces, you can decouple your code from its dependencies, making it easier to test in isolation. Instead of calling concrete implementations, you can call interfaces that define the behavior you need, for example:

```go
type repository interface {
  GetRecipe(recipeID string) (domain.Recipe, error)
  CreateRecipe(recipe domain.Recipe) error
  UpdateRecipe(recipe domain.Recipe) error
}
```

**Mock dependencies** To test code that relies on external dependencies, use mocks to simulate the behavior of those dependencies. This approach allows you to test your code in isolation, without relying on external resources.

```go
import (
    "testing"

    "github.com/stretchr/testify/assert"
)

func Test_getFromDB(t *testing.T) {
    mockDB := NewMockDB(t)
		mockDB.On("GetFlavor").Return("Chocolate", nil)
    flavor := getFromDB(mockDB)
    assert.Equal(t, "chocolate", flavor)
}
```

**Avoid file I/O**

File I/O can be slow and unreliable, making it difficult to test code that relies on it. Instead, consider using interfaces to abstract away file I/O and using mocks to simulate file operations during testing.

```go
func Test_getFromDB(t *testing.T) {
		mockReader := reader.NewMock()
		mockReader.On("Read", mock.Anything).Return(100, nil)
    scanSvc := scan.NewInstance(mockReader)

		expectedSize := 100
    assert.Equal(t, scan.size(), expectedSize)
}
```

**Avoid API calls** Like file I/O, API calls can be slow and unreliable, making it difficult to test code that relies on them. Instead, consider using interfaces to abstract away API calls and using mocks to simulate API responses during testing.

```go
func Test_AcceptJobRequest(t *testing.T) {
		mockEmailGwy := new(email.MockGateway)
		mockEmailGwy.On("SendEmailWithTemplate", mock.Anything, mock.Anything).Return(nil)

		workerCtrl := worker.New(mockEmailGwy)
		err := workerCtrl.acceptAndNotify()
		// Some asserts here
}
```

### Covering edge cases and boundary conditions

As we all know, this is a basic test strategy, but this reveals most of the potential bugs. Because humans usually break the rule, and that would break the happy flow. It's important to cover edge cases and boundary conditions to ensure that your code can handle extreme or unexpected values. Here are some tips for covering edge cases and boundary conditions in your tests:

- **Test extreme values**: Be sure to test extreme values, such as the maximum and minimum values that your code can handle.
- **Test unexpected input**: Be sure to test any weird input values or characters that might look like it would affect the test.
- **Test corner cases**: Be sure to test corner cases, such as scenarios where multiple inputs or conditions intersect. This approach can help you catch issues with complex logic or interactions between different parts of your code.

## Test coverage

Test coverage is defined as a metric in Software Testing that measures the amount of testing performed by a set of tests. It will include gathering information about which parts of a program are executed when running the test suite to determine which branches of conditional statements have been taken.

In simple terms, it is a technique to ensure that your tests are testing your code or how much of your code you exercised by running the test.

- **Very Poor: 0-20% coverage**. This means that very few or no unit tests have been written to test the code, which can result in bugs and errors going unnoticed.
- **Poor: 21-40% coverage**. This means that some unit tests have been written, but a significant amount of code remains untested.
- **Acceptable: 41-60% coverage**. This means that a reasonable number of unit tests have been written to test the code, but there is still room for improvement.
- **Good: 61-80% coverage**. This means that a large percentage of the code has been covered by unit tests, and most potential bugs and errors have been caught.
- **Very Good: 81-100% coverage**. This means that almost all code has been covered by unit tests, and the likelihood of bugs and errors slipping through is very low. However, achieving 100% coverage may not always be practical or necessary, depending on the nature and complexity of the code.

Although it depends on the project's status, ideally, we recommend aiming for a test coverage between **61-80%**. However, don't become obsessed with the number; the primary goal is to write tests that help us catch bugs effectively.

## Tooling and library

Golang possesses a robust testing framework; however, employing supplementary tools can enhance the development experience and reduce code creation efforts.

- **Mocking**: Rather than creating mock code manually, consider utilizing a mocking library, such as gomock or mockery, which supports mocking and generates mocks from interfaces, thereby reducing time expenditure.
- **Assert**: The default Golang testing framework has limited assertion capabilities. Alternatively, tools like testify provide improved support and more user-friendly assertions.

## Conclusion

In this article, we've covered the basics of testing in Golang and explored some strategies for writing effective and maintainable tests. We've seen best practices for using interfaces, avoiding file I/O and API calls, automating unit tests, and covering edge cases and boundary conditions.

By following these strategies and best practices, you can write tests that are easier to run, understand, and maintain. By investing time in testing, you can catch bugs and errors earlier in the development process, which can save time and improve the overall quality of your code.

Remember, testing is not a one-time task but an ongoing process that should be integrated into your development workflow. With the right tools, frameworks, and mindset, testing can become a natural and valuable part of your development process, helping you build more reliable and maintainable software.

## References

[Overview of testing package in Golang](https://www.geeksforgeeks.org/overview-of-testing-package-in-golang/)

[5 tips for better unit testing in golang](https://blog.devgenius.io/5-tips-for-better-unit-testing-in-golang-b25f9e79885a)

[https://learn.microsoft.com/en-us/dotnet/core/testing/unit-testing-best-practices](https://learn.microsoft.com/en-us/dotnet/core/testing/unit-testing-best-practices)

[https://www.testim.io/blog/unit-testing-best-practices/](https://www.testim.io/blog/unit-testing-best-practices/)

[https://www.freecodecamp.org/news/a-beginners-guide-to-testing-implement-these-quick-checks-to-test-your-code-d50027ad5eed/](https://www.freecodecamp.org/news/a-beginners-guide-to-testing-implement-these-quick-checks-to-test-your-code-d50027ad5eed/)

[https://testing.googleblog.com/2020/08/code-coverage-best-practices.html](https://testing.googleblog.com/2020/08/code-coverage-best-practices.html)


Unit Testing Best Practices In Golang


![](assets/use-go-selenium-to-crawl-data_f963144e3cfac24481dbfeb02cf6a0e6_md5.webp)

## Crawl data

Crawl is a widespread issue occurring in making software. News, discount news, film ticket, etc are some examples of crawl. To be simple, it is analytics HTML, read cards, and extract data. The Go library I usually use is [goquery](https://github.com/PuerkitoBio/goquery).

However, crawling an original HTML will not work in some cases: data loaded by ajax (when reading HTML, we will only see wrapper, not data), or must login when entering a page need crawl.

In this article, take crawling [Amazon deal](https://www.amazon.com/gp/goldbox/all-deals/ref=gbps_ftr_s-3_3022_wht_541966?ie=UTF8&*Version*=1&*entries*=0&gb_f_GB-SUPPLE=sortOrder%3ABY_SCORE%2CenforcedCategories%3A3760911%2C2335752011%2C541966&pf_rd_p=2292853022&pf_rd_s=slot-3&pf_rd_t=701&pf_rd_i=gb_all&pf_rd_m=ATVPDKIKX0DER&pf_rd_r=14CQSB5TF4GTC2RNHDAG) into consideration. In this page, javascript will call ajax taking data and then pour it into DOM. When using goquery read HTML, we will not see div cards like inspecting elements.

To these types, I use selenium to run the web in a real browser, take action to have fully loaded HTML before extracting data.

Selenium running in JVM is quite famous in automation test. It allows me to run script test in a real browser. My method will be: Use selenium to run the Amazon page, wait for javascript to load, and then crawl the data normally.

## How to setup

Firstly, you go to [seleniumhq](https://selenium.dev/downloads/) link to download and set up seleniumhq. Selenium plays a role like a server, receiving requests sent from my code Go.

To run it, we go to the folder containing file jar and run the command:

```plain_text
java -jar selenium-server-standalone-2.50.1.jar -port 8081
```

![](assets/use-go-selenium-to-crawl-data_79536b2784ffffd405fdcbd54b56927f_md5.webp)

\=> We have server selenium running at port 8081. Next, you pull Go-selenium in by Go get:

```javascript
go get sourcegraph.com/github.com/sourcegraph/go-selenium
```

After that, we need to set up a browser. I choose Firefox. Remember, when running locally, we only need to set up Firefox on the web. In contrast, running on the host we need to set up Firefox by Shell script. You can refer to [how to set up Selenium on Ubuntu 14.04](https://gist.github.com/curtismcmullan/7be1a8c1c841a9d8db2c) Done! Now let’s code.

We need:

- Remote to server selenium
- Access to Amazon deal link
- Conduct analytics HTLM to get information. I will print page title and the image of the first product

```javascript
func main() {
    var webDriver selenium.WebDriver
    var err error
    // set browser as firefox
    caps := selenium.Capabilities(map[string]interface{}{"browserName": "firefox"})
    // remote to selenium server
    if webDriver, err = selenium.NewRemote(caps, "http://localhost:8081/wd/hub"); err != nil {
        fmt.Printf("Failed to open session: %s\n", err)
        return
    }
    defer webDriver.Quit()

    err = webDriver.Get(URL_AMAZON_DEAL)
    if err != nil {
        fmt.Printf("Failed to load page: %s\n", err)
        return
    }
    // sleep for a while for fully loaded javascript
    time.Sleep(4 * time.Second)
    // get title
    if title, err := webDriver.Title(); err == nil {
        fmt.Printf("Page title: %s\n", title)
    } else {
        fmt.Printf("Failed to get page title: %s", err)
        return
    }

    var elem selenium.WebElement
    elem, err = webDriver.FindElement(selenium.ByCSSSelector, "#widgetContent")
    if err != nil {
        fmt.Printf("Failed to find element: %s\n", err)
        return
    }

    var firstElem selenium.WebElement
    firstElem, err = elem.FindElement(selenium.ByCSSSelector, ".a-section .dealContainer")
    if err != nil {
        fmt.Printf("Failed to find element: %s\n", err)
        return
    }
    // get image
    image, err := firstElem.FindElement(selenium.ByCSSSelector, "img")
    if err == nil {
        img, _ := image.GetAttribute("src")
        fmt.Println(img)
    }
}
```

Run the code, we have

```javascript
Page title: Gold Box Deals | Today's Deals - Amazon.com
https://images-na.ssl-images-amazon.com/images/I/51eU5JrGAXL.\_AA210\_.jpg
```

Well, we got all the needed information.

## Conclusion

Above is my knowledge when having problems with crawl in developing software. Here is Go software programming language. Selenium also helps us in other cases, like pages need login, web pages request captcha, etc. If anyone has other experiences, I hope to hear from you.


Learn how to use Go with Selenium to crawl dynamic web pages, overcoming challenges like AJAX-loaded content and login requirements. This guide demonstrates setting up Selenium, writing Go code to interact with web elements, and extracting data from Amazon's deal page.

Use Go Selenium To Crawl Data


## [Go Features By Version](https://antonz.org/which-go/)

- Context:

  - Go is released every six months.
  - Each major Go release is supported until there are two newer major releases.
  - Critical problems are fixed by issuing minor revisions.

- Solution:

  ![](assets/go-features-by-version.png)

## [Make Your Programs Use Less Memory](https://github.com/dkorunic/betteralign)

- Context:

  - Package [fieldalignment](https://pkg.go.dev/golang.org/x/tools/go/analysis/passes/fieldalignment) defines an Analyzer that detects structs that would use less memory if their fields were sorted.

- Solution:

  - betteralign is fork of fieldalignment with:
    - skips over generated files, either files with known "generated" suffix (\_generated.go, \_gen.go, .gen.go, .pb.go, .pb.gw.go) or due to package-level comment containing Code generated by... DO NOT EDIT. string,
    - skips over test files (files with \_test.go suffix),
    - skips over structs marked with comment betteralign:ignore,
    - doesn't lose comments (field comments, doc comments, floating comments or otherwise) but the comment position heuristics is still work in progress,
    - does very reliable atomic file I/O with strong promise not to corrupt and/or lose contents upon rewrite (not on Windows platform),
    - has more thorough testing in regards to expected optimised vs golden results,
    - integrates better with environments with restricted CPU and/or memory resources (Docker containers, K8s containers, LXC, LXD etc).

## [A Go-powered Minecraft 1.21 server](https://github.com/ZeppelinMC/Zeppelin)

- Context:

  - Blazingly fast, highly optimized server implementation written in Go for Minecraft 1.21

## [A Source Code Editor in Pure Go](https://github.com/jmigpin/editor)

- Auto-indentation of wrapped lines.
- No code coloring (except comments and strings).
- Many TextArea utilities: undo/redo, replace, comment, ...
- Handles big files.
- Start external processes from the toolbar with a click, capturing the output to a row.
- Drag and drop files/directories to the editor.
- Detects if files opened are changed outside the editor.
- Plugin support
  - examples such as gotodefinition and autocomplete below.
- Golang specific:
  - Calls goimports if available when saving a .go file.
  - Clicking on .go files identifiers will jump to the identifier definition (needs gopls).
  - Debug utility for go programs (GoDebug cmd).
    - allows to go back and forth in time to consult code values.
- Language Server Protocol (LSP) (code analysis):
  - -lsproto cmd line option
  - basic support for gotodefinition and completion
  - mostly being tested with clangd and gopls
- Inline complete
  - code completion by hitting the tab key (uses LSP).

![](assets/go-editor-image.png)

## [Ollama 0.3 Quickly Run Large Language Models](https://github.com/ollama/ollama)

![](assets/ollama.png)

---

- https://antonz.org/which-go
- https://github.com/dkorunic/betteralign
- https://pkg.go.dev/golang.org/x/tools/go/analysis/passes/fieldalignment
- https://github.com/ZeppelinMC/Zeppelin
- https://github.com/jmigpin/editor
- https://github.com/ollama/ollama


Explore Go version features, memory optimization techniques, a Go-powered Minecraft server, a pure Go source code editor, and a tool for running large language models.

#5 Go 1.23 features, Memory, Minecraft, and More

Go Commentary #5: Features, Memory Optimization, Minecraft Server, Code Editor, and LLM Tool


## [Cogent Core: A New GUI Framework for Go](https://www.cogentcore.org/blog/initial-release)

- GUI framework written in Go that allows you Code Once, Rune Everywhere (Core) (macOS, Windows, Linux, iOS, Android and web)

- The same Cogent Core app running on many devices using the same code:

![](assets/cogent-core-multi-devices.png)

- Simple Hello World app:

```go
package main

import "cogentcore.org/core/core"

func main() {
	b := core.NewBody()
	core.NewButton(b).SetText("Hello, World!")
	b.RunMainWindow()
}
```

- Supports all usual GUI widgets:

```go
core.NewButton(b).SetText("Send").SetIcon(icons.Send).OnClick(func(e events.Event) {
	core.MessageSnackbar(b, "Message sent")
})
core.NewText(b).SetText("Name:").SetTooltip("Enter your name in the text field")
core.NewTextField(b).SetPlaceholder("Jane Doe")
value := 0.5
spinner := core.Bind(&value, core.NewSpinner(b))
slider := core.Bind(&value, core.NewSlider(b))
spinner.OnChange(func(e events.Event) {
	slider.Update()
})
slider.OnChange(func(e events.Event) {
	spinner.Update()
})
core.NewColorButton(b).SetColor(colors.Orange)
type language struct {
	Name   string
	Rating int
}
sl := []language{{"Go", 10}, {"Python", 5}}
core.NewTable(b).SetSlice(&sl).OnChange(func(e events.Event) {
	core.MessageSnackbar(b, fmt.Sprintf("Languages: %v", sl))
})
```

![](assets/cogent-core-widgets.png)

- Interactive plots of data

```go
type Data struct {
	Time   float32
	Users  float32
	Profit float32
}
plotcore.NewPlotEditor(b).SetSlice([]Data{
	{0, 500, 1520},
	{1, 800, 860},
	{2, 1600, 930},
	{3, 1400, 682},
})
```

![](assets/cogent-core-plots.png)

- Key features:

  - A full set of GUI widgets, with built-in support for most elements of [Material 3](https://m3.material.io/) standard

    - tooltips
    - drag-and-drop
    - sprites
    - popup completion
    - full text editor with code highlighting
    - ...

  - Extension styling properties makes styling easy

  - Responsive widget elements enables 1 codebase run across platforms

  - Transparent implementation of widgets makes customization easy

  - Dynamic color system

  - ...

- Full software ecosystem

  ![](assets/cogent-core-neural-network.png)

  ![](assets/cogent-core-cogent-code.png)

  ![](assets/cogent-core-cogent-canvas.png)

## [Russ Cox Steps Down as Tech Lead of Go](https://groups.google.com/g/golang-dev/c/0OqBkS2RzWw/m/GzWvX5u6AQAJ?pli=1)

- Russ stepped down as Tech Lead of Go after 12 years, as Austin Clements taking the role.

- Now Russ is fulling working on Oscar - an AI agent system to help maintain OSS (including Gaby)

- Sources:
  - https://go.googlesource.com/oscar/+/refs/heads/master/README.md
  - https://go.googlesource.com/oscar/+/refs/heads/master/internal

---

- https://www.cogentcore.org/blog/initial-release
- https://groups.google.com/g/golang-dev/c/0OqBkS2RzWw/m/GzWvX5u6AQAJ?pli=1
- https://go.googlesource.com/oscar/+/refs/heads/master/README.md
- https://go.googlesource.com/oscar/+/refs/heads/master/internal


Introducing Cogent Core, a new GUI framework for Go, and announcing Russ Cox stepping down as Tech Lead of Go after 12 years.

#6 Cogent Core, Russ Cox stepping down

Go Commentary #6: GUI Framework, Leadership Change


## [Go 1.23 Release Note](https://go.dev/doc/go1.23)

- Full notes that you should skim through to get full-fledged of Go 1.23

## [A New Home for nhooyr/websocket](https://coder.com/blog/websocket)

- [nhooyr/websocket](https://github.com/nhooyr/websocket) is adopted by [Coder](https://coder.com/), CDE - Cloud Development Environment

## [Go structs are copied on assignment ](https://jvns.ca/blog/2024/08/06/go-structs-copied-on-assignment/)

- Inspired by [Common Go Mistakes](https://100go.co/)

```go
type Thing struct {
    Name string
}

func main() {
  thing := Thing{"record"}
  other_thing := thing
  other_thing.Name = "banana"
  fmt.Println(thing)            // {record}
}
```

```go
type Thing struct {
  Name string
}

func findThing(things []Thing, name string) *Thing {
  for _, thing := range things {
    if thing.Name == name {
      return &thing
    }
  }
  return nil
}

func main() {
  things := []Thing{Thing{"record"}, Thing{"banana"}}
  thing := findThing(things, "record")
  thing.Name = "gramaphone"
  fmt.Println(things)           // [{record} {banana}]
}
```

=> fix:

```go
func findThing(things []Thing, name string) *Thing {
  for i := range things {
    if things[i].Name == name {
      return &things[i]
    }
  }
  return nil
}
```

```go
func main() {
	x := []int{1, 2, 3, 4, 5}
	y := x[2:3]
	fmt.Println(y)
	y = append(y, 555)          // y = {3, 555}
	fmt.Println(x)              // {1, 2, 3, 555, 5}
}
```

=> fix:

```go
func main() {
	x := []int{1, 2, 3, 4, 5}
	y := x[2:3:3]
	fmt.Println(y)
	y = append(y, 555)          // y = {3, 555}
	fmt.Println(x)              // {1, 2, 3, 4, 5}
}
```

---

- https://go.dev/doc/go1.23
- https://coder.com/blog/websocket
- https://github.com/coder/websocket
- https://jvns.ca/blog/2024/08/06/go-structs-copied-on-assignment/
- https://100go.co/


Exploring Go 1.23 release notes, the new home for nhooyr/websocket, and common mistakes with Go structs and slices.

#7 Go 1.23, Websockets, and Structs

Go Commentary #7: Releases, Websockets, and Struct Behavior


## [GoNB: A Go Notebook Kernel for Jupyter](https://github.com/janpfeifer/gonb)

- Auto-complete and contextual help while coding.
- Rich content display: HTML, markdown (with latex), images, javascript, svg, videos, etc.

![](assets/go-commentary-aug-23_gonb-auto-complete.webp)

- Widgets (sliders, buttons) support: interact using HTML elements
- Plotly integration using [go-plotly](https://github.com/go-echarts/go-echarts)
- Apache ECharts integration using [gonb-echarts](https://github.com/janpfeifer/gonb-echarts) and [go-echarts](https://github.com/go-echarts/go-echarts)

![](assets/go-commentary-aug-23_gonb-chart.webp)

## [kubetrim: Trim Your KUBECONFIG Automatically](https://github.com/alexellis/kubetrim)

```cli
$ kubectx

default
do-lon1-openfaas-cluster
kind-2
kind-ingress

$ kubetrim

kubetrim (dev) by Alex Ellis

Loaded: /home/alex/.kube/config. Checking..
  - kind-2: ✅
  - kind-ingress: ❌ - (failed to connect to cluster: Get "https://127.0.0.1:40349/api/v1/nodes": dial tcp 127.0.0.1:40349: connect: connection refused)
  - default: ✅
  - do-lon1-openfaas-cluster: ❌ - (failed to connect to cluster: Get "https://da39a3ee5e6b4b0d3255bfef95601890afd80709.k8s.ondigitalocean.co.uk/api/v1/nodes": dial tcp: lookup da39a3ee5e6b4b0d3255bfef95601890afd80709.k8s.ondigitalocean.co.uk on 127.0.0.53:53: no such host)
Updated: /home/alex/.kube/config (in 364ms).

$ kubectx

default
kind-2
```

## [12 Talks from GopherCon UK 2024](https://youtube.com/playlist?list=PLDWZ5uzn69ezR6D6FUj_iBSOyRc9xaZFP&si=IdFGYzOivQqFRsVb)

- Key highlights:
  1. **Performance Testing Tools**: One of the standout presentations was about a home-grown performance testing tool used to replay HTTP access logs against infrastructure setups. This tool was crucial for performance testing at scale, particularly during low-traffic periods, ensuring the system’s responsiveness without disrupting overnight operations.
  2. **Software Design and Complexity**: A session by Shivam Acharya and Peter Chai delved into the complexities of software design, discussing how complexity should be embraced and managed thoughtfully. They explored how decisions in API design can either simplify or complicate the interaction between different system components.
  3. **Event-Driven Workflows**: Another notable talk covered the implementation of event-driven workflows, highlighting the challenges and decisions in designing a technology-agnostic library that could work with various technologies like Kafka, Redux, Postgres, and MySQL.
  4. **AI and the Future of Work**: There was also a provocative discussion on how AI is influencing the future of jobs in tech, encouraging developers to think about how they can stay relevant as automation and AI continue to evolve.

---

- https://github.com/janpfeifer/gonb
- https://github.com/go-echarts/go-echarts
- https://github.com/janpfeifer/gonb-echarts
- https://github.com/go-echarts/go-echarts
- https://github.com/alexellis/kubetrim
- https://youtube.com/playlist?list=PLDWZ5uzn69ezR6D6FUj_iBSOyRc9xaZFP&si=IdFGYzOivQqFRsVb


Explore GoNB for Jupyter notebooks, kubetrim for KUBECONFIG management, and key highlights from GopherCon UK 2024 talks, covering performance testing, software design, event-driven workflows, and AI's impact on tech jobs.

#8 GoNB, kubetrim, and GopherCon UK 2024

Go Commentary #8: Jupyter Notebooks, Kubernetes Tools, GopherCon Talks


## [TinyGo 0.33.0: The Go Compiler for 'Small Places'](https://github.com/tinygo-org/tinygo/releases/tag/v0.33.0)

- Context:

  - TinyGo is a Go compiler designed for small environments like microcontrollers, WebAssembly (wasm/wasi), and command-line tools.

  - It utilizes Go language libraries and LLVM to offer an alternative method for compiling Go programs

- Changelog highlights:

  - **Go 1.23 support** (including the new range-over-func language feature)
  - ...

## [sqlite-vec: A Vector Search Extension for SQLite](https://github.com/asg017/sqlite-vec)

- Context:

  - An extremely small, "fast enough" vector search SQLite extension that runs anywhere.
    - Store and query float, int8, and binary vectors in vec0 virtual tables
    - Written in pure C, no dependencies, runs anywhere SQLite runs (Linux/MacOS/Windows, in the browser with WASM, Raspberry Pis, etc.)
    - Pre-filter vectors with rowid IN (...) subquerie

- Usage:

```
go get -u github.com/asg017/sqlite-vec-go-bindings/ncruces
```

```go
package main

import (
	_ "embed"
	"log"

	_ "github.com/asg017/sqlite-vec-go-bindings/ncruces"
	"github.com/ncruces/go-sqlite3"
)

func main() {
	db, err := sqlite3.Open(":memory:")
	if err != nil {
		log.Fatal(err)
	}

	stmt, _, err := db.Prepare(`SELECT vec_version()`)
	if err != nil {
		log.Fatal(err)
	}

	stmt.Step()
	log.Printf("vec_version=%s\n", stmt.ColumnText(0))
	stmt.Close()
}
```

```sql
.load ./vec0

create virtual table vec_examples using vec0(
  sample_embedding float[8]
);

-- vectors can be provided as JSON or in a compact binary format
insert into vec_examples(rowid, sample_embedding)
  values
    (1, '[-0.200, 0.250, 0.341, -0.211, 0.645, 0.935, -0.316, -0.924]'),
    (2, '[0.443, -0.501, 0.355, -0.771, 0.707, -0.708, -0.185, 0.362]'),
    (3, '[0.716, -0.927, 0.134, 0.052, -0.669, 0.793, -0.634, -0.162]'),
    (4, '[-0.710, 0.330, 0.656, 0.041, -0.990, 0.726, 0.385, -0.958]');


-- KNN style query
select
  rowid,
  distance
from vec_examples
where sample_embedding match '[0.890, 0.544, 0.825, 0.961, 0.358, 0.0196, 0.521, 0.175]'
order by distance
limit 2;
/*
┌───────┬──────────────────┐
│ rowid │     distance     │
├───────┼──────────────────┤
│ 2     │ 2.38687372207642 │
│ 1     │ 2.38978505134583 │
└───────┴──────────────────┘
*/
```

## [Permify 1.0: Open Source Authorization as a Service](https://github.com/Permify/permify)

- Permify is an open-source authorization as a service inspired by [Google Zanzibar](https://storage.googleapis.com/pub-tools-public-publication-data/pdf/41f08f03da59f5518802898f68730e247e23c331.pdf).

- **Centralize & Standardize Your Authorization**: Abstract your authorization logic from your codebase and application logic to easily reason, test, and debug your authorization. Behave your authorization as a sole entity and move faster with in your core development.

- **Build Granular Permissions For Any Case You Have**: You can create granular (resource-specific, hierarchical, context aware, etc) permissions and policies using Permify's domain specific language that is compatible with RBAC, ReBAC and ABAC.

- **Set Authorization For Your Tenants By Default**: Set up isolated authorization logic and custom permissions for your vendors/organizations (tenants) and manage them in a single place.

- **Scale Your Authorization As You Wish**: Achieve lightning-fast response times down to 10ms for access checks with a proven infrastructure inspired by Google Zanzibar.

---

- https://github.com/tinygo-org/tinygo/releases/tag/v0.33.0
- https://github.com/asg017/sqlite-vec
- https://github.com/Permify/permify/releases/tag/v1.0.0
- https://storage.googleapis.com/pub-tools-public-publication-data/pdf/41f08f03da59f5518802898f68730e247e23c331.pdf


Exploring TinyGo's latest release, a new vector search extension for SQLite, and an open-source authorization service inspired by Google Zanzibar.

#9 TinyGo, SQLite vector search, and Permify

Go Commentary #9: TinyGo, SQLite Vector Search, and Authorization


## [Draft Release Notes Go 1.24](https://tip.golang.org/doc/go1.24)

### Go 1.24 is not yet released. These are work-in-progress release notes. Go 1.24 is expected to be released in February 2025.

- Fully supports *generic type aliases*: a type alias may be parameterized like a defined type

```go
type (
	nodeList = []*Node  // nodeList and []*Node are identical types
	Polar    = polar    // Polar and polar denote identical types
)
```

  => Specs:

```go
type set[P comparable] = map[P]bool
type A[P any] = P    // illegal: P is a type parameter
```

  => the feature can be disabled by setting `GOEXPERIMENT=noaliastypeparams`; but the `aliastypeparams` setting will be removed for Go 1.25.

- Some updates for tools `go tool` or `-tool` flag, new `tests` analyzer for `go vet`

- Some updates for Cgo, Runtime, Compiler, Linker, Bootstrap

- Some updates for stdlibs:

  - The new `os.Root` type provides the ability to perform filesystem operations within a specific directory.

  - The `os.OpenRoot` function opens a directory and returns an `os.Root`. Methods on `os.Root` operate within the directory and do not permit paths that refer to locations outside the directory, including ones that follow symbolic links out of the directory.

    `os.Root.Open` opens a file for reading.

    `os.Root.Create` creates a file.

    `os.Root.OpenFile` is the generalized open call.

    `os.Root.Mkdir` creates a directory.

  - Some updates for `crypto`, `hash` package

  - Some updates for `net/http` including support for HTTP/2 protocol settings in `Transport` and `Server`

  - New implementation for `sync.Map` that will improve overall performance and resolve some long-standing issues


## [Weak Pointers in Go: Why They Matter Now](https://victoriametrics.com/blog/go-weak-pointer/)

### Definition:

A type of reference to an object that does not prevent the object from being garbage collected

=> `weak` package:

```go
type Pointer[T any] struct {
	u unsafe.Pointer
}

// Make creates a weak pointer from a strong pointer to some value of type T.
func Make[T any](ptr *T) Pointer[T] {
  //...
}
```

- If the memory they’re pointing to gets cleaned up, the weak pointer automatically becomes `nil` — so there’s no risk of accidentally pointing to freed memory => they are safe

```go
type T struct {
  a int
  b int
}

func main() {
  a := new(string)
  println("original:", a)

  // make a weak pointer
  weakA := weak.Make(a)

  runtime.GC()

  // use weakA
  strongA := weakA.Strong()
  println("strong:", strongA, a)

  runtime.GC()

  // use weakA again
  strongA = weakA.Strong()
  println("strong:", strongA)
}

// Output:
// original: 0x1400010c670
// strong: 0x1400010c670 0x1400010c670
// strong: 0x0
```

- After the first garbage collection `runtime.GC()`, the weak pointer weakA still points to the memory because we’re still using the variable a in the `println("strong:", strongA, a)` line. The memory can’t be cleaned up yet since it’s in use.
- But when the second garbage collection runs, the strong reference (a) isn’t used anymore. That means the garbage collector can safely clean up the memory, leaving `weakA.Strong()` to return `nil`.

### Use case

- Canonicalization maps, where you only want to keep one copy of a piece of data around 
- Initial purpose is in internal packages, e.g: `unique`

```go
func main() {
	h1 := unique.Make("Hello")
	h2 := unique.Make("Hello")
	w1 := unique.Make("World")

	fmt.Println("h1:", h1)
	fmt.Println("h2:", h2)
	fmt.Println("w1:", w1)
	fmt.Println("h1 == h2:", h1 == h2)
	fmt.Println("h1 == w1:", h1 == w1)
}

// Output:
// h1: {0x14000090270}
// h2: {0x14000090270}
// w1: {0x14000090280}
// h1 == h2: true
// h1 == w1: false
```

### How does it work

- Having a in-between tiny object (8bytes)

![](assets/weak-pointer-indirection-reference.webp)

- This setup lets the garbage collector clean up weak pointers to a specific object all at once, efficiently: 
  - the collector only needs to set the pointer in the indirection object to nil (or 0x0). No need to go around updating each weak pointer individually.

![](assets/weak-pointer-gc-reclaim.webp)

---

https://tip.golang.org/doc/go1.24

https://victoriametrics.com/blog/go-weak-pointer/  

Draft Release Notes for Go 1.24 and the incoming of weak pointers in Go

#23 Draft Release Notes for Go 1.24 and weak pointers in Go

Go Commentary #23: Draft Release Notes for Go 1.24 and weak pointers in Go


## [Coming in Go 1.24: testing/synctest experiment for time and concurrency testing](https://danp.net/posts/synctest-experiment/)

### Context

```go
func Test(t *testing.T) {
    before := time.Now()
    time.Sleep(time.Second)
    after := time.Now()
    if d := after.Sub(before); d != time.Second {
        t.Fatalf("took %v", d)
    }
}
```

- Traditional hack

```go
func Test(t *testing.T) {
    before := time.Now()
    time.Sleep(time.Second)
    after := time.Now()
    if d := after.Sub(before); d >= 2*time.Second {
        t.Fatalf("took %v", d)
    }
}
```

- It's still flaky because it depends on the system clock.



### Solution

- The `testing/synctest` package is an experiment to provide a more deterministic way to test time and concurrency in Go.

```go
import (
	"testing"
	"testing/synctest"
	"time"
)

func Test(t *testing.T) {
	synctest.Run(func() {
		before := time.Now()
		time.Sleep(time.Second)
		after := time.Now()
		if d := after.Sub(before); d != time.Second {
			t.Fatalf("took %v", d)
		}
	})
}
```

- And then use [gotip](https://pkg.go.dev/golang.org/dl/gotip) with `GOEXPERIMENT=synctest`

### Extending to concurrency

```go
func Test(t *testing.T) {
	ctx := context.Background()

	ctx, cancel := context.WithCancel(ctx)

	var hits atomic.Int32
	go func() {
		tick := time.NewTicker(time.Millisecond)
		defer tick.Stop()
		for {
			select {
			case <-ctx.Done():
				return
			case <-tick.C:
				hits.Add(1)
			}
		}
	}()

	time.Sleep(3 * time.Millisecond)
	cancel()

	got := int(hits.Load())
	if want := 3; got != want {
		t.Fatalf("got %v, want %v", got, want)
	}
}
```

- It's flaky because of the initial delay of the Ticker

- Wrap the test in `synctest.Run` to make it deterministic

```go
func Test(t *testing.T) {
	synctest.Run(func() {
		ctx := context.Background()

		ctx, cancel := context.WithCancel(ctx)

		var hits atomic.Int32
		go func() {
			tick := time.NewTicker(time.Millisecond)
			defer tick.Stop()
			for {
				select {
				case <-ctx.Done():
					return
				case <-tick.C:
					hits.Add(1)
				}
			}
		}()

		time.Sleep(4 * time.Millisecond)
		cancel()

		got := int(hits.Load())
		if want := 3; got != want {
			t.Fatalf("got %v, want %v", got, want)
		}
	})
}
```

### Conclusion

- It seems that `testing/synctest` will significantly improve testing code that involves time or concurrency. Example in go source: [https://go-review.googlesource.com/c/go/+/630382](https://go-review.googlesource.com/c/go/+/630382)

- You can try it yourself now by using `gotip` and setting `GOEXPERIMENT=synctest`. When Go 1.24 comes out GOEXPERIMENT=synctest will still be required.

- Review the [main proposal](https://github.com/golang/go/issues/67434) and share any experience you have.

---

https://danp.net/posts/synctest-experiment/

https://go-review.googlesource.com/c/go/+/630382

https://github.com/golang/go/issues/67434

Go 1.24 testing/synctest experiment for time and concurrency testing

#24 Go 1.24 testing/synctest experiment for time and concurrency testing

Go Commentary #24: Coming in Go 1.24: testing/synctest experiment for time and concurrency testing



## [Why People are Angry over Go 1.23 Iterators](https://www.gingerbill.org/article/2024/06/17/go-iterator-design/)

- Context:
    - Most languages provide standardized way to iterate over values stored in [containers](https://en.wikipedia.org/wiki/Container_(abstract_data_type)).
    
- Problem:

    - Go has `for` `range` for maps, slices, strings, arrays and channels but no generic mechanism for user-written containers.

    - Short list of some non-generic iterators:
        - *runtime.CallersFrames*: returns `runtime.Frames` iterates over stack frames; `Frames` has a `Next` method and a bool to check if there are more frames.

        - *bufio.Scanner*: iterates through an `io.Reader`; `Scan` method advances to the next value, `Bytes` method to return the value and `Err` to return error.

        - *database/sql.Rows*: iterates through the results of a query; also has `Scan` method.
    
    - Before Generics were introduced, no way to write an interface that described an iterator that would cover all of the use cases...

- Solution:

    - Go 1.22 proposal of adding package `iter` can range over integers and functions:

    ```go
    type (
	    Seq[V any]     func(yield func(V) bool) bool
	    Seq2[K, V any] func(yield func(K, V) bool) bool
    )
    ```
    (Seq2 represents a sequence of paired values: key-value, index-value or value-error)
    
    - An iterator is a function that passes successive elements of a sequence to a callback function `yield`, it returns true if it should continue, false if it should stop.

    ```go
    type (
        Yield[V any] func(V bool)
        Yield2[K, V any] func(K,V) bool
    )
    ```

    - Examples:

    ```go
    func Backward[E any](s []E) func(func(int, E) bool) {
        return func(yield func(int, E) bool) {
            for i := len(s)-1; i >= 0; i-- {
                if !yield(i, s[i]) {
                    // Where clean-up code goes
                    return
                }
            }
        }
    }

    s := []string{"a", "b", "c"}
    for _, el in range Backward(s) {
	    fmt.Print(el, " ")
    }
    // c b a
    ```

- Conclusion:
    - The `for range` is getting complex:
        - *return true* => *continue*
        - *return false* => *break*
    - Feels too functional than an imperative language
    - Still the purpose is good, let's wait to see the adoption in community after awhile

---

- https://www.gingerbill.org/article/2024/06/17/go-iterator-design/
- https://github.com/golang/go/issues/61897
- https://en.wikipedia.org/wiki/Container_(abstract_data_type)


Exploring the new Go package for iterators - learn what it is, what the controversy is about, and how to use it.

#2 Go 1.23 Iterators

Go Weekly #2: Go 1.23 Iterators


## [Writing generic collection types in Go: the missing documentation](https://www.dolthub.com/blog/2024-07-01-golang-generic-collections/#the-solution)

- Context:

  - Generics were released in Go 1.18 (~2y), is not used very much - only in some good cases for methods in _slices_ and _maps_ packages.

- Problem:

  - Wanted to write a sortable Set (slice or map) of any type but the documents for a generic collection are poor and immature.

  ```go
  type Sortable[T comparable] interface {
      Less(member T) bool
  }

  type Name struct {
      First string
      Last string
  }

  func (n Name) Less(member Name) bool {
      return n.First < member.First || n.First == member.First && n.Last < member.Last
  }

  var _ Sortable[Name] = Name{}
  ```

  ```go
  type SortableSet[T Sortable] interface { // cannot use generic type Sortable[T comparable] without instantiation
      Add(member T)
      Size() int
      Contains(member T) bool
      Sorted() []T
  }
  ```

  - Can try what built-in slices does in slices.go

  ```go
  func Index[S ~[]E, E comparable](s S, v E) int {
      for i := range s {
          if v == s[i] {
              return i
          }
      }
      return -1
  }
  ```

  ```go
  type SortableSet[T Sortable[E], E comparable] interface {
      Add(member T)
      Size() int
      Contains(member T) bool
      Sorted() []T
  }
  ```

  - But not work for map:

  ```go
  type MapSet[T Sortable[E], E comparable] struct {
      members map[T]struct{} // invalid map key type T (missing comparable constraint)
  }
  ```

- Solution:

  ```go
  // Type set used only for constraints, not vars
  type SortableConstraint[T comparable] interface {
      comparable
      Sortable[T]
  }

  type SortableSet[T SortableConstraint[T]] interface {
      Add(member T)
      Size() int
      Contains(member T) bool
      Sorted() []T
  }

  type SliceSet[T SortableConstraint[T]] struct {
      members []T
  }

  type MapSet[T SortableConstraint[T]] struct {
      members map[T]struct{}
  }
  ```

- Conclusion:

  - Current Google rearch results are not really helpful since the immaturity

  - The official Go documentations (proposal/ language spec) are too long

  - This article is very handy as considerably pioneering in real life implementation at [@Dolt](https://github.com/dolthub/dolt)

  - Cheat sheet found in spec by the author:

  ```
  type argument      type constraint                // constraint satisfaction

  int                interface{ ~int }              // satisfied: int implements interface{ ~int }
  string             comparable                     // satisfied: string implements comparable (string is strictly comparable)
  []byte             comparable                     // not satisfied: slices are not comparable
  any                interface{ comparable; int }   // not satisfied: any does not implement interface{ int }
  any                comparable                     // satisfied: any is comparable and implements the basic interface any
  struct{f any}      comparable                     // satisfied: struct{f any} is comparable and implements the basic interface any
  any                interface{ comparable; m() }   // not satisfied: any does not implement the basic interface interface{ m() }
  interface{ m() }   interface{ comparable; m() }   // satisfied: interface{ m() } is comparable and implements the basic interface interface{ m() }

  ```

## [Russ Cox is working on a bot](https://github.com/golang/go/discussions/67901)

- Context:

  - As there is [@gopherbot](https://github.com/gopherbot) to try to help automate: labeling, commenting, changing issues' status...

  - Gaby (Go AI Bot) is being built (experiment) for what LLMs can be used effectively (including identifying what they should not be used for).

  - Source is not official nor open, can help snoop around in [this search](https://github.com/golang/go/issues?q=is%3Aissue+is%3Aopen+commenter%3Agabyhelp) and help send feedback.

---

- https://www.dolthub.com/blog/2024-07-01-golang-generic-collections/#the-solution
- https://go.dev/ref/spec
- https://github.com/golang/go/discussions/67901
- https://pkg.go.dev/rsc.io/gaby


Implementing generic collections in Go: challenges and solutions, with examples of sortable sets and constraints. Updates on Go's generics support and insights on a new AI bot being developed for the Go community. Stay current with Go's evolving ecosystem and best practices for using generics effectively.

#3 Generic Collections, Generics Constraints, AI Bot

Go Commentary #3: Generic Collections, Generics Constraints, AI Bot


## [Using Go to send 500 Million HTTP Requests to 2.5 Million Hosts](https://www.moczadlo.com/2024/how-i-sent-500-million-http-requests-in-under-24h)

- Context:

  - (Ethical hacking) To send 500 million of non RFC HTTP/1.1 requests to 2.5 million hosts in approx. a couple of hours.

  - Chose Go because of great concurrency support and fast. (The author did try Rust but too hard to understand)

- Problems:

  - If you use `curl` to send the requests (0.5s per request) 1 by 1 from single machine => 7.9 years.

  - Data transfer perspective (not that much):

    - 500 million requests \* 1 KB (average request size) ≈ 478 GB
    - 500 million responses \* 5 KB (average response size) ≈ 2.33 TB

  - Break down what HTTP client does in each call:

  ```go
  resp, err := http.Get("https://example.com")
  ```

        1. Resolve DNS
        2. TCP connect to another machine
        3. TLS handshake (generate and exchange cryptographic keys)
        4. Prepare HTTP request to send (normalize, encode)
        5. Wait for response and read it
        6. Parse response (decode, normalize and parse)
        7. Close connection (optional)

  - Measured results of sending HTTP/1.1 requests to subdomains of `*.wordpress.com` (wordpress.com has [wildcard dns](https://en.wikipedia.org/wiki/Wildcard_DNS_record) to prevent caching) shows that:

  ![](assets/http-request-timings.webp)

  - Resolving DNS and opening new TLS connection is slow (~160ms) but probably not notice how slow it is because the browser will open connect once and reuse it for many requests and HTTP/2 or HTTP/3 would be faster.

  - Can't rely on reusing connections since the need of sending many requests to many different hosts in different networks.

  - Would get rate-limited or banned quickly.

- Solution: Spread the load on many servers

  - Remove what we can from the steps: `Request parsing` and `DNS resolution`. (Simple crafted HTTP/1.1 requests by hand and [massdns](https://github.com/blechschmidt/massdns) to resolve thousands of DNS records in a couple of seconds)

  - Design the HTTP/1.1 sending mechanism:

    - Multiple worker pools that piped together:

      1. Request generation pool
      2. Sender pool
      3. Response pool

    ![](assets/cannon-diagram.webp)

    - Use a concurrency-safe [queue](https://github.com/enriquebris/goconcurrentqueue) to seperate the worker pools and to reuse the objects and memory as much as possible.

  - Choose `fasthttp` over `net/http` to get x10 faster according to the [benmark](https://github.com/valyala/fasthttp?tab=readme-ov-file#http-client-comparison-with-nethttp)

  - Optimize the fasthttp's client to get rid of normalization step (fork and edit the lib)

    ```go
    req := rawfasthttp.AcquireRequest()
    resp := rawfasthttp.AcquireResponse()

    rawBytes := []byte("GET / HTTP/1.1\r\nHost: example.com\r\n\r\n")

    req.SetRequestRaw(rawBytes)

    err := client.Do(req, resp)
    ```

  - Override the `Dial` function to use the resolved IP addresses:

    ```go
    // single instance of custom dialer
    customDialer = &rawfasthttp.TCPDialer{}

    req := rawfasthttp.AcquireRequest()
    resp := rawfasthttp.AcquireResponse()

    resolved_ip := "127.0.0.1"

    req.SetDial(func(addr string) (net.Conn, error) {
        return customDialer.Dial(resolved_ip)
    })

    // ...
    ```

  - Optimize TLS handshake to prevent wasting CPU cycles by hardcoding the keys but had not enough time to change the fork so skipped.

  - Split 2.5 million hosts into chunks with 200 hosts per chunk (not take more than a couple of minutes to complete but big enough to not waste time on creating new connections). (200 is optimal for the usecase, the worker pods were prone to fail => minimize the lost requests/ retries)

  - Chose DigitalOcean with K8s to scale (cheapest one - 2TB+ of bandwidth per droplet). Also DigitalOcean gives new public IP for each droplet to avoid being banned from Cloudfare.

  - Wrote a auto-scroller to scale the deployment up and down based on the targets in queue (0 to 60 pods in a couple of minutes)

- Conclusion:

  - The final results:

    - Each pod achieved 100-400 requests per second
    - Scaled to 60 pods
    - Sent 500 million HTTP/1.1 requests to 2.5 million hosts in just a couple of hours

  - Although there will be a next post for a further detailed result,

## [How I build simple Mac apps using Go](https://dev.to/progrium/how-i-build-simple-mac-apps-using-go-104j)

- Context:

  - There were no bindinds to native Mac APIs for Go => [DarwinKit](https://github.com/progrium/darwinkit)

    - Bindings for [33 frameworks](https://pkg.go.dev/github.com/progrium/darwinkit/macos@main#section-directories) with near complete coverage:
      - 2,353 classes
      - 23,822 methods and properties
      - 9,519 constants/enums
      - 543 structs
    - Automatic conversion and use of native Go builtin types in APIs
    - Support for block arguments as Go functions with properly typed arguments
    - Pre-made delegate implementations you can simply set Go functions on
    - 1-to-1 mapping to Objective-C symbols while still idiomatic to Go
    - Documentation for all symbols including a link to official Apple docs on that symbol
    - Growing collection of high-quality example starter apps for sponsors

- Future:

  - There are no bindings to Apple framework functions. The team are working on generating native Go function bindings for every framework function. Meanwhile, there is a workaround that involves using CGO (which DarwinKit is trying to help you avoid).

  - On the way of making DarwinKit not use CGO at all! Using [purego](https://github.com/ebitengine/purego), we can call into Apple frameworks without involving CGO. This will improve build time, make smaller binaries, and allow DarwinKit to be used in programs that need to avoid CGO for whatever reason.

  - For iOS and mobile devs, generated bindings are for MacOS for now. Any contribution is welcome.

---

- https://www.moczadlo.com/2024/how-i-sent-500-million-http-requests-in-under-24h
- https://dev.to/progrium/how-i-build-simple-mac-apps-using-go-104j
- https://github.com/progrium/darwinkit


This post explores two cutting-edge applications of Go programming. First, it details an ethical hacking project that successfully sent 500 million HTTP requests to 2.5 million hosts using Go's concurrency features and custom optimizations. The article then introduces DarwinKit, a powerful Go library for creating native Mac applications without Objective-C or Swift. Both examples demonstrate Go's versatility in handling high-performance networking tasks and cross-platform development, showcasing its potential for complex, scalable projects in cybersecurity and application development.

#4 Ethical Hacking, HTTP Requests, Mac App Development

Go Commentary #4: Ethical Hacking, HTTP Requests, Mac App Development


## [An Applied Introduction to eBPF with Go](https://sazak.io/articles/an-applied-introduction-to-ebpf-with-go-2024-06-06)

- Context:

  - We usually write software in user space (outside the OS's kernel, e.g: user apps like utilities, programming languages, GUI...).

  ![](assets/user-space-vs-kernel-space-basic-system-calls.png)

- Problem:

  - When profiling the application: if we do it from user space (e.g: [pprof](https://go.dev/blog/pprof)) => result is not reliable because there will be some overhead of each layer on top of the CPU/ memory.

  - 2 initial options:
    - _Edit the Kernel Source Code_: not really practical since the usecase is trivial amount. (pending for years to be adopted by distros)
    - _Write a Kernel Model_: this is more practical however, regular maintenance is inevitable as keeping up with the new kernel versions + risking corrupting the Kernel (e.g: if the module has a bug => crash the whole system)

- Solution:

  - **BPF** was originally used in Linux to filter network packets.
  - **eBPF (Extended Berkeley Packet Filter)** allows to trace syscalls, user space/ library functions, network packets... => system performance, monitoring, security...

  ![](assets/ebpf_overview.webp)

  - _How it works:_
    - Pre-defined hooks: system calls, function entry/exit, kernel tracepoints, network events...
    - eBPF programs are event-driven, run at certain hook point then:
      - Safe checked by Verifier
      - Then compiled by JIT compiler from bytecode to instructions
    - Execute the desired code right before actual system calls

  ![](assets/ebpf.png)

- Conclusion:
  - Powerful tool to dive deep in Kernel therefore many applicable usecase: systems programming, observability, security...
  - For profiling usecase, can use existing projects: [Parca](https://www.parca.dev/docs/overview/), [Pyroscope](https://pyroscope.io/) or [PGO](https://go.dev/doc/pgo) (from Go 1.20) for convenience.

## [The Profile-Guided Optimization Experience at Grab](https://engineering.grab.com/profile-guided-optimisation)

- Context:

  - PGO (Profile-Guided Optimization) is introduced in Go version 1.20, a.k.a FDO (feedback-directed optimization), a technique collects and feeds the profile data back to the next compilier build.
  - From the 2nd build/release, expectedly improving 2-14% performance (on-going for future builds)

  ![](assets/high-level-pgo.png)

- Problem:

  - Grab wanted to experiment this to some of their services: use self-managed database [TalariaDB](https://github.com/grab/talaria), orchestrated service and a monorepo one.

- Results:

  - In a service cluster's image that uses TalariaDB, add `-PGO=./talaria.PGO` to the `go build` command: 10% CPU usage reduction, 30% memory usage reduction and 38% volume usage reduction.
  - On the orchestrated service: the reduction is only around 5% << the effort the enable PGO => not substaintial
  - Monorepo service is currently not supported since needing a seperated pprof service and a build process supporting PGO arguments to attach/retrieve pprof file

- Conclusion:
  - Applicable on simple, yet low-effort deployed services

---

- https://sazak.io/articles/an-applied-introduction-to-ebpf-with-go-2024-06-06
- https://www.redhat.com/en/blog/architecting-containers-part-1-why-understanding-user-space-vs-kernel-space-matters
- https://ebpf.io/what-is-ebpf/
- https://www.parca.dev/docs/overview/
- https://pyroscope.io/

- https://engineering.grab.com/profile-guided-optimisation
- https://go.dev/doc/pgo


Explore advanced Go optimization techniques using eBPF for kernel-level insights and Profile-Guided Optimization (PGO) for compiler enhancements. Learn how to boost performance and efficiency in Go applications.

#1 eBPF and PGO Optimization Techniques

Go Weekly #1: Mastering Go Performance - eBPF and PGO Optimization Techniques


## [Fuzz Testing Go HTTP Services](https://packagemain.tech/p/fuzzing-http-services-golang)

Context:

- You can't envision all of the possible inputs your code could receive => You can only find bugs that you expect to find

Solution:

- Since Go 1.18, fuzzing was added to Go's std testing package

```
The steps to create a fuzz test in Go are the following:

In a _test.go file create a function that starts with Fuzz which accepts *testing.F

Add corpus seeds using f.Add() to allow fuzzer to generate the data based on it.

Call fuzz target using f.Fuzz() by passing fuzzing arguments which our target function accepts.

Start the fuzzer using regular go test command, but with the –fuzz=Fuzz flag
```

- Example:

```go
func Equal(a []byte, b []byte) bool {
  for i := range a {
    // can panic with runtime error: index out of range.
    if a[i] != b[i] {
      return false
    }
  }

  return true
}
```

```go
// Fuzz test
func FuzzEqual(f *testing.F) {

  // Seed corpus addition
  f.Add([]byte{'f', 'u', 'z', 'z'}, []byte{'t', 'e', 's', 't'})

  // Fuzz target with fuzzing arguments
  f.Fuzz(func(t *testing.T, a []byte, b []byte) {
    // Call our target function and pass fuzzing arguments
    Equal(a, b)
  })
}
```

- Fuzzing HTTP Services

```go
type Request struct {
  Limit  int `json:"limit"`
  Offset int `json:"offset"`
}

type Response struct {
  Results    []int `json:"items"`
  PagesCount int   `json:"pagesCount"`
}
```

```go
func ProcessRequest(w http.ResponseWriter, r *http.Request) {
  var req Request

  // Decode JSON request
  if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
    http.Error(w, err.Error(), http.StatusBadRequest)
    return
  }

  // Apply offset and limit to some static data
  all := make([]int, 1000)
  start := req.Offset
  end := req.Offset + req.Limit
  res := Response{
    Results:    all[start:end],
    PagesCount: len(all) / req.Limit,
  }

  // Send JSON response
  if err := json.NewEncoder(w).Encode(res); err != nil {
    http.Error(w, err.Error(), http.StatusInternalServerError)
    return
  }

  w.WriteHeader(http.StatusOK)
}
```

```go
func FuzzProcessRequest(f *testing.F) {
  // Create sample inputs for the fuzzer
  testRequests := []Request{
    {Limit: -10, Offset: -10},
    {Limit: 0, Offset: 0},
    {Limit: 100, Offset: 100},
    {Limit: 200, Offset: 200},
  }

  // Add to the seed corpus
  for _, r := range testRequests {
    if data, err := json.Marshal(r); err == nil {
      f.Add(data)
    }
  }

  // Create a test server
  srv := httptest.NewServer(http.HandlerFunc(ProcessRequest))
  defer srv.Close()

  // Fuzz target with a single []byte argument
  f.Fuzz(func(t *testing.T, data []byte) {
    var req Request
    if err := json.Unmarshal(data, &req); err != nil {
      // Skip invalid JSON requests that may be generated during fuzz
      t.Skip("invalid json")
    }

    // Pass data to the server
    resp, err := http.DefaultClient.Post(srv.URL, "application/json", bytes.NewBuffer(data))
    if err != nil {
      t.Fatalf("unable to call server: %v, data: %s", err, string(data))
    }

    defer resp.Body.Close()

    // Skip BadRequest errors
    if resp.StatusCode == http.StatusBadRequest {
      t.Skip("invalid json")
    }

    // Check status code
    if resp.StatusCode != http.StatusOK {
      t.Fatalf("non-200 status code %d", resp.StatusCode)
    }
  })
}
```

```
go test --fuzz=Fuzz -fuzztime=30s
--- FAIL: FuzzProcessRequest (0.02s)
    --- FAIL: FuzzProcessRequest (0.00s)
        runtime error: integer divide by zero
        runtime error: slice bounds out of range
```

Conclusion:

- Can detect hard-to-spot bugs with weird unexpected inputs


---

https://packagemain.tech/p/fuzzing-http-services-golang




Understanding how to use Fuzz Testing in Go

#18 Fuzz Testing Go HTTP Services

Go Commentary #18: Fuzz Testing Go HTTP Services


## [Writing secure Go code](https://jarosz.dev/article/writing-secure-go-code/)

**Context**: 

  What does it mean to keep security in mind when writing Go code?

**Solution**:

  Answer these questions:

    - How do we stay informed about the Go security announcements?
    - How do we keep our Go code patched and up to date?
    - How do we test our Go code focusing on security and robustness?
    - What are CVEs, and where do we learn about the most common software vulnerabilities?

  How do we stay informed about the Go security announcements?

   - Subscribe to `golang-announce@googlegroups.com` to get all critical security information right from the source.

  How do we keep our Go code patched and up to date?

   - Keeping Go version up to date: even though we don’t use the latest and greatest language features, bumping the Go version gives us all security patches for discovered vulnerabilities. Also, the new Go version ensures compatibility with newer dependencies. It protects our applications from potential integration issues.

   - Check accordingly which security issues and CVEs addressed in what Go releases and update `go.mod`.

   - Check for compatibility and dependency problems.

  How do we test our Go code focusing on security and robustness? 

   - Use Go tooling for static code analysers:
  
     - Old school `go vet` to detect syntax errors, unused variables, unreachable areas of codebase, goroutine mistakes...

     - `staticcheck`

        ```
        go install honnef.co/go/tools/cmd/staticcheck@latest
        ```

        e.g: test on NGIX Agent cloned repo

        ```
        ➜  agent git:(main) ✗ staticcheck ./...
        ```

        to detect packages, methods or functions are deprecated:

        ```bash...
        src/core/metrics/sources/cpu.go:111:9: times.Total is deprecated: Total returns the total number of seconds in a CPUTimesStat Please do not use this internal function. (SA1019)
        ...
        test/component/nginx-app-protect/monitoring/monitoring_test.go:15:8: "github.com/golang/protobuf/jsonpb" is deprecated: Use the "google.golang.org/protobuf/encoding/protojson" package instead. (SA1019)
        ```
        
        to detect unused variables and fields:

        ```bash
        src/core/metrics/sources/nginx_plus.go:74:2: field endpoints is unused (U1000)
        src/core/metrics/sources/nginx_plus.go:75:2: field streamEndpoints is unused (U1000)
        src/core/metrics/sources/nginx_plus_test.go:94:2: var availableZones is unused (U1000)
        ```
        
        to detect code quality problems:

        ```bash
        src/core/nginx.go:791:4: ineffective break statement. Did you mean to break out of the outer loop? (SA4011)
        ```

      - `golangci-lint`

        ```
        go install github.com/golangci/golangci-lint/cmd/golangci-lint@latest
        ```

        e.g: test on NGIX Agent cloned repo

        ```
        ➜  agent git:(main) ✗ golangci-lint run ./...
        ```

        suggestions for improving the code:

        ```
        src/extensions/nginx-app-protect/monitoring/processor/nap_test.go:60:14: S1025: the argument is already a string, there's no need to use fmt. Sprintf (gosimple)
        logEntry: fmt.Sprintf(`%s`, func() string {
        ^
        ```

        ```
        src/plugins/common.go:85:5: S1009: should omit nil check; len() for []string is defined as zero (gosimple)
        if loadedConfig.Extensions != nil && len(loadedConfig.Extensions) > 0 {
            ^
        ```

      - Detect race conditions

        ```
        go test -race
        ```

   - Scanning source code for vulnerabilities

     - `govulncheck`

        ```
        go install golang.org/x/vuln/cmd/govulncheck@latest
        ```

        ```
          ➜  habit git:(main) ✗ govulncheck
          No vulnerabilities found.
        ```

        ```
        ➜  habit git:(main) ✗ govulncheck -mode binary -show verbose habit
        ```

        ```
        Scanning your binary for known vulnerabilities...

        Fetching vulnerabilities from the database...

        Checking the binary against the vulnerabilities...

        === Symbol Results ===

        No vulnerabilities found.

        === Package Results ===

        Vulnerability #1: GO-2023-2186
            Incorrect detection of reserved device names on Windows in path/filepath
          More info: https://pkg.go.dev/vuln/GO-2023-2186
          Standard library
            Found in: path/filepath@go1.20.5
            Fixed in: path/filepath@go1.20.11

        === Module Results ===

        Vulnerability #1: GO-2024-3107
            Stack exhaustion in Parse in go/build/constraint
          More info: https://pkg.go.dev/vuln/GO-2024-3107
          Standard library
            Found in: stdlib@go1.20.5
            Fixed in: stdlib@go1.22.7
        ...

        Vulnerability #18: GO-2023-1878
            Insufficient sanitisation of Host header in net/http
          More info: https://pkg.go.dev/vuln/GO-2023-1878
          Standard library
            Found in: stdlib@go1.20.5
            Fixed in: stdlib@go1.20.6

        Your code is affected by 0 vulnerabilities.
        This scan also found 1 vulnerability in packages you import and 18
        vulnerabilities in modules you require, but your code doesn't appear to call
        these vulnerabilities.
        ```

     - `gosec`
      
       ```
       go install github.com/securego/gosec/v2/cmd/gosec@latest
       ```

       e.g: test on [brutus](https://github.com/CyberRoute/bruter) repo - an open-source experimental [OSINT](https://en.wikipedia.org/wiki/Open-source_intelligence) app for testing web server configuration.

       ```
       gosec ./...
       ```

       spotted [CWE-295](https://cwe.mitre.org/data/definitions/295.html) 

       ```
       ...

       [/.../bruter/pkg/fuzzer/randomua.go:69] - G404 (CWE-338): Use of weak random number generator (math/rand or math/rand/v2 instead of crypto/rand) (Confidence: MEDIUM, Severity: HIGH)
           68:
         > 69:  randomIndex := rand.Intn(len(userAgents))
           70:  return userAgents[randomIndex]

       ...

       [/.../bruter/pkg/server/config.go:40] - G402 (CWE-295): TLS InsecureSkipVerify set true. (Confidence: HIGH, Severity: HIGH)
           39:  customTransport := &http.Transport{
         > 40:   TLSClientConfig: &tls.Config{InsecureSkipVerify: true},
           41:  }

       ...
       ```

   - Fuzzing

     - Extremely helpful in finding potential security flaws like buffer overflows, SQL injections, DoS attacks and XSS attacks

     - Further read on [Fuzzing test HTTP services Golang](./nov-01.md)

---

https://jarosz.dev/article/writing-secure-go-code/

https://github.com/CyberRoute/bruter

https://en.wikipedia.org/wiki/Open-source_intelligence

https://cwe.mitre.org/data/definitions/295.html

#19 Writing secure Go code

Go Commentary #19: Writing secure Go code


## [Go Turns 15](https://go.dev/blog/15years)

- So much has changed since [Go's 10 year anniversery](https://go.dev/blog/10years)

  - Go’s user base x3 in the past five years, making it one of the fastest growing languages. 
  - From its beginnings just fifteen years ago, Go has become a top 10 language and the language of the modern cloud.

- It's a year of for loop:

  - [Go 1.22 in February](https://go.dev/blog/go1.22)

    - Fixed the accidental sharing of loop variables between iterations:

    ```go
    func main() {
      done := make(chan bool)

      values := []string{"a", "b", "c"}
      for _, v := range values {
          go func() {
              fmt.Println(v)
              done <- true
          }()
      }

      // wait for all goroutines to complete before exiting
      for _ = range values {
          <-done
      }
    }
    ```

    - Support for ranging over integer

    ```go
    package main

    import "fmt"

    func main() {
        for i := range 10 {
            fmt.Println(10 - i)
        }
        fmt.Println("go1.22 has lift-off!")
    }
    ```

    - Improved performance

      - Memory optimization in the Go runtime improves CPU performance by 1-3%, while also reducing the memory overhead of most Go programs by around 1%.

      - In Go 1.21, we shipped profile-guided optimization (PGO) for the Go compiler and this functionality continues to improve. One of the optimizations added in 1.22 is improved devirtualization, allowing static dispatch of more interface method calls. Most programs will see improvements between 2-14% with PGO enabled.

    - Standard library additions

      - A new **math/rand/v2** package provides a cleaner, more consistent API and uses higher-quality, faster pseudo-random generation algorithms. See the proposal for additional details.

      - The patterns used by **net/http.ServeMux** now accept methods and wildcards.

        For example, the router accepts a pattern like *GET /task/{id}/*, which matches only GET requests and captures the value of the {id} segment in a map that can be accessed through Request values.

      - A new *Null[T]* type in **database/sql** provides a way to scan nullable columns.

      - A Concat function was added in package **slices**, to concatenate multiple slices of any type.

  - [Go 1.23 in August](https://go.dev/blog/go1.23)

    - Range expressions in a *“for-range”* loop may now be iterator functions, such as ```func(func(K) bool)```. This supports user-defined iterators over arbitrary sequences. There are several additions to the standard slices and maps packages that work with iterators, as well as a new iter package. As an example, if you wish to collect the keys of a map m into a slice and then sort its values, you can do that in Go 1.23 with slices.Sorted(maps.Keys(m)).

    - Preview support for generic type aliases.

    - Tool improvements
      
      - Starting with Go 1.23, it’s possible for the Go toolchain to collect usage and breakage statistics to help understand how the Go toolchain is used, and how well it is working. This is *Go telemetry*, an opt-in system. Please consider opting in to help us keep Go working well and better understand Go usage. 

      - The go command has new conveniences. For example, running ```go env -changed``` makes it easier to see only those settings whose effective value differs from the default value, and ```go mod tidy -diff``` helps determine the necessary changes to the *go.mod* and *go.sum* files without modifying them. Read more on the Go command in the release notes.
      
      - The ```go vet``` subcommand now reports symbols that are too new for the intended Go version. 

- For next 15 years:

  - Evolving Go to better leverage the capabilities of current and future hardware

  - Go 1.24 will have a totally new map implementation under the hood that’s more efficient on modern CPUs. 
  
  - Prototyping new garbage collection algorithms designed around the capabilities and constraints of modern hardware. 

  - Some improvements will be in the form of new APIs and tools so Go developers can better leverage modern hardware.

  - Working on making Go better for AI—and AI better for Go (LangChainGo & Genkit)
  


---

https://go.dev/blog/15years

https://go.dev/blog/10years

https://go.dev/blog/go1.22

https://go.dev/blog/go1.23

The 15th anniversary of the Go open source release

#20 Go Turns 15

Go Commentary #20: Go Turns 15


## [Go sync.Once is simple... Is it really?](https://victoriametrics.com/blog/go-sync-once/)

### What is sync.Once?

- "Once is an object that will perform exactly one action"

```go
var once sync.Once
var conf Config

func GetConfig() Config {
    once.Do(func() {
        conf = fetchConfig()
    })
    return conf
}
```

- If `GetConfig()` is called multiple times, `fetchConfig()` is executed only once.

```go
type Singleton struct {
    // fields
}

var (
    instance *Singleton
    once     sync.Once
)

func GetSingleton() *Singleton {
    once.Do(func() {
        instance = &Singleton{}
    })
    return instance
}
```

- The benefit: it delays certain operations until they are first needed (lazy-loading), which can improve runtime performance and reduce initial memory usage.

```go
var once sync.Once

func main() {
    once.Do(func() {
        fmt.Println("This will be printed once")
    })

    once.Do(func() {
        fmt.Println("This will not be printed")
    })
}

// Output:
// This will be printed once
```

- No built-in way to reset a sync.Once, so if the function passed to Once.Do panics while running, the future calls to `Do(f)` won't run f again and will be tricky to catch the panic and handle the error afterward

```go
var once sync.Once
var config Config

func GetConfig() (Config, error) {
    var err error
    once.Do(func() {
        config, err = fetchConfig()
    })
    return config, err
}
```

- From Go 1.21, we get: `OnceFunc`, `OnceValue` and `OnceValues`

```go
// If f panics, the returned function will panic with the same value on every call. (cached)
func OnceFunc(f func()) func() { 
  ...
}

// returns the value returned by f
func OnceValue[T any](f func() T) func() T {
  ...
}

// returns the values returned by f
func OnceValues[T1, T2 any](f func() (T1, T2)) func() (T1, T2) {
  ...
}
```

- Example use:

```go
var config Config

var getConfigOnce = sync.OnceValues(fetchConfig)

func main() {
  var err error

  config, err = getConfigOnce()
  if err != nil {
    log.Fatalf("Failed to fetch config: %v", err)
  }
  ...
}
```

### How it works?

- The implementation of Once

```go
type Once struct {
	done atomic.Uint32
	m    Mutex
}
```

- Simply use mutex to lock and allows only 1 goroutine to enter; then if `done` is still 0 (function f hasn't run yet) set done to 1 and runs f()

=> Original version of sync.Once, written by Rob Pike in 2010
```go
func (o *Once) Do(f func()) {
	o.m.Lock()
	defer o.m.Unlock()

	if o.done.Load() == 0 {
		o.done.Store(1)
		f()
	}
}
```

- Not performant since it always locks first (goroutines wait on each other) whenever Do(f) is called.

=> check the flag done first before the lock

```go
func (o *Once) Do(f func()) {
  if atomic.LoadUint32(&o.done) == 1 {
    return
  }

  // slow path
  o.m.Lock()
  defer o.m.Unlock()

  if o.done.Load() == 0 {
    o.done.Store(1)
    f()
  }
}
```

- This introduces the race condition

![](assets/go-sync-once-done-mistake.webp)

=> add defer to when setting flag

```go
func (o *Once) Do(f func()) {
  if o.done.Load() == 1 {
    return
  }

  // slow path
  o.m.Lock()
  defer o.m.Unlock()

  if o.done.Load() == 0 {
    defer o.done.Store(1)
    f()
  }
}
```

- Since Go compiler supports inlining (taking the function's code and paste it directly to where the function is called) optimization

```go
func (o *Once) Do(f func()) {
	if o.done.Load() == 0 {
		o.doSlow(f)
	}
}

func (o *Once) doSlow(f func()) {
	o.m.Lock()
	defer o.m.Unlock()

	if o.done.Load() == 0 {
		defer o.done.Store(1)
		f()
	}
}
```

---

https://victoriametrics.com/blog/go-sync-once/

https://go.dev/ref/mem

Understanding Go's sync.Once - A Deep Dive into Single-Execution Guarantees and Atomic Operations

#21 Go sync.Once is Simple

Go Commentary #21: Go sync.Once is Simple


## [GoMLX: ML in Go without Python](https://eli.thegreenplace.net/2024/gomlx-ml-in-go-without-python)

### How ML models are implemented

- Written in Python, using frameworks like TensorFlow, JAX or Pytorch that take care of:

  - Expressive way to describe the model architecture, including auto-differentiation for training.
  - Efficient implementation of computational primitives on common HW: CPUs, GPUs and TPUs.

![](assets/openxla-diagram-with-gopher.png)

- The frameworks that provide high-level primitives to define and translate ML models to a common interchange format called StableHLO (High-Level Operations).

- The OpenXLA system, which includes two major components: the XLA compiler translating HLO to HW machine code, and PJRT - the runtime component responsible for managing HW devices, moving data (tensors) between the host CPU and these devices, executing tasks, sharding and so on.

- HW that executes these models efficiently. (C/C++ hidden complexity)

### GoMLX

- Wraps XLA - access to all building blocks TF and JAX use

### Examples

- a CNN (convolutional neural network) without any Python, training it on [CIFAR-10](https://www.cs.toronto.edu/~kriz/cifar.html)

as expected, Go code is longer and more explicit
```go
// define the model graph
func C10ConvModel(mlxctx *mlxcontext.Context, spec any, inputs []*graph.Node) []*graph.Node {
  batchedImages := inputs[0]
  g := batchedImages.Graph()
  dtype := batchedImages.DType()
  batchSize := batchedImages.Shape().Dimensions[0]
  logits := batchedImages

  layerIdx := 0
  nextCtx := func(name string) *mlxcontext.Context {
    newCtx := mlxctx.Inf("%03d_%s", layerIdx, name)
    layerIdx++
    return newCtx
  }

  // Convolution / activation layers
  logits = layers.Convolution(nextCtx("conv"), logits).Filters(32).KernelSize(3).PadSame().Done()
  logits.AssertDims(batchSize, 32, 32, 32)
  logits = activations.Relu(logits)
  logits = layers.Convolution(nextCtx("conv"), logits).Filters(32).KernelSize(3).PadSame().Done()
  logits = activations.Relu(logits)
  logits = graph.MaxPool(logits).Window(2).Done()
  logits = layers.DropoutNormalize(nextCtx("dropout"), logits, graph.Scalar(g, dtype, 0.3), true)
  logits.AssertDims(batchSize, 16, 16, 32)

  logits = layers.Convolution(nextCtx("conv"), logits).Filters(64).KernelSize(3).PadSame().Done()
  logits.AssertDims(batchSize, 16, 16, 64)
  logits = activations.Relu(logits)
  logits = layers.Convolution(nextCtx("conv"), logits).Filters(64).KernelSize(3).PadSame().Done()
  logits.AssertDims(batchSize, 16, 16, 64)
  logits = activations.Relu(logits)
  logits = graph.MaxPool(logits).Window(2).Done()
  logits = layers.DropoutNormalize(nextCtx("dropout"), logits, graph.Scalar(g, dtype, 0.5), true)
  logits.AssertDims(batchSize, 8, 8, 64)

  logits = layers.Convolution(nextCtx("conv"), logits).Filters(128).KernelSize(3).PadSame().Done()
  logits.AssertDims(batchSize, 8, 8, 128)
  logits = activations.Relu(logits)
  logits = layers.Convolution(nextCtx("conv"), logits).Filters(128).KernelSize(3).PadSame().Done()
  logits.AssertDims(batchSize, 8, 8, 128)
  logits = activations.Relu(logits)
  logits = graph.MaxPool(logits).Window(2).Done()
  logits = layers.DropoutNormalize(nextCtx("dropout"), logits, graph.Scalar(g, dtype, 0.5), true)
  logits.AssertDims(batchSize, 4, 4, 128)

  // Flatten logits, and apply dense layer
  logits = graph.Reshape(logits, batchSize, -1)
  logits = layers.Dense(nextCtx("dense"), logits, true, 128)
  logits = activations.Relu(logits)
  logits = layers.DropoutNormalize(nextCtx("dropout"), logits, graph.Scalar(g, dtype, 0.5), true)
  numClasses := 10
  logits = layers.Dense(nextCtx("dense"), logits, true, numClasses)
  return []*graph.Node{logits}
}
```

```go
// the classifier
func main() {
  flagCheckpoint := flag.String("checkpoint", "", "Directory to load checkpoint from")
  flag.Parse()

  mlxctx := mlxcontext.New()
  backend := backends.New()

  _, err := checkpoints.Load(mlxctx).Dir(*flagCheckpoint).Done()
  if err != nil {
    panic(err)
  }
  mlxctx = mlxctx.Reuse() // helps sanity check the loaded context
  exec := mlxcontext.NewExec(backend, mlxctx.In("model"), func(mlxctx *mlxcontext.Context, image *graph.Node) *graph.Node {
    // Convert our image to a tensor with batch dimension of size 1, and pass
    // it to the C10ConvModel graph.
    image = graph.ExpandAxes(image, 0) // Create a batch dimension of size 1.
    logits := cnnmodel.C10ConvModel(mlxctx, nil, []*graph.Node{image})[0]
    // Take the class with highest logit value, then remove the batch dimension.
    choice := graph.ArgMax(logits, -1, dtypes.Int32)
    return graph.Reshape(choice)
  })

  // classify takes a 32x32 image and returns a Cifar-10 classification according
  // to the models. Use C10Labels to convert the returned class to a string
  // name. The returned class is from 0 to 9.
  classify := func(img image.Image) int32 {
    input := images.ToTensor(dtypes.Float32).Single(img)
    outputs := exec.Call(input)
    classID := tensors.ToScalar[int32](outputs[0])
    return classID
  }

  // ...
}
```

- [a Gemma2 from Kaggle](https://www.kaggle.com/models/google/gemma-2) example

```go
var (
  flagDataDir   = flag.String("data", "", "dir with converted weights")
  flagVocabFile = flag.String("vocab", "", "tokenizer vocabulary file")
)

func main() {
  flag.Parse()
  ctx := context.New()

  // Load model weights from the checkpoint downloaded from Kaggle.
  err := kaggle.ReadConvertedWeights(ctx, *flagDataDir)
  if err != nil {
    log.Fatal(err)
  }

  // Load tokenizer vocabulary.
  vocab, err := sentencepiece.NewFromPath(*flagVocabFile)
  if err != nil {
    log.Fatal(err)
  }

  // Create a Gemma sampler and start sampling tokens.
  sampler, err := samplers.New(backends.New(), ctx, vocab, 256)
  if err != nil {
    log.Fatalf("%+v", err)
  }

  start := time.Now()
  output, err := sampler.Sample([]string{
    "Are bees and wasps similar?",
  })
  if err != nil {
    log.Fatalf("%+v", err)
  }
  fmt.Printf("\tElapsed time: %s\n", time.Since(start))
  fmt.Printf("Generated text:\n%s\n", strings.Join(output, "\n\n"))
}
```

### Conclusion

- Using GoMLX can help implement ML inference in Go without Python

- Since it's a relatively new project, it may be a little risky for production uses for now.

---

https://eli.thegreenplace.net/2024/gomlx-ml-in-go-without-python

https://www.cs.toronto.edu/~kriz/cifar.html

https://www.kaggle.com/models/google/gemma-2

Running Machine Learning inference in Go without Python

#22 GoMLX: ML in Go without Python

Go Commentary #22: GoMLX: ML in Go without Python


## [Prep: Golang comptime. Pure blasphemy](https://github.com/pijng/prep)

- A small Go tool that enables compile-time function evaluation. By using `prep.Comptime`, you can evaluate functions at build time, replacing them with their computed results. Just like `comptime` from Zig. Except it's not.

- Features
  - Compile-Time Evaluation: Replace function calls with their computed results at build time.
  - Simple Integration: Use prep as both a Go library and a standalone executable.
  - Tooling Support: Easily integrate prep with your Go build process using -toolexec.

```go
package main

import (
  "fmt"
  "github.com/pijng/prep"
)

func main() {
  // This will be evaluated at compile-time
  result := prep.Comptime(fibonacci(300))

  fmt.Println("Result:", result)
}

func fibonacci(n int) int {
  fmt.Printf("calculating fibonacci for %d\n", n)

  if n <= 1 {
    return n
  }

  return fibonacci(n-1) + fibonacci(n-2)
}
```

- Build `go build -a -toolexec="prep <absolute/path/to/project>" main.go`

- Limitations

  - Currently, prep.Comptime only supports basic literals as arguments.

  ```go
  // Pass a basic literal directly
  func job() {
    prep.Comptime(myFunc(1))
  }

  // Use a variable with the value of basic literal from the same scope as wrapped function
  func job() {
    x := 1
    y := 2
    prep.Comptime(myFunc(x, y))
  }
  ```

  - Only functions that can be fully resolved with the provided literal arguments can be evaluated at compile-time, therefore it is impossible to use any values from IO operations.

## [go-sqlite3: Go bindings to SQLite using wazero](https://github.com/ncruces/go-sqlite3)

- Go module **github.com/ncruces/go-sqlite3** is a cgo-free SQLite wrapper. It provides a **database/sql** compatible driver, as well as direct access to most of the C SQLite API.

- It wraps a Wasm build of SQLite, and uses wazero as the runtime. Go, [wazero](https://github.com/tetratelabs/wazero) and x/sys are the only runtime dependencies.

```go

import "database/sql"
import _ "github.com/ncruces/go-sqlite3/driver"
import _ "github.com/ncruces/go-sqlite3/embed"

var version string
db, _ := sql.Open("sqlite3", "file:demo.db")
db.QueryRow(`SELECT sqlite_version()`).Scan(&version)
```

---

https://github.com/pijng/prep

https://github.com/ncruces/go-sqlite3

https://github.com/tetratelabs/wazero


A quick toolings for compile-time evaluation and SQLite wrapper with WebAssembly runtime for Go

#14 Compile-time eval & SQLite with wazero

Go Commentary #14: Golang compile-time evaluation and Go bindings to SQLite using wazero


## [Using Go Embed](https://www.bytesizego.com/blog/go-embed)

- The ```go:embed``` directive tells the Go compiler to include files and folders into the compiled binary at build time. This means your application can access these resources directly from memory without needing to read from the disk at runtime.

- Usage: 

  - with a single file message.txt ("hello from bytesizego!")

  ```go
  package main

  import (
    _ "embed"
    "fmt"
  )

  //go:embed message.txt
  var message string

  func main() {
    fmt.Println(message) // hello from bytesizego!
  }

  ```

  - with multiple files

  ```go
  package main

  import (
    _ "embed"
    "fmt"
  )

  //go:embed messages/*.txt
  var messages embed.FS

  func main() {
    files, _ := messages.ReadDir("messages")
    for _, file := range files {
      data, _ := messages.ReadFile("messages/" + file.Name())
      fmt.Printf("File: %s\nContent: %s\n\n", file.Name(), data)
    }
  }
  ```

  - with a directory (the path specified in ReadFile is relative to the embedded root.)

  ```go
  package main

  import (
    "embed"
    "fmt"
  )

  //go:embed static
  var staticFiles embed.FS

  func main() {
    data, _ := staticFiles.ReadFile("static/index.html")
    fmt.Println(string(data))
  }
  ```


- Limitations:

  - File Size: Embedding large files can significantly increase your binary size.
  - File Changes: Changes to the embedded files require recompilation.

## [Reflecting on Go Reflection](https://www.dolthub.com/blog/2024-10-04-reflecting-on-reflect/)

  - Context: using generative AI tooling, generated code using Reflect package

  ```go
  bsVal := reflect.ValueOf(blockStore).Elem()

  tables := bsVal.FieldByName("tables")

  typ := tables.Type()
  fmt.Printf("tables.Type: %v\n", typ)
  for i := 0; i < typ.NumField(); i++ {
    fmt.Printf("tables %d: %s\n", i, typ.Field(i).Name)
  }
  for i := 0; i < typ.NumMethod(); i++ {
    fmt.Printf("method %d: %s\n", i, typ.Method(i).Name)
  }
  ```

  - [Laws of Reflection](https://go.dev/blog/laws-of-reflection)

    - Reflection goes from interface value to reflection object

    - Reflection goes from reflection object to interface value

    - To modify a reflection object, the value must be settable

    => In short, the *Interface* method is the inverse of the *ValueOf* function, except that its result is always of static type interface{}.
    Reiterating: Reflection goes from interface values to reflection objects and back again.

  - Zeroth Law: Use reflect at your own peril. Misuse it, and it will *panic* with no regrets.

---

https://www.bytesizego.com/blog/go-embed

https://www.dolthub.com/blog/2024-10-04-reflecting-on-reflect/


#15 Go embed and Reflect

Go Commentary #15: Using Go embed, and Reflect


## [Go sync.Map: The Right Tool for the Right Job](https://victoriametrics.com/blog/go-sync-map/index.html)

- Context:

  ```go
  func main() {
      m := make(map[string]int)

      go func() {
          for {
              m["blog"] = 1
          }
      }()

      go func() {
          for {
              fmt.Println(m["blog"])
          }
      }()

      select{} // block-forever trick
  }
  // fatal error: concurrent map read and map write

  ```

- sync.Map:

  - **sync.Map** takes care of all that locking (or atomic operations) for you - so no manual locking needed, and no worrying about race conditions.

  - reading, writing, and deleting keys faster

  ```go
    func main() {
      var syncMap sync.Map

      // store a key-value pair
      syncMap.Store("blog", "VictoriaMetrics")

      // load a value by key "blog"
      value, ok := syncMap.Load("blog")
      fmt.Println(value, ok)

      // delete a key-value pair by key "blog"
      syncMap.Delete("blog")
      value, ok = syncMap.Load("blog")
      fmt.Println(value, ok)
    }

    // Output:
    // VictoriaMetrics true
    // <nil> false
  ```

  ```go
  func (m *Map) Load(key any) (value any, ok bool)

  func (m *Map) Store(key, value any)
  func (m *Map) LoadOrStore(key, value any) (actual any, loaded bool)

  func (m *Map) Delete(key any)
  func (m *Map) LoadAndDelete(key any) (value any, loaded bool)
  func (m *Map) CompareAndDelete(key, old any) (deleted bool)

  func (m *Map) Swap(key, value any) (previous any, loaded bool)
  func (m *Map) CompareAndSwap(key, old, new any) (swapped bool)

  func (m *Map) Range(f func(key, value any) bool)
  func (m *Map) Clear()
  ```

- even when iterate through a map while writing is not safe

  ```go
    func main() {
      m := make(map[string]int)

      go func() {
          for {
              m["blog"] = 1
          }
      }()

      go func() {
          for {
              for range m {
                  fmt.Println("iterating")
              }
          }
      }()

      select{} // block-forever trick
    }

    // fatal error: concurrent map iteration and map write
  ```

  - With **sync.Map.Range**, it’s designed to handle concurrent reads and writes during iteration without locking up the entire map. The trade-off, though, is that you might not get a perfectly consistent snapshot of the map while you’re iterating.


- How it works:

  - two separate native maps: the readonly map and the dirty map.

  ```go
    type Map struct {
      mu Mutex
      read atomic.Pointer[readOnly]
      dirty map[any]*entry
      misses int
    }

    type readOnly struct {
      m       map[any]*entry
      amended bool // true if the dirty map contains some key not in m.
    }

    type entry struct {
      p atomic.Pointer[any]
    }
  ```

  - readonly map is where the fast, lock-free lookups happen; built around an atomic.Pointer, which lets multiple goroutines access it without needing to lock anything. (ideal for scenarios where data is mostly being read and not frequently modified) 
  => the readonly map might not always hold the most up-to-date data, therefore dirty map

  - dirty map stores any new entries that get added while the readonly map is still being used for lookups

  ![syncmap_structure](assets/syncmap_structure.png)

  => dirty map contains all the data from the readonly map, along with any new entries that haven’t yet been promoted to the readonly map

  - when you update a value, all you need to do is update this pointer. Since both the readonly and dirty maps point to the same entry

  ![double_pointer_indirection_syncmap](assets/double_pointer_indirection_syncmap.png)

  - The behavior of the pointer in the entry struct defines the state of the entry in the map, and there are 3 possible states:

    - **Normal state**: This is when the entry is valid. The pointer p is pointing to a real value, and the entry exists in those maps, meaning it’s actively in use and can be read without any issues.

    - **Deleted state**: When an entry is deleted from a sync.Map, it’s not immediately removed from the readonly maps. Instead, the pointer p is simply set to nil, signaling that the entry has been deleted but still exists in the maps.

    - **Expunged state**: This is a special state where the key is fully removed. The entry is marked with a special sentinel value that indicates it’s been completely deleted.

  ![state_chart_syncmap](assets/state_chart_syncmap.png)


---

https://victoriametrics.com/blog/go-sync-map/index.html


Understanding sync.Map and using the right tools for atomic operations in Go.

#16 Understand sync.Map

Go Commentary #16: Understand sync.Map


## [Leveraging benchstat Projections in Go Benchmark Analysis!](https://www.bwplotka.dev/2024/go-microbenchmarks-benchstat/)

Context: (golang.org/x/perf/cmd/benchstat)

  - Old-school: 

    1. Creating the benchmark test code:

      ```go
        func BenchmarkFoo(b *testing.B) {
          b.Run(...)
        }
      ```
      
    2. Running the benchmark for the version A of your code

    ```bash
      export bench=v1 && go test \
      -run '^$' -bench '^BenchmarkFoo' \
      -benchtime 5s -count 6 -cpu 2 -benchmem -timeout 999m \
      | tee ${bench}.txt 
    ```

    3. Running the benchmark for the version B of your code 

    4. Analyze the A/B benchmark results
    ```bash
      benchstat base=v1.txt new=v2.txt
    ``` 

  Example:

  - to compare the encoding efficiency of the [Remote Write 1.0](https://prometheus.io/docs/specs/remote_write_spec/) protocol to the [2.0 version](https://prometheus.io/docs/specs/remote_write_spec_2_0/) for different sample sizes, ideally across different compressions and two different Go protobuf encoders (marshallers).

    ```go
      package across_versions

      // ...

      /*
        export bench=v2 && go test \
          -run '^$' -bench '^BenchmarkEncode' \
          -benchtime 5s -count 6 -cpu 2 -benchmem -timeout 999m \
        | tee ${bench}.txt
      */
      func BenchmarkEncode(b *testing.B) {
        for _, sampleCase := range sampleCases {
          b.Run(fmt.Sprintf("sample=%v", sampleCase.samples), func(b *testing.B) {
            batch := utils.GeneratePrometheusMetricsBatch(sampleCase.config)

            // Commenting out what we used in v1.txt
            //msg := utils.ToV1(batch, true, true)
            msg := utils.ToV2(utils.ConvertClassicToCustom(batch))

            compr := newCompressor("zstd")
            marsh := newMarshaller("protobuf")

            b.ReportAllocs()
            b.ResetTimer()
            for i := 0; i < b.N; i++ {
              out, err := marsh.marshal(msg)
              testutil.Ok(b, err)

              out = compr.compress(out)
              b.ReportMetric(float64(len(out)), "bytes/message")
            }
          })
        }
      }
    ```

    ```bash
    $ benchstat base=v1.txt new=v2.txt
    goos: darwin
    goarch: arm64
    pkg: go-microbenchmarks-benchstat/across_versions
                          │     base     │                new                 │
                          │    sec/op    │   sec/op     vs base               │
    Encode/sample=200-2      264.7µ ± 3%   107.0µ ± 4%  -59.58% (p=0.002 n=6)
    Encode/sample=2000-2    2672.9µ ± 3%   900.3µ ± 3%  -66.32% (p=0.002 n=6)
    Encode/sample=10000-2   13.335m ± 4%   3.299m ± 6%  -75.26% (p=0.002 n=6)
    geomean                  2.113m        682.4µ       -67.70%

                          │     base      │                 new                  │
                          │ bytes/message │ bytes/message  vs base               │
    Encode/sample=200-2      5.964Ki ± 1%    5.534Ki ± 0%   -7.21% (p=0.002 n=6)
    Encode/sample=2000-2     45.88Ki ± 0%    33.45Ki ± 0%  -27.08% (p=0.002 n=6)
    Encode/sample=10000-2    227.4Ki ± 0%    122.0Ki ± 3%  -46.33% (p=0.002 n=6)
    geomean                  39.62Ki         28.27Ki       -28.66%

                          │     base      │                 new                 │
                          │     B/op      │     B/op      vs base               │
    Encode/sample=200-2     336.76Ki ± 0%   64.02Ki ± 0%  -80.99% (p=0.002 n=6)
    Encode/sample=2000-2    1807.7Ki ± 0%   370.8Ki ± 0%  -79.49% (p=0.002 n=6)
    Encode/sample=10000-2    9.053Mi ± 0%   1.322Mi ± 0%  -85.40% (p=0.002 n=6)
    geomean                  1.739Mi        317.9Ki       -82.14%

                          │    base     │                 new                 │
                          │  allocs/op  │ allocs/op   vs base                 │
    Encode/sample=200-2      2.000 ± 0%   2.000 ± 0%        ~ (p=1.000 n=6) ¹
    Encode/sample=2000-2    10.000 ± 0%   2.000 ± 0%  -80.00% (p=0.002 n=6)
    Encode/sample=10000-2   16.000 ± 0%   2.000 ± 0%  -87.50% (p=0.002 n=6)
    geomean                  6.840        2.000       -70.76%
    ¹ all samples are equal

    ```

    -> Limitations:

      - **Difficult to track changes**: easy to lost track of when current optimizations are not helping, and need to revert to previous states.

      - **Accidental benchmark changes**: Unintentional modifications to the benchmark code can lead to unreliable comparisons and are hard to notice in this flow.

      - **Limited collaboration**: hard to share/replicate, esp for bigger projects, where reviews need to ensure the reliability of the author’s benchmark/claimed results.

  - New-school:


  ```bash
    export bench=allcases && go test \
    -run '^$' -bench '^BenchmarkFoo' \
    -benchtime 5s -count 6 -cpu 2 -benchmem -timeout 999m \
    | tee ${bench}.txt 
  ```

  ```go
  package across_cases

  // ...

  /*
    export bench=allcases && go test \
      -run '^$' -bench '^BenchmarkEncode' \
      -benchtime 5s -count 6 -cpu 2 -benchmem -timeout 999m \
    | tee ${bench}.txt
  */
  func BenchmarkEncode(b *testing.B) {
    for _, sampleCase := range sampleCases {
      b.Run(fmt.Sprintf("sample=%v", sampleCase.samples), func(b *testing.B) {
        for _, compr := range compressionCases {
          b.Run(fmt.Sprintf("compression=%v", compr.name()), func(b *testing.B) {
            for _, protoCase := range protoCases {
              b.Run(fmt.Sprintf("proto=%v", protoCase.name), func(b *testing.B) {
                for _, marshaller := range marshallers {
                  b.Run(fmt.Sprintf("encoder=%v", marshaller.name()), func(b *testing.B) {
                    msg := protoCase.msgFromConfigFn(sampleCase.config)

                    b.ReportAllocs()
                    b.ResetTimer()
                    for i := 0; i < b.N; i++ {
                      out, err := marshaller.marshal(msg)
                      testutil.Ok(b, err)

                      out = compr.compress(out)
                      b.ReportMetric(float64(len(out)), "bytes/message")
                    }
                  })
                }
              })
            }
          })
        }
      })
    }
  }

  var (
    sampleCases = []struct {
      samples int
      config  utils.GenerateConfig
    }{
      {samples: 200, config: generateConfig200samples},
      {samples: 2000, config: generateConfig2000samples},
      {samples: 10000, config: generateConfig10000samples},
    }
    compressionCases = []*compressor{
      newCompressor(""),
      newCompressor(remote.SnappyBlockCompression),
      newCompressor("zstd"),
    }
    protoCases = []struct {
      name            string
      msgFromConfigFn func(config utils.GenerateConfig) vtprotobufEnhancedMessage
    }{
      {
        name: "prometheus.WriteRequest",
        msgFromConfigFn: func(config utils.GenerateConfig) vtprotobufEnhancedMessage {
          return utils.ToV1(utils.GeneratePrometheusMetricsBatch(config), true, true)
        },
      },
      {
        name: "io.prometheus.write.v2.Request",
        msgFromConfigFn: func(config utils.GenerateConfig) vtprotobufEnhancedMessage {
          return utils.ToV2(utils.ConvertClassicToCustom(utils.GeneratePrometheusMetricsBatch(config)))
        },
      },
    }
    marshallers = []*marshaller{
      newMarshaller("protobuf"), newMarshaller("vtprotobuf"),
    }
  )

  ```

  - In Jan 2023, benchstat is [rewritten](https://cs.opensource.google/go/x/perf/+/02c55175bb825ade4507ee5d459ea6a1ab6e0af5)

  ```bash
  benchstat -row ".name /sample /compression /encoder" -filter "/compression:zstd /encoder:protobuf" -col /proto allcases.txt
  ```

  ```bash
  goos: darwin
  goarch: arm64
  pkg: go-microbenchmarks-benchstat/across_cases
                            │ prometheus.WriteRequest │   io.prometheus.write.v2.Request   │
                            │         sec/op          │   sec/op     vs base               │
  Encode 200 zstd protobuf                 268.8µ ± 2%   103.3µ ± 7%  -61.57% (p=0.002 n=6)
  Encode 2000 zstd protobuf               2671.4µ ± 5%   877.4µ ± 4%  -67.16% (p=0.002 n=6)
  Encode 10000 zstd protobuf              12.834m ± 2%   3.059m ± 8%  -76.16% (p=0.002 n=6)
  geomean                                  2.097m        652.1µ       -68.90%

                            │ prometheus.WriteRequest │    io.prometheus.write.v2.Request    │
                            │      bytes/message      │ bytes/message  vs base               │
  Encode 200 zstd protobuf                5.949Ki ± 0%   5.548Ki ±  0%   -6.73% (p=0.002 n=6)
  Encode 2000 zstd protobuf               45.90Ki ± 0%   33.49Ki ±  0%  -27.03% (p=0.002 n=6)
  Encode 10000 zstd protobuf              227.8Ki ± 1%   121.4Ki ± 25%  -46.70% (p=0.002 n=6)
  geomean                                 39.62Ki        28.26Ki        -28.68%

                            │ prometheus.WriteRequest │   io.prometheus.write.v2.Request    │
                            │          B/op           │     B/op      vs base               │
  Encode 200 zstd protobuf               336.00Ki ± 0%   64.00Ki ± 0%  -80.95% (p=0.002 n=6)
  Encode 2000 zstd protobuf              1799.8Ki ± 1%   368.0Ki ± 0%  -79.55% (p=0.002 n=6)
  Encode 10000 zstd protobuf              9.015Mi ± 2%   1.312Mi ± 0%  -85.44% (p=0.002 n=6)
  geomean                                 1.732Mi        316.3Ki       -82.17%

                            │ prometheus.WriteRequest │   io.prometheus.write.v2.Request    │
                            │        allocs/op        │ allocs/op   vs base                 │
  Encode 200 zstd protobuf                  2.000 ± 0%   2.000 ± 0%        ~ (p=1.000 n=6) ¹
  Encode 2000 zstd protobuf                10.000 ± 0%   2.000 ± 0%  -80.00% (p=0.002 n=6)
  Encode 10000 zstd protobuf               16.000 ± 0%   2.000 ± 0%  -87.50% (p=0.002 n=6)
  geomean                                   6.840        2.000       -70.76%
  ¹ all samples are equal

  ```

  -> Limitations:

  - Rerunning benchmarks with a large amount of cases takes significantly time (slower feedback loop!).
  - It yields more complex benchmarking code, which makes it hard to iterate on, and spot places where you benchmark the testing code vs the portion of the code you wanted to.
  - For continuous production use, it does not make sense to commit that benchmark with all cases, which are no longer being continued. It fits better to capture such a benchmark in some remote branch for future reference though.

Conclusion:

  - Should use both in a hybrid approach, depending on your goals. 

  - Can even use more features like `-format csv` to export to sheets and render charts

## [Go Plan9 Memo, Speeding Up Calculations 450%](https://pehringer.info/go_plan9_memo.html)

Context 

- want more power than Go's concurrency, encounter SIMD - Same Instruction Muliple Data, that many languages either have compiler optimizations use simd or libs that support it.

- "I just want a package that offers a thin abstraction layer over arithmetic and bitwise simd operations."

- Go's assembler uses [Plan9](https://9p.io/plan9/)'s assemblers guidance which uses target platforms instructions and registers with slight modifications to their names and usage. This means that x86 Plan9 is different then say arm Plan9.

  ```
  example
  ┣━ AddInts_amd64.s
  ┗━ main.go
  ```

  ```
  // +build amd64

  TEXT ·AddInts(SB), 4, $0
      MOVL    left+0(FP), AX
      MOVL    right+8(FP), BX
      ADDL    BX, AX
      MOVL    AX, int+16(FP)
      RET
  ```

  ```go
  package main

  import "fmt"

  func AddInts(left, right) int

  func main() {
      fmt.Println("1 + 2 = ", AddInts(1, 2))
  }
  ```

  **LINE 1**: The file contains amd64 specific instructions, so we need to include a Go build tag to make sure Go does not try to compile this file for non x86 machines.

  **LINE 3**: You can think of this line as the functions declaration. TEXT declares that this is a function or text section. ·AddInts(SB) specifies our functions name. 4 represents “NOSPLIT” which we need for some reason. And $0 is the size of the function’s stack frame (used for local variables). It’s zero in this case because we can easily fit everything into the registers.

  **LINE 4 & 5**: Go’s calling convention is to put the function arguments onto the stack. So we MOVe both Long 32-bit values into the AX and BX registers by dereferencing the frame pointer (FP) with the appropriate offsets. The first argument is stored at offset 0. The second argument is stored at offset 8 (int’s only need 4 bytes but I think Go offsets all arguments by 8 to maintain memory alignment).

  **LINE 6**: Add the Long 32-bit value in AX (left) with the Long 32-bit value in BX. And store the resulting Long 32-bit value in AX.

  **LINE 7 & 8**: Go’s calling convention (as far as I can tell) is to put the function return values after its arguments on the stack. So we MOVe the Long 32-bit values in the AX register onto the stack by dereferencing the frame pointer (FP) with the appropriate offset. Which is 16 in this case.

  ![](assets/SmallVectorsFloat32Addition.png)
  ![](assets/MediumVectorsFloat32Addition.png)
  ![](assets/LargeVectorsFloat32Addition.png)

Conclusion:

- There is roughly a 200-450% speed up depending on the number of elements using Plan9. Hope this inspires others to use it.

- The package currently supports x84 only, hopefully arm in future.

---

https://www.bwplotka.dev/2024/go-microbenchmarks-benchstat/

https://pehringer.info/go_plan9_memo.html




Understanding benchstat usage in Go benchmark and Go Plan9 on boosting up performance

#17 Leveraging benchstat Projects in Go benchmark and Go Plan9 memo on 450% speeding up calculations

Go Commentary #17: Leveraging benchstat Projects in Go benchmark and Go Plan9 memo on 450% speeding up calculations


## [Script: Making it easy to write shell-like scripts in Go](https://github.com/bitfield/script)

- Go library that simplifies writing shell-like scripts by providing a fluent API for common operations like file manipulation, text processing, and command execution.

- Key features:

  - Chainable methods for piping operations
  - Easy file and directory operations
  - Text processing functions (grep, sed-like replacements)
  - Command execution and output handling
  - Error handling integrated into the API

- Usage:

  ```go
  // Read the contents of a file as a string
  contents, err := script.File("test.txt").String()

  // Count the number of lines that match a pattern
  numErrors, err := script.File("test.txt").Match("Error").CountLines()

  // Filter all the results through some arbitrary Go function
  script.Stdin().Match("Error").FilterLine(strings.ToUpper).Stdout()


  // Append the first 10 arguments to a file
  script.Args().Concat().Match("Error").First(10).AppendFile("/var/log/errors.txt")

  // Simple HTTP request
  script.Get("https://wttr.in/London?format=3").Stdout()
  // Output:
  // London: 🌦   +13°C
  ```

## [Go Telemetry](https://go.dev/blog/gotelemetry)

- Context:

  - the Go toolchain can collect usage and breakage statistics that help the Go team understand how the Go toolchain is used and how well it is working.

  ![](assets/dataflow.png)

- Usage:

  ```
  go telemetry on

  go telemetry off

  // revert to the default mode of local-only telemetry
  go telemetry local
  ```

---

https://github.com/bitfield/script

https://go.dev/blog/gotelemetry

https://go.dev/doc/telemetry


Exploring a Go library for shell-like scripting and Go's telemetry feature.

#10 Script, Telemetry

Go Commentary #10: Script, Telemetry


## [Go Survey](https://google.qualtrics.com/jfe/form/SV_ei0CDV2K9qQIsp8?s=b)

- The feedbacks from us - Gophers will help Go Team to understand:
  - How Go is being used across various industries and organizations
  - The challenges you face as a Go developer
  - The features and improvements you’d like to see in future releases
  - How we can better support the thriving Go community

## [Building LLM-powered applications in Go](https://go.dev/blog/llmpowered)

Ah, the sweet smell of progress mixed with the stench of hype. Welcome to the brave new world of Go, where Large Language Models reign supreme and every developer suddenly fancies themselves an AI expert. But before we dive headfirst into this cesspool of buzzwords and overengineered solutions, let's take a moment to examine what's really going on in our beloved gopher-land.

The LLM Gold Rush: Go's Ticket to Relevance? It seems the Go team has finally woken up to the fact that LLMs are the new darling of the tech world. Their recent blog post on "Building LLM-powered applications in Go" reads like a desperate attempt to stay relevant in a landscape dominated by Python frameworks. But here's the kicker - they might actually be onto something.

Go's strengths in concurrency and networking make it a natural fit for the distributed nature of LLM applications. It's like watching a middle-aged dad suddenly discover he's got a knack for TikTok dances - unexpected, slightly uncomfortable, but oddly compelling.

Let's look at their RAG server example:

```Go
func main() {
	ctx := context.Background()
	wvClient, err := initWeaviate(ctx)
	if err != nil {
		log.Fatal(err)
	}

	apiKey := os.Getenv("GEMINI_API_KEY")
	genaiClient, err := genai.NewClient(ctx, option.WithAPIKey(apiKey))
	if err != nil {
		log.Fatal(err)
	}
	defer genaiClient.Close()

	server := &ragServer{
		ctx:      ctx,
		wvClient: wvClient,
		genModel: genaiClient.GenerativeModel(generativeModelName),
		embModel: genaiClient.EmbeddingModel(embeddingModelName),
	}

	mux := http.NewServeMux()
	mux.HandleFunc("POST /add/", server.addDocumentsHandler)
	mux.HandleFunc("POST /query/", server.queryHandler)

	port := cmp.Or(os.Getenv("SERVERPORT"), "9020")
	address := "localhost:" + port
	log.Println("listening on", address)
	log.Fatal(http.ListenAndServe(address, mux))
}
```

[full version at](https://github.com/golang/example/blob/master/ragserver/ragserver/main.go)

Simple, clean, and to the point. No fancy decorators, no convoluted class hierarchies - just good old Go simplicity. It's almost refreshing in a world of over-abstracted Python monstrosities.

But here's where things get interesting. The Go team isn't content with just providing raw tools; they're pushing frameworks like LangChainGo and Genkit. It's as if they've looked at the Python ecosystem and thought, "Hey, we can create incomprehensible abstractions too!"

LangChainGo: Because We Needed Another Layer of Abstraction LangChainGo promises to be the silver bullet for all your LLM needs. Want to switch vector databases without rewriting your entire codebase? LangChainGo's got you covered:

```Go
type VectorStore interface {
    AddDocuments(ctx context.Context, docs []schema.Document, options ...Option) ([]string, error)
    SimilaritySearch(ctx context.Context, query string, numDocuments int, options ...Option) ([]schema.Document, error)
}
```

It's a beautiful interface, really. So clean, so abstract. But let's be real - how often are you actually switching vector databases? This is solution in search of a problem, the software equivalent of a Swiss Army knife when all you needed was a bottle opener.

Genkit: Google's Answer to... Everything? Not to be outdone, Google throws its hat into the ring with Genkit. It's like LangChain, but with that special Google touch that screams, "We'll deprecate this in two years, but trust us for now!"

Genkit promises "production features" and "integrated developer tooling." Because apparently, what the world really needed was another way to manage prompts and deployments. It's as if Google looked at the mess of AI tooling and thought, "You know what this needs? More complexity!"

### [The Ergo Framework: Erlang's Ghost Haunts Go](https://github.com/ergo-services/ergo)

Just when you thought we couldn't possibly need another framework, along comes Ergo. It's bringing the actor model to Go, because apparently, we all miss the days of Erlang and its byzantine approach to concurrency.

Ergo boasts features like _Network Transparency_, _Observability_ and a _Supervisor Tree_. It's like they've taken every buzzword from distributed systems and thrown them into a blender. The result? A framework that promises to solve problems you didn't even know you had.

Here's a taste of their "Quick Start":

```bash
$ ergo -init MyNode \
      -with-app MyApp \
      -with-sup MyApp:MySup \
      -with-actor MySup:MyActor \
      -with-web MyWeb \
      -with-actor MyActor2 \
      -with-observer
```

Because nothing says "simplicity" like a command line that looks like it was designed by a committee of enterprise architects.

The Gopher's Dilemma So here we are, standing at the crossroads of Go's future. On one side, we have the simplicity and performance that made Go great. On the other, we have a smorgasbord of frameworks and abstractions promising to turn Go into a one-stop shop for all your LLM needs.

The question is, do we really need all this? Are we solving real problems, or are we just creating new ones in the name of "progress"?

Don't get me wrong - it's exciting to see Go evolving and adapting to new challenges. But let's not lose sight of what made Go great in the first place. We don't need to become Python or Erlang. We need to be the best damn Go we can be.

As we forge ahead into this brave new world of LLMs and AI, let's remember the virtues of simplicity and pragmatism. By all means, let's embrace new technologies and paradigms. But let's do it the Go way - with clear, concise code that solves real problems, not imaginary ones.

The future of Go in the LLM era is bright, but it's up to us to ensure it doesn't become a tangled mess of frameworks and abstractions. Let's build tools that empower developers, not confuse them. After all, isn't that what Go was all about in the first place?

Remember, in the world of software development, the only constant is change. But that doesn't mean we have to lose our way. Stay sharp, stay critical, and above all, stay Go.

---

https://go.dev/blog/survey2024-h2

https://go.dev/blog/llmpowered

https://github.com/ergo-services/ergo


A critical look at Go's evolving role in the LLM ecosystem and the frameworks shaping its future

#11 Actors, Frameworks, and the Future of Go

Go Commentary #11: The Gopher's LLM Revolution - Actors, Frameworks, and the Future of Go


## [kl: An interactive Kubernetes log viewer for your terminal](https://github.com/robinovitch61/kl)

An interactive Kubernetes log viewer for your terminal.

```
// Example usage of kl
kl --context my-context,other-context -n default,other-ns
```

![](assets/kl1.png)

![](assets/kl2.png)

![](assets/kl3.png)

![](assets/kl4.png)

- This tool allows you to view logs across multiple containers, pods, and even clusters. It's like kubectl logs on steroids, which begs the question: why isn't this functionality built into kubectl itself? The fragmentation of the Kubernetes ecosystem continues unabated, with each new tool solving a problem that arguably shouldn't exist in the first place.

## [Restish](https://rest.sh/#/)

- A "CLI for interacting with REST-ish HTTP APIs." Because apparently, cURL and HTTPie weren't enough. Restish boasts features like automatic API discovery and generated commands:

```
# Perform an HTTP GET request
$ restish api.rest.sh/types

# Above is equivalent to:
$ restish GET https://api.rest.sh/types
```

```https
HTTP/2.0 200 OK
Content-Length: 278
Content-Type: application/cbor
Date: Tue, 19 Apr 2022 21:17:58 GMT

{
  $schema: "https://api.rest.sh/schemas/TypesModel.json"
  boolean: true
  integer: 42
  nullable: null
  number: 123.45
  object: {
    binary: 0xdeadc0de
    binary_long: 0x00010203040506070809...
    date: 2022-04-23
    date_time: 2022-04-23T21:41:58.20449651Z
    url: "https://rest.sh/"
  }
  string: "Hello, world!"
  tags: ["example", "short"]
}
```

While it's undeniably clever, one has to wonder: are we solving real problems, or are we just creating more abstraction layers to satisfy our insatiable appetite for "developer experience"? The irony of using a CLI to interact with REST APIs – which were designed for machine-to-machine communication – is not lost on me.

## [A fast full-text cli reader (works also with lobste.rs articles content)](https://github.com/piqoni/cast-text)

- A "zero latency, easy-to-use full-text news terminal reader." Yes, you read that correctly. In 2024, we're excited about reading RSS feeds in the terminal. It's as if we've come full circle, rejecting the rich multimedia experiences of modern web browsers in favor of monospaced fonts and ANSI color codes.

```
// Reading lobste.rs with cast-text
cast-text -rss https://lobste.rs/rss
```

![](assets/cast-text.png)

The Pendulum Swings This trend towards CLI tools in Go is part of a larger pendulum swing in our industry. We've gone from command-line interfaces to graphical UIs, from desktop applications to web apps, and now we're seeing a resurgence of terminal-based tools. It's as if we're collectively suffering from option paralysis, overwhelmed by the complexity of modern software stacks and yearning for the perceived simplicity of text-based interfaces.

But here's the rub: these new CLI tools are often just as complex as their graphical counterparts. They're built on layers of abstractions, requiring knowledge of specific command syntaxes and flags. We haven't simplified; we've just shifted the complexity to a different domain.

---

https://github.com/robinovitch61/kl

https://rest.sh/#/

https://github.com/piqoni/cast-text


A critical examination of Go's resurgence in CLI tool development, exploring recent projects in Kubernetes log viewing, REST API interaction, and terminal-based readers, while questioning the industry's shift back to command-line interfaces.

#12 CLI Tools for K8s, REST, and Terminals

Go Commentary #12: CLI Renaissance with Kubernetes, REST, and Terminal Readers in the Age of Complexity


## [Register Allocation in the Go Compiler](https://developers.redhat.com/articles/2024/09/24/go-compiler-register-allocation#go_s_register_allocator__a_high_level_view)

Red Hat has graced us with a deep dive into Go's register allocation in the compiler. It's a fascinating peek under the hood, if you're into that sort of thing. But let's be real: how many of us are actually going to benefit from understanding the intricacies of register allocation? It's like knowing the exact chemical composition of the asphalt you're driving on – interesting, but ultimately irrelevant to most people's daily commute.

The Go team's obsession with compiler speed is admirable, I suppose. They've managed to create a register allocator that's "very fast," taking up to 20% of the entire optimization pipeline's time. Bravo. But at what cost?

```go
// Imagine this is your codebase after Go's "fast" register allocation
func someFunction() {
    // Oops, your variable got spilled into a loop
    for i := 0; i < 1000000; i++ {
        // Load from memory, use, store back to memory
        // Repeat ad nauseam
    }
}
```

Sure, your compile times are blazing fast. But your runtime? Well, that's a different story. The lack of a global view in the register allocator means you might end up with code that's about as efficient as a government bureaucracy.

But hey, at least it compiles quickly, right? Because that's what really matters in production – how fast you can push out potentially suboptimal code.

## [BBQvec: An open-source, embedded vector index for Rust and Go](https://blog.daxe.ai/p/bbqvec-a-scalable-vector-search-library)

Speaking of optimizations, let's talk about the latest darling of the AI world: vector search. Daxe has thrown their hat into the ring with BBQvec, a "scalable vector search library." Because clearly, what the world needs is another way to find the nearest neighbor in high-dimensional space.

Don't get me wrong, vector search is useful. But the way the industry is salivating over it, you'd think it was the second coming of sliced bread. Every startup and their dog is now implementing some form of vector search, often without really understanding why or if they even need it.

```go
// The modern tech stack, apparently
type ModernAIStartup struct {
    VectorSearch    *FancyVectorLib
    LLM             *ChatGPT
    ActualProduct   *WhoNeedsThis
}
```

BBQvec claims to be all about scale, handling "many billions of vectors." That's great, but let's pause for a moment. How many companies actually need to search through billions of vectors? And of those that do, how many are doing it for anything more than vanity metrics or to impress VCs?

The algorithm itself is clever, I'll give them that. Using random orthonormal basis sets and bitmaps for indexing is an interesting approach. But it's telling that their big selling point is how fast they can build the index, not necessarily how accurate or fast the actual searches are.

---

https://developers.redhat.com/articles/2024/09/24/go-compiler-register-allocation#go_s_register_allocator__a_high_level_view

https://blog.daxe.ai/p/bbqvec-a-scalable-vector-search-library


A scathing look at Go's compiler internals and the vector search gold rush, exposing the industry's obsession with speed over substance

Playground > Go