Go Weekly #1: Mastering Go Performance - eBPF and PGO Optimization Techniques

An Applied Introduction to eBPF with Go

Context:
- We usually write software in user space (outside the OS’s kernel, e.g: user apps like utilities, programming languages, GUI…).
Problem:
- When profiling the application: if we do it from user space (e.g: pprof) => result is not reliable because there will be some overhead of each layer on top of the CPU/ memory.
- 2 initial options:
  - Edit the Kernel Source Code: not really practical since the usecase is trivial amount. (pending for years to be adopted by distros)
  - Write a Kernel Model: this is more practical however, regular maintenance is inevitable as keeping up with the new kernel versions + risking corrupting the Kernel (e.g: if the module has a bug => crash the whole system)
Solution:
- BPF was originally used in Linux to filter network packets.
- eBPF (Extended Berkeley Packet Filter) allows to trace syscalls, user space/ library functions, network packets… => system performance, monitoring, security…
- How it works:
  - Pre-defined hooks: system calls, function entry/exit, kernel tracepoints, network events…
  - eBPF programs are event-driven, run at certain hook point then:
    - Safe checked by Verifier
    - Then compiled by JIT compiler from bytecode to instructions
  - Execute the desired code right before actual system calls
Conclusion:
- Powerful tool to dive deep in Kernel therefore many applicable usecase: systems programming, observability, security…
- For profiling usecase, can use existing projects: Parca, Pyroscope or PGO (from Go 1.20) for convenience.

The Profile-Guided Optimization Experience at Grab

Context:
- PGO (Profile-Guided Optimization) is introduced in Go version 1.20, a.k.a FDO (feedback-directed optimization), a technique collects and feeds the profile data back to the next compilier build.
- From the 2nd build/release, expectedly improving 2-14% performance (on-going for future builds)
Problem:
- Grab wanted to experiment this to some of their services: use self-managed database TalariaDB, orchestrated service and a monorepo one.
Results:
- In a service cluster’s image that uses TalariaDB, add -PGO=./talaria.PGO to the go build command: 10% CPU usage reduction, 30% memory usage reduction and 38% volume usage reduction.
- On the orchestrated service: the reduction is only around 5% « the effort the enable PGO => not substaintial
- Monorepo service is currently not supported since needing a seperated pprof service and a build process supporting PGO arguments to attach/retrieve pprof file
Conclusion:
- Applicable on simple, yet low-effort deployed services

Mentioned in

No mentions found

Unable to load mentions

Subscribe to Dwarves Memo

Receive the latest updates directly to your inbox.

Go Weekly #1: Mastering Go Performance - eBPF and PGO Optimization Techniques

fuatto

Mint this entry as an NFT to add it to your collection.

Verification This entry has been permanently stored onchain and signed by its creator.

transaction
nft address: Loading...
content digest