reveal-md

# Building Haskell with Buck2

by Andreas Herrmann

<small>Haskell Implementors Workshop (Haskell Symposium) ICFP – 7th September 2024</small>

Note:
- introduction (SBG lead, Tweag - OSPO Modus Create)
- focus: scalable, polyglot build systems - Bazel / Buck2
- topic: Buck2 for Haskell
- ---
- ! indicates next fragment (slide animation)

---

### Collaboration

- Mercury, I.W. Kim
- Tweag by Modus Create
  - T. Schmits, C. Shao, S. Visscher (GHC)
  - A. Herrmann, C. Bley (build)
- Well-Typed, M. Pickering, Z. Duggal

Note:
- part of collaboration
  - Mercury (Ian-Woo Kim, spoke yesterday)
  - Tweag SBG (Claudio Bley, myself)
  - Tweag GHC (Torsten Schmits, Cheng Shao, Sjoerd Visscher)
  - Well Typed (Matthew Pickering, Zubin Duggal)
- focus on build

---

## Motivation

- fast feedback 
- distributed cache 
- distributed build 
- extensible 
- cross plaform

Note:
- why scalable build system - Buck2 / Bazel
- ! fast feedback loop
- what they offer
- ! dist cache (branch-switch, do/undo, use CI builds)
- ! dist build (scale)
- ! general & extensible (polyglot & code-gen)
- ! cross platform/compilation

---

## Background

- ‌  [tweag/rules_haskell](https://github.com/tweag/rules_haskell) 2017/18
- ‌  [Buck2 contributors](https://www.tweag.io/blog/2023-07-06-buck2/)

Note:
- relevant background - why us?
- Haskell Bazel rule authors
- contributors to Buck2 open sourcing

---

## Codebase

- $1.1\mathrm{M}$ lines Haskell 
- $10.2\mathrm{k}$ modules (single package) 
- TypeScript & code-generation

Note:
- target project (as Ian mentioned)
- large Haskell code base
  - ! 1.1M LoC (approx.)
  - ! 10.2k modules (single package)
- ! complex: polyglot (TS) & code gen
- ---
- (Cabal build times 8min 30s)
  - 20min w/o byte code
  - on 10 core Mac M1
- future proofing

---

## Demo
### Amazonka

- GitHub:  [aherrmann/2024-hiw-amazonka](https://github.com/aherrmann/2024-hiw-amazonka)
- $3.8\mathrm{M}$ lines Haskell 
- $35\mathrm{k}$ modules

Note:
- target proprietary
- open source demo project
- ! Amazonka
- ! example code on GitHub
- ! even larger !
- automated migration
- flatter dependency graph

---

### Amazonka

```mermaid
graph TD
  amazonka-sts --> amazonka-core
  amazonka-sso --> amazonka-core
  amazonka --> amazonka-core
  amazonka --> amazonka-sso
  amazonka --> amazonka-sts
  amazonka-s3 --> amazonka-core
  others["..."] ~~~ amazonka-core
  amazonka-ec2 --> amazonka-core

style others fill-opacity: 0, stroke-opacity: 0;
```

Note:
- only 1-2 packages deep

---

## Buck2 Overview

- Meta (2023) 
- v2 of Buck (2013) 
- inspired by Blaze (Bazel)

Note:
- What's Buck2
- ! build system by Meta
- open source 2023
- ! v2 of Buck (2x faster)
- ! inspired by Blaze (Google's internal Bazel)

---

### Targets

```python
haskell_library(
    name = "amazonka-s3",
    deps = [":amazonka-core"],
    ...
)

haskell_library(
    name = "amazonka-core",
    ...
)
```

```mermaid
graph TD
  amazonka-s3 --> amazonka-core
```

</div>

Note:
- separate target and action graphs
- ! target: user level
  - e.g. Haskell library (rule)
- ! target dependencies → target graph
  - deps attribute

---

### Actions

```python
haskell_library(
    name = "amazonka-core",
    srcs = [
        "src/Amazonka/Types.hs",
        "src/Amazonka/Core.hs",
    ],
    ...
)
```

```mermaid
%%{init: { "flowchart": { "nodeSpacing": 5 } } }%%
graph LR
  subgraph amazonka-core
    direction TB;
    Amazonka.Types(compile Amazonka.Types) --> Amazonka.Core(compile Amazonka.Core)
    Amazonka.Core & Amazonka.Types --> link_core(link library)
  end
```

</div>

Note:
- action: build level
- ! rules issue actions
  - ! e.g. compile or link action

---

### Buck2 Actions

- explicit inputs 
- explicit outputs 
- action graph 
- remote builds 
- remote cache

</div>

<pre style="flex: 1"><code class="python" data-fragment-index="0" data-line-numbers="|2,5|3,4|"
>cmd = cmd_args(
  ghc, "-c", src,
  "-ohi", iface.as_output(),
  "-o", obj.as_output(),
  hidden = dep_ifaces,
)
ctx.actions.run(cmd)</code></pre></div>

Note:
- actions have
- ! explicit inputs
- ! explicit outputs
- ! artifacts → action graph
  - scheduling / parallelism
- ! remote execution
  - which inputs to send
  - which outputs to collect
- ! content addressed cache
  - what to hash
  - when to invalidate

---

## Haskell Builds

Note:
- topic: Haskell builds

---

### Module Order Compilation

```mermaid
graph RL
  Core -->|import| Types
```

```mermaid
graph LR
  Types.hs --> compile_types{compile} --> Types.o & Types.hi

Types.hi --> compile_core{compile} --> Core.o & Core.hi
  Core.hs ----> compile_core

style Types.hi fill: darkslateblue
```

Note:
- you all know
- compile modules in dependency order
- ! if `Core` imports `Types`
- ! compile `Types` before `Core`
  - interface `Types.hi` needed
- how to map to Buck2?

---

### Static Target Graph

```python
haskell_library(
    name = "amazonka-core",
    srcs = [
        "src/Amazonka/Types.hs",
        "src/Amazonka/Core.hs",
    ],
    ...
)
```

Note:
- Bazel & Buck2: static target graph
- ! user specified target
- no build time target generation
- ensure fast query & dependency resolution
- and, explicit action inputs/outputs
- ---
- Bazel: need per module targets (Gazelle)

---

### Coarse Grained Actions

```mermaid
graph LR
  Types.hs & Core.hs --> compile{ghc --make}
  subgraph run_all[ctx.actions.run]
    direction LR

compile --> Types.o & Types.hi
    compile ---> Core.o & Core.hi
  end
```

Note:
- option: coarse grained actions
- ! i.e. use GHC make mode in single action
- upside:
  - `ghc --make` is smart
- downside:
  - not incremental (change single module)
  - remote build & cache
- ---
- Buck2 incremental actions
  - not for remote execution
  - hermeticity risk

---

### Fine Grained Actions

```mermaid
graph LR
  subgraph run_types[ctx.actions.run]
    direction LR
    compile_types --> Types.o & Types.hi
  end

subgraph run_core[ctx.actions.run]
    Types.hi --> compile_core --> Core.o & Core.hi
  end

Types.hs --> compile_types{ghc -c}
  Core.hs ----> compile_core{ghc -c}
```

Note:
- want: fine grained actions
- ! one action per module compilation
  - ghc one-shot mode
- incremental builds
  - change `Core` don't rebuild `Types`.
- remote build & cache
- downside: one-shot mode overhead
- but: how to teach Buck2?

---

### Dynamic Action Graph

```mermaid
%%{init: { "flowchart": { "nodeSpacing": 20 } } }%%
graph LR
  Types.hs & Core.hs --> dynamic_container

subgraph dynamic_container[dynamic]
    tbd{‌<br><br><br><br><br><br><br><br>‌}
  end

dynamic_container --> Types.o & Types.hi & Core.o & Core.hi

style tbd fill-opacity: 0, stroke-opacity: 0;
```

Note:
- Buck2: dynamic action graph
  - benefit over Bazel
  - reason to choose Buck2
- ! dynamic output action
- black box action "dynamic"
  - scoped inputs and outputs

---

### Dynamic Action Graph

```mermaid
%%{init: { "flowchart": { "nodeSpacing": 20 } } }%%
graph LR
  md{ghc -M} --> deps[graph] --> dynamic_container
  Types.hs & Core.hs --> md

subgraph dynamic_container[dynamic]
    tbd{‌<br><br><br><br><br><br><br><br>‌}
  end

style tbd fill-opacity: 0, stroke-opacity: 0;
```

Note:
- special input: dependency graph (`ghc -M`)
- parsed at build time

---

### Dynamic Action Graph

```mermaid
%%{init: { "flowchart": { "nodeSpacing": 10 } } }%%
graph LR
  Types.hs --> compile_types{ghc -c} ----> Types.o
  Types.hi -..-> Types.hi.out[Types.hi]
  Core.hs ----> compile_core{ghc -c} --> Core.o & Core.hi

subgraph dynamic_container[dynamic]
    direction LR
    compile_types{ghc -c} --> Types.hi --> compile_core{ghc -c}
  end
```

Note:
- generates new actions at build time
- scoped: pre-declared inputs & outputs

---

### GHC Metadata Output

[GHC#24384](https://gitlab.haskell.org/ghc/ghc/-/issues/24384)
```json [1]
$ ghc -M -dep-json dep.json ...
{ "Amazonka.Types": {
    "sources": [".../src/Amazonka/Types.hs"],
    "modules": ["Amazonka.Prelude", ...],  // imports
    "packages": [  // package deps
      { "name": "base",
        // cross package imports
        "modules": [ "Control.Concurrent", ... ],
        ... }, ... ],
    "options": [],  // ghc options
    ...  // other properties
  }
```

Note:
- To support that
- added structured dependency output to GHC
- See GHC issue 24384

---

## Package Boundaries

```mermaid
graph RL
  subgraph amazonka-core
    direction RL
    Amazonka.Core & Amazonka.Response
  end
  
  subgraph amazonka-s3
    direction RL
    Amazonka.S3.Types -->|import| Amazonka.Core
  end
```

Note:
- look at packages
- module imports cross packages
- `S3` does not depend on `Response`
- build should neither
- but dynamic action
  - scoped to target
- want
  - module-granular
  - incremental builds
  - across packages

---

### Buck2 Dynamic Values

[facebook/buck2#619](https://github.com/facebook/buck2/pull/619)

Note:
- want: share data across dynamic actions
- ! introduced "dynamic values" to Buck2
  - akin to "future"

---

### Module Granular Build

```shell [1-2|4|6-7]
$ buck2 build //.../amazonka-s3[...][Amazonka.S3.Types]
Commands: 195

# edit .../Amazonka/Response.hs

$ buck2 build //.../amazonka-s3[...][Amazonka.S3.Types]
Commands: 1
```

Note:
- benefit: per module build
- build individual module
- no linking
- only compile dependency modules
  - and `ghc -M` and package-db
- ! edit non-dependency module
- ! rerun build
- no build actions
  - only `ghc -M` action invalidated

---

### Eliminated Package Boundaries

```mermaid
%%{init: { "flowchart": { "nodeSpacing": 15, "rankSpacing": 20, "padding": 15 } } }%%
graph LR
  link-a --> link-b
  comp-a1 ----> comp-b3
  comp-a3 ----> comp-b1

subgraph pkg-a[ ]
    direction LR
    comp-a1 & comp-a2 & comp-a4 --> link-a{ld}
    comp-a1{ } --> comp-a2{ } & comp-a3{ } --> comp-a4{ }
    comp-a3 --> link-a
  end

subgraph pkg-b[ ]
    direction LR
    comp-b1 & comp-b2 & comp-b3 & comp-b4 --> link-b{ld}
    comp-b4{ } --> comp-b3{ }
    comp-b1{ } --> comp-b2{ }
  end

classDef pkg-b fill: darkslateblue
  class comp-b1,comp-b2,comp-b3,comp-b4,link-b pkg-b
```

Note:
- and: eliminate package boundaries
- important for scheduling
- package boundary bottlenecks
  - linking/package-db/compile starvation

---

### Eliminated Package Boundaries

```mermaid
%%{init: { "flowchart": { "nodeSpacing": 15, "rankSpacing": 20, "padding": 15 } } }%%
graph LR
  link-a --> link-b
  comp-a1 & comp-a2 & comp-a4 & comp-a3 --> link-a{ld}
  comp-b1 & comp-b2 --> link-b{ld}
  comp-b3 ----> link-b
  comp-b4 -----> link-b

comp-a1 & comp-b4{ } --> comp-b3{ }
  comp-a1{ } --> comp-a2{ } & comp-a3{ } --> comp-a4{ }
  comp-a3 --> comp-b1
  comp-b1{ } --> comp-b2{ }

classDef pkg-b fill: darkslateblue
  class comp-b1,comp-b2,comp-b3,comp-b4,link-b pkg-b
```

Note:
- compile does not depend on link
- compile actions "float left"
- ---
- package-db: purely metadata

---

## Current Performance

- Cabal: 1h 6min (local, 8 core) 
- Buck2: 2h 3min (local, 8 core) 
- Buck2:   15min (remote, 100 core)

Note:
- benchmarks
- warning: preliminary
- clean build
- ! local Cabal 1h
- ! local Buck2 2h
  - room for improvement
- ! remote Buck2 15min
  - but scaling works

---

### Full Local Build

![](assets/full_clean_local_cabal_load.png) 
![](assets/full_clean_local_buck2_load.png)

Note:
- Cores permanently saturated
- Mostly one-shot/rules mode overhead

---

### Full Remote Build

![](assets/full_clean_remote_buck2_trace.png)

Note:
- 100 remote nodes
  - mostly saturated
  - some gaps
  - queued (probably competing build)

---

### Small Local Build

```mermaid
graph TD
  amazonka-sts --> amazonka-core
  amazonka-sso --> amazonka-core
  amazonka --> amazonka-core
  amazonka --> amazonka-sso
  amazonka --> amazonka-sts
```

- Cabal: 34s 
- Buck2: 23s

Note:
- Build only `amazonka` (& `amazonka-core|sts|sso`)
- ! Cabal 34s
- ! Buck2 23s
- despite one-shot overhead
- (cherry-picked)

---

### Small Local Build

![](assets/small_clean_local_cabal_load.png) 
![](assets/small_clean_local_buck2_load.png)

Note:
- Buck2 has better utilization
- ---
- Cabal `-j2 --ghc-options="-j4"`
- Cabal `-jsem` not available

---

### Incremental Build

```mermaid
graph TD
  amazonka-sts --> amazonka-core["amazonka-core<br>(edit Amazonka.Bytes)"]
  amazonka-sso --> amazonka-core
  amazonka --> amazonka-core
  amazonka --> amazonka-sso
  amazonka --> amazonka-sts
```

- Cabal: 7.7s 
- Buck2: 1.4s

Note:
- edit `amazonka-core`
  - non-ABI changing
- rebuild `amazonka`
- ! Cabal 7.7s
- ! Buck2 1.4s

---

## Other Topics

Note:
- not enough time for details

---

### Template Haskell

- GHC  loads `.o` / `.so`
  - link dependency packages 
- one-shot  `-fprefer-byte-code`
  - only  `.hi` needed

[GHC#13042](https://gitlab.haskell.org/ghc/ghc/-/merge_requests/13042)

Note:
- Template Haskell
- ! GHC loads objects / libraries
  - ! compile depends on link
  - causes package bottleneck
- ! enable byte-code interpreter in one-shot mode
  - ! no dependency on link
- ! GHC MR 13042
- cross package object loading
  - catch me in the break

---

### Recompilation Avoidance

- content addressed 
- ABI hash 
-  Recompilation avoidance in rules_haskell <br>[tweag.io/blog 3rd November 2022](https://www.tweag.io/blog/2022-11-03-blog_recompilation/)
- expose build plan

Note:
- GHC avoids unnecessary recompilation
- ! Buck2: content hash
  - any `.hi` change
  - want `.hi|.o` determinism (Matt/Zubin)
- ! ABI hash
- ! same trick as in Bazel
  - mark `.hi` as unused input
  - separate ABI hash input
- ! future: expose GHC build plan
  - improved recompilation avoidance

---

### Compilation Server

- one-shot overhead 
  - GHC startup 
  - package-db 
  - linking 
  - interfaces 
- persistent worker

Note:
- ! one-shot mode overhead
  - ! GHC startup time
  - ! package-db loading
  - ! library linking (esp. MacOS)
  - ! interface loading
  - repeated for every module!
- ! use compilation server
  - Buck2/Bazel: persistent worker
  - amortize across modules

---

## Conclusion

- extensible & scalable 
- cross package boundaries 
- incremental & parallel 
- one-shoto mode overhead 
- upstream contributions

Note:
- ! Buck2: extensible & scalable
- ! cross package  module granular build
- ! incremental (caching)
  - and parallel (remote)
- ! one-shot overhead
  - need compilation server
- ! upstream (benefit OS)
  - GHC
  - Buck2
  - Buck2 rules (planned)

---

# Thank You

Note:
- thank you