dicer-go

An idiomatic Go port of databricks/dicer — an auto-sharder that dynamically assigns contiguous ranges of hashed keys ("slices") to Kubernetes pods.

This port preserves the user-facing surface (port defaults 24500/24510, the TargetConfig schema, the dicer-assigner / dicer-demo-{server,client} CLI triple) and the algorithmic semantics (constraint → deallocate → merge → place → split, with the documented imbalance/reservation thresholds) while reorganising the implementation along Go-idiomatic lines: context.Context plumbing, log/slog, errgroup supervisors, and stdlib net/http fronting gRPC on a shared listener.

Status

Initial port of the v1.0.0 upstream release. The public types (SliceKey, Slice, Assignment, Generation, Squid, the Clerk / Slicelet interfaces, and the TargetConfig YAML schema), the in-memory Store, the assignment generator, the watch stream, the load accumulator, and three CLI binaries are implemented.

What is intentionally simplified vs. upstream Scala:

• The wire protocol uses hand-rolled gRPC service definitions (the upstream uses Databricks' internal RPC wrapper, not vanilla gRPC). The framing and the resource model match; the message bytes do not.
• The algorithm covers the five core phases described in the upstream paper, with the documented load-imbalance thresholds (DEFAULT ≈ 10 %, TIGHT ≈ 2.5 %, LOOSE ≈ 40 %) and reservation tiers (1 / 10 / 20 / 40 % of max_load_hint). Hot-key isolation lives in the splitter; state transfer and homomorphic assignment for "key of death" scenarios are deferred.
• The etcd-backed store is stubbed; the in-memory store is fully functional.
• TargetConfigP is rendered as YAML rather than Google textproto. Field set and semantics are unchanged.

Layout

cmd/                          CLI entrypoints (dicer-assigner, dicer-demo-server, dicer-demo-client)
pkg/dicer/                    Public library API: SliceKey, Slice, Assignment, Clerk, Slicelet, TargetConfig
internal/algorithm/           Five-phase assignment generation
internal/assigner/            Control-plane server, generator loop, subscriber manager
internal/client/              Clerk + Slicelet implementations + load accumulator
internal/watch/               Shared watch server helper (server-streaming gRPC)
internal/store/               Memory (full) and Etcd (stub) stores
internal/config/              TargetConfig YAML loader and ConfigScope matcher
internal/metrics/             Prometheus collectors (dicer_assigner_* / dicer_client_*)
internal/proto/dicerpb/       gRPC service + message definitions
deploy/helm/                  Helm chart for the assigner

Quick start

make build

mkdir -p /tmp/dicer/target_config
cat > /tmp/dicer/target_config/demo.yaml <<'EOF'
owner_team_name: caching
default_config:
  primary_rate_metric_config:
    max_load_hint: 30000
    imbalance_tolerance_hint: DEFAULT
EOF

./bin/dicer-assigner \
  --target-config-dir=/tmp/dicer/target_config \
  --rpc-port=24500 --info-port=7777 \
  --location=kubernetes-cluster:dev/local/public/local/dev/01 &

POD_IP=127.0.0.1 POD_UID=$(uuidgen 2>/dev/null || cat /proc/sys/kernel/random/uuid) \
LOCATION=kubernetes-cluster:dev/local/public/local/dev/01 \
./bin/dicer-demo-server --target=demo --assigner=localhost:24500 --rpc-port=24510 &

./bin/dicer-demo-client --target=demo --slicelet=localhost:24510 --assigner=localhost:24500

Configuration

The assigner reads *.yaml files from --target-config-dir. Each file describes one target and follows the TargetConfigP shape from upstream dicer/external/proto/target.proto:

owner_team_name: my-team
default_config:
  primary_rate_metric_config:
    max_load_hint: 30000             # required, units are application-defined
    imbalance_tolerance_hint: DEFAULT # DEFAULT | TIGHT | LOOSE
    uniform_load_reservation_hint: NO_RESERVATION
  key_replication_config:
    min_replicas: 1
    max_replicas: 1
overrides:
  - override_scopes:
      - env: prod
        cloud: aws
        region: us-east-1
    override_config:
      primary_rate_metric_config:
        max_load_hint: 60000

Environment variables

POD_IP        Slicelet's externally-reachable IP (mirrors upstream)
POD_UID       Slicelet's stable identifier (mirrors upstream)
LOCATION      kubernetes-cluster:<env>/<cloud>/<domain>/<region>/<cluster-type>/<instance>
DB_CONF       Optional path to a properties file overlaying config flags

License

Apache 2.0. See LICENSE.