Skip to content

chore(helm/charts): import erpc from morpho-infra-helm#90

Merged
rguichard merged 12 commits into
morpho-mainfrom
feature/pla-1455-move-app-related-deployment-config-into-application-repos
Jul 9, 2026
Merged

chore(helm/charts): import erpc from morpho-infra-helm#90
rguichard merged 12 commits into
morpho-mainfrom
feature/pla-1455-move-app-related-deployment-config-into-application-repos

Conversation

@rguichard

@rguichard rguichard commented Jun 24, 2026

Copy link
Copy Markdown
Collaborator

Summary

This PR brings the eRPC Helm chart and its production environment configuration into this repository, consolidating application-level deployment config alongside the application code rather than keeping it in a separate infrastructure repository. The migration covers the full production topology: multiple eRPC instances, HAProxy smart load balancers with Prometheus-driven weight and fallback controllers, Redis HA, and a CNPG-managed PostgreSQL cluster.

Two companion fixes are included: security hardening of the validation scripts based on an Aikido review, and bumping the eRPC and erpc-validator image tags to 0.1.3 to match the latest production values.

Changes

New Helm chart — helm/charts/erpc/

  • Base application chart with Deployment, HPA (autoscaling/v2), PDB, ClusterIP services (HTTP + metrics + optional headless), and ServiceAccounts.
  • Vault-based secret pipeline: a pre-install Job renders a config template from Vault using a built-in AWK renderer that resolves __SECRET_<KEY>__ placeholders and generates per-key auth strategies from API_KEY_* entries; an optional dedicated validation init-container (vault.validationImage) runs erpc validate before the secret is stored in Kubernetes.
  • CNPG PostgreSQL cluster (pg-erpc.yaml) with volume-snapshot or Barman S3 backup, custom autovacuum/checkpoint tuning, and a recovery health monitoring ConfigMap.
  • Security hardening gate (securityHardening.enabled) implementing the Pod Security Standards restricted profile (non-root, read-only rootfs, dropped capabilities, seccomp RuntimeDefault).
  • Local development support: docker-compose.yml, .env.example, and test-local.sh.

Production environment wrapper — helm/environments/prd/erpc/

  • Five eRPC sub-chart instances (erpc, erpc-dev, erpc-fallback, erpc-processing, erpc-router), each with independent Vault config paths, resource profiles, and per-instance runtime ServiceAccounts.
  • Three HAProxy deployments (erpc-haproxy, erpc-processing-haproxy, erpc-router-haproxy), each carrying two sidecar controllers: a weight-controller that adjusts per-pod backend weights using goroutine count, P99 latency, request rate, and CPU metrics; and a fallback-controller that gradually activates the hot-standby fallback pool on sustained degradation and immediately on total outage.
  • Redis HA (redis-ha + HAProxy + prometheus-redis-exporter) with Vault-managed password.
  • PrometheusRule with recording rules across histogram families, counters, and gauges; latency percentile rules; fallback traffic share rules; and alerting rules for high fallback traffic and actionable failure rates.
  • Vault secret materialization Jobs for Redis and DB secrets, with matching RBAC.

CI/CD and repo governance

  • .github/CODEOWNERS: platform-engineers and @0x666c6f as mandatory reviewers on all patterns including /helm/.
  • .github/workflows/helm-charts-validation.yaml: three-job pipeline — chart discovery, parallel helm lint + kubeconform validation (with database chart skip logic), summary gate.
  • .github/workflows/slack-pr-notification.yaml: Slack notification on merge to morpho-main, skipping bot authors.
  • .github/workflows/wiz-iac-scan.yml: Wiz CLI IaC scan triggered on PRs touching helm/.
  • validate-helm-charts.sh: local script mirroring CI validation, with GNU parallel support and sequential fallback.

Aikido security fixes

  • validate-helm-charts.sh: process_chart uses return instead of exit so the sequential loop always reaches the summary; single mktemp -d with unified cleanup via rm -rf; chart path passed directly to helm dependency update / helm template instead of cd-ing; helm template | kubeconform pipeline split for correct exit-code propagation.
  • test-local.sh: LABELS_SERVICE_API_KEY redacted when echoing the auth response; eth_getLogs block numbers corrected to a valid 100 000-block range on Arbitrum.

Image version bump

  • values.yaml: erpc and erpc-validator image tags updated from 0.0.77 to 0.1.3.

Testing

Local validation script

# Install prerequisites: helm >= 3.16.4, kubeconform v0.6.1
./validate-helm-charts.sh          # parallel (requires GNU parallel)
./validate-helm-charts.sh -s       # sequential fallback

Both the base chart and the prd environment wrapper should report lint and kubeconform passes; erpc-db and any *postgres* charts are intentionally skipped by kubeconform.

CI
Push or open a PR touching helm/** — the Helm Chart Validation workflow runs chart discovery, parallel validation, and a summary job that fails the check if any chart fails.

Local Docker Compose (eRPC chart)

cd helm/charts/erpc
cp .env.example .env  # fill in API keys
docker-compose up -d
./test-local.sh

All seven test cases (health, Ethereum/Arbitrum/Base RPC, auth enforcement, eth_getLogs, DynamoDB note) should pass.

Edge cases to verify

  • A chart with dependencies: in Chart.yaml correctly triggers helm dependency update before template rendering.
  • process_chart failures in sequential mode do not abort validation of subsequent charts (the return fix).
  • Requests to eRPC without a secret are rejected when LABELS_SERVICE_API_KEY is configured; the rejection response does not echo the key back in test output.

Codex follow-up

  • Fixed agent-harness repo-map drift by making scripts/review-repo-map.sh use deterministic case-insensitive sorting: LC_ALL=C sort -f -u.
  • Fixed Aikido checkout-token findings by setting persist-credentials: false in Helm validation and Wiz IaC workflows.
  • Verified locally with ./scripts/agent-harness/update-repo-map.sh and make agent-check.

@wiz-c998a0ef2b

wiz-c998a0ef2b Bot commented Jun 24, 2026

Copy link
Copy Markdown

Wiz Scan Summary

Scanner Findings
Vulnerability Finding Vulnerabilities -
Data Finding Sensitive Data -
Secret Finding Secrets -
IaC Misconfiguration IaC Misconfigurations 1 Medium 1 Low
SAST Finding SAST Findings -
Software Management Finding Software Management Findings -
Total 1 Medium 1 Low

View scan details in Wiz

To detect these findings earlier in the dev lifecycle, try using Wiz Code VS Code Extension.

Comment thread validate-helm-charts.sh Outdated
Comment thread helm/charts/erpc/test-local.sh Outdated
Comment thread validate-helm-charts.sh Outdated
Comment thread validate-helm-charts.sh Outdated
@rguichard rguichard force-pushed the feature/pla-1455-move-app-related-deployment-config-into-application-repos branch from 3db6a38 to 631f9ad Compare June 24, 2026 08:59
Comment thread validate-helm-charts.sh
Comment thread helm/charts/erpc/test-local.sh Outdated
@rguichard rguichard force-pushed the feature/pla-1455-move-app-related-deployment-config-into-application-repos branch from 631f9ad to 7cad0d6 Compare June 24, 2026 13:13
Comment thread validate-helm-charts.sh Outdated
Comment thread .github/workflows/helm-charts-validation.yaml
Comment thread .github/workflows/pr-commands.yml Outdated
Comment thread .github/workflows/update-image-tag.yaml
Comment thread .github/CODEOWNERS
Comment thread helm/charts/erpc/values.yaml Outdated
@rguichard rguichard force-pushed the feature/pla-1455-move-app-related-deployment-config-into-application-repos branch from 7cad0d6 to 7cadc0e Compare June 24, 2026 13:33
Comment thread validate-helm-charts.sh Outdated
Comment thread validate-helm-charts.sh Outdated
Comment thread validate-helm-charts.sh
rguichard and others added 3 commits June 25, 2026 12:21
Apply the fixes from the per-finding review branches directly:

validate-helm-charts.sh:
- process_chart now returns instead of exit, so the sequential loop
  validates every chart and reaches the summary (the parallel path is
  unaffected).
- Use a single `mktemp -d` dir (lint/template files) instead of an
  unused base temp file; clean up with one `rm -rf`.
- Pass the chart path to `helm dependency update` / `helm template`
  instead of cd-ing into the dir; drop the now-unused INITIAL_DIR.
- Split the `helm template | kubeconform` pipeline for readability
  (keeping the `if !` guard, which is pipefail-safe).

test-local.sh:
- Redact LABELS_SERVICE_API_KEY when echoing the auth response.
- Correct the eth_getLogs decimal block numbers (0x1254048f =
  307,496,079, 0x12558b2f = 307,596,079, range = 100,000).

The Aikido suggestion to move should_skip_validation before `helm lint`
was intentionally not applied: lint is meant to run for all charts;
only kubeconform is skipped for database charts.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- CODEOWNERS: add @0x666c6f as a mandatory codeowner alongside the
  teams, on every pattern (incl. /helm/) so he is a required reviewer
  repo-wide, not only for files matching the default `*` rule.
- values.yaml: bump erpc and erpc-validator image tags 0.0.77 -> 0.1.3
  to match the latest prd values in morpho-org/morpho-infra-helm.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@rguichard rguichard force-pushed the feature/pla-1455-move-app-related-deployment-config-into-application-repos branch from 6553bf2 to a8d49f2 Compare June 25, 2026 10:21
@rguichard rguichard requested a review from 0x666c6f June 25, 2026 10:21
@rguichard

Copy link
Copy Markdown
Collaborator Author

@0x666c6f do you know why agent-harness is failing ? I ran ./scripts/agent-harness/update-repo-map.sh but the file was already up to date

Comment thread .github/workflows/helm-charts-validation.yaml
Comment thread .github/workflows/wiz-iac-scan.yml
@0x666c6f

0x666c6f commented Jun 29, 2026

Copy link
Copy Markdown
Collaborator

Fixed in 7dd92d7.

Root cause: scripts/review-repo-map.sh used locale-sensitive sort -u. Linux CI sorted uppercase top-level files differently than the local macOS run, so make agent-check saw a generated review/repo-map.md diff even though it looked up to date locally.

Fix: force deterministic case-insensitive ordering with LC_ALL=C sort -f -u. I verified with:

./scripts/agent-harness/update-repo-map.sh
make agent-check

rguichard added 3 commits July 6, 2026 19:55
The import predates the latest bumps in morpho-infra-helm (images 0.1.3,
main replicaCount 12); cutting ArgoCD over on the stale copy would roll
production back.
Anchor the nine ECR image tags (erpcImageTag for the erpc runtime,
erpcValidatorImageTag for the validator job image) so the
update-image-tag workflow can bump them all in one place. The upstream
ghcr.io digest pins are intentionally left out of the anchors.
Replicate the morpho-api in-repo image bump flow, adapted to erpc:

- update-image-tag.yaml: workflow_dispatch with a semver version_tag
  (validated), prd/morpho-main only (erpc has no dev environment or
  branch), bumps the erpcImageTag and erpcValidatorImageTag anchors,
  creates a GitHub-signed commit via createCommitOnBranch with the
  morpho-infra-deployer app token, opens a PR and — when
  vars.ERPC_PRD_AUTOMERGE is true — waits for required checks with the
  workflow token then merges with admin bypass. Auto-merge defaults to
  off: production PRs require manual approval.
- only-values-yaml-guard.yml: same guard as morpho-api targeting
  morpho-main, scoping the bot's review bypass to values.yaml-only
  changes.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR migrates the eRPC Helm chart and the production (prd) environment wrapper into this repository, alongside CI workflows and helper scripts for chart validation and IaC scanning.

Changes:

  • Added a new base Helm chart (helm/charts/erpc) plus a production environment wrapper chart (helm/environments/prd/erpc) with HAProxy/Redis/CNPG Postgres resources and Prometheus rules.
  • Introduced Helm chart validation tooling locally (validate-helm-charts.sh) and in CI (.github/workflows/helm-charts-validation.yaml), plus Wiz IaC scanning and governance updates (CODEOWNERS, values-only guard, Slack notifications).
  • Updated repo-map tooling output determinism and refreshed review/repo-map.md accordingly.

Reviewed changes

Copilot reviewed 39 out of 46 changed files in this pull request and generated 7 comments.

Show a summary per file
File Description
validate-helm-charts.sh Adds a local Helm lint + kubeconform validation script for selected charts.
scripts/review-repo-map.sh Makes repo-map generation deterministic with case-insensitive sorting.
review/repo-map.md Updates generated repo map to reflect newly added top-level paths/files.
helm/environments/prd/erpc/values.yaml Adds production configuration for multiple eRPC instances, HAProxy controllers, Redis HA, and DB settings.
helm/environments/prd/erpc/templates/serviceaccount-vault.yaml Creates a dedicated ServiceAccount for Vault materialization hooks in prd wrapper.
helm/environments/prd/erpc/templates/rbac-vault-redis.yaml Adds RBAC allowing the Vault job to create/update the Redis auth secret.
helm/environments/prd/erpc/templates/prometheus-rules.yaml Adds PrometheusRule recording + alerting rules for eRPC and HAProxy/fallback behavior.
helm/environments/prd/erpc/templates/job-vault-redis.yaml Adds a hook Job to materialize Redis credentials from Vault into a Kubernetes Secret.
helm/environments/prd/erpc/templates/_helpers.tpl Provides naming helpers for the prd wrapper chart templates.
helm/environments/prd/erpc/Chart.yaml Defines the prd wrapper chart and its dependencies (erpc, haproxy, redis-ha, exporter).
helm/environments/prd/erpc/Chart.lock Locks dependency versions/digests for the prd wrapper chart.
helm/environments/prd/.gitkeep Keeps the prd environments directory tracked.
helm/charts/erpc/values.yaml Adds default values for the base eRPC chart, including Vault rendering/validation knobs.
helm/charts/erpc/test-local.sh Adds a local docker-compose based smoke-test script for eRPC.
helm/charts/erpc/templates/serviceaccount.yaml Implements runtime + Vault-creator ServiceAccount behavior (token automount disabled by default).
helm/charts/erpc/templates/service.yaml Adds the main ClusterIP Service for eRPC HTTP.
helm/charts/erpc/templates/service-metrics.yaml Adds a dedicated metrics Service.
helm/charts/erpc/templates/service-headless.yaml Adds an optional headless Service for DNS-based discovery (e.g., HAProxy server-template).
helm/charts/erpc/templates/scheduled-backup.yaml Adds a CNPG ScheduledBackup resource when Postgres is enabled.
helm/charts/erpc/templates/rbac-vault.yaml Adds RBAC for Vault hook jobs to manage secrets.
helm/charts/erpc/templates/podmonitor.yaml Adds a PodMonitor for scraping eRPC metrics.
helm/charts/erpc/templates/podmonitor-erpc-db.yaml Adds a PodMonitor for CNPG metrics when Postgres is enabled.
helm/charts/erpc/templates/pg-erpc.yaml Adds CNPG Cluster manifest with tuning + backup configuration.
helm/charts/erpc/templates/pdb.yaml Adds a PDB for eRPC pods.
helm/charts/erpc/templates/monitoring-configmap.yaml Adds CNPG custom queries ConfigMap for recovery/replication health monitoring.
helm/charts/erpc/templates/job-vault-db.yaml Adds a hook Job to materialize DB credentials from Vault into a Kubernetes Secret.
helm/charts/erpc/templates/job-vault-config.yaml Adds a hook Job that renders config from Vault secrets, validates it, then stores it as a Secret.
helm/charts/erpc/templates/image-catalog-pg18.yaml Adds a CNPG ImageCatalog for Postgres 18 images.
helm/charts/erpc/templates/hpa.yaml Adds an HPA for the eRPC Deployment.
helm/charts/erpc/templates/deployment.yaml Adds the eRPC Deployment, including hardening options and secret-mounted config.
helm/charts/erpc/templates/_helpers.tpl Adds common naming/label helpers and ServiceAccount name logic for the base chart.
helm/charts/erpc/README.md Adds documentation for local dev/testing and Vault placeholder rendering.
helm/charts/erpc/docker-compose.yml Adds a docker-compose file to run eRPC locally with a mounted config.
helm/charts/erpc/Chart.yaml Adds the base eRPC chart definition.
helm/charts/erpc/.env.example Adds an example environment file for local eRPC docker-compose testing.
helm/charts/.gitkeep Keeps the charts directory tracked.
.github/workflows/wiz-iac-scan.yml Adds a Wiz IaC scan workflow gated on helm/** changes.
.github/workflows/update-image-tag.yaml Adds an automation workflow to update production values.yaml image tags via PR.
.github/workflows/slack-pr-notification.yaml Adds Slack notification on merge to morpho-main.
.github/workflows/only-values-yaml-guard.yml Adds a required guard to ensure deployer-bot bypasses reviews only on values.yaml-only PRs.
.github/workflows/helm-charts-validation.yaml Adds CI Helm lint + kubeconform validation for charts under helm/**.
.github/CODEOWNERS Adds/updates CODEOWNERS, including mandatory reviewers for Helm and scripts.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread validate-helm-charts.sh
Comment thread helm/charts/erpc/templates/hpa.yaml
Comment thread helm/charts/erpc/templates/pdb.yaml
Comment thread helm/charts/erpc/templates/deployment.yaml
Comment thread helm/charts/erpc/README.md Outdated
Comment thread helm/environments/prd/erpc/templates/job-vault-redis.yaml
Comment thread helm/charts/erpc/templates/job-vault-db.yaml Outdated
rguichard and others added 3 commits July 8, 2026 17:58
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Signed-off-by: Romain Guichard <romain@particule.io>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Signed-off-by: Romain Guichard <romain@particule.io>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Signed-off-by: Romain Guichard <romain@particule.io>
Comment thread helm/environments/prd/erpc/values.yaml
Comment thread .github/workflows/helm-charts-validation.yaml
@rguichard rguichard merged commit b02a628 into morpho-main Jul 9, 2026
18 of 20 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants