Skip to content

feat(concurrency): add opt-in per-domain concurrency limit scope#1267

Open
izrail09 wants to merge 2 commits into
conductor-oss:mainfrom
izrail09:feature/per-domain-concurrency-limit
Open

feat(concurrency): add opt-in per-domain concurrency limit scope#1267
izrail09 wants to merge 2 commits into
conductor-oss:mainfrom
izrail09:feature/per-domain-concurrency-limit

Conversation

@izrail09

@izrail09 izrail09 commented Jul 9, 2026

Copy link
Copy Markdown

Pull Request type

  • Bugfix
  • Feature
  • Refactoring (no functional changes, no api changes)
  • Build related changes
  • WHOSUSING.md
  • Other (please describe):

NOTE: Please remember to run ./gradlew spotlessApply to fix any format violations.

Changes in this PR

Adds an opt-in per-domain scope for task concurrency limits.

Problem. concurrentExecLimit is counted only by task-definition name — a single cluster-wide cap shared across every domain a task runs under. But the task queue is already partitioned by domain (domain:taskType), so when the same task is used by multiple workflows/tenants separated by domain, one busy domain can starve the others: they all contend on a single in-progress count keyed by name.

What's new. A new TaskDef.concurrencyLimitScope enum:

  • TASK_DEF (default) — unchanged: the limit is a single cap counted per task_def_name across all domains.
  • DOMAINconcurrentExecLimit is counted per (task_def_name, domain), so each domain gets an independent concurrency budget. Tasks with no domain fall into the legacy no-domain bucket.

Backward compatibility. The field defaults to TASK_DEF; absent JSON values deserialize to TASK_DEF; the default code path is byte-for-byte the existing behavior, so existing task definitions are unaffected. Because DOMAIN is opt-in, the Redis key-layout change applies only to tasks that explicitly opt in (new behavior) — there is no rolling-upgrade impact on existing tasks.

Persistence coverage.

  • Postgres / MySQL / SQLite — a nullable domain column is added to task_in_progress (migrations V17 / V11 / V5). DOMAIN scope counts and orders by (task_def_name, domain) using each dialect's null-safe match (IS NOT DISTINCT FROM / <=> / IS). NULL is the legacy no-domain bucket; the default TASK_DEF path runs the original SQL unchanged (no PK change).
  • Redis (redis-persistence and the optional redis-concurrency-limit module) — DOMAIN scope keys the in-progress set / limit bucket by (task_def_name, domain); TASK_DEF keeps the legacy per-name key.
  • CassandraDOMAIN scope currently falls back to per-task-definition (global) counting. Enforcing it per domain requires a task_def_limit schema change and Cassandra has no automatic migration path; left as a follow-up (consistent with
    getInProgressTaskCount already being unsupported there).

Tests. ExecutionDAOTest#testTaskExceedsLimitPerDomain (shared across Postgres, MySQL, SQLite, and Redis) verifies per-domain isolation and the null-domain bucket. The existing testTaskExceedsLimit (default scope) continues to pass on all backends.

Deliberately out of scope (follow-ups): protobuf/gRPC mapping for the new field (consistent with inputSchema/outputSchema, which are also non-proto today), and per-domain concurrency metrics in WorkflowMonitor.

Issue #1266

Alternatives considered

  • Always domain-aware (no flag) — count per (task_def_name, domain) unconditionally, treating a null domain as the global bucket. Simpler, but silently changes behavior for existing users who combine task domains with concurrentExecLimit (each domain would suddenly get its own full limit). Rejected to avoid a backward-incompatible semantic change.
  • Global config toggle — one server property to switch all task defs to per-domain. Rejected: all-or-nothing, causes config drift (the same exported definition behaves differently per cluster), and the semantics don't travel with the definition.
  • Opt-in enum on TaskDef (chosen) — per-definition, backward compatible by default, travels with the definition on export/import, and matches the existing per-TaskDef config idiom (retryLogic, timeoutPolicy, ...).

izrail09 and others added 2 commits July 9, 2026 15:25
Adds TaskDef.concurrencyLimitScope (TASK_DEF default | DOMAIN). When set to
DOMAIN, concurrentExecLimit is counted per (task_def_name, domain) so each
task domain gets an independent concurrency budget; TASK_DEF preserves the
existing cluster-wide, per-task-name behavior.

Enforcement is implemented across the SQL and Redis persistence backends:
- Postgres/MySQL/SQLite: a nullable `domain` column on task_in_progress
  (Flyway migrations), counted with each dialect's null-safe match
  (IS NOT DISTINCT FROM / <=> / IS). NULL is the legacy no-domain bucket and
  the default TASK_DEF path runs the original SQL unchanged.
- Redis (redis-persistence + redis-concurrency-limit): domain-qualified
  in-progress/limit keys for DOMAIN scope only; TASK_DEF keeps legacy keys.
- Cassandra falls back to per-task-definition counting for now (no automatic
  schema-migration path); documented as a follow-up.

Backward compatible: the field defaults to TASK_DEF, the default code path is
unchanged, and DOMAIN's Redis key change applies only to opt-in tasks.

Tests: shared ExecutionDAOTest#testTaskExceedsLimitPerDomain runs for
Postgres, MySQL, SQLite, and Redis.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant