Skip to content

feat: add LiteLLM as AI gateway provider#1249

Open
RheagalFire wants to merge 5 commits into
conductor-oss:mainfrom
RheagalFire:feat/add-litellm-provider
Open

feat: add LiteLLM as AI gateway provider#1249
RheagalFire wants to merge 5 commits into
conductor-oss:mainfrom
RheagalFire:feat/add-litellm-provider

Conversation

@RheagalFire

@RheagalFire RheagalFire commented Jul 3, 2026

Copy link
Copy Markdown

Pull Request type

  • Feature

./gradlew spotlessApply has been run.

Changes in this PR

Adds LiteLLM as a new AI gateway provider, giving Conductor users access to 100+ LLM providers through a single unified OpenAI-compatible interface.

LiteLLM is a user-deployed AI gateway/proxy that unifies 100+ providers (OpenAI, Anthropic, Azure, Bedrock, Vertex AI, Mistral, Cohere, etc.) behind one OpenAI-compatible endpoint. Users deploy it locally, via Docker, Kubernetes, or on AWS/GCP/Azure using Terraform modules, and point Conductor at it. This lets users route requests through one config, add cost tracking/rate limiting/fallbacks, and switch providers without changing workflow definitions.

Files added:

  • ai/src/main/java/.../providers/litellm/LiteLLM.java - new provider implementing AIModel via OpenAICompatChatModel (same pattern as Grok/Perplexity). Zero new dependencies.
  • ai/src/main/java/.../providers/litellm/LiteLLMConfiguration.java - Spring @ConfigurationProperties(prefix = "conductor.ai.litellm"). base-url is required (LiteLLM is user-deployed, no public SaaS endpoint). api-key is optional (many deployments run without auth).
  • ai/README.md - added LiteLLM to supported providers table (now 13), configuration docs, environment variables, and drop_params guidance.

Configuration:

conductor.ai.litellm.base-url=http://litellm-proxy:4000
conductor.ai.litellm.api-key=${LITELLM_API_KEY}  # optional, only if proxy has auth enabled

Full E2E (Conductor server in Docker + Postgres + LiteLLM proxy -> Azure Foundry / Anthropic Claude Sonnet 4.6):

POST /api/workflow/execute/litellm_e2e_test/1

Workflow: COMPLETED
Task: LLM_CHAT_COMPLETE -> COMPLETED
  Result: 4
  Tokens: 25
  Finish: STOP

Complete chain verified: Conductor workflow -> LLM_CHAT_COMPLETE task -> LiteLLM provider -> OpenAICompatChatModel -> HTTP to LiteLLM proxy -> Azure Foundry -> Anthropic Claude -> response parsed back through the full Conductor pipeline.

Example workflow:

{
  "llmProvider": "litellm",
  "model": "gpt-4o",
  "messages": [{"role": "user", "message": "What is the capital of France?"}],
  "temperature": 0.7,
  "maxTokens": 500
}

Set drop_params: true in your LiteLLM proxy config (litellm_settings) so provider-unsupported parameters (e.g. frequency_penalty for Anthropic) are silently dropped instead of causing 400 errors.

@RheagalFire

Copy link
Copy Markdown
Author

cc @v1r3n @manan164

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants