Skip to content

fix(ai): attach user media for Cohere (vision) requests#1246

Open
ling-senpeng13 wants to merge 7 commits into
mainfrom
fix/cohere-media-input
Open

fix(ai): attach user media for Cohere (vision) requests#1246
ling-senpeng13 wants to merge 7 commits into
mainfrom
fix/cohere-media-input

Conversation

@ling-senpeng13

@ling-senpeng13 ling-senpeng13 commented Jul 2, 2026

Copy link
Copy Markdown
Contributor

Problem

Cohere is vision-capable — command-a-vision-07-2025 (and Aya Vision) accept image input, and Cohere's v2 chat API accepts image_url content parts (base64 data URI or URL). But conductor-ai dropped image input entirely:

  1. CohereChatModel.toCohereMessage() built the message from message.getText() only — getMedia() was ignored.
  2. Deeper: the request DTO CohereApi.ChatMessage.content was a bare String, so it couldn't even represent content parts.

Same class of bug as #1238 (Anthropic), #1241 (Gemini), #1243 (Grok/Perplexity), #1245 (HuggingFace) — Cohere was the last provider still dropping media.

Fix

  • CohereApi.ChatMessage: widen content from String to Object (String or List<ContentPart>); add a ContentPart type (text / image_url + ImageUrl) and a ChatMessage.user(List<ContentPart>) factory.
  • CohereChatModel.toCohereMessage(): for a UserMessage with media, emit text + one image_url content part per media item (base64 data: URI for bytes). Text-only messages unchanged.

Test

Three layers:

  • CohereChatModelMediaTest (mocked HTTP, always runs — CI has no provider keys): captures the outgoing ChatCompletionRequest and asserts the user message serializes to a content-part list with an image_url part carrying the base64 data URI (plus the text part). FAILS before the fix, PASSES after.
  • CohereAITest.IntegrationTests#testChatCompletionWithImageMedia (live API, self-skips without COHERE_API_KEY): drives the adapter against Cohere's real v2 chat API with a repo-committed image embedding a machine-unguessable token (MELON7391) and asserts the vision model transcribes it — a token, unlike a color, cannot be answered correctly by luck, so a pass proves the image arrived (pattern from test(e2e): Python media-input suite (image → vision model) agentspan-ai/agentspan#301).
  • LLMChatCompleteTests#cohere_vision_imageMediaInMessages_modelSeesTheImage (live-API e2e, self-skips without COHERE_API_KEY): full server path — workflow messages[].media URL → HttpDocumentLoader download → adapter → live API — asserting the same token transcription. The image is committed in this repo (e2e/src/test/resources/assets/melon7391.png) and referenced at a raw URL pinned to its immutable commit SHA, so there is no third-party image host. COHERE_API_KEY is forwarded through the e2e docker-compose files like the other provider keys.

Also includes LLMChatCompleteTests#anthropic_haiku_vision_imageMediaInMessages_modelSeesTheImage — the same e2e shape through Anthropic, as a regression lock for the already-merged #1238 media fix, which shipped without an e2e test — plus its counterfactual (anthropic_haiku_vision_withoutMedia_tokenIsAbsent: same prompt, no media, token must be absent — proving the token can only come from the image). Both gated on ANTHROPIC_API_KEY and verified against the live API: the positive case transcribes MELON7391, the counterfactual completes without it.

The shared server pipeline for the e2e shape was verified against a live provider (Anthropic: same messages+URL input returns "Red" with image tokens counted in the prompt). The Cohere-gated tests are pending a COHERE_API_KEY — no key is available in CI or locally today, which is why the mocked test stays as the always-running regression guard.

Note (out of scope, provider-agnostic): workflow media passed as bare base64 never reaches any provider adapter — FileSystemDocumentLoader.supports() claims any string without :// and the document access policy rejects it. URL media is the supported end-to-end shape.

./gradlew :conductor-ai:test --tests "org.conductoross.conductor.ai.providers.cohere.*"
BUILD SUCCESSFUL

Notes

Image support is model-dependent (use a Cohere vision model such as command-a-vision-07-2025). This makes conductor forward the media in Cohere's documented image_url format.

🤖 Generated with Claude Code

@ling-senpeng13 ling-senpeng13 self-assigned this Jul 2, 2026
ling-senpeng13 added a commit to agentspan-ai/agentspan that referenced this pull request Jul 2, 2026
Correcting an earlier omission: Cohere IS vision-capable (e.g.
command-a-vision-07-2025) and its v2 chat API accepts image_url content
parts. Enabled in conductor-ai by conductor-oss/conductor#1246. Support is
model-dependent. Added to both provider docs.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@ling-senpeng13 ling-senpeng13 requested a review from v1r3n July 5, 2026 18:27
ling-senpeng13 and others added 2 commits July 8, 2026 17:09
Cohere is vision-capable (e.g. command-a-vision-07-2025) and its v2 chat
API accepts image_url content parts (base64 data URI or URL). But
conductor-ai dropped image input: CohereChatModel.toCohereMessage() used
message.getText() only, and the request DTO CohereApi.ChatMessage.content
was a bare String that couldn't even represent content parts.

Widen ChatMessage.content to Object (String or List<ContentPart>), add a
ContentPart type (text / image_url) and a ChatMessage.user(List) factory,
and forward UserMessage media as image_url parts (base64 data URI for
bytes). Same class of fix as #1238/#1241/#1243.

Adds CohereChatModelMediaTest, which captures the outgoing request and
asserts the user message carries an image_url content part with the
base64 data URI. Verified it fails before the fix (bare string content)
and passes after. Image support is model-dependent.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Fixes the build (spotlessJavaCheck) failure on the Cohere media-input PR.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@ling-senpeng13 ling-senpeng13 force-pushed the fix/cohere-media-input branch from 819cbff to c29091c Compare July 9, 2026 00:11
ling-senpeng13 and others added 5 commits July 8, 2026 17:31
…ERE_API_KEY

Two live-API layers on top of the existing mocked serialization test, both
self-skipping when COHERE_API_KEY is absent (it is not available in CI or
locally today, so the mocked test remains the always-running guard):

- CohereAITest.IntegrationTests#testChatCompletionWithImageMedia: drives the
  adapter directly against Cohere's live v2 chat API with a solid-red PNG and
  asserts the vision model actually sees it.
- LLMChatCompleteTests#cohere_vision_imageMediaInMessages_modelSeesTheImage:
  full server path (workflow messages[].media URL -> HttpDocumentLoader
  download -> adapter -> live API). Media is a URL pinned to an immutable
  commit SHA because that is the workflow-media shape the server supports
  end-to-end; bare base64 is rejected upstream of any provider by the
  document access policy (pre-existing, provider-agnostic).

The shared server pipeline for the e2e shape was verified live via Anthropic
(same messages+URL input answers "Red" with image tokens counted), so the
only part that runs unverified until a key exists is Cohere's side of the
contract. COHERE_API_KEY is forwarded through the e2e docker-compose files
the same way as the other provider keys.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…1238 fix

PR #1238 fixed Anthropic dropping user media but shipped without an e2e
test. Reuses this PR's messages+media harness (registerMessagesWorkflow +
pinned image URL); gated on ANTHROPIC_API_KEY like the other Anthropic
tests, and verified against the live API: the model answers "red" for the
red logo, proving the image survives workflow input -> download -> adapter.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ON7391)

Referenced by the live-API media tests at an immutable raw URL pinned to
this commit's SHA. The token is machine-unguessable, so transcription
proves the model saw the image — unlike a color question, which a model
can answer correctly by luck without any image attached.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…anscription

Replaces the "dominant color" assertion with transcription of a
machine-unguessable token (MELON7391) rendered in a repo-committed image
(pinned raw URL at the asset commit SHA — no third-party host). A color
can be guessed correctly without the image ever arriving; the token
cannot, so a pass now proves the model actually saw the image. Adds the
counterfactual case (same prompt, no media) asserting the token is absent,
proving it cannot leak from the prompt. Pattern borrowed from
agentspan-ai/agentspan#301.

Verified live via Anthropic: positive case transcribes the token,
counterfactual completes without it. Cohere cases remain key-gated.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Records in the TOKEN_IMAGE_URL javadoc why local alternatives don't work:
loopback URLs are rejected by DocumentAccessPolicy (SSRF protection), bare
base64/data URIs are claimed by FileSystemDocumentLoader and denied as file
paths, and file:// paths break the dockerized flow. A remote HTTPS URL is
also the production media shape, so the test covers the real path.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant