fix(ai): attach user media for Cohere (vision) requests#1246
Open
ling-senpeng13 wants to merge 7 commits into
Open
fix(ai): attach user media for Cohere (vision) requests#1246ling-senpeng13 wants to merge 7 commits into
ling-senpeng13 wants to merge 7 commits into
Conversation
ling-senpeng13
added a commit
to agentspan-ai/agentspan
that referenced
this pull request
Jul 2, 2026
Correcting an earlier omission: Cohere IS vision-capable (e.g. command-a-vision-07-2025) and its v2 chat API accepts image_url content parts. Enabled in conductor-ai by conductor-oss/conductor#1246. Support is model-dependent. Added to both provider docs. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This was referenced Jul 2, 2026
Cohere is vision-capable (e.g. command-a-vision-07-2025) and its v2 chat API accepts image_url content parts (base64 data URI or URL). But conductor-ai dropped image input: CohereChatModel.toCohereMessage() used message.getText() only, and the request DTO CohereApi.ChatMessage.content was a bare String that couldn't even represent content parts. Widen ChatMessage.content to Object (String or List<ContentPart>), add a ContentPart type (text / image_url) and a ChatMessage.user(List) factory, and forward UserMessage media as image_url parts (base64 data URI for bytes). Same class of fix as #1238/#1241/#1243. Adds CohereChatModelMediaTest, which captures the outgoing request and asserts the user message carries an image_url content part with the base64 data URI. Verified it fails before the fix (bare string content) and passes after. Image support is model-dependent. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Fixes the build (spotlessJavaCheck) failure on the Cohere media-input PR. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
819cbff to
c29091c
Compare
…ERE_API_KEY Two live-API layers on top of the existing mocked serialization test, both self-skipping when COHERE_API_KEY is absent (it is not available in CI or locally today, so the mocked test remains the always-running guard): - CohereAITest.IntegrationTests#testChatCompletionWithImageMedia: drives the adapter directly against Cohere's live v2 chat API with a solid-red PNG and asserts the vision model actually sees it. - LLMChatCompleteTests#cohere_vision_imageMediaInMessages_modelSeesTheImage: full server path (workflow messages[].media URL -> HttpDocumentLoader download -> adapter -> live API). Media is a URL pinned to an immutable commit SHA because that is the workflow-media shape the server supports end-to-end; bare base64 is rejected upstream of any provider by the document access policy (pre-existing, provider-agnostic). The shared server pipeline for the e2e shape was verified live via Anthropic (same messages+URL input answers "Red" with image tokens counted), so the only part that runs unverified until a key exists is Cohere's side of the contract. COHERE_API_KEY is forwarded through the e2e docker-compose files the same way as the other provider keys. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…1238 fix PR #1238 fixed Anthropic dropping user media but shipped without an e2e test. Reuses this PR's messages+media harness (registerMessagesWorkflow + pinned image URL); gated on ANTHROPIC_API_KEY like the other Anthropic tests, and verified against the live API: the model answers "red" for the red logo, proving the image survives workflow input -> download -> adapter. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ON7391) Referenced by the live-API media tests at an immutable raw URL pinned to this commit's SHA. The token is machine-unguessable, so transcription proves the model saw the image — unlike a color question, which a model can answer correctly by luck without any image attached. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…anscription Replaces the "dominant color" assertion with transcription of a machine-unguessable token (MELON7391) rendered in a repo-committed image (pinned raw URL at the asset commit SHA — no third-party host). A color can be guessed correctly without the image ever arriving; the token cannot, so a pass now proves the model actually saw the image. Adds the counterfactual case (same prompt, no media) asserting the token is absent, proving it cannot leak from the prompt. Pattern borrowed from agentspan-ai/agentspan#301. Verified live via Anthropic: positive case transcribes the token, counterfactual completes without it. Cohere cases remain key-gated. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Records in the TOKEN_IMAGE_URL javadoc why local alternatives don't work: loopback URLs are rejected by DocumentAccessPolicy (SSRF protection), bare base64/data URIs are claimed by FileSystemDocumentLoader and denied as file paths, and file:// paths break the dockerized flow. A remote HTTPS URL is also the production media shape, so the test covers the real path. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Cohere is vision-capable —
command-a-vision-07-2025(and Aya Vision) accept image input, and Cohere's v2 chat API acceptsimage_urlcontent parts (base64 data URI or URL). But conductor-ai dropped image input entirely:CohereChatModel.toCohereMessage()built the message frommessage.getText()only —getMedia()was ignored.CohereApi.ChatMessage.contentwas a bareString, so it couldn't even represent content parts.Same class of bug as #1238 (Anthropic), #1241 (Gemini), #1243 (Grok/Perplexity), #1245 (HuggingFace) — Cohere was the last provider still dropping media.
Fix
CohereApi.ChatMessage: widencontentfromStringtoObject(String orList<ContentPart>); add aContentParttype (text/image_url+ImageUrl) and aChatMessage.user(List<ContentPart>)factory.CohereChatModel.toCohereMessage(): for aUserMessagewith media, emit text + oneimage_urlcontent part per media item (base64data:URI for bytes). Text-only messages unchanged.Test
Three layers:
CohereChatModelMediaTest(mocked HTTP, always runs — CI has no provider keys): captures the outgoingChatCompletionRequestand asserts the user message serializes to a content-part list with animage_urlpart carrying the base64 data URI (plus the text part). FAILS before the fix, PASSES after.CohereAITest.IntegrationTests#testChatCompletionWithImageMedia(live API, self-skips withoutCOHERE_API_KEY): drives the adapter against Cohere's real v2 chat API with a repo-committed image embedding a machine-unguessable token (MELON7391) and asserts the vision model transcribes it — a token, unlike a color, cannot be answered correctly by luck, so a pass proves the image arrived (pattern from test(e2e): Python media-input suite (image → vision model) agentspan-ai/agentspan#301).LLMChatCompleteTests#cohere_vision_imageMediaInMessages_modelSeesTheImage(live-API e2e, self-skips withoutCOHERE_API_KEY): full server path — workflowmessages[].mediaURL →HttpDocumentLoaderdownload → adapter → live API — asserting the same token transcription. The image is committed in this repo (e2e/src/test/resources/assets/melon7391.png) and referenced at a raw URL pinned to its immutable commit SHA, so there is no third-party image host.COHERE_API_KEYis forwarded through the e2e docker-compose files like the other provider keys.Also includes
LLMChatCompleteTests#anthropic_haiku_vision_imageMediaInMessages_modelSeesTheImage— the same e2e shape through Anthropic, as a regression lock for the already-merged #1238 media fix, which shipped without an e2e test — plus its counterfactual (anthropic_haiku_vision_withoutMedia_tokenIsAbsent: same prompt, no media, token must be absent — proving the token can only come from the image). Both gated onANTHROPIC_API_KEYand verified against the live API: the positive case transcribesMELON7391, the counterfactual completes without it.The shared server pipeline for the e2e shape was verified against a live provider (Anthropic: same messages+URL input returns "Red" with image tokens counted in the prompt). The Cohere-gated tests are pending a
COHERE_API_KEY— no key is available in CI or locally today, which is why the mocked test stays as the always-running regression guard.Note (out of scope, provider-agnostic): workflow media passed as bare base64 never reaches any provider adapter —
FileSystemDocumentLoader.supports()claims any string without://and the document access policy rejects it. URL media is the supported end-to-end shape.Notes
Image support is model-dependent (use a Cohere vision model such as
command-a-vision-07-2025). This makes conductor forward the media in Cohere's documentedimage_urlformat.🤖 Generated with Claude Code