feat(http): native SSE/text streaming support for http_json targets#324
Open
placerda wants to merge 10 commits into
Open
feat(http): native SSE/text streaming support for http_json targets#324placerda wants to merge 10 commits into
placerda wants to merge 10 commits into
Conversation
Replace the long README with a short GitHub landing page focused on outcomes and the public docs. Move Foundry runtime dependencies into the default package so new users can install AgentOps without choosing a separate Foundry extra, and update generated templates and guidance accordingly. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Patch release: make 'agentops eval init' resilient to the azure.ai.agents extension 0.1.40 rename of 'azd ai agent eval init' -> 'generate' by trying 'generate' first and falling back to legacy 'init' on older extensions. See CHANGELOG [0.4.2]. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Release v0.4.2
Add response_mode (json|sse|text), an optional stream block (text_field,
done_marker, strip_leading_token), and a configurable auth header
(auth_header_name / auth_value_template) to the http_json evaluation
target so AgentOps can evaluate SSE/streaming agents such as the
gpt-rag-orchestrator /orchestrator endpoint directly.
response_mode defaults to json and reproduces the existing single
json.loads behavior byte-for-byte. Streaming reuses the same stdlib
urllib transport and 3-try backoff as the JSON path. The auth header
defaults to Authorization: Bearer {token}, preserving current behavior,
while allowing a raw shared-secret header (e.g. X-API-KEY) sourced from
an env var. A JSON-parse failure on a text/event-stream response now
suggests response_mode: sse|text.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- assert run: default to a fresh inference stage with a --cached opt-out and render the per-dimension summary as clean/total clean OK - redteam run: pass skip_upload=True to the scanner so local and CI runs do not emit Foundry blob AuthorizationFailure warnings when upload is not permitted Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…P, workflow gates Adds assert.env passthrough to the AgentOpsConfig schema, materializes a runtime ASSERT config so committed configs avoid absolute paths, resolves assert-ai in the active venv with optional AAD token and GPT-5 token shim, and wires the generated PR/dev workflows to install and run ASSERT and Red Team gates against live HTTP orchestrator targets. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Reasoning judge models (gpt-5, o1/o3/o4) are detected by model name via AZURE_OPENAI_MODEL_NAME so the evaluator switches to max_completion_tokens. Generated GitHub and Azure DevOps pipelines now forward this variable to the eval and Red Team steps. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Reconcile main (0.4.2 azd eval generate/init fallback, simplified README) with the HTTP governance work on feat. Conflict resolutions: - src/agentops/services/azd_eval_init.py: keep the smart evaluator selection (evaluator_selection) from feat and the generate/init subcommand fallback from main, invoking via _run_eval_subcommand. - CHANGELOG.md: keep 0.4.4 and 0.4.3, restore 0.4.2 from main, add 0.4.5. - README.md: keep the tutorial-linked Get started section from feat. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…defaults The _apply_http_redteam_defaults annotation referenced AgentOpsConfig without importing it, tripping ruff F821 in the release build. Add a TYPE_CHECKING-guarded import; no runtime cost. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The HTTP governance commit (77c4467) renamed the agent prompt label to 'Agent / orchestrator endpoint' and made 'my-agent:1' a placeholder that forces a re-prompt, but did not update test_run_wizard_reconfigure_forces_questions_even_when_defaults_present. With the placeholder agent and an injected prompt that always returns empty, the agent prompt looped forever, hanging the release test job. Use a non-placeholder agent value and assert the current question label. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Native SSE/text streaming for
http_jsoneval targetsLets AgentOps evaluate streaming agents (Server-Sent Events or raw text)
directly, without writing and hosting a thin non-streaming adapter. The
motivating case is the gpt-rag-orchestrator
/orchestratorendpoint, whichresponds with
text/event-streamand prefixes its first chunk with theconversation_id.Defaults are unchanged:
response_modedefaults tojsonand the existingsingle-
json.loadspath is byte-for-byte compatible.What changed
response_mode: json|sse|textonhttp_jsontargets selects the parser.streamblock for aggregation:text_field(dotted path to tokentext when each
data:line is JSON),done_marker(e.g.[DONE]),strip_leading_token(drops the leading whitespace-delimited token, e.g. theorchestrator's
conversation_id).auth_header_name(defaultAuthorization) andauth_value_template(defaultBearer {token}), so shared-secret gates suchas
X-API-KEYwork without hardcoding the secret.{token}is replaced bythe
auth_header_envvalue at run time.urllibtransport and 3-try backoff as theJSON path. A failed JSON parse on a
text/event-streamresponse now suggestssetting
response_mode: sse|text.Files
src/agentops/core/agentops_config.py:ResponseMode,StreamConfig, newfields, extended http-only validation gate.
src/agentops/pipeline/invocations.py:_http_request_stream,_strip_leading_token,_aggregate_stream, rewritten auth + streaming branchin
_invoke_http_json.tests/unit/test_http_streaming.py(new),tests/unit/test_agentops_config.py(gate tests).
CHANGELOG.md,AGENTS.md, and theagentops-evalSKILL (plugin + template).Validation
Full unit suite: 991 passed, 1 skipped (14 net-new tests, no regressions).
Note on base branch
This PR is based on the
v0.4.3line (simplify-readme-foundry-default), whichis the latest released code.
origin/mainandorigin/developare still behindv0.4.3, so this PR targets that branch to keep the diff to the streaming change
only. The
v0.4.4tag and GitHub release are cut from the head of this branch.