feat: Flexprice integration

[TEJASNARAYANS]

What Changed?

Adding flexprice changes

Why?

Explain the problem this solves and why this approach was chosen.

How to Test?

List clear steps for reviewers to verify the change.

Release Label

Select one semantic version bump intent for this PR:

• major - breaking change, next release bumps major version
• minor - backward-compatible feature, next release bumps minor version
• fix - backward-compatible bug fix, next release bumps patch version
• No label (defaults to patch release)

If you do not have permission to apply labels, mention the intended release label here and a maintainer will set it.

Checklist

• I have read the CONTRIBUTING.md guide.
• My code follows the project's style guidelines.
• I have added tests that prove my fix is effective or my feature works.
• I have updated documentation where needed.

[greptile-apps]

Greptile Summary

This PR adds Flexprice usage metering across the application. The main changes are:

• New Flexprice service helpers and configuration.
• Billing events added to API routes and background workers.
• Call-import evaluation billing watermark storage.
• Meter setup and cleanup scripts.
• Tests for Flexprice helpers and selected billing flows.

Confidence Score: 4/5

This should be fixed before merging.

• Partial evaluator dispatches can bill more runs than were queued.
• Playground evaluation events can still report zero metrics when metrics actually run.
• The issues affect usage data sent to billing and analytics.

app/api/v1/routes/evaluators.py, app/api/v1/routes/playground.py

Important Files Changed

Filename	Overview
app/api/v1/routes/evaluators.py	Adds requested-run billing for evaluator dispatches, but the emitted quantity can exceed the number of queued tasks.
app/api/v1/routes/playground.py	Adds playground billing events, but evaluation metric counts are still hardcoded to zero on the changed paths.
app/services/billing/flexprice_service.py	Introduces Flexprice event wrappers and shared event ingestion behavior.
app/workers/tasks/evaluate_call_import_row.py	Adds call-import evaluation delta billing based on completed-row watermarks.

_{Reviews (3): Last reviewed commit: "feat: updating logs" | Re-trigger Greptile}

[greptile-apps]

Failed Dispatches Are Billed

The dispatch loop catches per-evaluator queue failures and continues, but this event bills len(request.evaluator_ids) instead of the number of tasks that were actually queued. A request where one evaluator queues and another raises in run_evaluator_task.delay(...) returns success with one task while recording usage for both requested evaluators.

Suggested change

	background_tasks.add_task(
	record_evaluator_run_requested,
	organization_id,
	uuid4(),
	workspace_id=workspace_id,
	quantity=len(request.evaluator_ids),
	)
	background_tasks.add_task(
	record_evaluator_run_requested,
	organization_id,
	uuid4(),
	workspace_id=workspace_id,
	quantity=len(task_ids),
	)

[greptile-apps]

Metric Count Always Zero

This records every custom websocket evaluation with metric_count=0 even though the worker evaluates the configured metrics for the call. Any downstream meter or report that uses this property will under-report these evaluations as having no metrics.

[greptile-apps]

Metric Count Always Zero

This records every re-evaluation with metric_count=0 after queueing the evaluation worker. If the billing catalog or analytics uses this property to track evaluated metrics, re-evaluated calls are reported as zero-metric runs even when metrics are actually executed.

[greptile-apps]

Partial Dispatch Overbills When one evaluator queues successfully and a later evaluator fails inside the dispatch loop, the endpoint still returns success because task_ids is non-empty. This line then records the full requested count, so Flexprice bills evaluators that were never queued and can never produce results. Use the count of successfully queued tasks for this event.

Suggested change

	quantity=len(request.evaluator_ids),
	quantity=len(task_ids),

[greptile-apps]

Metric Count Dropped This records every custom websocket evaluation with metric_count=0, but the worker can evaluate the organization’s enabled agent metrics for the call. Since the later completion event does not include the actual count, any Flexprice meter or analytics using this property will report these evaluated calls as zero-metric runs. Move the metered event to the worker where the metric count is known, or compute the same selected metric count before enqueueing.

[greptile-apps]

Reevaluation Count Dropped Re-evaluations still send metric_count=0 immediately after queueing the worker. A re-evaluated call can run the enabled agent metrics, and no later event sends the real count, so these runs are reported as zero-metric evaluations. Emit this after metric execution or pass the actual selected metric count into the billing call.