[AIT-1146] Add first spec-derived UTS integration and proxy tests via rewritten uts-to-swift skill

[sacOO7]

• Fixes https://ably.atlassian.net/browse/AIT-1146

🎯 What this PR does

Rewrites the uts-to-swift skill (v2.0.0) to mirror ably-java's uts-to-kotlin step for step. The skill previously translated one spec file at a time with hand-maintained path tables and an eyeball-only review; it now drives the whole module-directory workflow the kotlin skill established, backed by two deterministic scripts:

/uts-to-swift <path-to-uts-module-directory>     # e.g. <cloned-ably-specification-repo-path>/uts/realtime

Built on top of the Test/UTS integration/proxy infrastructure introduced by #2223 (hence the base branch).

The PR also ships the skill's first real output: three spec-derived integration tests that replace the interim smoke tests, with all env-var gating dropped and the UTS docs brought to block-level parity with ably-java — see the spec-derived tests section below.

🧭 The new two-phase flow

Phase 1 — Selection (scripted, byte-for-byte deterministic):

1. scripts/resolve_uts.py validates the module directory, reads uts-package-mapping.json, and emits per-tier target directories plus every candidate spec with its derived Swift class name (connection_recovery_test.md → ConnectionRecoveryTests).
2. The user then confirms/creates the module mapping, picks a tier (unit / integration / proxy), picks which specs, and chooses translate-only vs translate-and-evaluate.

Phase 2 — Per-spec translation (Steps 1–7): read spec (+ per-module translation notes) → resolver-provided output path → read the tier's infra → generate → compile → run (evaluate mode only) → audit-driven review.

📦 What's new alongside SKILL.md

File	Purpose
scripts/resolve_uts.py	Module validation + tier/spec/class-name resolution; --create maintains the mapping. Minimal-diff port of kotlin's (only deltas: Tests suffix, no Kotlin package concept, cocoa wording).
scripts/audit_translation.py	Step 7 review aid: Test-ID coverage (missingInSwift / orphanInSwift from // UTS: tags) + a per-test spec-line ledger with an assertion-shortfall tripwire.
uts-package-mapping.json	Spec module → Test/UTS tier-directory mapping (realtime, rest, objects).
references/objects-mapping.md	Intentionally-empty placeholder for the future ably-js → ably-cocoa LiveObjects type map (authored when objects translation starts).

✍️ SKILL.md highlights

• Structure matches kotlin heading-for-heading — the only structural deltas are Swift necessities, each deliberate: no package concept, scoped-resource base cases instead of @BeforeAll/finally teardown, and JSON forced explicitly on the proxy tier.
• All cocoa-specific translation content retained: the import Ably.Private internals-access ladder, Captured<T>/Swift-6-Sendable rules, paired pseudocode→Swift mock examples, and per-tier file templates (unit / direct-sandbox / proxy).
• Evaluation model upgraded: every run ends in one of three legal terminal states — spec-correct pass, documented SDK deviation (env-gated/adapted), or UTS spec error failing fast (Issue.record) so the spec gets fixed at source. The accommodate-both pattern stays banned.
• Required reading is fetched, not remembered: writing-derived-tests.md up front, and uts/docs/proxy.md before any proxy-tier translation (with a local-checkout fallback while the upstream docs/ move is unmerged).
• Documents two known gaps honestly rather than papering over them: the WS mock lacks respondWithTimeout/respondWithDNSError (extend-or-record instruction), and fixture-driven specs (e.g. msgpack_interop.md) legitimately yield specCount: 0 + an expected orphan tag in the audit.

🔍 One deliberate improvement over the kotlin original

audit_translation.py skips comment-only lines when counting assertions — a commented-out // #expect(…) is not an assertion, and counting it (as kotlin's script does with a commented assertEquals) would silently mask exactly the dropped-assertion case the tripwire exists to catch. Worth upstreaming to uts-to-kotlin.

✅ Verification (initial skill commit)

• Resolver: exercised against the real spec checkout for all three modules; all five error codes (NOT_A_UTS_MODULE_PATH, DIR_NOT_FOUND, NO_TIER_DIRS, MAPPING_NOT_FOUND, BAD_TARGET_NAME) verified; --create round-trip preserves notes and LF endings.
• Audit: clean ID coverage on both existing spec/test pairs (ConnectionRecoveryTests, TimeTests — the one shortfall it flags is TimeTests' documented annotated omission); mutation tests confirm missing-test, orphan-tag, and shortfall detection; swept over the entire spec corpus (146 files) with zero extractor failures.
• Skill: heading skeleton diffs against uts-to-kotlin with only the documented deltas; every snippet API verified against the actual Test/UTS/infra signatures.
• swift test --filter UTS passes; make lint clean; the initial commit touches nothing under Source/ or Test/ (later commits in this PR do add tests under Test/UTS/ — see below).

🔁 Review follow-ups (7e383be3, 455f4ca7)

All ten review comments across two rounds are addressed in the threads; the code changes:

• Split assert/await accounting in the audit (swiftAssertionCount vs swiftAwaitCount, assertionShortfall primary / awaitShortfall advisory) — a wait surplus could previously mask deleted assertions (proven on RTN16f).
• Pseudo-only fence extraction — only ```pseudo-labelled blocks feed the spec ledger (corpus-verified output-neutral today).
• duplicateInSwift in idCoverage + the exit-2 gate — a duplicated // UTS: tag previously made a method invisible to the audit (reported, not merged; not a hard failure).
• BAD_MAPPING fail-fast for a blank/invalid testRoot (validated before --create can persist corruption) and expanduser() for ~-paths in the resolver.
• Docstring escape cleanup (Ruff W605) and username-neutral example paths (<cloned-ably-specification-repo-path>/uts/objects).
• One suggestion deliberately rejected with rationale in-thread: full lexical comment/string stripping in the audit (a mini-lexer in an explicitly advisory review aid).

🧪 First spec-derived integration tests (58fa0f5e)

The rewritten skill was then used for real (translate-and-evaluate mode) — this PR includes its first output, replacing the interim infra "smoke" tests per their TODOs:

Test	Spec	Covers
integration/standard/realtime/ChannelHistoryTests.swift	channel_history_test.md (RTL10d)	Cross-client history, newest-first, once per protocol variant (JSON + msgpack)
integration/standard/realtime/TokenRequestTests.swift	token_request_test.md (RSA9/RSA9a/RSA9g)	Locally signed TokenRequest accepted by the real server, with/without clientId
integration/proxy/realtime/AuthReauthTests.swift	auth_reauth.md (RTN22/RTC8a)	Injected server-initiated AUTH → re-auth via authCallback + client→server AUTH frame in the proxy log, connection undisrupted. macOS-only

• IntegrationSmokeTest / ProxyInfraSmokeTests removed — the spec-derived tests cover their ground end-to-end (sandbox provisioning, real clients, proxy chain, log assertions).
• All env-var gating dropped (UTS_INTEGRATION_*): every tier now runs directly from swift test and the Xcode test runner UI; tier selection is by --filter, mirroring ably-java's runUtsUnitTests/runUtsIntegrationTests split. UTS_PROXY_LOCAL_PATH (config override) and RUN_DEVIATIONS (deviation mechanism) remain.
• Test/UTS/README.md brought to block-level parity with ably-java's uts/README.md: §2 tier table with "Example in this repo" paths, §7 walkthrough restructured to numbered sub-sections, §10 rewritten around ungated per-suite runs, §11.4/§11.5 now walk the real ChannelHistoryTests/AuthReauthTests with code excerpts at the same points as ably-java's §10/§11, §11.3 control-API endpoints + common-actions list, §12 cheat-sheet build-client snippets / Test-ID format / decision-tree entries.
• SKILL.md: reference examples now point at the spec-derived suites (matching kotlin's ChannelHistoryTest reference), and generated tests are instructed to stay ungated.

Verification: all three suites pass ungated against the real sandbox (TokenRequest 2/2, ChannelHistory 2/2 JSON+msgpack, AuthReauth 1/1 incl. proxy binary sync); unit suites 11/11; swift build --build-tests and make lint clean; zero smoke/env-gate references remain in code or docs.

Summary by CodeRabbit

• New Features

  • Added deterministic tooling to resolve UTS module directories into Swift test targets.
  • Added an automated faithfulness audit that reports missing/orphan/duplicate test identifiers and assertion shortfalls.
  • Added UTS-to-Swift package mappings for realtime, objects, and REST across unit/integration/proxy tiers.

• Documentation

  • Substantially rewrote the UTS-to-Swift translation guide, including two-phase workflow, tighter naming/assertion rules, and Swift 6 mocking concurrency guidance.
  • Added an objects mapping reference placeholder.

[coderabbitai]

Walkthrough

The UTS-to-Swift workflow now resolves module directories and tiers before translation, provides expanded mocking and integration guidance, adds deterministic audit tooling, and includes new direct-sandbox and proxy reference tests.

Changes

UTS-to-Swift translation workflow

Layer / File(s)	Summary
Module resolution and package mapping .claude/skills/uts-to-swift/scripts/resolve_uts.py, .claude/skills/uts-to-swift/uts-package-mapping.json, .claude/skills/uts-to-swift/references/objects-mapping.md	Resolves module tiers, filters spec candidates, maps targets, and supports optional translation notes.
Module-driven translation workflow .claude/skills/uts-to-swift/SKILL.md	Replaces single-spec handling with resolver-based selection, tier-specific templates, evaluation guidance, and integration infrastructure rules.
Deterministic translation audit .claude/skills/uts-to-swift/scripts/audit_translation.py	Reports test ID coverage, pseudocode lines, Swift assertions, waits, duplicates, and assertion shortfalls as JSON.
Integration reference tests and documentation Test/UTS/README.md, Test/UTS/integration/...	Adds channel history, token request, and proxy re-authentication tests while replacing the former integration smoke tests and updating guidance.

Estimated code review effort: 3 (Moderate) | ~30 minutes

Sequence Diagram(s)

sequenceDiagram
  participant resolve_uts.py
  participant UTS module
  participant Swift test target
  resolve_uts.py->>UTS module: discover tiers and spec files
  resolve_uts.py->>Swift test target: resolve mapped output directories
  Swift test target-->>UTS module: translate selected specs

Loading

Possibly related PRs

• ably/ably-cocoa#2213: Introduces the initial UTS-to-Swift skill that this workflow expands.

Suggested reviewers: maratal

Poem

“Hop,” said the rabbit, “the modules align,
Audits count assertions in orderly time.
Proxy sparks messages, sandboxes gleam,
Swift tests now follow each UTS dream.
Carrots for coverage—crunch, crunch, hooray!”

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title accurately summarizes the main change: rewritten uts-to-swift work plus new spec-derived integration and proxy tests.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch uts-to-swift-module-skill

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 4

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (2)

.claude/skills/uts-to-swift/SKILL.md (2)

304-315: 🎯 Functional Correctness | 🔴 Critical | ⚡ Quick win

Fix Swift 6 concurrency violation in mock setup example.

The variable attemptCount is mutated inside a mock handler closure, which is marked @Sendable in Swift 6. This will result in a compiler error (Mutation of captured var 'attemptCount' in concurrently-executing code).

As recommended later in the document, use the thread-safe Captured<T> helper to branch logic on the attempt count instead.

🐛 Proposed fix

-var attemptCount = 0
-let wsProvider = MockWebSocketProvider(onConnectionAttempt: { connection in
-    attemptCount += 1
-    if attemptCount == 1 {
+let capturedAttempts = Captured<MockWebSocket>()
+let wsProvider = MockWebSocketProvider(onConnectionAttempt: { connection in
+    capturedAttempts.append(connection)
+    if capturedAttempts.count == 1 {
         connection.respondWithSuccess()
         connection.sendToClient(.connected(connectionId: "conn-s", connectionKey: "key-s", connectionStateTtl: 2))
     } else {
         connection.respondWithRefused()
     }
 })

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @.claude/skills/uts-to-swift/SKILL.md around lines 304 - 315, Update the mock
setup example around MockWebSocketProvider to replace the captured mutable
attemptCount variable with the thread-safe Captured<Int> helper, and use its
value/update operations inside the `@Sendable` connection handler to preserve
first-attempt success and subsequent-attempt refusal behavior.

---

576-586: 🎯 Functional Correctness | 🔴 Critical | ⚡ Quick win

Fix Swift syntax error and leaked continuation.

This snippet contains two issues:

1. The guard let date else { ... } block lacks a scope exit (e.g., return or throw), which will cause a guard body must not fall through compiler error.
2. If a return is simply added, the continuation will leak, causing the test task to hang indefinitely.

Since the documentation text above instructs to "make the test async throws", update this helper to use async throws and withCheckedThrowingContinuation to resolve both issues idiomatically.

🐛 Proposed fix

-private func awaitTime(_ rest: ARTRest, sourceLocation: SourceLocation = `#_sourceLocation`) async -> Date {
-    await withCheckedContinuation { (continuation: CheckedContinuation<Date, Never>) in
+private func awaitTime(_ rest: ARTRest, sourceLocation: SourceLocation = `#_sourceLocation`) async throws -> Date {
+    try await withCheckedThrowingContinuation { continuation in
         rest.time { date, error in
-            if let error { Issue.record("time() failed: \(error)", sourceLocation: sourceLocation) }
-            guard let date else { Issue.record("time() failed: date should not be nil", sourceLocation: sourceLocation) }
+            if let error {
+                Issue.record("time() failed: \(error)", sourceLocation: sourceLocation)
+                continuation.resume(throwing: error)
+                return
+            }
+            guard let date else {
+                Issue.record("time() failed: date should not be nil", sourceLocation: sourceLocation)
+                continuation.resume(throwing: CancellationError())
+                return
+            }
             continuation.resume(returning: date)   // resume exactly once
         }
     }
 }

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @.claude/skills/uts-to-swift/SKILL.md around lines 576 - 586, Update
awaitTime to be async throws and use withCheckedThrowingContinuation with a
CheckedContinuation<Date, Error>. In the rest.time callback, throw or resume the
continuation with the received error, throw a descriptive error when date is
nil, and resume with the date on success, ensuring every path completes exactly
once.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In @.claude/skills/uts-to-swift/scripts/audit_translation.py:
- Around line 128-139: Update the fence-tracking logic around FENCE_RE and
in_fence so only triple-backtick fences explicitly labeled pseudo open an
extraction block; ignore Swift, JSON, shell, and unlabeled fences entirely.
Preserve the existing heading, preamble, and line collection behavior for
content inside pseudo fences.
- Around line 167-172: Update the assertion extraction loop in
audit_translation.py to lexically remove Swift block comments, trailing
comments, and string literals before applying SWIFT_ASSERT_RE, rather than only
skipping comment-only lines. Count assertion and poll-helper calls exclusively
from executable Swift text while preserving the existing assertionCalls and
assertionCount output.
- Around line 59-63: Separate assertion and await detection in
audit_translation.py: update SWIFT_ASSERT_RE and the related counting logic to
track `#expect/`#require independently from awaitConnectionState,
awaitChannelState, awaitState, pollUntil, and poll helpers. Compare the
resulting Swift assertion count with specAssertCount and the Swift await count
with specAwaitCount, including the corresponding logic around the additional
reported location.
- Line 13: Remove the unnecessary backslash escapes before backticks in the
documentation text within audit_translation.py, including the Test-ID coverage
description. Keep the backticks unescaped so the file passes Ruff W605 without
changing the documented meaning.

---

Outside diff comments:
In @.claude/skills/uts-to-swift/SKILL.md:
- Around line 304-315: Update the mock setup example around
MockWebSocketProvider to replace the captured mutable attemptCount variable with
the thread-safe Captured<Int> helper, and use its value/update operations inside
the `@Sendable` connection handler to preserve first-attempt success and
subsequent-attempt refusal behavior.
- Around line 576-586: Update awaitTime to be async throws and use
withCheckedThrowingContinuation with a CheckedContinuation<Date, Error>. In the
rest.time callback, throw or resume the continuation with the received error,
throw a descriptive error when date is nil, and resume with the date on success,
ensuring every path completes exactly once.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: d55f5b3a-3f92-4430-8f4a-35e626f9625c

📥 Commits

Reviewing files that changed from the base of the PR and between 10ae406 and 539081b.

📒 Files selected for processing (5)

• .claude/skills/uts-to-swift/SKILL.md
• .claude/skills/uts-to-swift/references/objects-mapping.md
• .claude/skills/uts-to-swift/scripts/audit_translation.py
• .claude/skills/uts-to-swift/scripts/resolve_uts.py
• .claude/skills/uts-to-swift/uts-package-mapping.json

[Copilot]

Pull request overview

This PR rewrites the .claude/skills/uts-to-swift skill to operate on an entire UTS module directory (mirroring the uts-to-kotlin workflow) rather than a single spec file, and adds deterministic resolver/audit scripts to drive selection and translation review.

Changes:

• Add resolve_uts.py to validate a UTS module directory, resolve tier targets via uts-package-mapping.json, and list candidate specs with derived Swift class names.
• Add audit_translation.py to mechanically compare spec Test IDs / pseudocode ledger vs generated Swift (// UTS: tags + assertion/wait calls).
• Update SKILL.md to document the new two-phase module-directory flow and introduce module mapping + audit steps (plus a placeholder objects mapping reference).

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
.claude/skills/uts-to-swift/uts-package-mapping.json	Defines module→tier target directory mapping under Test/UTS, plus optional per-module notes reference.
.claude/skills/uts-to-swift/SKILL.md	Replaces single-spec flow with module-directory selection + per-spec translation/evaluation + audit-driven review guidance.
.claude/skills/uts-to-swift/scripts/resolve_uts.py	New deterministic resolver for module validation, tier discovery, target-dir resolution, and spec/class-name enumeration.
.claude/skills/uts-to-swift/scripts/audit_translation.py	New deterministic audit tool for Test-ID coverage and pseudocode-vs-Swift completeness tripwires.
.claude/skills/uts-to-swift/references/objects-mapping.md	Placeholder notes file for future objects (LiveObjects) mapping work.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.

[ttypic]

LGTM