Ruby: Convert CodeQL .qlref tests to inline expectations

[Copilot]

• Audit remaining Ruby .qlref tests and classify each as convertable or skip with reason
• Convert standard Ruby problem/path-problem tests to inline expectations with minimal source and .qlref edits
• Preserve existing .expected files and existing postprocessors, and keep a final skip list for unconverted tests
• Run focused validation and static checks, then report results and risks

Original prompt

Follow these instructions for Ruby

qlref conversion instructions

Here’s the repeatable recipe for converting a CodeQL language’s .qlref tests to inline expectation tests.

Overall Workflow

1. cd into the language folder.

2. Run grep -rL "InlineExpectationsTestQuery" --include="*.qlref".

3. Ignore generated .actual files, .testproj copies, and other generated artifacts.

4. For each .qlref, resolve the referenced query and read its metadata.

5. Classify each test before editing:

  - convert now for standard @kind problem and @kind path-problem

  - skip for anything else, with a short concrete reason

6. Inspect the existing .expected file before converting:

  - If it is empty and the query is a standard problem or path-problem query, usually convert by adding the postprocessor only. No source comments are needed.

  - If it is non-empty, add inline expectation comments to the source files at the expected result locations, then add the postprocessor to the .qlref.

7. Keep the existing .expected file. Do not edit generated expected files by hand, even when converting.

8. Edit only source files and .qlref files.

9. Maintain a skip list for tests not converted, with exact paths and short reasons.

10. Do not make a PR.

.qlref Formatting

For a single postprocessor, use the compact form:

query: path/to/Query.ql

postprocess: utils/test/InlineExpectationsTestQuery.ql

If the .qlref already has other postprocessors, use list form:

query: path/to/Query.ql

postprocess:

  - utils/test/PrettyPrintModels.ql

  - utils/test/InlineExpectationsTestQuery.ql

If the original .qlref was a plain single-line query reference, convert it to query: form only when adding postprocess:.

Always preserve existing postprocessors.

Expectation Comments

Use the language’s supported inline comment syntax from its inline expectation implementation. Typical tags are:

$ Alert

$ Source

$ Sink

For example, in Java or Kotlin:

sink(userInput); // $ Alert

For problem queries, add Alert at each expected result location.

For path-problem queries, add:

• Alert at the alert location.

• Source at the source location, unless the source is on the same line as the alert.

• Sink at the sink location when it is a distinct relevant location.

Multi-line result locations should be annotated on the end line.

If more than one result is expected on the same line for the same query, just use a single tag in the comment, for example:

foo(bar); // $ Alert

Only add comments to real source files that belong to the test. Do not annotate generated copies, shared vendored fixtures, or shared stubs unless that is clearly the right ownership boundary for the test.

Empty .expected Files

If a problem or path-problem test already has an empty .expected file, the default conversion is:

1. Leave .expected untouched.

2. Add postprocess: utils/test/InlineExpectationsTestQuery.ql to the .qlref.

3. Add no source comments unless the test actually has expected results that are currently represented elsewhere.

This is common in zero-result tests.

Query Ids

Do not add [query-id] by default.

Only qualify tags when needed, usually when more than one inline expectation test in the same effective test folder or database can see the same source comments. Then use the query’s metadata id:

response.write(x); // $ Alert[java/xss]

Apply this to Alert, Source, and Sink only for the comments that need disambiguation.

If you add query ids in a shared folder, validate the owner queries for those comments as well, not just the query you were editing.

Good Skip Reasons

Skip, or revert to non-inline, when conversion would make the test misleading, brittle, or noisy. Common reasons:

• The query is not @kind problem or @kind path-problem.

• The query is a telemetry, metrics, table, definitions, diagnostic, extractor-information, stub-generation, or tool-specific query.

• The test is an AST dump or similar structural output test, such as PrintAst.

• The selected result text is inside a string literal, raw string, heredoc, text block, XML literal, or multiline literal and adding a comment changes the selected text.

• The expected location is inside documentation syntax where adding a comment changes what the query reports, such as Javadoc tags.

• The .expected output is nonstandard enough that mechanical conversion is risky.

• Results are in shared stubs, generated copies, vendored files, or synthetic/test-project paths where inline comments would pollute shared fixtures.

• The alert, source, or sink locations are too dense or ambiguous to make the source understandable.

• The language’s inline expectation implementation does not support the source file or comment style needed.

• The query uses a special result kind or framework behavior, such as alert-s...

Created from VS Code.

[Copilot]

Pull request overview

This PR migrates a large set of Ruby CodeQL query tests from .expected-driven validation to inline expectation validation by adding the InlineExpectationsTestQuery postprocessor to .qlref files and annotating Ruby test fixtures with $ Alert / $ Source (and where needed, query-id qualified) markers.

Changes:

• Converted many Ruby .qlref tests to run utils/test/InlineExpectationsTestQuery.ql as a postprocessor.
• Added inline expectation markers across Ruby security/performance/experimental test fixtures to mirror prior .expected results.
• Updated one .expected file in a way that captures postprocess failures (this should be avoided; see PR comments).

Show a summary per file

File	Description
ruby/ql/test/query-tests/security/decompression-api/DecompressionApi.qlref	Add inline-expectations postprocess.
ruby/ql/test/query-tests/security/decompression-api/decompression_api.rb	Add inline expectation markers.
ruby/ql/test/query-tests/security/cwe-918/ServerSideRequestForgery.rb	Add inline expectation markers.
ruby/ql/test/query-tests/security/cwe-918/ServerSideRequestForgery.qlref	Add inline-expectations postprocess.
ruby/ql/test/query-tests/security/cwe-915/test.rb	Add inline expectation markers.
ruby/ql/test/query-tests/security/cwe-915/MassAssignment.qlref	Add inline-expectations postprocess.
ruby/ql/test/query-tests/security/cwe-912/HttpToFileAccess.qlref	Add inline-expectations postprocess.
ruby/ql/test/query-tests/security/cwe-912/http_to_file_access.rb	Add inline expectation markers.
ruby/ql/test/query-tests/security/cwe-807-user-controlled-bypass/ConditionalBypass.rb	Add inline expectation markers.
ruby/ql/test/query-tests/security/cwe-807-user-controlled-bypass/ConditionalBypass.qlref	Add inline-expectations postprocess.
ruby/ql/test/query-tests/security/cwe-798/HardcodedCredentials.rb	Add inline expectation markers.
ruby/ql/test/query-tests/security/cwe-798/HardcodedCredentials.qlref	Add inline-expectations postprocess.
ruby/ql/test/query-tests/security/cwe-732/WeakFilePermissions.qlref	Add inline-expectations postprocess.
ruby/ql/test/query-tests/security/cwe-732/WeakCookieConfiguration.qlref	Add inline-expectations postprocess.
ruby/ql/test/query-tests/security/cwe-732/FilePermissions.rb	Add inline expectation markers (with query-id qualification).
ruby/ql/test/query-tests/security/cwe-732/app/config/application.rb	Add inline expectation markers (with query-id qualification).
ruby/ql/test/query-tests/security/cwe-611/xxe/Xxe.qlref	Add inline-expectations postprocess.
ruby/ql/test/query-tests/security/cwe-611/xxe/Nokogiri.rb	Add inline expectation markers.
ruby/ql/test/query-tests/security/cwe-611/xxe/LibXmlRuby.rb	Add inline expectation markers.
ruby/ql/test/query-tests/security/cwe-611/libxml-backend/Xxe.qlref	Add inline-expectations postprocess.
ruby/ql/test/query-tests/security/cwe-611/libxml-backend/LibXmlBackend.rb	Add inline expectation markers.
ruby/ql/test/query-tests/security/cwe-601/UrlRedirect.rb	Add inline expectation markers.
ruby/ql/test/query-tests/security/cwe-601/UrlRedirect.qlref	Add inline-expectations postprocess.
ruby/ql/test/query-tests/security/cwe-598/SensitiveGetQuery.qlref	Add inline-expectations postprocess.
ruby/ql/test/query-tests/security/cwe-598/app/controllers/users_controller.rb	Add inline expectation markers.
ruby/ql/test/query-tests/security/cwe-506/tst.rb	Add inline expectation markers.
ruby/ql/test/query-tests/security/cwe-506/HardcodedDataInterpretedAsCode.qlref	Add inline-expectations postprocess.
ruby/ql/test/query-tests/security/cwe-502/unsafe-deserialization/UnsafeDeserialization.rb	Add inline expectation markers.
ruby/ql/test/query-tests/security/cwe-502/unsafe-deserialization/UnsafeDeserialization.qlref	Add inline-expectations postprocess.
ruby/ql/test/query-tests/security/cwe-502/ox-global-options/UnsafeDeserialization.qlref	Add inline-expectations postprocess.
ruby/ql/test/query-tests/security/cwe-502/ox-global-options/OxGlobalOptions.rb	Add inline expectation markers.
ruby/ql/test/query-tests/security/cwe-502/oj-global-options/UnsafeDeserialization.qlref	Add inline-expectations postprocess.
ruby/ql/test/query-tests/security/cwe-502/oj-global-options/OjGlobalOptions.rb	Add inline expectation markers.
ruby/ql/test/query-tests/security/cwe-352/railsapp/config/environments/production.rb	Add inline expectation markers (with query-id qualification).
ruby/ql/test/query-tests/security/cwe-352/railsapp/config/environments/development.rb	Add inline expectation markers (with query-id qualification).
ruby/ql/test/query-tests/security/cwe-352/railsapp/config/application.rb	Add inline expectation markers (with query-id qualification).
ruby/ql/test/query-tests/security/cwe-352/railsapp/app/controllers/users_controller.rb	Add inline expectation markers (with query-id qualification).
ruby/ql/test/query-tests/security/cwe-352/railsapp/app/controllers/application_controller.rb	Add inline expectation markers (with query-id qualification).
ruby/ql/test/query-tests/security/cwe-352/railsapp/app/controllers/alternative_root_controller.rb	Add inline expectation markers (with query-id qualification).
ruby/ql/test/query-tests/security/cwe-352/CSRFProtectionNotEnabled.qlref	Add inline-expectations postprocess.
ruby/ql/test/query-tests/security/cwe-352/CSRFProtectionDisabled.qlref	Add inline-expectations postprocess.
ruby/ql/test/query-tests/security/cwe-327/WeakSensitiveDataHashing.qlref	Add inline-expectations postprocess.
ruby/ql/test/query-tests/security/cwe-327/weak_hashing.rb	Add inline expectation markers (with query-id qualification).
ruby/ql/test/query-tests/security/cwe-327/BrokenCryptoAlgorithm.qlref	Add inline-expectations postprocess.
ruby/ql/test/query-tests/security/cwe-327/broken_crypto.rb	Add inline expectation markers (with query-id qualification).
ruby/ql/test/query-tests/security/cwe-312/logging.rb	Add inline expectation markers (with query-id qualification).
ruby/ql/test/query-tests/security/cwe-312/CleartextStorage.qlref	Add inline-expectations postprocess.
ruby/ql/test/query-tests/security/cwe-312/CleartextLogging.qlref	Add inline-expectations postprocess.
ruby/ql/test/query-tests/security/cwe-312/app/models/user.rb	Add inline expectation markers (with query-id qualification).
ruby/ql/test/query-tests/security/cwe-312/app/controllers/users_controller.rb	Add inline expectation markers (with query-id qualification).
ruby/ql/test/query-tests/security/cwe-295/Typhoeus.rb	Add inline expectation markers.
ruby/ql/test/query-tests/security/cwe-295/RestClient.rb	Add inline expectation markers.
ruby/ql/test/query-tests/security/cwe-295/RequestWithoutValidation.qlref	Add inline-expectations postprocess.
ruby/ql/test/query-tests/security/cwe-295/OpenURI.rb	Add inline expectation markers.
ruby/ql/test/query-tests/security/cwe-295/NetHttp.rb	Add inline expectation markers.
ruby/ql/test/query-tests/security/cwe-295/HttpClient.rb	Add inline expectation markers.
ruby/ql/test/query-tests/security/cwe-295/Httparty.rb	Add inline expectation markers.
ruby/ql/test/query-tests/security/cwe-295/Faraday.rb	Add inline expectation markers.
ruby/ql/test/query-tests/security/cwe-295/Excon.rb	Add inline expectation markers.
ruby/ql/test/query-tests/security/cwe-209/StackTraceExposure.rb	Add inline expectation markers.
ruby/ql/test/query-tests/security/cwe-209/StackTraceExposure.qlref	Add inline-expectations postprocess.
ruby/ql/test/query-tests/security/cwe-134/TaintedFormatString.qlref	Add inline-expectations postprocess.
ruby/ql/test/query-tests/security/cwe-134/tainted_format_string.rb	Add inline expectation markers.
ruby/ql/test/query-tests/security/cwe-1333-regexp-injection/RegExpInjection.rb	Add inline expectation markers.
ruby/ql/test/query-tests/security/cwe-1333-regexp-injection/RegExpInjection.qlref	Add inline-expectations postprocess.
ruby/ql/test/query-tests/security/cwe-1333-polynomial-redos/PolynomialReDoS.rb	Add inline expectation markers.
ruby/ql/test/query-tests/security/cwe-1333-polynomial-redos/PolynomialReDoS.qlref	Add inline-expectations postprocess.
ruby/ql/test/query-tests/security/cwe-1333-polynomial-redos/lib/index.rb	Add inline expectation markers.
ruby/ql/test/query-tests/security/cwe-1333-exponential-redos/tst.rb	Add inline expectation markers.
ruby/ql/test/query-tests/security/cwe-1333-exponential-redos/ReDoS.qlref	Add inline-expectations postprocess.
ruby/ql/test/query-tests/security/cwe-117/LogInjection.qlref	Add inline-expectations postprocess.
ruby/ql/test/query-tests/security/cwe-117/app/controllers/users_controller.rb	Add inline expectation markers.
ruby/ql/test/query-tests/security/cwe-116/IncompleteSanitization/tst.rb	Add inline expectation markers.
ruby/ql/test/query-tests/security/cwe-116/IncompleteSanitization/IncompleteSanitization.qlref	Add inline-expectations postprocess.
ruby/ql/test/query-tests/security/cwe-116/BadTagFilter/test.rb	Add inline expectation markers.
ruby/ql/test/query-tests/security/cwe-116/BadTagFilter/BadTagFilter.qlref	Add inline-expectations postprocess.
ruby/ql/test/query-tests/security/cwe-094/UnsafeCodeConstruction/UnsafeCodeConstruction.qlref	Add inline-expectations postprocess (needs reconsideration for this test).
ruby/ql/test/query-tests/security/cwe-094/UnsafeCodeConstruction/UnsafeCodeConstruction.expected	Contains generated postprocess failure output (should not be committed).
ruby/ql/test/query-tests/security/cwe-094/UnsafeCodeConstruction/impl/unsafeCode.rb	Add inline expectation markers; includes heredoc case that can’t be annotated.
ruby/ql/test/query-tests/security/cwe-089/SqlInjection.qlref	Add inline-expectations postprocess.
ruby/ql/test/query-tests/security/cwe-089/PgInjection.rb	Add inline expectation markers.
ruby/ql/test/query-tests/security/cwe-089/ArelInjection.rb	Add inline expectation markers.
ruby/ql/test/query-tests/security/cwe-089/ActiveRecordInjection.rb	Add inline expectation markers (one marker needs syntax fix).
ruby/ql/test/query-tests/security/cwe-078/UnsafeShellCommandConstruction/UnsafeShellCommandConstruction.qlref	Add inline-expectations postprocess.
ruby/ql/test/query-tests/security/cwe-078/UnsafeShellCommandConstruction/impl/unsafeShell.rb	Add inline expectation markers.
ruby/ql/test/query-tests/security/cwe-078/UnsafeShellCommandConstruction/impl/sub/other2.rb	Add inline expectation markers.
ruby/ql/test/query-tests/security/cwe-078/UnsafeShellCommandConstruction/impl/sub/other.rb	Add inline expectation markers.
ruby/ql/test/query-tests/security/cwe-078/UnsafeShellCommandConstruction/impl/sub/notImported.rb	Add inline expectation markers.
ruby/ql/test/query-tests/security/cwe-078/NonConstantKernelOpen/NonConstantKernelOpen.rb	Add inline expectation markers.
ruby/ql/test/query-tests/security/cwe-078/NonConstantKernelOpen/NonConstantKernelOpen.qlref	Add inline-expectations postprocess.
ruby/ql/test/query-tests/security/cwe-078/KernelOpen/KernelOpen.rb	Add inline expectation markers.
ruby/ql/test/query-tests/security/cwe-078/KernelOpen/KernelOpen.qlref	Add inline-expectations postprocess.
ruby/ql/test/query-tests/security/cwe-020/SuspiciousRegexpRange/suspicous_regexp_range.rb	Add inline expectation markers.
ruby/ql/test/query-tests/security/cwe-020/SuspiciousRegexpRange/OverlyLargeRangeQuery.qlref	Add inline-expectations postprocess.
ruby/ql/test/query-tests/security/cwe-020/MissingRegExpAnchor/MissingRegExpAnchor.qlref	Add inline-expectations postprocess.
ruby/ql/test/query-tests/security/cwe-020/MissingRegExpAnchor/missing_regexp_anchor.rb	Add inline expectation markers.
ruby/ql/test/query-tests/security/cwe-020/MissingFullAnchor/MissingFullAnchor.qlref	Add inline-expectations postprocess.
ruby/ql/test/query-tests/security/cwe-020/MissingFullAnchor/impl/miss-anchor.rb	Add inline expectation markers.
ruby/ql/test/query-tests/security/cwe-020/IncompleteUrlSubstringSanitization/tst-IncompleteUrlSubstringSanitization.rb	Add inline expectation markers.
ruby/ql/test/query-tests/security/cwe-020/IncompleteUrlSubstringSanitization/IncompleteUrlSubstringSanitization.qlref	Add inline-expectations postprocess.
ruby/ql/test/query-tests/security/cwe-020/IncompleteHostnameRegExp/tst-IncompleteHostnameRegExp.rb	Add inline expectation markers.
ruby/ql/test/query-tests/security/cwe-020/IncompleteHostnameRegExp/IncompleteHostnameRegExp.qlref	Add inline-expectations postprocess.
ruby/ql/test/query-tests/security/cwe-020/IncompleteHostnameRegExp/hosttest.rb	Add inline expectation markers.
ruby/ql/test/query-tests/performance/UseDetect/UseDetect.rb	Add inline expectation markers.
ruby/ql/test/query-tests/performance/UseDetect/UseDetect.qlref	Add inline-expectations postprocess.
ruby/ql/test/query-tests/experimental/XPathInjection/XPathInjection.qlref	Add inline-expectations postprocess.
ruby/ql/test/query-tests/experimental/XPathInjection/RexmlInjection.rb	Add inline expectation markers.
ruby/ql/test/query-tests/experimental/XPathInjection/NokogiriInjection.rb	Add inline expectation markers.
ruby/ql/test/query-tests/experimental/XPathInjection/LibxmlInjection.rb	Add inline expectation markers.
ruby/ql/test/query-tests/experimental/weak-params/WeakParams.rb	Add inline expectation markers.
ruby/ql/test/query-tests/experimental/weak-params/WeakParams.qlref	Add inline-expectations postprocess.
ruby/ql/test/query-tests/experimental/TemplateInjection/TemplateInjection.qlref	Add inline-expectations postprocess.
ruby/ql/test/query-tests/experimental/TemplateInjection/SlimInjection.rb	Add inline expectation markers.
ruby/ql/test/query-tests/experimental/TemplateInjection/ErbInjection.rb	Add inline expectation markers.
ruby/ql/test/query-tests/experimental/manually-check-http-verb/ManuallyCheckHttpVerb.rb	Add inline expectation markers.
ruby/ql/test/query-tests/experimental/manually-check-http-verb/ManuallyCheckHttpVerb.qlref	Add inline-expectations postprocess.
ruby/ql/test/query-tests/experimental/LdapInjection/LdapInjection.rb	Add inline expectation markers.
ruby/ql/test/query-tests/experimental/LdapInjection/Ldapinjection.qlref	Add inline-expectations postprocess.
ruby/ql/test/query-tests/experimental/InsecureRandomness/InsecureRandomness.rb	Add inline expectation markers.
ruby/ql/test/query-tests/experimental/InsecureRandomness/InsecureRandomness.qlref	Add inline-expectations postprocess.
ruby/ql/test/query-tests/experimental/ImproperLdapAuth/ImproperLdapAuth.rb	Add inline expectation markers.
ruby/ql/test/query-tests/experimental/ImproperLdapAuth/ImproperLdapAuth.qlref	Add inline-expectations postprocess.
ruby/ql/test/query-tests/experimental/CWE-522-DecompressionBombs/zipBombs.rb	Add inline expectation markers.
ruby/ql/test/query-tests/experimental/CWE-522-DecompressionBombs/gzipBombs.rb	Add inline expectation markers.
ruby/ql/test/query-tests/experimental/CWE-522-DecompressionBombs/DecompressionBombs.qlref	Add inline-expectations postprocess.
ruby/ql/test/query-tests/experimental/cwe-502/UnsafeYamlDeserialization.rb	Add inline expectation markers.
ruby/ql/test/query-tests/experimental/cwe-502/UnsafeYamlDeserialization.qlref	Add inline-expectations postprocess.
ruby/ql/test/query-tests/experimental/cwe-347/MissingJWTVerification.rb	Add inline expectation markers (with query-id qualification).
ruby/ql/test/query-tests/experimental/cwe-347/MissingJWTVerification.qlref	Add inline-expectations postprocess.
ruby/ql/test/query-tests/experimental/cwe-347/EmptyJWTSecret.rb	Add inline expectation markers (with query-id qualification).
ruby/ql/test/query-tests/experimental/cwe-347/EmptyJWTSecret.qlref	Add inline-expectations postprocess.
ruby/ql/test/query-tests/experimental/cwe-176/UnicodeBypassValidation.qlref	Add inline-expectations postprocess.
ruby/ql/test/query-tests/experimental/cwe-176/unicode_normalization.rb	Add inline expectation markers.
ruby/ql/test/query-tests/experimental/cwe-022-ZipSlip/ZipSlip.qlref	Add inline-expectations postprocess.
ruby/ql/test/query-tests/experimental/cwe-022-ZipSlip/zip_slip.rb	Add inline expectation markers.

Copilot's findings

• Files reviewed: 136/136 changed files
• Comments generated: 3

[hvitved]

LGTM, but I pushed a small improvement to get rid of the testFailure in UnsafeCodeConstruction.expected.

[owen-mc]

@hvitved Thanks for that code to allow some alerts to be ignored for inline expectation tests. It made two other tests fail where there were inline expectations comments in a heredoc, so I was using copilot to explore if we could make it ignore fewer cases, and then it pointed out that we actually could use the same trick used in one of them (add #{# $ Alert\n} at the end of the line with the alert) to fix the test I thought couldn't be fixed. So I've updated the commit "Work around problem with comments in heredocs" and undone your changes, as they aren't needed in this case.