Backward-compatibility regression tests for the renderer#15
Merged
Conversation
…rence images Restructures #7 per review: all files live under tests/pixel_renderer/regression, the test imports the rendering helpers from create_sample_images.py instead of duplicating them, and reference images are committed under assets/ so no SAMPLE_IMAGES_DIR env variable is needed. Reference images were generated on Ubuntu 24.04 (pango 1.52.1) to match CI; regenerate with: python tests/pixel_renderer/regression/create_sample_images.py Co-authored-by: İlker Kesen <ilkerksn@gmail.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: ⛔ Files ignored due to path filters (206)
📒 Files selected for processing (4)
📝 WalkthroughWalkthroughAdds tooling to generate language-specific regression PNGs, records rendering-stack versions, and introduces pixel-exact compatibility tests gated by Pango and Cairo versions. ChangesPixel rendering regression
Estimated code review effort: 3 (Moderate) | ~20 minutes 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Supersedes #7 by @ilkerkesen (all credit for the approach, the SIB-200 samples, and the original implementation goes to İlker — carried over here with review feedback applied, since we couldn't push to the fork). Related to #2.
What this does
Adds pixel-exact regression tests that re-render 5 SIB-200 sentences for each of 205 languages and compare the result against committed reference images.
Everything lives under
tests/pixel_renderer/regression/:samples.tsv— 5 example sentences per language from SIB-200create_sample_images.py— regenerates the reference images (python tests/pixel_renderer/regression/create_sample_images.py, no arguments needed)assets/— 205 committed reference PNGs (5 MB) plusversions.jsonrecording the rendering stack they were generated withtest_backward_compatibility.py— one parametrized test per languageChanges relative to #7
tests/pixel_renderer/regression/; the test imports the rendering helpers fromcreate_sample_images.pyinstead of duplicating them, and both share a singlebuild_processor()so the generator and verifier can't drift apartSAMPLE_IMAGES_DIRenv variable is gone — paths are relative to the test fileversions.jsonis load-bearing: since pixel-exact output depends on pango/cairo versions, the tests skip with an actionable "regenerate references" message (including the exact version delta) when the local rendering stack differs from the one that generated the references, instead of failing all 205 tests inscrutablyVerification
Reference images were generated on Ubuntu 24.04 (pango 1.52.1, cairo 1.18.0) to match CI. In that environment: all 205 tests pass in ~10s, regeneration is byte-identical (rendering is deterministic within a fixed stack), and simulating a pango version bump correctly skips all tests with the regenerate hint. Note that running locally on macOS will skip (Homebrew ships a different pango) — that is the drift guard working as intended.
🤖 Generated with Claude Code
Note
Low Risk
Test-only additions with no production renderer logic changes; main operational note is maintaining reference PNGs and version metadata when the rendering stack changes.
Overview
Adds pixel-exact regression coverage for
PixelRendererProcessorundertests/pixel_renderer/regression/, re-rendering five SIB-200 sentences per language and comparing against committed reference PNGs (~205 languages).create_sample_images.pyis the single rendering path: sharedbuild_processor(), TSV loading, vertical image stitching, andversions.json(pango/cairo and related stack versions).test_backward_compatibility.pyimports those helpers so generators and tests cannot diverge; it skips when local pango/cairo differ from the reference stack (with a regenerate hint) and reports size or bbox on mismatch instead of opaque failures.Also adds
samples.tsv,assets/versions.json, and the committed reference images workflow described in the PR (regenerate viacreate_sample_images.py).Reviewed by Cursor Bugbot for commit 2c5a094. Bugbot is set up for automated code reviews on this repo. Configure here.
Summary by CodeRabbit
New Features
Tests