Bug: Final Score Collision — Multiple Candidates Share Identical Scores

[gurunesh30]

Bug: Final Score Collision — Multiple Candidates Share Identical Scores

Labels: bug ranking submission-compliance
Priority: High
Component: backend/main.py → _evaluate_rankings_streaming

---

Summary

The ranking pipeline produces multiple candidates with identical final_score values in the output. This violates the submission constraint in validate_submission.py which requires scores to be strictly non-increasing across ranks 1–100, and causes the validator to flag tie-break ordering errors.

---

Steps to Reproduce

1. Start the backend with a candidate pool of ~1000+ entries
2. Upload a job description and trigger a ranking run via POST /api/rank/start
3. Download the exported team_viltrumites.csv
4. Run the official validator:

python validate_submission.py team_viltrumites.csv

Expected: Submission is valid.
Actual:

Validation failed (N issue(s)):
- score must be non-increasing by rank: rank 12 (67.4) < rank 13 (67.4).
- Equal scores at ranks 12 and 13: tie-break requires candidate_id ascending ...

---

Root Cause

The composite score formula (S1 × 0.60) + (S2 × 0.20) + (S3 × 0.20) collapses many distinct candidate inputs into the same output value. This happens for three reasons:

1. Stage 2 (Qwen STAR) returns integers only
The LLM is constrained to output a single integer (0–100). With 150 candidates going through Stage 2, many receive the same integer score (e.g. 50, 60, 70), which becomes a dominant 20% weight in the composite.

2. Stage 3 (telemetry) has discrete buckets
Notice period scoring is a step function: ≤30 days → 50 pts, ≤60 days → 40 pts, else → 20 pts. Combined with binary flags (open_to_work, verified_email), Stage 3 only produces a small set of distinct values.

3. Stage 1 (semantic) can also produce ties
Candidates with identical or highly similar skill sets against the same JD receive identical cosine similarity scores from the C++ ranker.

When all three stages independently produce the same values for different candidates, the weighted composite is identical — and round(..., 2) further collapses scores by truncating any sub-cent differences.

Previous fix attempt that failed:
A seen_scores dict was used to detect and nudge duplicate scores by +0.001. This failed because:

• composite_arr is a NumPy array — in-place mutations weren't reflected in already-computed round() lookups against the dict
• A third candidate with the same raw score would look up the original key, find offset 0.001, and nudge to the same value as the second candidate — creating a new collision
• Cascading collisions were never resolved

---

Fix Implemented

File: backend/main.py — _evaluate_rankings_streaming, Step 5

Replaced the broken seen-set approach with a deterministic position-based offset:

# Sort all candidates by (-score, candidate_id) to establish stable order
scored_items = [
    (float(composite_arr[i]), candidate_ids[i], i)
    for i in range(n_cands)
]
scored_items.sort(key=lambda x: (-x[0], x[1]))

# Each position gets score - (position * 0.001)
# Position 0 → unchanged, position 1 → score - 0.001, ...
EPSILON = 0.001
unique_composite = np.copy(composite_arr)
for position, (_, _, orig_idx) in enumerate(scored_items):
    raw = float(composite_arr[orig_idx])
    adjusted = max(round(raw - position * EPSILON, 4), 0.0)
    unique_composite[orig_idx] = adjusted

Why this is correct:

• Every position gets a different offset → guaranteed uniqueness across all 150 candidates
• The maximum adjustment is 149 × 0.001 = 0.149 points — negligible as a signal
• Ordering is preserved: a higher raw score at position 0 always stays above a lower raw score at position 1 (the offset on position 0 is smaller)
• Equal raw scores are broken by candidate_id ascending — deterministic and reproducible
• Works correctly regardless of how many candidates share the same raw score

Maximum score delta introduced: 0.149 points out of 100
Precision of stored scores: 4 decimal places (round(..., 4))

---

Validation After Fix

The exported CSV now passes all checks in validate_submission.py:

• ✅ Exactly 100 data rows
• ✅ Header: candidate_id,rank,score,reasoning
• ✅ Rank integers 1–100, each used exactly once
• ✅ Scores strictly non-increasing by rank
• ✅ No equal-score tie-break violations
• ✅ All candidate_id values match CAND_[0-9]{7} pattern

---

Related Files

File	Change
backend/main.py	Replaced seen_scores dict with position-based epsilon offset
backend/app/exporter.py	Writes full float score (no round(..., 2) truncation)
frontend/ui/src/pages/Dashboard.jsx	Displays scores with toFixed(1) — shows float, not integer