Migration Pilot: Test backfill pipeline on 50 works

Summary

Validates the backfill pipeline end-to-end on a small sample before scaling to production. Tests Phase 1 (dump scan) and Phase 2 (dry-run tag assignment) for both genres and subgenres.

Linked to: Issue #14 (Processing Phase: Schema Update + Batch Backfill for Genre Tags)

What Was Tested

Phase 1: Scanning dumps for matched keys

Scanned the full Open Library works dump (40,792,262 works) to find works whose subjects match our tag mappings:

a). Ran the scan on dump to find how many works were correctly matched genres

python scripts/backfill_tags.py --dump ol_dump_works_latest.txt.gz --type genres > work_keys.txt

output:
Genres: 869,461 works matched

b). Ran the scan on dump to find how many works were correctly matched subgenres

python scripts/backfill_tags.py --dump ol_dump_works_latest.txt.gz --type subgenres > subgenre_work_keys.txt

output:
Subgenres: 51,526 works m**atched**

Phase 2: Backfilling tag keys (Dry run)

Extracted 50 works from each Phase 1 output to test tag assignment without writing to production:

head -50 work_keys.txt > test_folder/pilot_keys.txt
head -50 subgenre_work_keys.txt > test_folder/pilot_subgenre_keys.txt

Why 50 works? The full scan matched 869K+ genres and 51K+ subgenres. Processing all of them via API would take hours and isn't necessary for validation. 50 works provides enough samples to verify correctness while keeping runtime manageable.

a). Ran dry-run on 50 genre candidate works to verify tag assignment

python scripts/backfill_tags.py --keys test_folder/pilot_keys.txt --type genres --dry-run

output:

Genres: 50/50 works matched with tag keys (100% match rate - last terminal output)

b). Ran dry-run on 50 subgenre candidate works to verify tag assignment

python scripts/backfill_tags.py --keys test_folder/pilot_subgenre_keys.txt --type subgenres --dry-run

output:

Subgenres: 50/50 works matched with tag keys (100% match rate - last terminal output)

Verification

Manually verified 3 genre works and 2 subgenre works on Open Library to confirm correct tag assignments.

Genre Works Verified

JSON API checks:
https://openlibrary.org/works/OL11322801W.json
https://openlibrary.org/works/OL11327358W.json
https://openlibrary.org/works/OL111086W.json

Tag verification:
https://openlibrary.org/tags/OL174T
https://openlibrary.org/tags/OL174T
https://openlibrary.org/tags/OL170T/Historical

Then , confirmed each tag key maps to the correct genre name in tag_types/genres/vocabulary.json.

Subgenre Works Verified

JSON API checks:
https://openlibrary.org/works/OL1942303W.json

https://openlibrary.org/works/OL17101607W.json

Tag verification:
https://openlibrary.org/tags/OL273T

Confirmed tag key maps to the correct subgenre name in tag_types/subgenres/vocabulary.json.

Errors Encountered and Fixes

Error 1: Field Index Mismatch in Dump Parsing

Error: AttributeError: 'int' object has no attribute 'get'

Cause: The Open Library works dump has 5 tab-separated fields, not 3 as originally assumed:

field[0]: /type/work
field[1]: /works/OL10000159W
field[2]: 3                    ← revision number (int)
field[3]: 2010-04-28T06:54:19  ← timestamp
field[4]: {"title": "..."}     ← JSON blob

The script was parsing parts[2] (the revision number) instead of parts[4] (the JSON).

Fix: Changed json.loads(parts[2]) → json.loads(parts[4]) on line 62 of scripts/backfill_tags.py.

Error 2: Type Filter Not Applied in Phase 1 Scan

Symptom: Both work_keys.txt and subgenre_work_keys.txt contained the exact same 50 works, despite running with --type genres and --type subgenres respectively.

Cause: scan_dump_for_matched_keys() accepted a tag_type parameter but never used it. It called migrator.classify_subject(s) which checks all tag types, so both scans matched the same works.

Fix: Added a type filter check after classify_subject():

result = migrator.classify_subject(s)
if result and result[0] == tag_type:
    print(parts[1].strip())
    matched += 1
    break

After the fix:

Genres: 869,461 works (down from 3.25M before fix)
Subgenres: 51,526 works (properly filtered)

Files Changed

scripts/backfill_tags.py — Fixed field index and added type filter to Phase 1 scan

Reviewers

@mekarpeles