Phase 2: Work Migration — Subject Classification Pipeline by Chisomnwa · Pull Request #41 · Open-Book-Genome-Project/tags

Chisomnwa · 2026-07-04T12:35:01Z

Part of the Phase 2 work for #14.

What this PR adds

Three new files that form the core of the tag migration pipeline:

tags/utils.py — slug_to_tag_key() helper
scripts/migrate_work.py — WorkMigrator class + CLI
backfill_genre_tags.py — Two-phase batch backfill script (Phase 1: dump scan, Phase 2: fetch + migrate + save)

Together they implement the mapping chain that transforms a Work's messy subject strings into clean OL Tag keys.

The mapping chain

subject "science fiction"
→ mappings.json → slug "sci-fi" (messy → canonical)
→ vocabulary.json → key "/tags/OL179T" (slug → Tag key)
→ work.genres → ["/tags/OL179T"] (to be written)

File 1: tags/utils.py

A pure function that reads a type's vocabulary.json and returns the Tag key for a given slug.

How to test:

from tags.utils import slug_to_tag_key

print(slug_to_tag_key("genres", "sci-fi"))     # /tags/OL179T
print(slug_to_tag_key("genres", "fantasy"))    # /tags/OL169T
print(slug_to_tag_key("genres", "nonexistent"))  # None

Output:

Input	Result
slug_to_tag_key("genres", "sci-fi")	/tags/OL179T
slug_to_tag_key("genres", "fantasy")	/tags/OL169T
slug_to_tag_key("genres", "mystery")	/tags/OL175T
slug_to_tag_key("genres", "nonexistent")	None

File 2: scripts/migrate_work.py

A class WorkMigrator that loads all controlled type mappings from tag_types/<name>/mappings.json, classifies every subject in a Work, converts matching slugs to Tag keys via slug_to_tag_key(), and returns a dict of typed key lists.

Also includes a CLI for testing on real OL works.

How to test:

python scripts/migrate_work.py --work OL82563W --dry-run

Output:

=== OL82563W: Harry Potter and the Philosopher's Stone ===
genres:
- /tags/OL169T # Fantasy
- /tags/OL164T # Adventure
- /tags/OL175T # Mystery
- /tags/OL179T # Sci-Fi

Subject (raw)	Mapped Type	Slug	Tag Key	Tag Name
"fantasy"	genres	fantasy	/tags/OL169T	Fantasy
"fantasy fiction"	genres	fantasy	/tags/OL169T	Fantasy
"Adventure"	genres	adventure	/tags/OL164T	Adventure
"Adventure and adventurers, fiction"	genres	adventure	/tags/OL164T	Adventure
"Mystery"	genres	mystery	/tags/OL175T	Mystery
"Science fiction & fantasy"	genres	sci-fi	/tags/OL179T	Sci-Fi

(Duplicate keys are deduplicated — multiple subjects mapping to the same Tag key only appear once in the final list.)

File 3: scripts/backfill_tags.py

A two-phase batch backfill script that finds works with genre-matching subjects and writes the corresponding Tag keys to their typed fields.

Phase 1 — Scan dump (no API calls):

python scripts/backfill_tags.py --dump ol_dump_works_latest.txt.gz --type genres > work_keys.txt

Reads the gzipped OL works dump line by line (tab-separated, 3rd field is JSON). For each work, checks its subjects against our genre mappings. If any subject matches, prints the work key (e.g. /works/OL82563W) to stdout. Just a filter — no API calls, no auth needed.

Phase 2 — Fetch, migrate, save:

python scripts/backfill_tags.py --keys work_keys.txt --type genres --dry-run

Reads work keys from Phase 1 output. For each work:

Fetches its JSON from the OL API via requests.get()
Runs WorkMigrator.migrate() to compute Tag keys
If --dry-run: prints what would change, does nothing
If not dry-run: sets work.genres = [...] and saves via ol.save_many() in batches of 50

CLI reference:

Argument	Purpose
--dump	Phase 1: path to gzipped OL works dump
--keys	Phase 2: path to work keys file (one per line)
--type genres	Which tag type to backfill (default: genres, future: subgenres)
--dry-run	Phase 2: preview changes without writing

Test output:

$ python scripts/backfill_tags.py --keys test_keys.txt --type genres --dry-run

/works/OL82563W: genres = ['/tags/OL169T', '/tags/OL164T', '/tags/OL175T', '/tags/OL179T']

Remaining for Phase B:

Batch backfill — Phase 2 is complete in dry-run mode, but actual writes to OL are blocked on cdrini adding the genres (and eventually subgenres) field to the Work schema (work.schema.json) and Infogami type registry (/type/work).
Extend to subgenres — Once the team agrees on whether subgenres get their own field or share the genres field, add --type subgenres support to the same script.

Prerequisites

S3 keys configured in ~/.config/ol.ini for the openlibrarytagsbot account
openlibrary-client installed (pip install -e /path/to/openlibrary-client)

Caveats / open questions

Phase 2 blocked on cdrini's Infogami schema — writes to work.genres will be silently dropped until the field exists in /type/work. Dry-run works now.
WorkMigrator import — since both scripts live in scripts/, importing from migrate_work import WorkMigrator works as long as you run from repo root.
Batch size — save_many() accepts groups. 50 works per batch is a safe starting point.
Dump format — OL dumps are tab-separated: /type/work\t/works/OL123W\t{...json...}\n. We parse with gzip.open() + line-by-line JSON parsing.
Progress tracking — Phase 1 tracks line count; Phase 2 can accept a --continue flag to resume from a checkpoint file.

Reviewers:

@mekarpeles

… vocabulary.json

…IME SKIP tag creation test script'

…keys

…tags.py

mekarpeles · 2026-07-07T17:26:10Z

+# Phase 1 - Scan dump for matched work keys
+# ---------------------------------------------------------------------------
+def phase1(dump_path: str, tag_type: str):


We could name according to the functionality def scan_dump_for_matched_keys

get_work_keys_of_migration_candidates_from_monthly_dump

mekarpeles · 2026-07-07T17:37:04Z

+#---------------------------------------------------------------------------
+# Phase 2 - Fetch,. migrate, save
+# ---------------------------------------------------------------------------
+def phase2(keys_path: str, tag_type: str, dry_run: bool, batch_size: int = 50):


Q: do we want dry_run to have some default kwarg value? e.g. dry_run: bool = True

mekarpeles · 2026-07-07T17:43:14Z

For reference, we have a similar pipeline for fixing or updating entries in our Reading Log database so that every few days we ensure that, when multiple works get merged, reading logs are updated to point to the latest correct work:
https://github.com/internetarchive/openlibrary/blob/f1b8fbad022cec0a9c0a78136a4990f87fc8148c/openlibrary/core/models.py#L794-L835

It similarly uses batches, API calls and other patterns that may be useful for reference (this is in no way implying our PR is doing anything wrong, it's just useful context)

mekarpeles · 2026-07-07T17:44:35Z

+    cfg = Config().get_config()
+    s3 = cfg["s3"]
+    ol = OpenLibrary(credentials=Credentials(access=s3[0], secret=s3[1]))
+    ol.session.headers.update({"Content-Type": "application/json"})


I believe this session is now handled by the ol object (it should now set the content type for us internally)

mekarpeles · 2026-07-07T17:49:03Z

+    for i, key in enumerate(keys):
+        # Fetch the work JSON from Open Library
+        try:
+            resp = requests.get(f"https://openlibrary.org{key}.json")
+            resp.raise_for_status()
+            work = resp.json()
+        except Exception as e:
+            logger.error(f"Error fetching {key}: {e}")
+            continue


This may actually be quite slow and produce a lot of traffic for the API;

We have an endpoint called /query.json (to fetch all 50 docs at once)

mekarpeles · 2026-07-07T17:50:22Z

+
+        if dry_run:
+            # Preview mode: log what we would write
+            logger.info(f"{key}: {tag_type} = {tag_keys}")


Q: Do we want to run this either way? dry_run

mekarpeles · 2026-07-07T17:54:37Z

+            r = ol.save_many(batch, f"backfill {tag_type} tags from subject mapping")
+            if r.status_code == 200:
+                updated += len(batch)
+            else:
+                logger.error(f"save_many error: {r.status_code} {r.text[:200]}")
+            batch = []


This block can likely be DRY'd up for the batch case and the final case (i.e. share code)

mekarpeles · 2026-07-07T17:58:08Z

+cfg = Config().get_config()
+s3 = cfg["s3"]
+ol = OpenLibrary(credentials=Credentials(access=s3[0], secret=s3[1]))
+ol.session.headers.update({"Content-Type": "application/json"})


We might want to DRY up the fetching of an authenticated ol object (i.e. just have a single function rather than repeat this across files)

e.g.

def get_ol_session(): cfg = Config().get_config() s3 = cfg["s3"] ol = OpenLibrary(credentials=Credentials(access=s3[0], secret=s3[1])) ol.session.headers.update({"Content-Type": "application/json"}) return ol

mekarpeles · 2026-07-07T17:59:15Z

+cfg = Config().get_config()
+s3 = cfg["s3"]
+ol = OpenLibrary(credentials=Credentials(access=s3[0], secret=s3[1]))
+ol.session.headers.update({"Content-Type": "application/json"})


Suggested change

ol.session.headers.update({"Content-Type": "application/json"})

This should now be merged into olclient

…ollowup)

Address PR #41 review: shared auth helper, remove olclient Content-Type, rename functions

Chisomnwa added 4 commits July 3, 2026 21:00

Create Tag objects for all 10 controlled types and write keys back to…

1d32319

… vocabulary.json

git commit -m 'Uncomment the main part of the script and remove the T…

e69a05a

…IME SKIP tag creation test script'

Add slug_to_tag_key() helper for vocabulary Tag key lookups

1c3265f

Add WorkMigrator to classify work subjects into typed Tag keys

766e057

Chisomnwa marked this pull request as draft July 4, 2026 12:36

Add backfill_tags.py — two-phase batch backfill script for genre Tag …

4050ab0

…keys

Chisomnwa marked this pull request as ready for review July 5, 2026 07:20

Chisomnwa added 2 commits July 7, 2026 14:01

Replace print with logging, make batch-size configurable in backfill_…

2181d3d

…tags.py

Replace print with logging and fix http bug in migrate_work.py

b7d8b71

mekarpeles reviewed Jul 7, 2026

View reviewed changes

Comment thread scripts/backfill_tags.py

mekarpeles reviewed Jul 7, 2026

View reviewed changes

Comment thread scripts/backfill_tags.py

mekarpeles reviewed Jul 7, 2026

View reviewed changes

Comment thread scripts/backfill_tags.py Outdated

fix typo

1ea4c45

mekarpeles reviewed Jul 7, 2026

View reviewed changes

mekarpeles merged commit 5c48e5d into Open-Book-Genome-Project:main Jul 7, 2026
1 check passed

Chisomnwa mentioned this pull request Jul 9, 2026

Address PR #41 review: shared auth helper, remove olclient Content-Type, rename functions #43

Merged

Chisomnwa changed the title ~~Phase B: Work Migration — Subject Classification Pipeline~~ Phase 2: Work Migration — Subject Classification Pipeline Jul 13, 2026

mekarpeles pushed a commit that referenced this pull request Jul 14, 2026

Extract shared get_ol_session() helper, rename phase functions (#41 f…

dedf78e

…ollowup)

mekarpeles added a commit that referenced this pull request Jul 14, 2026

Merge pull request #43 from Chisomnwa/refactor/tag-migration-tooling

81d3ce3

Address PR #41 review: shared auth helper, remove olclient Content-Type, rename functions

Chisomnwa mentioned this pull request Jul 29, 2026

Phase 2 of Work Processing — Run Genre Backfill on all Matched Works #46

Open

Chisomnwa mentioned this pull request Aug 24, 2026

Parent Issue: Audience Tag Type — Full Workflow #52

Open

5 tasks

Navigation Menu

Uh oh!

Phase 2: Work Migration — Subject Classification Pipeline - #41

Phase 2: Work Migration — Subject Classification Pipeline#41
mekarpeles merged 8 commits into
Open-Book-Genome-Project:mainfrom
Chisomnwa:phase-b/tag-migration

Chisomnwa commented Jul 4, 2026 •

edited

Loading

Uh oh!

mekarpeles Jul 7, 2026

Uh oh!

mekarpeles Jul 7, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mekarpeles Jul 7, 2026

Uh oh!

mekarpeles commented Jul 7, 2026

Uh oh!

mekarpeles Jul 7, 2026

Uh oh!

mekarpeles Jul 7, 2026

Uh oh!

mekarpeles Jul 7, 2026

Uh oh!

mekarpeles Jul 7, 2026

Uh oh!

mekarpeles Jul 7, 2026

Uh oh!

mekarpeles Jul 7, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

Chisomnwa commented Jul 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What this PR adds

The mapping chain

File 1: tags/utils.py

File 2: scripts/migrate_work.py

File 3: scripts/backfill_tags.py

CLI reference:

Remaining for Phase B:

Prerequisites

Caveats / open questions

Reviewers:

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mekarpeles commented Jul 7, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Chisomnwa commented Jul 4, 2026 •

edited

Loading