Phase 1: Create Tag objects for all 10 controlled vocabulary types - #40
Conversation
|
Looks good overall! A questions about comments in our tag generation script |
…IME SKIP tag creation test script'
|
Hi @mekarpeles, thank you for the review! I have uncommented the main parts of the script. Everything is okay now. |
| def create_tag(name, tag_type, slug, description): | ||
| """ | ||
| Send a single tag to Open Library's /api/new endpoint. | ||
| Returns the auto-generated key (e.g. /tags/OL123T) or None on failure. | ||
| """ | ||
| payload = json.dumps([{ | ||
| "type": {"key": "/type/tag"}, | ||
| "name": name, | ||
| "tag_type": tag_type, | ||
| "slugs": [slug], | ||
| "tag_description": description, | ||
| }]) | ||
| headers = { | ||
| "Content-Type": "application/json", | ||
| "Opt": '"http://openlibrary.org/dev/docs/api"; ns=42', | ||
| "42-comment": f"create tag: {slug} ({tag_type})", | ||
| } | ||
| r = ol.session.post("https://openlibrary.org/api/new", data=payload, headers=headers) | ||
| if r.status_code == 200: | ||
| # /api/new returns a list of keys, one per input doc | ||
| return r.json()[0] | ||
| else: | ||
| print(f" Failed ({r.status_code}) for {name}: {r.text[:200]}") | ||
| return None |
There was a problem hiding this comment.
This should ultimately be DRY'd (Do Not Repeat ourselves) up into the openlibrary-client as there's a lot of boiler plate here and this is a common task that will be done over again. Not a blocker for this PR.
| print(f" Wrote keys to {os.path.relpath(vf, os.path.join(TAG_TYPES_DIR, '..'))}") | ||
|
|
||
| # Small delay so we don't hammer the API | ||
| time.sleep(0.5) |
There was a problem hiding this comment.
Timeouts based on ms are fragile as the load on the server is ever-changing. This would be better as a configurable parameter.
| if modified: | ||
| with open(vf, "w") as f: | ||
| json.dump(data, f, indent=2) | ||
| print(f" Wrote keys to {os.path.relpath(vf, os.path.join(TAG_TYPES_DIR, '..'))}") |
There was a problem hiding this comment.
Generally logging would be preferable to print statements -- this would give us control over where the IO goes (e.g. into sentry or some other aggregation program) and lets us also log different types like errors v. info.
Part of the Phase 1 work for #14.
What this does
Creates OL Tag objects for every entry in all 10 controlled
vocabulary.jsonfiles and writes the returnedOLxxxTkeys back. This is a one-time job — after this PR, the vocabulary files permanently store the Tag key for each term, which the migration script (Phase B) will use to populatework.genres.In other words, this PR:
scripts/create_tags.py— reads entries from all 10 controlledvocabulary.jsonfiles, creates a Tag viaPOST /api/newfor each, captures the auto-generatedOLxxxTkey, and writes it back to the filekeyfield to every entry in every controlledvocabulary.json(198 entries total)Files changed
scripts/create_tags.pytag_types/audience/vocabulary.jsonkeyfield to all 9 entries (OL94T–OL102T)tag_types/content_features/vocabulary.jsonkeyfield to all 17 entries (OL103T–OL119T)tag_types/content_formats/vocabulary.jsonkeyfield to all 24 entries (OL120T–OL143T)tag_types/content_warnings/vocabulary.jsonkeyfield to all 18 entries (OL144T–OL161T)tag_types/genres/vocabulary.jsonkeyfield to all 21 entries (OL162T–OL182T)tag_types/literary_form/vocabulary.jsonkeyfield to both entries (OL183T–OL184T)tag_types/literary_themes/vocabulary.jsonkeyfield to all 29 entries (OL185T–OL213T)tag_types/literary_tropes/vocabulary.jsonkeyfield to all 28 entries (OL214T–OL240T, OL291T)tag_types/moods/vocabulary.jsonkeyfield to all 26 entries (OL241T–OL266T)tag_types/subgenres/vocabulary.jsonkeyfield to all 24 entries (OL267T–OL290T)Verification
.jsonendpointcreate_tags.pyskips entries that already have a key (safe to re-run)Notes for future contributors
venv, you mustpip install -e /path/to/openlibrary-clientinside it, not just system-wide.save_manyvs/api/new:save_manyupdates existing docs (needs a key)./api/newcreates new ones (auto-generates key)."type": {"key": "/type/tag"}(dict), not"type": "/type/tag"(string).ol.session.headers.update({"Content-Type": "application/json"}).Reviewers
@mekarpeles