User talk:Difool

Welcome to Wikidata, Difool!

Wikidata is a free knowledge base that you can edit! It can be read and edited by humans and machines alike and you can go to any item page now and add to this ever-growing database!

Need some help getting started? Here are some pages you can familiarize yourself with:

• Introduction – An introduction to the project.
• Wikidata tours – Interactive tutorials to show you how Wikidata works.
• Community portal – The portal for community members.
• User options – including the 'Babel' extension, to set your language preferences.
• Contents – The main help page for editing and using the site.
• Project chat – Discussions about the project.
• Tools – A collection of user-developed tools to allow for easier completion of some tasks.

Please remember to sign your messages on talk pages by typing four tildes (~~~~); this will automatically insert your username and the date.

If you have any questions, don't hesitate to ask on Project chat. If you want to try out editing, you can use the sandbox to try. Once again, welcome, and I hope you quickly feel comfortable here, and become an active editor for Wikidata.

Best regards! --Epìdosis 07:19, 11 July 2023 (UTC)

---

Hi Difool, with your edit to the Bot permission request page [1], you removed some other requests from the list (including mine; see comparison [2]). It would be nice if you could revert/repair your edit :) Thanks! Parnswir (talk) 06:40, 24 October 2023 (UTC)

Hi Parnswir, sorry about that. I've corrected it now. Difool (talk) 07:31, 24 October 2023 (UTC)

Awesome, thanks! Parnswir (talk) 09:10, 24 October 2023 (UTC)

Date of death removal

Why did you remove this date of death? According to Wikidata:Requests for permissions/Bot/DifoolBot your bot should only work on references, not remove statements. Looking at what you put in Q13564452#P570: Don't ever use unknown value Help for date of birth unless you're working around the year 0. Instead put in a date with lower precision. See Help:Dates for more info. Multichill (talk) 21:10, 13 November 2023 (UTC)

The removal of the statement falls within "Remove ECARTICO references from outdated statements, or, if it's the only reference, remove the complete statement". I'll read up about your unknown value comment. Difool (talk) 02:25, 14 November 2023 (UTC)

Could you look at Q19911735, what date/precision would you put in the death date? I did it this way: Ecartico lists a birth year of "circa 1627" and a death year "after 1678". I added 100 years to the earliest listed year (1627) so the death year should be between 1678 and 1727. Because those years are not in the same century I used precision 6, millennium. Difool (talk) 02:43, 15 November 2023 (UTC)

Please have a look on reverted edits

I guess, 0.05% of the automated edits are wrong, so please have a look on reverted edits, it makes no sense to add a statement four times again, once it was reverted. See Q65558479. Florentyna (talk) 14:10, 27 December 2023 (UTC)

Hi @Florentyna, sorry about that. I've stopped the bot and will alter the code so this won't happen again. In the case of Andreas Springer (Q65558479), I think the value of J9U entity ID (P8189) is wrong; I would set the claim to 'deprecated rank' with qualifier reason for deprecated rank (P2241) to refers to different subject (Q28091153). Difool (talk) 15:38, 27 December 2023 (UTC)

For Andreas Springer (Q65558479) it is only a guessed birth year by me. Andreas Springer it is a very common German name. For such "common" cases, one hase to be very carefully. Elsewise your additions are quite good, no need to stop the bot, the error rate is in the range of a normal human being, because you are adding to quite unique names. --Florentyna (talk) 15:44, 27 December 2023 (UTC)

@Florentyna - the bot doesn't search in VIAF for a name or birth year; if you have a J9U entity ID (P8189), you can ask VIAF what the corresponding VIAF cluster ID (P214) is.

For Andreas Springer, you can do this with this query: https://viaf.org/viaf/sourceID/J9U|987007374268105171/justlinks.json (987007374268105171 is the J9U entity ID (P8189) on the Andreas Springer page) Difool (talk) 16:10, 27 December 2023 (UTC)

Like I already told, for common names I have seen already hundreds of wrong additions. Probably someone before you made a KI addition, that this is the real one Andreas Springer. Usually, I try to identify the primary source, but often the secondary ones (like you) seem to be the bad ones. --Florentyna (talk) 16:16, 27 December 2023 (UTC)

@Florentyna - Okay, I've added an ignore list and added Andreas Springer (Q65558479) to it and started the bot again. Luckily, your edit was the only reversal, but I'll keep an eye on the reversal notifications in the future.

I've also added the other Andreas Springer - indeed with a same birth year of 1966 - at Andreas Springer (Q124030875) - Difool (talk) 02:56, 28 December 2023 (UTC)

Thanks and happy new year!

Hi! I would just like to thank you for the last section you added to User:Difool/viaf already somewhere, it will be very useful for solving hundreds of duplications and conflations. I hope to have more time to deal with them from the second half of this month. And, of course, happy 2024! See you soon, --Epìdosis 14:00, 4 January 2024 (UTC)

Hi, @Epìdosis. Glad you like it, and a happy 2024 to you too! I'm going to add a 'score' column too, with the score based on year of birth and death, and country of birth and death. It still we be a lot of work, I'm sure; I'll just do some when I have the time. I'll let the bot do Union List of Artist Names ID (P245) again, and then start with GND ID (P227) and IdRef ID (P269) - Difool (talk) 02:17, 5 January 2024 (UTC)

Thanks Difool, it is very helpful. I have added it to my maintenance page Geagea (talk) 12:57, 7 January 2024 (UTC)

I have just finished User:Difool/viaf already somewhere#Union List of Artist Names ID and I will then pass to User:Difool/viaf already somewhere#National Library of Greece ID in the next days. Could you do next SBN author ID (P396)? I'm particularly interested in it. Thanks! --Epìdosis 23:39, 12 January 2024 (UTC)

Okay, I'll let the bot do the SBN author ID (P396) after it's finished with GND ID (P227), which will take about a week or so - Difool (talk) 03:40, 13 January 2024 (UTC)

Very good! I have finished this morning with NLG and I'm now on User:Difool/viaf already somewhere#Library of Congress authority ID from Union List of Artist Names ID. --Epìdosis 17:50, 13 January 2024 (UTC)

I have now finished User:Difool/viaf already somewhere#Library of Congress authority ID from Union List of Artist Names ID and I will slowly proceed on User:Difool/viaf already somewhere#Library of Congress authority ID. --Epìdosis 20:02, 22 January 2024 (UTC)

The bot finally finished with GND ID (P227) and I let it work through SBN author ID (P396), but VIAF mostly returned 'not found'; only three duplicates were found. It's now busy with NL CR AUT ID (P691) Difool (talk) 03:19, 27 January 2024 (UTC)

Thanks! After NKC I would suggest BAV (Vatican Library VcBA ID (P8034)) if possible. See you soon, --Epìdosis 08:13, 27 January 2024 (UTC)

Could you include WD:ISNI (ISNI (P213))? They also offer data download https://isni.org/page/linked-data/ . CV213 (talk) 14:55, 3 February 2024 (UTC)

Okay, I'll start it on Monday - Difool (talk) 01:41, 4 February 2024 (UTC)

Great, thank you! CV213 (talk) 10:47, 4 February 2024 (UTC)

VIAF counts - meaning

User:Difool/viaf counts it says ISNI (P213) : 21913 . What does that mean? CV213 (talk) 15:27, 3 February 2024 (UTC)

It means that wikidata has 21913 pages about humans with a ISNI (P213) but no VIAF cluster ID (P214). A row in the table includes an example page and three lookup links for that page; two by id, and one by name. The bot uses the lookup by id link, so that gives an indication how successful the bot will be. Difool (talk) 01:39, 4 February 2024 (UTC)

Thank you. Re lookup: It could be that the ISNI item contains one or even more VIAF IDs, even when the VIAF page doesn't have the ISNI. Maybe link the example ID to the primary source? But you will maybe analyse the ISNI data set separately. CV213 (talk) 10:45, 4 February 2024 (UTC)

The bot doesn't check the ISNI data, but only checks the VIAF data through the VIAF API. For example, if you have ISNI 0000000032172983, it will call https://viaf.org/viaf/sourceID/ISNI|0000000032172983/justlinks.json. Not all persons in ISNI are added to VIAF, so it will probably result in a lot of "not found"s. Difool (talk) 11:25, 4 February 2024 (UTC)

viaf already somewhere ISNI

Thanks a lot for User:Difool/viaf already somewhere#ISNI, I already merged some, mostly duplicates from CZ data set. Could you add the score column, since some have different from, e.g. this https://dicare.toolforge.org/wikidata-diff/?qids=Q12619817+Q48965567&language=en ? CV213 (talk) 22:01, 7 February 2024 (UTC)

Difool 3 translit

Hi Difool,

I don't want to spam the discussion page there, so may I continue the discussion here?

translitua: I've just noticed that I have installed this package in January 2013! It however lacks a Duden translit for Russian which would be required more often.

If you open a Github repo I'd contribute.

Please exclude German from this Bot run. I'm tired of the influx of bad (usually English) translits into my language.

Best wishes, Tadarrius Bean (talk) 13:14, 16 April 2024 (UTC)

Hi @Tadarrius Bean, the Github repo is here; What is your username? then I add you as a collaborator. Difool (talk) 14:45, 16 April 2024 (UTC)

Adriaan van den Santheuvel (Q110083298)

Hello Difool, since you added the image, you might wish to replace it by a larger one. Cheers. Lotje (talk) 05:28, 7 July 2024 (UTC)

@Lotje Thanks, indeed a better image - Difool (talk) 01:34, 9 July 2024 (UTC)

You are vey welcome Difool :-) Lotje (talk) 11:42, 17 July 2024 (UTC)

Bot problems

As explained in User:ASarabadani (WMF)/Growth of databases of Wikidata, the database is growing too quickly. In particular, the number of previous revisions of items.

Your bot is currently editing items repeatedly in a short space of time, e.g. on Special:History/Q455040 it made 8 edits in just over a minute. You should group these changes into fewer edits, ideally a single edit. Please stop the bot until you have done that.

I also recommend using the syntax [[Property:P268]] in the summary when referring to properties, instead of just an ID (e.g. P268) or a name (e.g. "reference url"). That will create a link like Bibliothèque nationale de France ID (P268), with the property name in the user's language.

- Nikki (talk) 09:59, 4 September 2024 (UTC)

@Nikki: Alright, I’ve stopped the bot. I’ll check out what you mentioned. Difool (talk) 10:27, 4 September 2024 (UTC)

@Nikki: I've updated the bot script and restarted it. Thanks for informing me, I didn't know the revisions were taking up so much storage space. Difool (talk) 02:06, 5 September 2024 (UTC)

Also, with this edit a reference was removed, and one reference was transferred to a different source without manual checking. --EncycloPetey (talk) 00:12, 7 September 2024 (UTC)

@EncycloPetey: It might appear that way because the difference viewer doesn't display the unchanged content. For instance, the place of birth originally had three references - one duplicate - and after the edit, it has two references. Difool (talk) 01:41, 7 September 2024 (UTC)

There was a reference for sex or gender from the GND, which is no longer present. That information was not a duplicate. --EncycloPetey (talk) 02:16, 7 September 2024 (UTC)

@EncycloPetey: The sex or gender claim included two references with the same value 118615688 for GND ID (P227). The stated in (P248) value of these references is disregarded because the GND ID has only one applicable 'stated in' value (P9073), which is Integrated Authority File (Q36578). Therefore, the script merges the two GND ID references. Difool (talk) 03:25, 7 September 2024 (UTC)

My mistake. --EncycloPetey (talk) 03:59, 7 September 2024 (UTC)

Withdrawn ID value

https://www.wikidata.org/w/index.php?title=Q127548319&diff=prev&oldid=2210581071 - it wasn't withdrawn, it is in an invalid format. ISNIplus (talk) 18:30, 9 September 2024 (UTC)

Ah, yes, that is of course also possible. I'll adjust the script accordingly. Difool (talk) 00:35, 10 September 2024 (UTC)

Thank you. The format constraint could be out of date, I just added "sj" [3] which was added to "format as a regular expression (Property:P1793)" on 20 July 2020 [4]. @Kolja21: constraint out of date at least more than 4 years. ISNIplus (talk) 01:18, 10 September 2024 (UTC)

I'll adjust the script to make no changes and only log the ID when it encounters a 404 - Not Found error from the BnF/IdRef/GND/LoC server. Then it must be manually checked, though fortunately, it doesn't happen too often. Difool (talk) 01:56, 10 September 2024 (UTC)

P813-only references

Hi! A little proposal of integration to your bot task: I sometimes find references containing only retrieved (P813); unless they are used on properties with datatype "URL" or "external identifier", they are obviously wrong;

• if they are the only reference to the statement, they should be removed (e.g.);
• if there is only one other reference which contains a "URL" or "external identifier" and no value of retrieved (P813), they should be merged with it (e.g.);
• in other cases, I would just avoid editing because probably there should be a manual fix.

This QLever query found nearly 30k cases of statements with datatype "time" and a reference containing only P813; surely, considering statements with other datatypes (excluding "URL" or "external identifier") other tens/hundreds of thousands will emerge. Could you add it to your bot task? No haste, of course. Thanks very much as always! --Epìdosis 08:33, 15 September 2024 (UTC)

Hi @Epìdosis, I've updated the code based on your suggestions. Some cases might be acceptable, so I'll run the code in debug mode and list any that I encounter:

Skipped:

• different from (P1889): Example
• copyright status as a creator (P7763): Example
• on focus list of Wikimedia project (P5008): Example

Not sure:

• member of (P463): Example - The meaning of 'retrieved' is here the date the user looked at the mentioned wikidata page and checked the claim

Done:

• https://www.wikidata.org/w/index.php?title=Q519082&diff=prev&oldid=2248717983
• https://www.wikidata.org/w/index.php?title=Q392189&diff=prev&oldid=2248719170 - but looks like member of (P463)
• https://www.wikidata.org/w/index.php?title=Q29855670&diff=prev&oldid=2248748994 - manner of death (P1196)
• https://www.wikidata.org/w/index.php?title=Q7027443&diff=prev&oldid=2248756283 - copyright representative (P6275). A retrieved for languages spoken, written or signed (P1412) was moved to CONOR.SI ID (P1280): probably not correct, only move to next reference if possible

Difool (talk) 03:15, 16 September 2024 (UTC)

Thanks very much! Skipping different from (P1889) (probably also the twin said to be the same as (P460)), copyright status as a creator (P7763), on focus list of Wikimedia project (P5008) makes sense; for {{|463}}, I think it would make sense improving the reference with inferred from (P3452) = value (e.g.); https://www.wikidata.org/w/index.php?title=Q519082&diff=prev&oldid=2248717983 is perfect; for child organization or unit (P355) the same as member of (P463) could be applied, I agree (i.e. improving the reference with inferred from (P3452)). If you have other dubious cases I'm happy to discuss them of course! Epìdosis 06:06, 16 September 2024 (UTC)

@Epìdosis Thanks! I let the script run now, but log and skip items where it wants to change retrieved references. I'll summarize the results tomorrow. Difool (talk) 06:57, 16 September 2024 (UTC)

Perfect! On the case of "A retrieved for languages spoken, written or signed (P1412) was moved to CONOR.SI ID (P1280): probably not correct, only move to next reference if possible": I think that, if he was a statement value with one P813-only reference and 2(+) non P813-only references, we can just remove the P813 reference without moving it into one of the existing references. I would move P813-only reference into another one only if 1) the value has only one other reference and 2) this reference has no preexisting P813 value. --Epìdosis 07:44, 16 September 2024 (UTC)

@Epìdosis: All items from the QLever query result are done now. Once QLever processes a new Wikidata dump, I'll run the script again. I ended up with a list of 163 properties (not datatype "time") permitting a single 'retrieved' reference, and a list of 126 properties from which such references are automatically removed. Statements with multiple references, which included a reference with only a 'retrieved' statement, have been handled by either merging or removing those single 'retrieved' statements, by the script or manually. Difool (talk) 03:50, 21 October 2024 (UTC)

Wonderful job! Epìdosis 03:54, 21 October 2024 (UTC)

@Epìdosis: Small update: Approximately 21k items in your QLever query are scholarly article (Q13442814) with the same edit [5]. I skip these for now. For claims with only a single retrieved reference, I've created a whitelist of properties where the reference can be removed (name properties like pseudonym (P742) and properties that issue a warning without a reference, like convicted of (P1399)). Additionally, I've made a skip list of properties to ignore (the list above plus said to be the same as (P460)). The script logs the rest, and I'll review these once the bot completes the list. For claims with multiple references, I've coded the reference list as follows:

• R: Reference with single retrieved (P813)
• S: Reference with single stated in (P248)
• W: Reference with an external ID, without a retrieved (P813)
• A: Reference with a retrieved (P813) and an external ID
• B: Reference with a retrieved (P813) and a stated in (P248)
• X: Others

If a reference list contains only R, A, and B, I'll remove the R references. For sequences like RS, SR, RW, WR, I'll merge them. For BWR, I'll merge the WR part, and for SRB, I'll merge the SR part. After running the script, I'll check the log and determine actions for other sequences. Difool (talk) 04:28, 19 September 2024 (UTC)

@Difool: it seems perfect! I think we can also fix the 21k items with the same edit in the future, but probably at the end, they are certainly not prioritary. Thanks very much also for the coding of the reference list, it is a great solution! Bye, Epìdosis 06:43, 19 September 2024 (UTC)

Mis-merging references

In this edit, the source was changed to be the same for three references, however, those three sources are not the same. They are all housed at the same base URL, but they are separate publications, with different names and authors. Merging the references to be all at the housing website is misleading, and loses information. --EncycloPetey (talk) 20:40, 24 September 2024 (UTC)

Hi @EncycloPetey, I see what you mean: the original 'stated in' information is lost, now that all the references point to Treccani's Enciclopedia on line (Q65921422). I'll have the bot skip these URLs for now and reach out to @Epìdosis for assistance. I noticed a similar edit from the bot. Difool (talk) 01:16, 25 September 2024 (UTC)

Looking closer, I see that URLs like https://www.treccani.it/enciclopedia/aristofane_(Enciclopedia-machiavelliana)/ match the URL match pattern (P8966) of multiple external id properties. I'll update the bot code to handle these cases and write a script to correct the already modified data in Wikidata. Difool (talk) 01:47, 25 September 2024 (UTC)

Hi! To clarify, Treccani ID (P3365) has been initially used generically both for the Enciclopedia Treccani online (whose articles have nothing in parenthesis) and for other Istituto Treccani works' like Enciclopedia dei ragazzi ID (P9983) and Enciclopedia machiavelliana ID (P11820), for which we are gradually creating new apposite properties, moving IDs there from generic P3365. Probably the URL match pattern (P8966) of P3365 should be modified to better fit these cases; also @Horcrux: who has proposed most of Treccani properties. Epìdosis 06:21, 25 September 2024 (UTC)

@Epìdosis: Done: [6] --Horcrux (talk) 07:28, 25 September 2024 (UTC)

@EncycloPetey, @Epìdosis I ran the script to correct the already modified data in Wikidata. Some values are not corrected (query), notably because URLs like https://www.treccani.it/enciclopedia/raffaello-sanzio_(Enciclopedia-Italiana) match with the URL match pattern (P8966) of Encyclopedia of Italian ID (P11586) and Treccani's Enciclopedia Italiana ID (P4223). @Horcrux: I think the URL match pattern of P11586 misses the dell' part Difool (talk) 04:16, 26 September 2024 (UTC)

My fault! --Horcrux (talk) 06:28, 26 September 2024 (UTC)

Withdrawn identifier value

Hi Difool, in some cases the (old) DNB catalog is not showing a GND. That does not mean the identifier value has been ithdrawn. It's a known problem with the current version of this catalog. (A new one is in preparation.) You can find the GND for example by using OGND. --Kolja21 (talk) 05:18, 4 October 2024 (UTC)

@Kolja21: Ow, that's good to know. The script isn't running right now, and I'm planning to rework this part before I start it up again. I'll likely switch it to log only, so if a server gives back a 'not found' message while searching for an identifier, it will need to be manually checked to see if the identifier has been withdrawn. Difool (talk) 06:31, 4 October 2024 (UTC)

Perfect. If you post a list with the problematic IDs I'll try to fix them. I don't know why they don't show up in the current version of the DNB catalog (the problem has been known for a year) but clicking on edit and saving them again mostly helps. @Epìdosis: FYI. --Kolja21 (talk) 15:10, 4 October 2024 (UTC)

Some missed reference URL on Q361509

Hi there! I see that the bot passed on September 12th on Adolf Faller (Q361509), however some references URL remain (e.g. the VIAF reference on languages spoken, written or signed (P1412) was missed, while the bot correctly adapted other references on the same statement). Would it be possible for you to check if anything went wrong? Thanks ! --Thomas Kerboul (BGE) (talk) 09:02, 21 October 2024 (UTC)

Hi @Thomas Kerboul (BGE), since that VIAF reference already has a VIAF cluster ID (P214), it is currently not changed by the script. The script now only removes a reference URL if the reference doesn't have a database identifier like VIAF cluster ID (P214), Bibliothèque nationale de France ID (P268) etc. or during a merge of references. However, I'm considering requesting bot permissions to remove such redundant reference URLs too. Difool (talk) 09:38, 21 October 2024 (UTC)

Thanks for the reply! Let me know if/when you ask for this permission, I'll support it. --Thomas Kerboul (BGE) (talk) 09:53, 21 October 2024 (UTC)

Changing Allgemeines Künstlerlexikon (Q15649499)

Hello Difool, to mention that first: I am a big fan of your work on references normalization, thanks a lot! A caveat regarding Allgemeines Künstlerlexikon (Q15649499): I think it should not be replaced like this because of the somewhat convoluted relation between Allgemeines Künstlerlexikon (Q15649499) and Artists of the World Online (Q41640909). For the cases it is applied to (I guess it aren't many and I might be one of their main sources) stated in (P248)Allgemeines Künstlerlexikon (Q15649499) is probably more precise than stated in (P248)Artists of the World Online (Q41640909). Would it be very complicated to revert those changes? If so, it isn't a disaster, but maybe worth to leave future cases as they are. Thanks again! --Marsupium (talk) 00:33, 14 November 2024 (UTC)

Hi @Marsupium, thanks and apologies for that! I'll work on retrieving and reverting all the changes. The bot now uses a whitelist of 'stated in' values that can be overwritten, and I added Allgemeines Künstlerlexikon (Q15649499) to that list yesterday. I think I've reverted all those values. But previously, the bot would always overwrite the 'stated in' value with the applicable 'stated in' value (P9073) if it changed a reference URL to an external ID. I'll need to put some effort into retrieving those, i.e. write a program, but that's no problem. Difool (talk) 04:44, 14 November 2024 (UTC)

Thanks a lot for your work! They are not the same, but if the values stay Artists of the World Online (Q41640909) it isn't a big deal I'd say.

And BTW, it is interesting that you use a no-change-list now. Could you maybe also already add Artists of the World Online (Q41640909) to that list? Because I am planning to change the formatter URL (P1630) and applicable 'stated in' value (P9073) soon for the new database/interface with (mostly) the same IDs, but the old stated in (P248) values should not be changed. --Marsupium (talk) 05:05, 14 November 2024 (UTC)

@Marsupium Yes, sure; I'll add Artists of the World Online to that list. I made the list BTW because while using the bot, I noticed that the applicable 'stated in' value can't be used for books or newspaper pages that are accessible online, such as those found on Google Books or Internet Archive. (like here and here) Instead, the 'stated in' value should be the book or newspaper itself. Difool (talk) 06:46, 14 November 2024 (UTC)

Great! Thank you! Actually that is the case here, Allgemeines Künstlerlexikon (Q15649499) is originally a printed work/series of books. Artists of the World Online (Q41640909) is a paywalled (incomplete) online edition of that book series and other works. --Marsupium (talk) 07:52, 14 November 2024 (UTC)

IPSB, Sejm-Wielki

Hmm... i've got an idea. Maybe your bot should also normalize many references from Sejm-Wielki.pl profile ID (P8172) which are in form:

• bare urls, e. g. https://www.sejm-wielki.pl/b/sw.661304
• id's only, e. g. Sejm-Wielki.pl profile ID (P8172) : sw.661304

Hm... 78.131.244.18 14:22, 3 December 2024 (UTC)

Yes, sure. I've added that URL to the list, so the bot should slowly normalize those references. Difool (talk) 14:47, 3 December 2024 (UTC)

A future task on P21

Hi Difool! While DifoolBot is going on with its great work on URLs in references (Wikidata:Requests for permissions/Bot/DifoolBot 5 and Wikidata:Requests for permissions/Bot/DifoolBot 6; BTW, do you have a rough esteem on the number of missing items and how many days will it take for them? I fear still many ...), I was thinking of future tasks. Surely doing again Wikidata:Requests for permissions/Bot/DifoolBot 2 for all VIAF sources will be worth, but I would like to work a bit on Property talk:P214/Duplicates/humans before re-launching it (ideally that page should be empty; but let's say at least below 20k bytes, which means a lot of work).

Anyway, recently in my spare time I have been fixing manually some tens of items having both (and non-deprecated) sex or gender (P21)female (Q6581072) and sex or gender (P21)male (Q6581097), which is quite strange, and I have made an interesting discovery about the way in which most of these cases originated: to the existing (and usually not-referenced) value of P21, which was correct, the other value (incorrect) was added in mid 2024 by your bot using as source IdRef ID (P269); I have reported all these cases here, where @Thomas Kerboul (BGE): gently double-checked and solved them, but more are still to come. This was a very interesting and counter-intuitive discovery, and gave me an idea: we could think, in the future, of analysing more extensively the divergences of "Genre" in IDREF and sex or gender (P21) values in Wikidata, to see how much they diverge and if (as I have experienced as of now) the divergence is usually because of mistakes on IDREF side. We could e.g. try to start from items with IdRef ID (P269) and unreferenced sex or gender (P21), using IdRef ID (P269) as reference to existing P21 or adding new P21 if needed, and see how many new P21 are added and then check them manually; the query runs only with LIMIT on WDQS, but as of now runs well without LIMIT on https://try.orbopengraph.com/ and gives 267108 results in 157824 ms. I think it would be worth, both for Wikidata and for IDREF. Good night! --Epìdosis 00:29, 29 January 2025 (UTC)

Hi @Epìdosis, I write a longer response later, but I wanted to let you know that you can query Wikidata and IdRef in one SPARQL query. For example:

SELECT ?item ?idref ?gender_wd ?gender_idref
WITH {
  SELECT ?item ?idref ?gender_wd WHERE {
    ?item wdt:P269 ?idref_id ;
          p:P21 ?gender_statement .
    MINUS { ?gender_statement prov:wasDerivedFrom ?ref . }
    ?item wdt:P21 ?gender .
    BIND(IRI(CONCAT("http://www.idref.fr/", ?idref_id, "/id")) AS ?idref)
    BIND(
      IF(?gender = wd:Q6581097, "male", 
         IF(?gender = wd:Q6581072, "female", "other")) AS ?gender_wd
    )
  }
  LIMIT 1000
} AS %wikidata
WITH {
  SELECT ?idref ?gender_idref WHERE {
    INCLUDE %wikidata
    SERVICE <https://data.idref.fr/sparql> {
      SELECT DISTINCT ?idref ?gender_idref WHERE {
        ?idref <http://xmlns.com/foaf/0.1/gender> ?gender_idref .
      }
    }
  }
} AS %idrefdata
WHERE {
  INCLUDE %wikidata .
  INCLUDE %idrefdata .
  FILTER (?gender_wd != ?gender_idref)
}

Try it!

Difool (talk) 04:26, 30 January 2025 (UTC)

Yay, more work for me! Jokes aside, thank you very much for the query, that will help a lot for improving IdRef. Would it be possible to have the same query with birth date and death date? --Thomas Kerboul (BGE) (talk) 10:18, 30 January 2025 (UTC)

Hi Difool, the query above is greatly useful, I thought it was possible in theory but I wouldn't imagine it could work so well.

I made a few tests: with the form above I got timeouts and it is presently not supported by QLever; in order not to obtain timeout, I removed the condition of sex or gender (P21) being not referenced in Wikidata, and I used this:

SELECT DISTINCT ?item ?idref ?gender_wd ?gender_idref
WITH {
  SELECT ?item ?idref ?gender_wd WHERE {
    ?item wdt:P269 ?idref_id ;
          wdt:P21 ?gender .
    BIND(IRI(CONCAT("http://www.idref.fr/", ?idref_id, "/id")) AS ?idref)
    BIND(
      IF(?gender = wd:Q6581097, "male", 
         IF(?gender = wd:Q6581072, "female", "other")) AS ?gender_wd
    )
  }
  LIMIT 10000
} AS %wikidata
WITH {
  SELECT ?idref ?gender_idref WHERE {
    INCLUDE %wikidata
    SERVICE <https://data.idref.fr/sparql> {
      SELECT DISTINCT ?idref ?gender_idref WHERE {
        ?idref <http://xmlns.com/foaf/0.1/gender> ?gender_idref .
      }
    }
  }
} AS %idrefdata
WHERE {
  INCLUDE %wikidata .
  INCLUDE %idrefdata .
  FILTER (?gender_wd != ?gender_idref)
}

Try it!

which works well in WDQS (about 100 results in 44 seconds). I also tried removing the LIMIT 10000: the query times out in WDQS (and it gets stuck in https://try.orbopengraph.com/). But trying only the %wikidata part with no limit, it succeeds in 42 seconds in WDQS with 926k results; obviously the query service of IDREF is unable to analyse these 926k IDs in just 18 seconds.

So, I was thinking: would it be possible, in some way (e.g. dividing the 926k Wikidata items having both sex or gender (P21) and IdRef ID (P269) in non-overlapping chunks of 5k or 10k), to obtain in a Wikidata table a complete list of cases of mismatches, so that we can then work on it in the next months, also making accurate statistics on how many times Wikidata was wrong and how many times IDREF was wrong? I think it would be an interesting project for us in this year 2025, maybe also resulting in a future publication. The table could be placed in Property talk:P269/Gender mismatches; it could contain six colums: the four columns of the query + two empty columns on the right (one for the judgement on who's wrong and one for a note on the resolution of the mistake on the other side). --Epìdosis 10:34, 30 January 2025 (UTC)

P.S. If the mistake rate is about 100 on 10k items, it could be about 1%; so on 926k items we could expect around 9k rows in the table, of maybe fewer (I suspect the real rate could be between 0.5 and 1%, so probably a range between 5k and 10k mismatches). Epìdosis 10:42, 30 January 2025 (UTC)

Probably using SERVICE bd:slide as in this query that you suggested me a few months ago it would be easy. Epìdosis 11:35, 30 January 2025 (UTC)

@Epìdosis Perfect, I will write some code later to fill the page using your suggestions and let you know when it is finished! Difool (talk) 13:24, 30 January 2025 (UTC)

@Epìdosis, @Thomas Kerboul (BGE) I've created the page, the bot found 8945 items, so just below the 1%. I'm going to look at birth date and death date, but that will probably be after the weekend. Difool (talk) 19:48, 30 January 2025 (UTC)

Great, and super-fast! So now we can start coordinating about how to manage this page @Thomas Kerboul (BGE):, of course with no haste (we can calmly take some months/years). So, we should first settle a few points:

1. subpages: Property talk:P269/Gender mismatches is 865k bytes, so very heavy, and {{Q}} works only up to row 4669 as of now for me; I think that for these two reasons splitting it in multiple subpages will be necessary; I would suggest: we could use Property talk:P269/Gender mismatches as coordination page and create subpages like /2025/1, /2025/2 etc. - how about 18 subpages, with 500 items each? Their dimension would be 48k bytes on average at start, but estimating that each row can grow of about 350 bytes after solution, they could increase of 175k bytes reaching a total of about 223k bytes, which is reasonable IMHO
2. solution: in the columns "Judgement" and "Solution" I propose to use mostly standardised judgements, to facilitate future counts and avoid making the pages uselessly heavier than necessary; specifically, I propose
   1. "Judgement": use "Wrong WD" / "Wrong IdRef" / "OK both", followed by a longer comment in parenthesis only if necessary (especially for the cases of "other" in WD), and then the user signature
   2. "Solution": use "Fixed WD" / "Fixed IdRef" / "OK both", followed by a longer comment in parenthesis only if necessary, and then the user signature
3. avoid edit conflicts: so basically whichever user can fill in "Judgement"; then if the judgement is "Wrong WD", the same user can also add "Fixed WD" to "Solution"; but, since in the majority of cases probably the judgement will be "Wrong IdRef", only you Thomas can add "Fixed IdRef" to "Solution"; so basically we can coordinate so as not to edit at the same subpage at the same time; we could e.g. agree that I fill in the judgement (and fix WD wherever necessary) in subpage 1 for the first 100 rows, then while you fill in the solution for them I do the first 100 rows in subpage 2, then while you fill in the solution for them I do the first 100 rows in subpage 3, etc.

After we agree on these, and we have split the subpages, we can slowly start working IMHO. Buona serata! Epìdosis 20:50, 30 January 2025 (UTC)

• @1: Yes, I agree. How about adding a subpage for the Wikidata 'Other' gender? I don't think IdRef has a gender label for those.
• @2: I could add the text as hidden comment, making it easier to include them.

While checking some items, I found an obvious mismatch: No. 246, George Curzon, 1st Marquess Curzon of Kedleston (Q271878), 119091364. Would it be worth it to add more data from IdRef, such as the name, to the table?

The code for the report generator is here. Difool (talk) 06:11, 31 January 2025 (UTC)

I'm fine with the above proposed steps and solutions. Note however that taking care of IdRef (and even less the whole of IdRef) isn't my only duty, so expect things to go slow on the IdRef fixing side. From the previous query's results I also randomly noticed that fixing gender will sometimes just be a part of what I will be doing to improve the record (like resolving duplicates or deduplicates), which will slow things further down. Two years of work seems an optimistic minimum.

Running select ?gender (count(?gender) as ?gender_count) where {?idref foaf:gender ?gender } GROUP BY ?gender on https://data.idref.fr/sparql I'm only getting "male" and "female". I know for sure that I can specify "Inconnu" and "Non applicable" (and also remove the gender field altogether), but in the first case it seems it doesn't show up in the rdf https://www.idref.fr/076553876.rdf ; not sure for the second, I don't remember ever seeing it. We can add names, but I feel that the mismatch will be obvious while trying to fix the gender mismatch. --Thomas Kerboul (BGE) (talk) 08:16, 31 January 2025 (UTC)

Thanks for the fundamental help!

1: yes, I think that having a subpage for Wikidata "other" (311 cases if I count correctly) would be useful, since they are probably the most complex cases: we could use subpage 1 for "other" and then divide the other cases in 17 subpages of a little more than 500 cases.

2: yes, having the text as hidden comment would be helpful.

And yes, having a column for the label of IdRef would effectively be helpful: a certain number of cases is surely due to a misplaced IdRef, as Nr 246. Epìdosis 08:18, 31 January 2025 (UTC)

Oh, conflicted with @Thomas Kerboul (BGE):: yes, I perfectly agree that it will be a very long work - also for me, in fact, it is one of the many things I do on Wikidata (and outside Wikidata). Yesterday evening I was thinking that with nearly 9000 cases a solution in 10 months would imply ca. 30 cases a day and in 20 months ca. 15 cases a day ... the second could be optimistically feasible, but we will see, anyway it will remain a middle-long term project.

So I think we now agree on the split of the big table, on the hidden texts and on the IdRef names: after the 18 subpages are ready, we can start our slow and patient work ;-) Epìdosis 08:24, 31 January 2025 (UTC)

I've added the subpages and added the names from IdRef to the table. This time, I set a limit of 8000 and got more items. Any items in the table without a name from IdRef are either already resolved or have an unknown gender. An idea is to add on focus list of Wikimedia project (P5008) or something similar to track the items, which would allow to write a simpler query without needing to process all items with an IdRef ID. A bot could then update the table if something changes in WikiData or IdRef. Difool (talk) 07:07, 3 February 2025 (UTC)

Great job! OK, we are now ready to start. I have copied in Property talk:P269/Gender mismatches the instructions and I have created a table for coordination. I slowly start the check from subpage 2 and let's see! I think we can discuss problematic cases in a dedicated subpage like Property talk:P269/Gender mismatches/Discussion or a similar title in the future, so as to collect everything under one place. @Thomas Kerboul (BGE):, this will be a very long work, but surely very interesting! Epìdosis 10:22, 3 February 2025 (UTC)

Great! There are some shenanigans currently with my IdRef account, so I'll only be able to start in a few weeks. At that time hopefully I'll also have accesses to other tools. I just spotted Marie Antoinette (Q47365) in the list (who might be the most well-known French queen in France), can't wait to see what other marvels we'll fix. --Thomas Kerboul (BGE) (talk) 11:24, 3 February 2025 (UTC)

@Thomas Kerboul (BGE): the third I saw was Elizabeth I of England (Q7207): Queen of England and Ireland from 1558 to 1603 (1533-1603), which is also a bit astonishing, so I left a small commentary "(!)" ... it will be very interesting! Epìdosis 08:34, 4 February 2025 (UTC)

Epìdosis asked about how many items there are left for DifoolBot to remove reference URLs from. I rerun the queries Difool had run in October, and here are some results: VIAF IDs have been reduced from 405k to 308k items, BnF IDs from 212k to 87k items, RKDartists ID from 170k to 106k items, and IdRef IDs from 58k to only 171 items(!!). The database dump on QLever is currently dated 2025-01-29. You might be interested in seeing what are the items left for IdRef URL removal, so here's a query for that: [7] :). Samoasambia ✎ 05:43, 5 February 2025 (UTC)

@Samoasambia, @Epìdosis My estimate is about 250k items related to paintings and prints, and about 500k items related to humans. The bot processes around 8500 items per day, so it will take approximately 90 days to complete this task. I've configured the bot to search for items with reference URLs containing specific fragments such as 'idref.fr' or 'bnf.fr'. These numbers are only for those fragments; the bot doesn't search for the URLs of senat.fr ID (P1808) or PACE member ID (P1331) for example, but I believe those will not add significantly to the total after the current 750k items are processed.

I think the remaining IdRef URLs fall into the following categories:

1. New reference URLs. After everything is done, I plan to run the script regularly, for example, once every two months.
2. Within a reference containing multiple reference URLs; the bot skips those, so they need to be manually checked.
3. On a page where the bot encountered an error. I regularly fix logged errors by changing the code or by modifying the Wikidata pages.
4. URLs not captured by the regular expression, such as 'https://www.idref.fr/085799599#070', where the '#070' directs the browser to a specific section of the page. For those URLs the regular expression can be changed or a new one can be made. If the reference URL should be kept, it's best to create a new regular expression, because I flag whether the reference URL should be kept or not per regular expression.

Difool (talk) 11:45, 5 February 2025 (UTC)

Thanks for the explanation! Do you plan to add new URL fragments after the current ones are finished or just make the bot to query for any reference URL? If you choose the latter, I hope there aren't many scholarly articles to go through since that could take many months. Samoasambia ✎ 06:44, 10 February 2025 (UTC)

@Samoasambia - my plan is to gradually add more URL fragments to keep it manageable and focused on pages about humans. Initially, when running this bot task, I experimented with queries that searched for duplicate references. I stopped with that, because I encountered things I hadn't taken into account. I feel the code is now more robust, so I might redo that.

My primary focus has been on pages about humans. These pages contain dynamic information, with inputs from both humans and bots. Information on scholarly articles and paintings tend to be more static, with the pages generated once and rarely changed afterward, so they are not so interesting to me. Difool (talk) 06:51, 12 February 2025 (UTC)

Sounds good :). I do have some suggestions for the next fragments when the time comes to resume. Samoasambia ✎ 14:47, 28 March 2025 (UTC)

For the substitution of reference URLs, I suggest Magyar életrajzi lexikon ID (P8998) for a check on URL patterns in references (e.g. birth and death date in https://www.wikidata.org/w/index.php?title=Q1214176&oldid=2245299121). Epìdosis 10:09, 23 April 2025 (UTC)

Removal of reference URLs

Please don't remove reference URLs when you add GND IDs (like here). With online references, both URLs and their content might change in the future, so adding a retrieval date and the URL is essential for later verification. Adding a refence URL also triggers archiving in the Internet Archive, which makes later verification easy. If you know only the GND ID (which might link to a different URL in the future), this information is lost. --Bruno413 (talk) 08:01, 4 March 2025 (UTC)

@Bruno413 - GND IDs on a Wikidata page are displayed as identifiers, like 1150332263, but still include a link that can be found by web crawlers from the Internet Archive. I think you typically check the current version of the GND page for verification. Why would you check if a claim was supported by GND in the past? For example, if a date of birth has a reference to a GND page, and upon checking the GND page the information isn't there, I remove the reference. I don't go to the Internet Archive to see if the information used to be on the GND page. Difool (talk) 14:10, 4 March 2025 (UTC)

There is a difference between outdated references and invalid ones. If the URL or the content it links to is changed but the retrieval date is not updated, you created an invalid reference. The Wikipedia foundation cooperates with the Internet Archive such that adding links in Wikipedia or Wikidata articles automatically triggers a crawling of the page but I tested myself that on Wikidata, this works only for reference URL properties, not for implicit links via GND IDs. --Bruno413 (talk) 15:48, 4 March 2025 (UTC)

I don't see that; I created the page Jacob Schotte (Q122066528) on 30 August 2023. It doesn't have a reference URL to GND, but it does include a GND ID. The Wikidata page was archived on January 23, 2025 at 05:34:24, see here, and the GND page was also archived a bit later, at 05:42:42, see here. Same for other pages I checked. Difool (talk) 01:11, 5 March 2025 (UTC)

That sounds good. I recently created a reference with only a GND ID and checked a few days later and the page was not archived. I don't remeber which page it was, hopefully this was an exception. However, I still think that there a no real reasons to remove a fixed reference URL and replace it only by a GND ID with a mutable link, but there are good reasons to explicitly keep them. If the content it links to changes, then per Wikidata's policy a new claim with a new URL/date reference should be added (potentially with preference or deprecation ranking). --Bruno413 (talk) 05:58, 5 March 2025 (UTC)

@Bruno413 - I don't think we can reach a comprise here; I think I did my part by going through the requests for permissions at Wikidata:Requests for permissions/Bot/DifoolBot 6, so if you timely start a discussion at Wikidata:Project chat or Wikidata:Requests for comment or somewhere else, then I will halt the bot until this is resolved. Difool (talk) 07:57, 5 March 2025 (UTC)

I did not ask you to halt the bot. I think that adding stated in/GND ID properties to references is a good thing. I only asked you not to remove the reference URL property if it was explicitely added. --Bruno413 (talk) 08:00, 5 March 2025 (UTC)

DifoolBot nice work

DifoolBot's work - replacing long URLs with IDs - is the most thoughtful and intelligent project in recent times. If I can support you in any way (mainly voting, expressing opinions), please ping me. Thanks for your sensible work, cheers. Pallor (talk) 00:52, 25 March 2025 (UTC)

Hi @Pallor, thanks for the kind words! I started a discussion on the Project chat. Difool (talk) 04:08, 1 April 2025 (UTC)

Born and died on the same day

Hi! In this interesting case that @Bargioni: reported me, DifoolBot correctly imported from LCNAF an unfortunately wrong death date which was identical to the birth date; I have deprecated it and reported the mistake to LCNAF (I hope they will fix it, although my previous reports to LCNAF have been ineffective). It would maybe be interesting if DifoolBot compiled somewhere a report of cases where, after adding a date of death (P570), the difference between date of birth (P569) and date of death (P570) is unusually small (< 10 years) or big (> 100 years). Thanks very much as always, --Epìdosis 14:47, 25 March 2025 (UTC)

Hi @Epìdosis, good catch from @Bargioni! The bot already checks for unusual dates, like this one, but not for weird combinations. I changed the code using your suggestion. At the moment, the only case in the log is Rune Lagneborg (Q55082421), but I'll make a report somewhere if more cases come up. Difool (talk) 07:17, 26 March 2025 (UTC)

Clap, clap!  Bargioni 🗣 08:37, 26 March 2025 (UTC)

Btw, yesterday I randomly caught the bot adding a wrong gender. Samoasambia ✎ 14:54, 28 March 2025 (UTC)

@Samoasambia: thanks for noticing, now fixed in IdRef! --Thomas Kerboul (BGE) (talk) 15:44, 28 March 2025 (UTC)

@Samoasambia Ah yes, I'll change the code to ignore the items on the Property talk:P269/Gender mismatches list for now. Difool (talk) 16:35, 28 March 2025 (UTC)

89017 mostly humans having ISNI and GND ID and DDB ID not having VIAF ID

1. https://web.archive.org/web/20250410070036/https://www.wikidata.org/w/index.php?search=haswbstatement%3AP213+haswbstatement%3AP227+haswbstatement%3AP13049+-haswbstatement%3AP214&title=Special%3ASearch&ns0=1&ns120=1
2. https://archive.ph/47Ff1

They could be added from GND dump, but maybe your bot could query the VIAF API by GND verify the ISNI is there and also collect (not add) other IDs from the VIAF cluster.

Including those that have no VIAF there are 275000+ missing VIAF https://www.wikidata.org/w/index.php?title=Wikidata:Database_reports/Constraint_violations/P13049&oldid=2336524576#%22Item_VIAF_cluster_ID_(P214)%22_violations but maybe more data available for items that have ISNI. VIAFmissing (talk) 07:39, 10 April 2025 (UTC)

(Note: the user has been blocked indefinitely) Samoasambia ✎ 08:20, 10 April 2025 (UTC)

Maintenance of abandoned and redirected VIAF clusters

Hi! Waiting for DifoolBot to end its current task of fixing reference URL (P854) into IDs and restart its task of importing missing VIAF cluster ID (P214) on the basis of VIAF members already present in the items, I was also reflecting on the issue of maintaining periodically our existing VIAF cluster ID (P214), which practically consists in the deletion of abandoned clusters and in changing the redirected clusters into the new clusters (if the new cluster is already present in the item, as it often happens, the redirected cluster is just deleted).

@KrBot: (code not public unfortunately), maintained by @Ivan A. Krestinin:, used to do this maintenance periodically (ca. monthly I think), with just one problem, i.e. it does not consider ranks, so it also edits deprecated IDs (for this reason we also have cases of items having the same VIAF values as both normal-ranked and deprecated); it is not easy to check its activity since it does ca. 1 M edits a month divided into its various tasks, but my impression is that its last edit on P214 was on 3 January 2025 (this); I think that after VIAF changed its interface (and APIs) on 10 January 2025, KrBot stopped doing maintenance on P214 (@Bargioni: asked a confirmation to Ivan in User talk:Ivan A. Krestinin#Update obsolete VIAF clusters, but he has not answered yet), and this is of course an issue, since VIAF clusters change frequently and having to update redirected and abandoned ones manually is very time-consuming for the few users that work hardly on the relationship between Wikidata and VIAF.

Do you think DifoolBot could take up the task of maintaining abandoned and redirected VIAF clusters in the future? And, BTW, are you able to verify with certainty that KrBot effectively stopped doing this after 10 January 2025? Thanks as always! --Epìdosis 13:11, 17 April 2025 (UTC)

Hi @Epìdosis, I was actually just restarting Task 2 yesterday: I will let the bot run through Union List of Artist Names ID (P245) and make a report, and then check the results before starting with other properties.

With approximately 3.6 million pages containing a VIAF statement, checking all of them within a year would require around 7 checks per second, a significant load. So, I've always wondered how KrBot selects which pages to check. I assume it relies on Wikidata backups to focus on pages with recent changes or something, so the bot might also have been impacted by the backup issues that began in early 2025 and were resolved this month.

Using the Edit Summary Search tool, I checked KrBot's edits and found that the last resolved redirect was on January 6, 2025.

I'll start write some code to check the VIAF IDs on pages returned by a query, for example all pages with multiple VIAF IDs, and then we can check how that works out. Difool (talk) 03:11, 18 April 2025 (UTC)

By the way, I'm now encountering an API rate limit exceeded error from VIAF. Difool (talk) 05:59, 18 April 2025 (UTC)

OK, it's great to know that yesterday DifoolBot restarted task 2. I will restart monitoring User:Difool/viaf already somewhere as in the old times ;-)

Yes, checking all VIAF statements is in fact unfeasible; or, it would be feasible only using a VIAF dump, but VIAF published the last in August 2024 (https://viaf.org/en/viaf/data) and now writes:

The files on this page were last updated in August 2024 and are currently not being updated.
We are working on VIAF security and production environment improvements to provide you with a better experience.
Once that work is complete, we will resume making updated files available.

Scraping each VIAF used in Wikidata is surely not a viable possibility; it would be worth, as you say, checking periodically at least the few thousands of items containing multiple VIAF IDs, I completely agree. And yes, the backups of Wikidata could of course help in seeing recently change items in Wikidata.

Great to know the Edit Summary Search tool, I had never used it but it will be very useful for me in the future.

Yes, VIAF in changing interface in January 2025 also introduced a rate limit of 1k per day and 10k per month ... it can be bypassed by changing IP after the rate limit is reached, but it is very annoying; probably it was done to protect VIAF from AI scrapers and/or because they are preparing to put VIAF partially under paywall or something similar :( Epìdosis 06:53, 18 April 2025 (UTC)

@Epìdosis Hmm, that's unfortunate. User VIAFmissing mentioned above that there are over 89,017 pages with a GND ID but no VIAF ID, which means Task 2 will take at least nine months to process the GND IDs. Difool (talk) 07:21, 18 April 2025 (UTC)

Yes ... unless we the bot changes IP whenever it reaches the rate limit. I don't know how much difficult it could be. Epìdosis 12:29, 18 April 2025 (UTC)

Less precise dates

Hi, I see that your bot has added the year of birth to this item, but has not marked the statement as deprecated. Please be more careful and deprecate less precise dates to avoid duplicate values. I would appreciate it if you could check your previous edits for such errors, as duplicate dates cause problems in local infoboxes. Thanks, Bencemac (talk) 14:03, 29 June 2025 (UTC)

Hi, @Bencemac, I'll update the bots code to do that. There are other bots that set the preferred rank to more precise dates, but I'll write some code to double check my previous edits. Difool (talk) 01:18, 30 June 2025 (UTC)

Thank you very much! Bencemac (talk) 05:58, 30 June 2025 (UTC)

I think at least Frettiebot (ping Frettie) fixes those but it comes quite late, something like a month after the edit(?) Samoasambia ✎ 19:54, 30 June 2025 (UTC)

The bot is now going through the list. Till now (9% done) it has only changed items with dates with month/decennium/century precision. Difool (talk) 04:53, 1 July 2025 (UTC)

Yes, we correct daily vs. annual accuracy (when the day has a ref), for both births and deaths. Sometimes, not regurarly.--Frettie (talk) 06:07, 1 July 2025 (UTC)

The bot has completed its task. It grouped date claims into groups. If a property [Date of birth/date of death] had only one group containing multiple claims, and none of those claims had qualifiers, the bot marked the most precise claim as preferred. Below are some statistics; between [] are the qualifiers for pages with 1 group or a group with Unknown value.

• Unknown value with qualifiers should be translated to some date.
• Unknown value without qualifiers and without source can probably be deprecated/removed.
• Unsourced best claim could be deprecated/removed.
• Same dates across calendar models is okay, for example: 21 January 1863 Julian and 2 February 1863 Gregorian.
• Same year/month/day, different calendar model is an error.
• This edit is somewhat questionable, as 30 December 1879 in the Julian calendar corresponds to 1880 in the Gregorian calendar.

# groups	Description	Count	Samples
3	Day, Day, Month	1	Q104723584
2	Century, Unknown value []	1	Q25420328
3	Year, Year, Unknown value []	1	Q4115520
4	Day, Day, Year, Year	1	Q14918072
2	Day, Unknown value [latest date]	1	Q104816273
3	Day, Year, Unknown value [latest date]	1	Q8248634
3	Year, Year, Unknown value [earliest date]	1	Q13032547
2	Decade, Unknown value [refine date, latest date]	1	Q5952884
1	Qualifiers: Day [day in year for periodic occurrence]	1	Q2662227
1	Qualifiers: Day [circa]	1	Q64761711
1	Qualifiers: Day [country]	1	Q11725403
1	Qualifiers: Day [presumably]	1	Q8961137
1	Qualifiers: Day [refine date]	1	Q94686061
1	Qualifiers: Century [refine date]	1	Q12210161
1	Qualifiers: Month [earliest date]	1	Q11731297
1	Qualifiers: Decade [Gregorian<1584]	1	Q5791906
1	Qualifiers: Year [circa, earliest date]	1	Q22994285
1	Qualifiers: Month [earliest date, latest date]	1	Q21589380
1	Qualifiers: Year [circa, earliest date, latest date]	1	Q112054585
1	Qualifiers: Century [earliest date, latest date, circa]	1	Q105679191
1	Qualifiers: Year [earliest date, latest date, unspecified calendar]	1	Q4199876
2	Day, Unknown value []	2	Q21001479, Q112348398
2	Month, Unknown value []	2	Q73205916, Q112347034
4	Year, Year, Year, Year	2	Q402807, Q7967017
2	Year, Unknown value [latest date]	2	Q9370440, Q100088169
1	Qualifiers: Month [circa]	2	Q22936406, Q28064283
1	Qualifiers: Year [refine date]	2	Q16120302, Q117326716
1	Qualifiers: Month [Julian>19XX]	2	Q8433304, Q26262573
1	Qualifiers: Century [Gregorian<1584]	2	Q4387115, Q18945386
1	Qualifiers: Year [latest date]	3	Q2086330, Q16503801, Q22986379
1	Qualifiers: Month [Gregorian<1584]	3	Q846349, Q3875874, Q16650432
1	Qualifiers: Century [earliest date]	3	Q1685310, Q16067153, Q72219111
1	Qualifiers: Year [Gregorian<1584, circa]	4	Q79203, Q3085025, Q5953620
3	Day, Year, Year	5	Q3767590, Q11713717, Q11922924
2	Year, Unknown value [earliest date, latest date]	5	Q3647004, Q4529318, Q12096150
1	Qualifiers: Day [age of subject at event]	5	Q11666615, Q16263965, Q22348433
3	Day, Day, Year	6	Q4094650, Q5490866, Q6915821
1	Qualifiers: Decade [earliest date, latest date]	6	Q11731747, Q27140220, Q27181038
2	Month, Month	8	Q3339589, Q4384891, Q6600659
2	Year, Unknown value [earliest date]	10	Q2919382, Q4104088, Q4111916
1	Qualifiers: Year [refine date, Islamic calendar]	10	Q2916804, Q3660738, Q4118171
2	Year, Century	11	Q2726403, Q4703257, Q5664711
1	Qualifiers: Day [Gregorian<1584]	12	Q1028786, Q2653389, Q2997518
2	Year, Decade	16	Q1120219, Q3284975, Q4190486
3	Year, Year, Year	17	Q4372816, Q4501515, Q5666972
1	Qualifiers: Day [Julian>19XX]	18	Q4162844, Q4409262, Q4462187
2	Year, Unknown value []	32	Q852415, Q1003490, Q2777938
2	Day, Month	43	Q1161528, Q2609391, Q3384385
2	Month, Year	44	Q197605, Q3384191, Q3916356
3	Day, Day, Day	45	Q1161654, Q1311440, Q3648780
	Redirect	86	Q17509089, Q20684084, Q22934655
1	Qualifiers: Year [circa]	87	Q65091, Q1188700, Q1931849
2	Different dates across calendar models	101	Q80529, Q191619, Q328549
1	Qualifiers: Year [Gregorian<1584]	120	Q387856, Q545278, Q1017968
2	Same year/month/day, different calendar model	197	Q603879, Q777090, Q947039
2	Same dates across calendar models	211	Q558838, Q1584750, Q1735737
1	Changed	696	Q94394, Q171202, Q716091
2	Day, Year	720	Q338178, Q347925, Q853791
2	Year, Year	1045	Q68234, Q128485, Q484901
2	Day, Day	1725	Q55745, Q72064, Q72071
1	Unsourced best claim	5718	Q54392, Q65301, Q130939
1	Nothing changed	120579	Q8015, Q39378, Q45142

Mini Wikimedia Hackathon

Hi, we’re organizing another Mini Wikimedia Hackathon in Utrecht (NL) on October 25. Maybe something for you? It’s small-scale, technically oriented, and in-person. If this suits you, you can sign up via the event page Daanvr (talk) 17:56, 13 October 2025 (UTC)

Wikimedia Hackathon Northwestern Europe 2026

Hello! I wanted to let you know about a hackathon we're organizing: the Wikimedia Hackathon Northwestern Europe 2026, on 13–14 March in Arnhem, Netherlands.

It's a two-day, technically oriented hackathon bringing together Wikimedians from the region. Registration closes mid-January or when full. Let me know if there are any questions. Hope to see you there! Daanvr (talk) 23:28, 4 January 2026 (UTC)

Date precision changes with references

Hi! When changing a date from year to day precision, you can't keep the same references - they only support the year. For example, see your edit and the bot's correction. Best practice is to add the day-date as a new value with its own references and mark it preferred, or else remove the year-date entirely before adding the day-date with correct references. Thanks!

(This is an automatic message generated by a bot to explain the correction.) DifoolBot (talk) 01:53, 5 January 2026 (UTC)

Appreciated! ELSchissel (talk) 16:45, 5 January 2026 (UTC)

Wikimedia Hackathon Northwestern Europe 2026

Hello! I noticed you're a bot operator, so I thought you might be interested in a hackathon we're organizing: the Wikimedia Hackathon Northwestern Europe 2026, on 13–14 March in Arnhem, Netherlands.

It's a two-day, technically oriented hackathon bringing together Wikimedians from the region. Whether you want to work on bot frameworks, tools, or other technical projects, this could be a great opportunity to collaborate with fellow developers. Registration closes mid-January or when full. Let me know if there are any questions. Hope to see you there! Daanvr (talk) 23:18, 11 January 2026 (UTC)

Hi @Daanvr, thanks for the invite. I'm not around the Netherlands these days, so I'll have to pass. Difool (talk) 01:22, 12 January 2026 (UTC)

Emeritus professors

Hi! If you are interested, this is not urgent but would be a useful standardisation of the data model: Wikidata:Bot requests#Standardise professor emeritus statements. Thanks very much as always, --Epìdosis 17:53, 4 February 2026 (UTC)

Hi @Epìdosis, I'll work on it next week or so, unless someone does it before me. Difool (talk) 01:20, 6 February 2026 (UTC)

@Epìdosis I left a message on the bot request page, but forgot to ping you. Difool (talk) 14:47, 12 February 2026 (UTC)

Q4164543 – Ma'ruf al-Rusafi

Hello,

Can you please explain this edit?

חזרתי (talk) 04:53, 7 February 2026 (UTC)

Hi @חזרתי, this edit is part of a project to find and undo changes where a date of birth or death, with sources, was changed from year-only to full day precision. The problem is that the sources only support the year, not the exact day.

For this step, I searched for birth/death dates with day precision that use a Trove reference, because Trove usually does not give exact days. For example, see the Trove page for Ma'ruf al-Rusafi. The bot checks which revision changed the claim from year to day precision with a Trove reference, then undoes that edit: It keeps the year-precision date, adds the day-precision date separately, and moves any new references to that day-precision date. This page shows the full log of the project.

For Q4164543, this edit was found. I only did some random checks and did not notice the big difference in years. I will now check other edits where there is a difference in years. Difool (talk) 13:24, 7 February 2026 (UTC)

Thanks. חזרתי (talk) 13:27, 7 February 2026 (UTC)

WikidataCleanup - minor issue

Thanks for WikidataCleanup, it is a very useful tool. I have, however, identified an edge case where the tool does not function correctly. This concerns the recording of financial data for companies whose fiscal year differs from the calendar year. In such cases, the relevant financial properties (P2139, P2295, P3362) use the start time and end time qualifiers to define the beginning and end of the fiscal year. As a result, a property may include an end time qualifier while still being the most up-to-date value (thus have preferred rank). The tool, however, incorrectly suggests that this value should be degraded to normal rank. Jklamo (talk) 07:37, 15 February 2026 (UTC)

@Jklamo @RVA2869 Is this only relevant when the fiscal year differs from the calendar year? Or do these only have end-time qualifiers?

I think the distinction between properties like total revenue (P2139) and spouse (P26) might be that, for spouse, once an end time such as a divorce is recorded, the current value is immediately clear: no spouse. Total revenue updates stepwise: the current value (calendar year 2026) cannot be known until the calendar year concludes. In such cases, using most recent value (Q71533355) as the reason for preferred rank (for calendar year 2025, with an end-time in the past) makes sense, but not so for properties like spouse or citizenship.

Because it's difficult to classify what type of property a given property is, I think the solution is that the tool should avoid making these suggestions when a most recent value (Q71533355) reason for preferred rank is already present, whatever the property is. Difool (talk) 01:55, 16 February 2026 (UTC)

An update on authority records' issues

Hi! An update from my side, today I have succeeeded in emptying Property_talk:P213/Duplicates/humans (it peaked above 1400 rows last December) for the first time since July 2023. A cleanup on a lot of old bad matches of CERL Thesaurus ID (P1871) is also starting, cf. Property talk:P1871. Best, --Epìdosis 14:09, 22 March 2026 (UTC) P.S. in cases of ISNI conflations I have reported to ISNI and obtained the correction of the conflated ISNI and the creation of proper ISNIs for the other involved persons, then I have updated Wikidata accordingly; this should make more difficult for these conflations to come back here

Hi @Epìdosis, that's a lot of work, well done! I assume many of the duplicates came from Tamawashi's mass GND import. The GND-VIAF detected duplicates were massive too, but I noticed that a lot have already been resolved. I updated the page User:Difool/viaf already somewhere with some GND duplicates, but I got busy with other things here on Wikidata.

Good to hear that ISNI is responsive to reports; I'll keep that in mind when spotting conflations. I'm not sure I've been consistent about marking them as conflated, so I'll try to do that. And good luck with the CERL P1871 cleanup; I may drop by if time allows! Difool (talk) 01:22, 26 March 2026 (UTC)

Submission for URL-deprecation blocklist

For the Sponsored or contributed content section, Rolling Stone's "Culture Council": https://www.rollingstone.com/culture-council/ Trivialist (talk) 23:52, 23 April 2026 (UTC)

Dates -01-01 and -00-00

Hi! I have a small request regarding Wikidata:Requests for permissions/Bot/DifoolBot 3: when it adds new dates, or it edits dates, with precision year (9), e.g., could the bot always use the probably most common format -01-01 instead of -00-00? Waiting for a general standardisation (phab:T310981), I think it is a better solution. Thanks as always! --Epìdosis 10:46, 5 May 2026 (UTC)

Wrong label imported from IdRef

Hi, would you be able to track down edits like this one? I guess the script got confused with the uppercase of the occupation and mistook it for the first name. I can correct this wrong uppercase but it would be easier with a list of impacted items. Thanks, --Thomas Kerboul (BGE) (talk) 12:41, 9 June 2026 (UTC)

Hi @Thomas Kerboul (BGE), yes, good catch; the script checks the skos:prefLabel, which can include text in parentheses. If it is all lowercase or contains numbers, it is ignored; otherwise, it is assumed to be the long given name, but this is apparently too lax as you found; it can also contain longer text like "auteur d'une thèse de Mathématiques Daïm". I’ll rethink how best to handle this.

I looked at the items done and found these problems (some might not have been done by my script):

SELECT ?item ?itemLabel ?alias WHERE {
  VALUES ?item {
    wd:Q110300323
    wd:Q102393559
    wd:Q110164300
    wd:Q111644604
    wd:Q102620958
    wd:Q112487143
    wd:Q63214284
    wd:Q7612049
    wd:Q112464122
    wd:Q110300323
    wd:Q102393559
    wd:Q106644590
  }
  ?item rdfs:label ?itemLabel, ?itemLabel.
  FILTER((LANG(?itemLabel)) = "fr")
  ?item skos:altLabel ?alias.
  FILTER((LANG(?alias)) = "fr")
}

Try it!

Difool (talk) 16:08, 9 June 2026 (UTC)

I'll leave it as is for now, so you can review it, and then make corrections [that is, I'll correct them here on wikidata] tomorrow or so. Difool (talk) 16:12, 9 June 2026 (UTC)

Thanks, I will correct them before the end of the week. I will contact ABES and try to solve as many other with the following query (lots of false positives but also lots of things to work on).

PREFIX foaf: <http://xmlns.com/foaf/0.1/> 
PREFIX skos: <http://www.w3.org/2004/02/skos/core#>

 select distinct ?p ?label ?valeur_actuelle where {
  ?p a foaf:Person ;
     skos:prefLabel ?label .
  FILTER(REGEX(?label, "\\([A-Z]"))
  BIND(STRBEFORE(STRAFTER(?label, "("),")") AS ?valeur_actuelle)
}

Try it!

As for the script, in case the item is rdf:type foaf:Person, wouldn't it be easier to retrieve foaf:name? --Thomas Kerboul (BGE) (talk) 17:02, 9 June 2026 (UTC)

Update on VIAF

Hi Difool! I am very happy to write you that I have just finished to empty again Property talk:P214/Duplicates/humans, and I am very near to emptying Property talk:P213/Duplicates/humans. I think it could be a good moment to launch again the task of inferring VIAF ID from the members. Thanks very much as always! --Epìdosis 16:48, 25 June 2026 (UTC)

Hi @Epìdosis, good to hear! I've started the task this morning, starting with P244 LoC, then I'll let it do P214 + the rest. The daily limit of 1000 still applies, but the monthly cap seems to have disappeared. BTW I'm now not running it, but do you think I should continue with Wikidata:Requests for permissions/Bot/DifoolBot 3, or is that less relevant now given the mass imports you recently carried out? Any plans to also do that for maybe BNF/IdRef/KRNLK? Difool (talk) 14:48, 26 June 2026 (UTC)

Hi @Difool:, perfect, thanks! Fingers crossed for VIAF, not having the monthly limit would be very good. Task 3 IMHO is still relevant: I made imports from GND mainly because it is 1) with a SPARQL endpoint 2) CC0 3) with very structured data; BNF and IdRef only have good SPARQL endpoints, but no CC0 (which anyway is not a problem for taking data from individual records) and no structured data for places and occupations, so I have no plans as of now to do massive imports from them. Epìdosis 16:00, 26 June 2026 (UTC)

BTW, a feedback on the section with VIAF cluster pointing to two GND IDs: it was very useful, all but one were GND duplicates which I have marked and reported. If you find other similar cases, it is surely interesting to collect them! Epìdosis 16:14, 27 June 2026 (UTC)

Hi @Epìdosis, yes I add those duplicates now to all sections; LoC has just ended, now starting with GND: it has 8506 items to check. Difool (talk) 01:17, 2 July 2026 (UTC)

I see that most of the GND returns are duplicates. I'll cap the list at 1000 duplicates to keep it manageable and then continue with the other identifiers. Difool (talk) 01:41, 3 July 2026 (UTC)

I will continue to check the GND reports (and also the others) during August. A small curiosity: which is the periodicity of the action "rescored Score column" performed by DifoolBot on User:Difool/viaf already somewhere? Thanks, Epìdosis 09:56, 1 August 2026 (UTC)

I now start it manually once in a while when I've checked some items, but I could set it to run daily or so for a while. What do you think? Difool (talk) 14:40, 1 August 2026 (UTC)

I think daily could be useful, at least in this period with a lot of work around the table. Epìdosis 16:27, 1 August 2026 (UTC)

Suggestions for URL-deprecation blocklist

• 24-7pressrelease.com: press release distributor
• accessnewswire.com: press release distributor
• alchetron.com: Wikipedia mirror
• einpresswire.com: press release distributor
• issuewire.com: press release distributor
• news.marketersmedia.com: press release distributor
• usanews.com: press release distributor
• wikispooks.com: far-right conspiracy "encyclopedia"

Trivialist (talk) 19:23, 20 September 2026 (UTC)