User talk:Theknightwho

Hey. You have to block me, I'm about to go fucking crazy Van Man Fan (talk) 22:36, 30 March 2023 (UTC)

I assume this is related to your etymology-language changes, but categories like CAT:German terms derived from Classical Latin are now empty (see CAT:Empty categories), and the category text for a category like Category:German terms derived from Late Latin redlinks the word 'Late Latin' to Category:Late Latin language when it should link to Category:Late Latin. Benwing2 (talk) 01:55, 1 April 2023 (UTC)

@Benwing2 Good catch - fixed. Theknightwho (talk) 02:04, 1 April 2023 (UTC)

I think you need to rethink your edits relative to this code. For instance, Philistine is a real language of unknown affinities, and has 11 lemmas. Many of them are questionable, but we can't rfv them- because your edits have messed up the processing of "und-phi", which is an exception code based on "und", not "und" itself.

Also, "und" has been used as a placeholder when the correct language code is unknown. Most of the time this is a bad idea, but there are some cases where it's at least arguable. For instance, while inheritance by "und" is clearly wrong, borrowing doesn't require any relationship between the two languages, so borrowing into a language with an unknown code can certainly happen. On a practical level, these "und" descendants have been added by very knowledgable people to deal with problems they couldn't solve. They aren't going to be easy to clean up. Chuck Entz (talk) 16:00, 1 April 2023 (UTC)

After spending a good part of my day on module errors, it looks like the problem with Philistine has nothing to do with the "und" part. Instead, it's due to it having nil instead of a family code. There are a number of languages like this, mostly in the Americas but also in Africa or the ancient Near East. What they have in common is extremely limited attestation due to having died out before a decent corpus could be collected or produced. With such a limited corpus, it's often impossible to determine whether a given language is related to others, or even whether it's an isolate. I've spot-checked a number of these, and they all throw the same error in the {{rfv}} template, due to the module's checking the family in order to decide which rfv forum to send it to. The simplest solution would be to just assume that anything with a null family code goes to RFVN, and skip the rest. We really do need to have {{rfv}} working for these, since unknowns tend to attract crackpots or overconfident idiots who are only too happy to fill in the blanks. See for instance the trouble we've been having with Illyrian (never mind that we're pretty sure it's Indo-European), though Philistine itself is a pretty good illustration, too. Most of the Philistine entries were added and/or edited either by BedrockPerson and socks or by ShlomoKatsav, who probably knows better by now. Interestingly, most of ShlomoKatsav's entries were deleted after an RFV initiated by what seems in retrospect to be a BedrockPerson IP sock. But I digress...

Also, the two Proto-Turkic entries in CAT:E seem to be the same thing going in the other direction: they're only attested in one very old manuscript covering a multitude of languages- so it's apparently impossible to tell exactly what language they are- but there's enough data to figure out which branch they belong to. Chuck Entz (talk) 02:43, 2 April 2023 (UTC)

@Chuck Entz You make some very good points, which I hadn't considered - I've removed the ban.

This is something that came up due to the problem of substrates, where (at least internally) they're treated as etym-only variants of und, but with the language family of whatever their parent is; as what distinguishes them is that they have families for parents, not regular languages. Before the major update to etym-only languages that just happened, they were being caught by the ban on using family codes in descendant templates, as the template would grab the parent, see that it's a family, and throw an error. However, after the update, they'd be getting processed as und instead - and therefore working.

To be honest, I suspect the old logic was simply an oversight, as substrates are hardly used. If we allow und for descendants, then there's no reason we shouldn't allow substrate descendants either. Theknightwho (talk) 16:05, 2 April 2023 (UTC)

@Chuck Entz I've fixed the family bug. The underlying issue was that the language's inFamily first checks if the language is in family X. If not, it then checks if the language's family is in family X, (and if not checks that family's family etc). It's that second step which was throwing the error, because you can't check what family nil is in. I've changed it so that it immediately returns false if a language doesn't have a family; the practical result being that Module:request-forum will see that these terms aren't Italic, so sends them to WT:RFVN. Theknightwho (talk) 23:05, 2 April 2023 (UTC)

I saw you had recent edits to Module:ltc-pron/data so I wonder if you know where the following bug is coming from.

The final in the Middle Chinese table (4th row) for certain rimes loads the wrong rime character but the right number.

Examples where you can see include: 溢 (showing 眞 instead of 質) and 沃 (showing 冬 instead of 沃)

It seems the items in the "data.fin_conv" variable in the module are the ones affected.

Zywxn (talk) 16:32, 2 April 2023 (UTC)

@Zywxn: these are in fact correct and not a bug: 質 is the entering tone equivalent of 眞, likewise for 冬/沃 and the other ones in data.fin_conv. The entering tone variant is by all means phonologically the same as the non-entering tone equivalent except that it ends in a stop and being an entering tone.

For the purpose of the module, saying that a character has 冬 rime and entering tone is sufficient for telling the reader that it has 沃 rime. – Wpi31 (talk) 19:25, 2 April 2023 (UTC)

See CAT:E. Benwing2 (talk) 03:34, 3 April 2023 (UTC)

@Benwing2 Thanks - was a metatable issue, where the parent now caches _type during the creation of an etym-only language due to running :hasType(), but that was interfering with the etym-only language's own :hasType method by giving a false positive when it checked for a cached _type table: Theknightwho (talk) 04:12, 3 April 2023 (UTC)

This is definitely a non-obvious interaction; can you add a comment by your change in ? Otherwise it won't be at all obvious why this is being done. Benwing2 (talk) 04:17, 3 April 2023 (UTC)

@Benwing2 Done. Theknightwho (talk) 04:21, 3 April 2023 (UTC)

My bot created a bunch of these, which is an indication that the category code isn't working quite right. I've stopped it from creating more but the category code needs fixing here. Benwing2 (talk) 05:50, 4 April 2023 (UTC)

BTW I can delete these once you fix the category code. Benwing2 (talk) 05:51, 4 April 2023 (UTC)

@Benwing2 Thanks - all of these have been dealt with. Theknightwho (talk) 10:54, 5 April 2023 (UTC)

Hi again, see CAT:E. I purged the category, which eliminated the Old Iranian errors, but the Middle Iranian ones remain. I assume these categories shouldn't exist at all; "Middle Iranian" and "Old Iranian" are not languages at all, but families (and not even clades at that). However, this is a sore subject for User:Vahagn Petrosyan, who insists that these "languages" should exist for ease in porting over Armenian etymologies, where it's apparently difficult to figure out which of several candidate Old and Middle Iranian languages are the right donors for various borrowed terms, and so the linguists working in this area lazily put "Middle Iranian" and "Old Iranian". I have difficulty understanding how this can possibly work, as the phonologies of different Old and Middle Iranian languages are radically different, but go figure. Benwing2 (talk) 06:31, 5 April 2023 (UTC)

You can make both Old Iranian and Middle Iranian codes categorize into "Iranian languages" if that will make things easier. For example, the etymology of սրահ (srah) would display "From Middle Iranian *srāh" but categorize into Category:Old Armenian terms borrowed from Iranian languages. Vahag (talk) 07:18, 5 April 2023 (UTC)

@Benwing2 @Vahagn Petrosyan So Old and Middle Iranian are the only two etymology-only families that we have. Unlike with substrates, where they behave like variants of und, that doesn’t seem appropriate here. I can fix them up, but the question still remains whether we should even have something like this at all. Theknightwho (talk) 10:54, 5 April 2023 (UTC)

I have come to the conclusion that despite some claims, Old Iranian borrowings in Armenian probably do not exist. The main reason for having separate "Middle Iranian" and "Old Iranian" has disappeared for me. I don't mind if the codes are deleted. You can bot-replace {{der|xcl|MIr.}}, {{der|xcl|OIr.}} with "Middle {{der|xcl|ira}}", "Old {{der|xcl|ira}}". Vahag (talk) 11:58, 5 April 2023 (UTC)

I find it okay too, even though it is certain that both Old and Middle Iranian borrowings exist in Aramaic. Not every language conceptualization with a name needs to have a code, if it is not even a language per se and etymological storytelling does not require it either. It is perfectly okay if Sarri.Greek has written “Hellenistic Koine Greek” on various places because that’s what she got to know while language codes may not do anything here. I used “Middle Iranian” because of analogy and Systemzwang more than inherent reason. Fay Freak (talk) 14:29, 5 April 2023 (UTC)

@Vahagn Petrosyan @Fay Freak From a technical perspective this is easy to fix - I agree it isn’t a true family, but it feels a bit crap to remove it entirely. This feels like the sort of thing we could keep for the purpose of categorisation, but nothing more.

That being said, I don’t know much about Indo-Iranian languages - I’m just commenting on the fact that there’s no technical limitation. Theknightwho (talk) 14:34, 5 April 2023 (UTC)

Middle Iranian and Old Iranian complicate the categorization. Now we have Category:Old Armenian terms borrowed from Middle Iranian languages whose members do not directly appear in Category:Old Armenian terms borrowed from Iranian languages, which is not optimal. Let's remove them. Vahag (talk) 16:06, 5 April 2023 (UTC)

See for example 南#Coordinate terms, where ] appears after 北 and 南. – Wpi31 (talk) 19:48, 6 April 2023 (UTC)

@Wpi31 Fixed. I forgot that Module:zh-translit needed to be tweaked after a change to Module:links: . Theknightwho (talk) 20:11, 6 April 2023 (UTC)

This doesn't look right. There are a lot of such categories with 'Unspecified' for the script, e.g. Category:Requests for Unspecified script for Old Uyghur terms. I suspect you should avoid generating such categories for 'Unspecified' script as well as probably for the 'Undetermined' language. Benwing2 (talk) 06:41, 7 April 2023 (UTC)

@Benwing2 Thanks - I’m on mobile right now, but I suspect this is down to substrates being handled as und. Will have a look when I get home. Theknightwho (talk) 20:23, 7 April 2023 (UTC)

If it helps any, these are completely unrelated to the Middle Persian issue. There seems to be something about Module:ka-form of in those entries that's not setting up the parameters for Module:links the way the latter is expecting since you changed the code. I checked the transclusion list for one of the entries: none of the Georgian-specific modules or Georgian-specific data in data-modules has been changed recently except for some changes to Module:Geor-translit on March 30. Chuck Entz (talk) 21:43, 7 April 2023 (UTC)

@Chuck Entz I know what’s causing the problem, as it’s down to Module:links destructively modifying the input. Normally this doesn’t matter, but it does sometimes matter in loops when the same data container gets used over and over. This came up with a Pali module as well.

It’s possible to solve by doing what’s called a “shallow copy” (where you create a copy of the container that also contains the originals, versus a deep copy that copies everything inside as well). That way, any modifications to to the container don’t affect the original one. However, this has a (very minor) memory impact, and also seems to break modules that are accidentally passing the input data as a global variable. I need to catch all of those first (such as the one I just fixed in Module:interproject). The memory impact is negligible, but will probably require some additional saving measures on extremely vulnerable pages like go. Unfortunately, there’s no real way around that impact, as the destructive modification is the most efficient way to implement multiple forms for links.

Incidentally, Module:headword has suffered from this same problem for a while, so I’ll look into fixing that, too. Theknightwho (talk) 21:54, 7 April 2023 (UTC)

I'm not sure exactly how to fix these, but I think the problem stems from assuming that a |tr= parameter is always an implicit request for a native-script equivalent. After weeding out a couple of incorrect language codes and fixing cases where the |tr= was an improvised substitute for using an empty first parameter with a display text in the second parameter to display without linking (I haven't figured out yet how to fix some of these in {{desc}}), there are a few cases where the |tr= is some kind of phonetic transcription meant to display alongside the term specified by the usual parameters. There might be a need for a new parameter for this function, but I'm not sure what to call it and it should be discussed first. At the very least I think you should avoid generating a native-script request category for Latin-script languages, since {{auto cat}} will throw a module error if you create it. It may also be desirable to display Latin-script |tr= text for Latin-script terms as if it were a transliteration, even though it technically can't be one and there's no translit module for it. Chuck Entz (talk) 21:01, 8 April 2023 (UTC)

@Chuck Entz I think this is what ts= is for, though using that still produces the request for native script terms. To be honest, I actually think it's worth lifting the ban on having that for the Latin script, as it's still an accurate description for something like {{cog|lus||ts=puŋᴴ}}, which produces Mizo (/⁠puŋᴴ⁠/). Theknightwho (talk) 01:17, 9 April 2023 (UTC)

I think Template:trans-see currently generates wrong markup.

See, for example, amenity, then Translations, then "convenience — see convenience".

Currently generated makup:

<div class="NavFrame"><div class="pseudo NavFrame">...</div></div>

Should be

<div class="NavFrame pseudo">...</div>

So it should be sinlge .NavFrame element and I believe "NavFrame" should also come before "pseudo". Disfated (talk) 12:41, 9 April 2023 (UTC)

@Disfated Thanks - I’ll take a look. Theknightwho (talk) 13:09, 9 April 2023 (UTC)

@Disfated Are you sure that's what you're seeing? I see this:

<div class="pseudo NavFrame"><div class="NavHead" ... >...</div></div>

Which matches the pre-Lua version: . Theknightwho (talk) 15:34, 9 April 2023 (UTC)

Actually, I've just spotted there's a third div outside of those two which shouldn't be there. My bad. Theknightwho (talk) 15:35, 9 April 2023 (UTC)

We need to be more selective with where we put this. Using it on something like Module:zh/data/st is like throwing a device in a bucket of water so we can be 100% certain we know what the problem is...

It was working fine for over a month, and I spent a long time working on that check. Something new has thrown it off, but it’s important that we keep it. Theknightwho (talk) 22:08, 9 April 2023 (UTC)

@Chuck Entz - ping failed the first time. Theknightwho (talk) 22:10, 9 April 2023 (UTC)

That's okay, I have you watchlisted. While you're at it, take a look at Category:Proto-Indo-European language- this is the first time we've ever had an out-of-memory error in {{auto cat}}. I suspect it has something to do with family trees, since PIE has the biggest one. Chuck Entz (talk) 22:15, 9 April 2023 (UTC)

@Chuck Entz If you check Module:zh/data/ts or Module:zh/data/st now, you’ll see there are tons of results, but it runs quickly with a relatively low overhead, all things considered. Seems the serialisation checks are causing problems, which is not surprising given how massive Module:Hani-sortkey/data/serialized is; but using it means we can sort 1.5k Chinese terms with a 1.5MB overhead instead of a 15-20MB one.

The {{auto cat}} issue I’m aware of. It’s because I just consolidated Module:etymology languages into Module:languages, as the modular separation between them was becoming untenable. On pages with very few calls (e.g. 1), require is actually way more memory efficient than mw.loadData - using around 50% less in some cases. Currently, we’re in a brief transitional stage where we’re doing both, so once that’s over these issues should go away. Should hopefully be tomorrow. On a related point, I also rewrote Module:family tree/nested data to be more comprehensive and flexible, but it’s now an even bigger memory hog than it was before. It’s probably why we’re hitting the limit for the first time, as the PIE tree uses 37MB alone. That should drop down to about 25MB with the changes, bringing us well within the buffer. Theknightwho (talk) 22:33, 9 April 2023 (UTC)

@Chuck Entz The memory footprint of family trees has now dropped significantly, with Category:Proto-Indo-European language now using 42MB; the tree on its own uses 27MB. Incidentally, I think Category:Proto-Niger-Congo language has the largest tree - it's nearly twice the size. Theknightwho (talk) 18:09, 10 April 2023 (UTC)

Just curious what the solution is for Middle Iranian and Old Iranian errors. We also have a ton of categories in Special:WantedCategories that reference 'Middle Iranian languages' and for which {{auto cat}} doesn't work. Benwing2 (talk) 07:34, 10 April 2023 (UTC)

@Benwing2 I keep meaning to get around to this. It's just me being slow, that's all. The solution is probably to just delete the codes, but I'll put a bar against the category modules creating categories like these if the object is etymology-only. Theknightwho (talk) 18:09, 10 April 2023 (UTC)

There are also a lot of 'Requests for native script for ETYMLANG terms' as well as 'Requests for Unspecified script for LANG terms' in Special:WantedCategories. The latter should definitely not be generated at all; for the former, either we need to avoid generating them or modify the Requests category code to allow for etym langs (which would be easy but it's not clear to me what the right thing to do is). Benwing2 (talk) 23:06, 10 April 2023 (UTC)

@Benwing2 I've dealt with the Middle and Old Iranian issue, which involved splitting them out into Module:families/data/etymology (as both families and etym-only languages need to be allowed for them to work). This means terms are now getting categorised in (e.g.) Category:Old Armenian terms derived from Middle Iranian languages. This touches on a wider point about etym-only languages/families, which is that they need to categorise terms into any parent language/family categories as well. That means all those terms would go in Category:Old Armenian terms derived from Iranian languages, as currently that doesn't happen.

This could also be how we handle request categories, but since they're a maintenance issue, we might want to limit it to the non-etym language only. Theknightwho (talk) 23:37, 10 April 2023 (UTC)

Thank you for dealing with this. I'm a bit confused though about your assertion that we need to categorize e.g. all terms in Category:Old Armenian terms derived from Middle Iranian languages into Category:Old Armenian terms derived from Iranian languages as well. I don't think this is currently how these categories work; in general if something is categorized as being derived from Late Latin, it isn't also placed into the corresponding derived-from-Latin category. Instead the category itself has the derived-from-Latin category as its parent. This doesn't quite seem to be happening with Category:Old Armenian terms derived from Middle Iranian languages, which instead has Category:Old Armenian terms derived from Indo-Iranian languages as its parent; I'm not sure why. Benwing2 (talk) 00:12, 11 April 2023 (UTC)

@Benwing2 It's because Vahag expressed dissatisfaction at the terms being divided up between the categories, and I'm inclined to agree; it's much more useful to have the terms in one place, in a similar way that "derived from" is a catch-all category.

The reason Category:Old Armenian terms derived from Middle Iranian languages is being put in the Indo-Iranian category is because Middle Iranian has Iranian as its parent, which means that its family is Indo-Iranian (not Iranian). Currently, the boiler is only looking for the family, but we need to make sure it handles the parent as well - which is how etym-only languages are handled. Presuambly this is because etym-only families haven't really been dealt with properly until now.

I also suggest we formally disentagle substrates from families, and just make their parent und. If they need to have a specific family, we can just set it in the data. This is now easier, because the etym language data has some of its keys automatically moved around before being returned, to make sure that it's fully compatible with the normal language data.

One last thing that occurs to me is that there's actually no reason why we couldn't use class inheritance during the creation of full language objects, too. For example, it might make sense to use it for Bokmål and Nynorsk, since we also have the code no for Norwegian. Theknightwho (talk) 00:32, 11 April 2023 (UTC)

Hmm. If we are to add all terms derived from an etym language to the category corresponding to the etym language's parent, I think we should bring this up in the BP first as some people could conceivably object. As for substrates, I have no strong opinions here; go ahead if you want to make changes. As for the catboiler code, the relevant code is around line 1080 of Module:category tree/poscatboiler/data/terms by etymology; I don't think it will be hard to fix it. For regular-language inheritance, sure, I guess if we are stuck with all three of 'no', 'nb' and 'nn' as regular languages, we can make the latter two inherit from the former (although in practical terms, what would this get us?). Benwing2 (talk) 01:46, 11 April 2023 (UTC)

@Benwing2 I see your point - it could get out of hand. I’ve fixed the issue in Module:category tree/poscatboiler/data/terms by etymology, so (e.g.) Category:Old Armenian terms derived from Middle Iranian languages is now getting categorised correctly. On a related note, I also changed it so that anything with und as a parent gets categorised in its family instead, which catches substrates, as well as a few others like Category:Terms derived from Xiongnu. Otherwise, they’d go in Category:Terms derived from Undetermined (which I don’t think should exist).

This could also come in handy with Norwegian, too: Category:Terms derived from Norwegian Bokmål doesn’t go in Category:Terms derived from Norwegian at the moment, and it probably should. Inheritance would solve that. Plus, it would simplify the data, as it’s a simple way of coordinating languages. Perhaps this is a way to solve the problems we have with langs like Chinese or Tibetan, where multiple L2s sometimes get placed under the same header. I appreciate that this starts to blur the boundary between language and family, but in practical terms we already do that. Theknightwho (talk) 18:51, 13 April 2023 (UTC)

Hello, your edits broke stuff again. {{cog-lite|en|test}} now shows as English test, likewise for the other etymology lite templates. – Wpi31 (talk) 04:31, 15 April 2023 (UTC)

Your edits to Module:zh-translit seem to be the cause. If it helps any, the error in Appendix:Sinitic Swadesh lists goes away if you remove the language code "hsn" from the template, and the error in water/translations goes away if you remove the line with the Xiang translation. I'm not sure why this is only surfacing on these two pages and Appendix:Xiang Swadesh list. A search for Xiang translations finds 99 pages, some of which are also translation subpages that use {{multitrans}} like water/translations does. Chuck Entz (talk) 06:00, 20 April 2023 (UTC)

@Chuck Entz Thanks. I introduced a change so that Module:zh-translit will check the term linked by {{zh-see}} if it can't find any instances of {{zh-pron}} on the page, which means it now works for simplified or variant terms as well. Occasionally, there'll be a chain (e.g. simplified forms of variants), so it iterates until it finds {{zh-pron}} (or neither of them). Theoretically, that means it can get into an infinite loop if you point a group of pages at each other with {{zh-see}} without ever using {{zh-pron}}, but of course that should never happen...

It's relevant to note that you will get different results with different lects of Chinese, as it depends on whether the lect is referred to in {{zh-pron}} or not. If it isn't, it moves on to {{zh-see}}. I suspect there isn't a real mistake on the entries here, but rather an instance of two pages missing a Xiang reading, which causes an infinite loop due to missing data. It's not unusual for very common characters to be variants for certain readings, so a pair of these would cause this. Theknightwho (talk) 06:08, 20 April 2023 (UTC)

I was right: 水 can be a variant of 媠 in Min Nan, which can be a variant of 嫷, which in turn can be a variant of 媠 for a different reading (ad infinitum). I'll put in something to stop that happening, as it's an edge case I hadn't considered. Theknightwho (talk) 06:23, 20 April 2023 (UTC)

In Module:languages/data/2, you wrote "FIXME: Not clear what's going on with the PUA characters here" apropos U+F700 and U+F70F. The basic issue is that ญ YO YING and ฐ THO THAN drop their bottom parts when combined with a mark below. As basic rendering of Thai is fairly old, predating OpenType, there arose the convention of using those two PUA characters for the glyphs thus modified. There's also what I think is the barbarous practice (@Octahedron80 for comment) of using those truncated glyphs for YO YING and THO THAN in all positions when writing Pali. (It's certainly not a universal convention for Pali.) Standards-oriented font creators tend to reject this glyph encoding; the glyphs are highly unlikely to be encoded in Unicode as separate characters on the basis of Pali.

My view is this glyph encoding should be expunged from Wiktionary whenever it crops up. --11:27, 21 April 2023 (UTC) RichardW57m (talk) 11:27, 21 April 2023 (UTC)

An example of unmodified and modified yo ying in the same word is วิญญู (viññū); an example of the instrumental plural suffixed with the quotative particle, วิญญูหีติ (viññūhīti), can be seen on line at the bottom right of the sample of p5 at https://www.google.co.uk/books/edition/%E0%B8%AB%E0%B8%99%E0%B8%B1%E0%B8%87%E0%B8%AA%E0%B8%B7%E0%B8%AD_%E0%B8%AA%E0%B8%A7%E0%B8%94%E0%B8%A1%E0%B8%99%E0%B8%95/Ap3wBQAAQBAJ?hl=en&gbpv=1 . --RichardW57m (talk) 12:03, 21 April 2023 (UTC)

At thwikt, these two dated letters will only show in headwords, as Thai fonts can natively show them. Traditionally, ญ & ฐ should be "unfooted" even without marks under them. Why? Because old people didn't write the "foots". Not only Pali, but also Sanskrit. When naming title, we just use normal letters. The dated letters are added in entry_name and change them to normal, in case if they are linked from somewhere.

Adding VS01 is the new method that could be used as variation sequences, but currently no standard for Thai is released and no font supports it yet. (The fonts would be updated after the standard.) PS: VS01 is used in other scripts. --Octahedron80 (talk) 14:20, 21 April 2023 (UTC)

First off, thanks for your recent work on mod:cmn-pron. While it is possible to improve the other modules in a similar vein, I think ultimately mod:zh-pron will need some major revampping, which I already have some basic idea what to do – this involves moving most of the repetitive lect-specific stuff into a data module or their respective modules (plus, the mixture of the labels and the if else conditionals is difficult to maintain), and standardizing the interface etc. This will be a time-consuming job, which hopefully I could start working on it in late May. In the meantime, I don't think there is much point in changing the modules, since they'll probably have to be redone to suit the new interface anyways. – Wpi31 (talk) 17:09, 26 April 2023 (UTC)

@Wpi31 Yes, you're right! My main concern was to develop a more sensible baseline transcription which all the other conversions can work from, which is essentially Pinyin with some of the quirks removed (iu --> iou etc). That's now done, which should make the task of simplifying other parts of the module much simpler.

I've been putting off working on Module:zh-pron as it's so huge, but I'll follow your lead. Theknightwho (talk) 17:13, 26 April 2023 (UTC)

Hi again,

I created 韶神星 for the asteroid (6) Hebe, but am having trouble with zh-forms. The derivation is not 韶 + 神 + 星, but an abbreviation of 韶華 + 神 + 星. Is there not an option for that in zh-forms, or am I just missing it? kwami (talk) 00:05, 6 May 2023 (UTC)

@Kwamikagami I don't think there's a dedicated way to do it, but I've changed the gloss for the first character in a way that hopefully makes it clear what's going on. Theknightwho (talk) 00:27, 6 May 2023 (UTC)

That works! Thanks. kwami (talk) 00:52, 6 May 2023 (UTC)

Hi,

Thanks for working on the Russian transliteration scraper. I am more keen about this possibility - multiword Thai, Khmer, etc. transliterations, be it with spaces (or ]]

Hello, hope you are well. I wanted to ask you if you would take a look at Template:zh-wikipedia. You can see it in use at 中國／中国 (Zhōngguó). I believe that the order of Wikipedias on the Zhongguo entry (that I just linked) is not academically or ethically justified. I believe that Wiktionary should follow the order of Wikipedias as at , and that Template:zh-wikipedia should "auto-sort" (automatically sort) the words in the order of the Wikipedias that Wikipedia uses, no matter what order the Wikipedias appear in the editor's view in any given Wiktionary entry's Template:zh-wikipedia. (1) Even if you don't agree, would it be technically feasible to code in an auto-sort or something like that on Template:zh-wikipedia? Maybe for a different order? (2) Is the Wikipedia order justified within Wikipedia but yet not justified for sorting the Wikipedias when linked from Wiktionary? (3) Is there perhaps another order that you can think of? Even ones you don't think would be good? I've always felt there was something wrong with what I was doing, and I'd like to see if that resonates at all with you (or anybody reading this). Thanks for any feedback. --Geographyinitiative (talk) 12:46, 12 May 2023 (UTC) (Modified)

Hi.

1. Proto-North Caucasian. I think we should have removed everything related to Proto-North Caucasian, everything that it contains in itself and Proto-North Caucasian itself. Simply on the grounds that this is not a family, but a superfamily, which, moreover, is not yet a proven superfamily.
2. Proto-Northeast Caucasian. There are still no good reconstructions about the Proto-Northeast Caucasian. Perhaps the only revision of the reconstruction of Starostin and Nikolaev (1994) was the work of Nichols (2003). However, she uses the # sign, which stands for pseudo-reconstructions, which was introduced by Williams (1989). Here everything (Appendix:Proto-Nakh-Daghestanian reconstructions) is incorrectly indicated by an asterisk, this can be misleading. This family has not been proven.
3. Proto-Northeast Caucasian splits into two branches Proto-Nakh and Proto-Daghestanian. Proto-Nakh is quite well reconstructed, which is not to say, of course, about the Proto-Daghestanian. For some reasons, the Proto-Daghestanian can be left and not deleted, although it has also not been proven. Although recently, for example, in the work Schrijver (2021), the Proto-Nakh are compared with the Proto-Tsezian and Proto-Avaro-Andian. However, the Proto-Avaro-Andian is not worked out properly. And besides, Proto-Tsezian forms is adjusted to the data of Proto-Avaro-Andian.
4. In accordance with paragraph 3, the modern comparison is not with the Proto-Nakh and Proto-Daghestanian, but directly with the Daghestanian groups (Proto-Tsezian and Proto-Avaro-Andian). Which makes one doubt the existence of Proto-Daghestanian.
5. Proposed trees:

• Daghestanian ngn 🌳 ngn-pro
  • Avaro-Andian ngn-ava 🌳 ngn-ava-pro
  • Dargic ngn-drg 🌳 ngn-drg-pro
  • Khinalug kjj
  • Lak lbe
  • Lezgic ngn-lzg 🌳 ngn-lzg-pro
  • Tsezic ngn-tsz 🌳 ngn-tsz-pro
• Nakh nkh 🌳 nkh-pro
  • Bats bbl
  • Vainakh nkh-vay 🌳 nkh-vay-pro

In a sense, such a division will go into contradiction for macrocomparative comparisons. Gnosandes ❀ (talk) 10:33, 13 May 2023 (UTC)

When you excluded those languages from diacritic categorization, did you also intend to cause upwards of 150 module errors? When I first saw the module errors, I thought they would only be there for an hour or two while you were working on the changes to switch things over, so I didn't say anything. Chuck Entz (talk) 21:37, 13 May 2023 (UTC)

The problem is that char_category() is local to a portion of the main function when it needs to be pulled out and moved up. Benwing2 (talk) 21:41, 13 May 2023 (UTC)

@Chuck Entz @Benwing2 Fixed. I've been a bit off my game today, and didn't check CAT:E. Theknightwho (talk) 21:41, 13 May 2023 (UTC)

Hi. I'm not sure if this concerns you, but I noticed it first after you added the function of automatically adding categories.

Under the entries 1:a, 1:e, and 1:o there was a red Category:Swedish terms spelled with 1. I added {autocat}; but it only show one term, 180, and none of them above.

I assume it is those colons that mess it up, it usually is. --Christoffre (talk) 17:04, 14 May 2023 (UTC)

@Christoffre I don't think so - numbers were excluded from all automatic categorisation (which was something that's been the case for a long time), but I've now changed that as I think they're rare enough to warrant categorising: people were manually categorising them anyway, so it's better to just do it properly.

Changes to modules take time to filter through to individual entries, as (literally) millions of pages can be affected. If you purge the cache on a page it'll do an immediate update, though. If you look at the category now, you'll see all three are in it. I'm sure others will filter through in time. 17:08, 14 May 2023 (UTC) Theknightwho (talk) 17:08, 14 May 2023 (UTC)

Ah, it seems like I was just impatient. Thanks for the help anyway! --Christoffre (talk) 17:12, 14 May 2023 (UTC)

Aside from the module errors, these look wrong. For some reason, they're displaying the vowels with iota subscripts as vowels followed by free-standing iotas. This is deceptive, since there are plenty of examples like Greek Αιγίνης (Aigínis), which aren't in Category:Greek terms spelled with ᾼ.

It must have something to do with fonts, since "Category:Greek terms spelled with ᾼ" displays with the iota subscript in the edit window for me, but not in the preview. What's more, if I use a Greek keyboard to type the Greek part in manually as separate characters, it displays as separate characters but doesn't link to the correct category: Category:Greek terms spelled with Αι even though the spelling looks identical to me in preview. For the record, I'm using a MacBook Pro with an old version of MacOS, and I get the same thing with both Firefox and Safari (not logged in on the latter). Chuck Entz (talk) 21:48, 14 May 2023 (UTC)

I just noticed that there's also Category:Greek terms spelled with ◌ͅ- why both? Chuck Entz (talk) 21:54, 14 May 2023 (UTC)

@Chuck Entz So I have a feeling I know what's behind this, but I need to dig through the guts of the category tree to work out exactly what's happening.

In terms of the display, it looks fine for me, but I think this is related to the fact that subscript iota (a combining character) capitalises as a non-combining iota. Something, somewhere is making the assumption that diacritics never change on capitalisation, but this is an instance where they do. However, if that's down to the font you've got, there's not a huge amount we can do.

The reason for having Category:Greek terms spelled with ◌ͅ is because I recently upgraded the standard characters function so that it's no longer restricted to atomic Unicode characters, as they're often pretty arbitrary and not linguistically interesting. Before, any terms in categories for diacritics were only there because the diacritic couldn't form an atomic character with the base character (e.g. there is no Unicode character B̊, so the function was seeing the individual character ̊ and dumping them there instead). Since I made the change, I realised that it's probably quite interesting to see the spread of diacritics, too, so basically did the inverse of the first change by making sure all the atomic characters got categorised by diacritic as well (e.g. Å). However, the diacritic categories only appear if the diacritic isn't otherwise used in the language: so French Ÿ gets the category Category:French terms spelled with Ÿ, but there is no category for French terms spelled with ◌̈, because ◌̈ is a standard diacritic of French. Theknightwho (talk) 22:30, 14 May 2023 (UTC)

Judging by iota_subscript#Computer_encoding, the matter of uppercasing iota subscripts is a big mess. Unicode says to change them to adscripts (what I see), but a lot of usage stays with the subscript (what you see). An adscript isn't the same as an independent letter, which is why my typing the Greek ended up with a redlink. I'm guessing the module errors are due to different parts of the system (module code?) disagreeing on what exact Unicode codepoint an uppercased iota subscript is supposed to be: the part that converts the lowercase iota-subscripted letter to uppercase is using a different codepoint than the part that checks for uppercase vs. lowercase. Chuck Entz (talk) 00:01, 15 May 2023 (UTC)

I'm sure you're already done for the day/evening, and I'll be gone most of the day tomorrow, but just to be thorough: the alpha with iota subscript copypasted from the displayed title on the pagen in σοφίᾳ (sophíāi) is the precomposed character U+1FB3 (see ᾳ for the unicode info that displays even without an entry), and the category page has the precomposed character U+1FBC (see ᾼ). The first one is described as "GREEK SMALL LETTER ALPHA WITH YPOGEGRAMMENI", where "ypogegrammeni" is the iota subscript, and the second one is "GREEK CAPITAL LETTER ALPHA WITH PROSGEGRAMMENI", where "prosegegrammeni" is the iota adscript. The "Composition" section, however, has "Composition: α + ◌ͅ " for the first and "Composition: Α + ◌ͅ " for the second. I have no idea how that translates to the codepoints the module sees, but I figure some extra data won't hurt. I would also note that {{uc:ᾳ}} > ΑΙ, which is the U+1FBC codepoint, and {{lc:ᾼ}} > ᾳ, which is the U+1FB3 codepoint. Chuck Entz (talk) 02:31, 15 May 2023 (UTC)

This is getting stale. I just added the precomposed letters to the standardChars of Grek under "el" at Module:languages/data/2, but apparently with no effect (I noticed there were other precomposed letters there already). That's about all I dare to do myself, and even that may need to be reverted if there are unanticipated side-effects. We need to do something, though. Chuck Entz (talk) 01:38, 20 May 2023 (UTC)

@Chuck Entz I've been putting this off, but I'll look at it now. Theknightwho (talk) 15:45, 20 May 2023 (UTC)

@Chuck Entz This is now solved. The issue was that Module:category tree/poscatboiler/data/characters checks if single-character inputs are upper or lowercase, and throws an error if they're lowercase (with one edge-case exception if the uppercase form is actually standard, as sometimes happens with ı and I). It does this by running mw.ustring.upper on the input and then checking it's the same, because usually if you run it on a capital letter then nothing happens.

This caused a problem with ᾼ, because the output from mw.ustring.upper is actually ΑΙ (i.e. it capitalises the subscript iota and splits it out as a separate full character). This isn't an error, as it matches the Unicode specification, but it isn't what we want in this context (because what we really want is title case, but there's no function for that). I've added special-case behaviour for subscript iota so that it remains unchanged.

I'll investigate if there are any other oddities like this: one other I've accounted for so far is ß, which capitalises as SS by default. Instead we use ẞ, which is the correct form for contexts like this. Theknightwho (talk) 14:07, 21 May 2023 (UTC)

Please take a look. There are a ton of categories here related to 'LANG terms spelled with X' that shouldn't or arguably shouldn't be there. Most blatantly, Cyrillic is entirely normal for Mongolian. I also don't think digits in terms is remarkable for English. Please review what's there and make appropriate fixes to remove these categories from being created, thanks! Benwing2 (talk) 09:04, 16 May 2023 (UTC)

@Benwing2 Macedonian's been dealt with (which was an error on my part). Regarding numerical entries, they're not that remarkable in English, but if we take the most frequent number (1) and assume everything in Category:English terms spelled with 1‏ is a lemma (it's not), that only comprises 0.03% of English lemmas. I'd say that's rare enough to be notable. That being said, we may want to consolidate the categories for numerals.

Other than those, I'd say the categories are all fine: Tagalog, Danish and Norwegian don't use "C" in native words, for example. I think there needs to be a special exclusion for entries with "..." from being categorised as having ".", so I will deal with that. Theknightwho (talk) 09:58, 16 May 2023 (UTC)

I think at most we want one 'terms spelled with numbers' category, although I still believe it's better not to have these; nothing terribly remarkable about a term with a number in it (probably English terms with Q and Z are fairly rare too but not remarkable). Do you mean both Mongolian and Macedonian? Benwing2 (talk) 10:27, 16 May 2023 (UTC)

Also, all the "terms spelled with ." should go; these are just abbreviations. Benwing2 (talk) 10:28, 16 May 2023 (UTC)

Also IMO Category:English terms spelled with &. Benwing2 (talk) 10:30, 16 May 2023 (UTC)

@Benwing2 The only Mongolian categories listed relate to the traditional script, and they're filtering out. We do have a few categories for Mongolian Cyrillic letters, but only for the rare ones used in borrowings: see Category:Mongolian terms by their individual characters.

My initial inclination was to exclude numbers and characters like "." as well, but people were adding them manually anyway. I judged it was better to just handle it automatically, as if we're going to have them we may as well do it properly, and I don't want want to waste time periodically deleting them. Ultimately, they're pretty harmless. Theknightwho (talk) 10:42, 16 May 2023 (UTC)

Not sure I agree on Category:English terms spelled with & - we don't usually have entries with it unless it's either obligatory or overwhelmingly common: R & B, &c., R&R and so on. Theknightwho (talk) 10:47, 16 May 2023 (UTC)

The problem is that every one of these extra categories adds spam to the category list at the bottom of the page. On a page with a period or number in it, there may be several such categories if we allow them. So they're not totally harmless. I have already done runs in the past removing the manually added number categories and can do it again. Benwing2 (talk) 18:20, 16 May 2023 (UTC)

Hi. The punctuation marks are deleted on links but they don't transliterate then. E.g. at how much does it cost#Translations the Chinese/Wu 多少鈔票／多少钞票 (¹tu-sau ⁵tshau-phiau) works but 多少鈔票？／多少钞票？ doesn't. Could you please have a look? Anatoli T. ^{(обсудить}/^вклад) 23:53, 17 May 2023 (UTC)

@Atitarev I'll take a look, yes. Theknightwho (talk) 16:27, 29 May 2023 (UTC)

See diff. Chuck Entz (talk) 19:19, 20 May 2023 (UTC)

Fixed. Theknightwho (talk) 19:59, 20 May 2023 (UTC)

Hi, could you add documentation on the changes you've made to Module:languages? They're extensive and not well-documented and I'm finding it hard to use. E.g. there used to be a function to return the type of a given language object, which seems to have been replaced with hasType(), but the usage isn't well documented and the interface itself may have issues (e.g. is there a simple way of telling whether a language object describes a regular language)? BTW something needs to be done about languages like Rudbari, which are both a regular language and an etymology language; this causes all sorts of problems. Benwing2 (talk) 05:30, 29 May 2023 (UTC)

@Benwing2 Yes, you're right that I need to do this. You can check if a language is regular with lang:hasType("regular"). In this case, I think the documentation points to Module:languages/data/2, which lists the three types, but it would be better to explain it more fully, as the regular type is only assigned if the language isn't one of the other types. Theknightwho (talk) 16:20, 29 May 2023 (UTC)

You're right about the languages which are both etym-only and regular - I don't really know what to do with them, as they're all languages I don't have any experience editing. Theknightwho (talk) 16:21, 29 May 2023 (UTC)

By "regular" does this exclude reconstructed and appendix-reconstructed languages? If so, this may be a mistaken choice. Traditionally "regular" languages are all languages that aren't etym-only. I would keep "regular" in this meaning and use "mainspace" to indicate languages that go in the mainspace (excluding those that go in Reconstruction: or the appendix). Benwing2 (talk) 22:43, 31 May 2023 (UTC)

I think these terms should never have the prefix reduced, with either a secondary stress or an unreduced marker, so the respelling "трехстру́нный" at трёхстру́нный (trjoxstrúnnyj) is quite rare and even unnatural, IMO. Anatoli T. ^{(обсудить}/^вклад) 01:27, 30 May 2023 (UTC)

@Atitarev What is this in reference to? The "fast/casual" pronunciation at трёхстру́нный (trjoxstrúnnyj) has been there a long time. Did User:Theknightwho recently push their translation-scraping code to production? If so and if the scraping is looking at pronunciations, maybe there needs to be a way of indicating a pronunciation should not be scraped (e.g. a boolean flag to {{ru-IPA}}). Benwing2 (talk) 22:47, 31 May 2023 (UTC)

@Benwing2: The fast/casual pronunciation was added in diff. I let it slide, although I don't fully agree. I thought for completeness, it's OK, then. I don't see why all possible incorrect rare pronunciations should be added, though, especially unsourced, so I might remove that.

I messaged Theknightwho, since he added a few words prefixed with трёх- (trjox-), all of them with a reduced pronunciation, he probably referred to that edit by KoreanQuoter as a model. In my opinion, the most accurate and common pronunciation of the prefix is or , respelled "трё̂х-" or "трё̀х" and (reduced) should go. Anatoli T. ^{(обсудить}/^вклад) 02:17, 2 June 2023 (UTC)

@Benwing2 @Atitarev Transliteration scraping isn’t live yet. Atitarev is right about what happened re the pronunciations, and of course him being a native speaker means I’ll defer to his expertise. Theknightwho (talk) 02:32, 2 June 2023 (UTC)

Hi, Theknightwho. I have a question and I've discovered I don't know where I can find out any information.

I've looked at my "User contributions" and discovered that there is an "edit count". When I looked at my edit count I discovered that there were 50 "Deleted edits". Wiktionary allows me to see what contributions I've made but I do not know how to access information on which edits have been deleted. This concerns me, because I don't know what it is referring to. Is it edits I've deleted myself? Edits that someone has been going through and deleting? How can I locate these "deleted edits"? Can you tell me anything about this? Any help would be greatly appreciated.

Bathrobe (talk) 20:26, 30 May 2023 (UTC)

@Bathrobe They're edits on pages which have been deleted. That may be because you created them, but it may also be if they're on someone's userpage which got deleted or something. Without knowing any of the details, it's not possible for me to say, but it's almost certainly nothing to worry about. Theknightwho (talk) 20:31, 30 May 2023 (UTC)

I had a look (as it seems admins can view a user's deleted contributions), and it was mostly pages like white/translations which have since been folded back into the main page. Theknightwho (talk) 20:43, 30 May 2023 (UTC)

Ok, thanks! As you may know, I'm not on very good terms with one of the editors, one of the grand old men of Wiktionary, an admin, and an extremely prolific contributor from way back, who added some poor translations in languages he knew very little about. I was concerned that such an editor might have been simply deleting my contributions. I'm relieved that it's nothing like this. Bathrobe (talk) 21:54, 30 May 2023 (UTC)

Hi. Go to User:Benwing2/test-ca-noun and click "Show preview" and you'll see a bunch of warnings for headwords that attempted to set displayTitle. This only applies to terms with accented characters in them for some reason. Any idea what's going on? Benwing2 (talk) 22:40, 31 May 2023 (UTC)

@Benwing2 Yes - it's Module:headword trying to set the display title. For some reason, it only does that if there are non-ASCII characters in the string, which seems a bit hacky to me. Theknightwho (talk) 22:53, 31 May 2023 (UTC)

Hmm. I see the code in question and I didn't add it. Where did it come from? Benwing2 (talk) 22:55, 31 May 2023 (UTC)

@Benwing2 Which bit do you mean? The module's been handling the display title since before I could edit the module, from the look of it.

I was considering moving it into Module:headword/data so that it only gets calculated once per page, but that ran into issues with determining the best script without a language, so that project is stalled for now. Theknightwho (talk) 22:58, 31 May 2023 (UTC)

The garbagey code in lines 732-770 of Module:headword. When I rewrote the module I added a FIXME and isolated the code in one place, but it may need a rewrite. The check for ASCII seems very hacky to me as well as some of the other stuff being done in that code. Benwing2 (talk) 23:04, 31 May 2023 (UTC)

@Benwing2 Yes, I agree. I did recently add a bit for unsupported titles, but it's a bit pointless unless the Javascript gadget can access it in some way. I remember doing the bit to keep the Han script region-neutral, too, but that was a while back. Theknightwho (talk) 23:07, 31 May 2023 (UTC)

Might I suggest that you check for lower(upper(x))=x before throwing an error? In template syntax that would be: {{lc:{{uc:ſ}}}}=ſ producing s=ſ. If false, don't throw the error. Chuck Entz (talk) 23:57, 2 June 2023 (UTC)

Can we do something to fix these? They've been in CAT:E literally for weeks, since you changed the category modules. They shouldn't be throwing errors at all, since the characters only exist in lowercase. Chuck Entz (talk) 13:27, 6 June 2023 (UTC)

Could 'digraph' or 'multigraph' be added as an option for Module:headword/data? ch for example might count as a 'letter' in Spanish orthography (at least it was considered a letter for a while, maybe not anymore), but it isn't one in English. Which means that technically we can have a Spanish entry for orthographic 'ch' but not an English one. Or is 'letter' already understood to include multigraphs?

This came up because of the (IMO) odd obsolete English digraph ée, which might deserve an entry, but I see that we don't have entries for ee or ea. kwami (talk) 06:59, 5 June 2023 (UTC)

It feels like every month somebody has to publicly and explicitly denounce you as a tyrant and the scourge of Wiktionary. I don’t get it. —(((Romanophile))) ♞ (contributions) 16:10, 8 June 2023 (UTC)

@Romanophile I think it's a manipulation tactic: accusing someone of abusing their power makes it much more difficult to hold you accountable, because it "proves" you right. Once one person does it, it doesn't take much for other people who don't like being held accountable to do the same. Theknightwho (talk) 16:16, 8 June 2023 (UTC)

Hi. Before things go on too much farther can we please use the type 'regular' to mean any non-etymology language and 'mainspace' to refer to what you have redefined 'regular' as? Formerly regular languages were any non-etymology languages but you redefined it to refer only to mainspace languages. There's not even any simple way any more of checking for regular languages. I can make this change myself but I'm not sure what code depends on the current definition of 'regular'. Thanks. Benwing2 (talk) 21:55, 10 June 2023 (UTC)

@Benwing2 I think the definition pre-dates any changes I made: compare line 58 of this version of Module:languages with the for-loop at the bottom of Module:languages/data/2. You can see it in use in line 41 of this version of Module:translations, for example, which is still there at line 32.

I'm in two minds as to whether we should change it to your suggestion: on the one hand, it makes explicit the distinction between full languages and etymology-only ones (i.e. variants), but on the other hand the current definition is a useful catch-all for any language with no special characterstics, and you can simply check for non-etym languages by doing if not lang:hasType("etymology-only") then .... Theknightwho (talk) 22:17, 10 June 2023 (UTC)

So you can't actually just check the way you mention; you need to also check for !family when calling getNonEtymological, which is why I make this suggestion. It is annoying and non-obvious to have to write two checks of the form "not A and not B" to get regular (full) languages, and if we introduce a new type of language-like entity in the future, it will break all such checks. Benwing2 (talk) 23:46, 10 June 2023 (UTC)

I see, yes I didn't realize the 'regular' language type in this defn goes back a way. If you'd rather we can call non-etymological languages "full" languages but IMO we do need a type for this. Benwing2 (talk) 23:49, 10 June 2023 (UTC)

@Benwing2 That's a good point, and I agree "full" is the best name here. Because types specified in the data are inheritable (i.e. etym-only children get all the types of their parents), the full and etymology-only types have had to be hard-coded into Module:languages instead, and get generated the first time :hasType gets called based on whether self:getNonEtymologicalCode() == self:getCode().

For the sake of completeness I've also implemented this for families, so if you only want languages you can do if lang:hasType("language", "full") then ....

I guess we could also modify the method so that you can check not-X as part of a multi-type check (e.g. if lang:hasType("language", "!etymology-only") then ...), but I'm not sure how useful this would be in practice. Theknightwho (talk) 14:11, 11 June 2023 (UTC)

Cool, thank you. Benwing2 (talk) 16:30, 11 June 2023 (UTC)

Hi. Many Spanish terms have origins in Lunfardo, which is a Buenos Aires argot derived by mixing Italian dialect words into Spanish grammar. It is given an etymological code es-lun, but that prevents deriving Spanish words from Lunfardo; as a result many existing etymologies wrongly use {{cog|Lunfardo|...}} in place of {{bor|es|es-lun|...}} or whatever. What is the correct solution here? Benwing2 (talk) 06:49, 14 June 2023 (UTC)

@Benwing2 Should we give it a new creole language code? Theknightwho (talk) 15:28, 14 June 2023 (UTC)

I don't think it's a creole, as creoles have their own grammar. Here the grammar is still Spanish. Wikipedia defines it as an "argot", which is its own class of thing. Benwing2 (talk) 15:34, 14 June 2023 (UTC)

Maybe it's similar to if Classical Latin borrowed terms from Vulgar Latin, which is entirely possible; it suggests we need a type of etymology language that runs concurrent with the non-etymology parent, so borrowing can go in both directions. Benwing2 (talk) 15:49, 14 June 2023 (UTC)

@Benwing2 Isn’t it only inheritance where there’s a restriction? Borrowing can go in both directions anyway, can’t it? Maybe I’m misunderstanding the issue. Theknightwho (talk) 17:25, 14 June 2023 (UTC)

Hmm, you are right. Benwing2 (talk) 17:30, 14 June 2023 (UTC)

I can see that you've been very busy with fixing sortkeys, but you really need to deal with the 500 (and counting) entries in CAT:E as a result of your adding error() to the scrape_page function in Module:Jpan-sortkey. The number has increased by almost a hundred in the time it took me to compose this message. Chuck Entz (talk) 19:24, 17 June 2023 (UTC)

I'm trying to clear the "...terms spelled with...read as..." cats. If I add "kun" in the first parameter of {{auto cat}} as the error message suggests, it displays "terms spelled with " ++" with its kun reading of", where should be the kanji in the name of the category, but is instead "Template:-l". I'm guessing the module is assuming that all of these language codes have counterparts to Japanese and its {{ja-l}}, since the display is what one would expect from putting a parameter in a non-existent template. Chuck Entz (talk) 21:08, 17 June 2023 (UTC)

@Chuck Entz Thanks. That bit of the function is little-used, and only there as a last resort. Thankfully that meant it was ~500 entries, not 100k.

The link template issue was very straightforward to solve by simply slotting in {{l}} instead, which means it'll work regardless of the language. They're just links to kanji with no transliterations, so the output is exactly the same as it would be. Theknightwho (talk) 21:20, 17 June 2023 (UTC)

Hi. Memory errors seem to be increasing again. Category:Proto-Niger-Congo language is now throwing a memory error. Can you let me know what you've been doing lately and if it is increasing memory load (and if so what we're gonna do about it)? Benwing2 (talk) 02:58, 19 June 2023 (UTC)

@Benwing2 I've not done anything major for a while, but Category:Proto-Niger-Congo language and Category:Proto-Atlantic-Congo language occasionally seem to throw errors for no reason at all. Scribunto memory usage varies each time the page is refreshed, so occasionally a page will start throwing memory errors for no reason. I suspect it's exacerbated by the fact it's almost all down to the family tree, as opposed to most pages close to the limit where it's hundreds of separate template calls. Atlantic-Congo is the largest language family we have, so it's not massively surprising. Theknightwho (talk) 13:31, 19 June 2023 (UTC)

OK, thanks. Benwing2 (talk) 18:52, 19 June 2023 (UTC)

I think you asked me awhile ago to obsolete {{zh-der}}? I can do that but I forget exactly what needs to be done. Do you know where the discussion was? Benwing2 (talk) 03:00, 19 June 2023 (UTC)

@Benwing2: It's at Wiktionary:Grease pit/2023/March#Deprecating Template:zh-der. I believe the following needs to be done:

• Replacing {{zh-der}} with {{col3|zh}} suffices for most pages
• If a derived term contains foo/bar (which is used to manually specify simplified forms), it should be replaced by foo//bar
• If a derived term contains foo:bar, this should be converted to either foo<tr:bar> or foo<t:bar>. The distinction between tr and t is determined via the regexes at Module:zh/link#L-63.
• If a derived term contains foo;bar, this should be converted to foo<q:bar>; when bar is the name of a lect, it needs to be linked with the langcode prefix, e.g. nan:foo or foo<q:Min Nan> for foo;Min Nan. (I'm not sure what the practice should be for this one, TKW has been doing the former and me the latter)
• Note that the ; and : syntaxes can (and usually) occur simultaneously. The two are not that common (I estimated that it's probably around 1400 in total, out of the 38k invocations of {{zh-der}}), so I think they could be done manually since they require extra care in most cases, and also because there seems to be sometimes confusion between the two syntaxes.
• If {{zh-der}} has |hide_pron=1, or if {{zh-der/fast}} is used, these need to be dealt with manually.
• |fold= can be ignored in any case; |title= can be copied wholesale into {{col3}}

– Wpi (talk) 19:34, 19 June 2023 (UTC)

@Wpi Thank you. I think the langcode prefix form should be used if possible in place of using a qualifier, because it supplies the info in a machine-readable format, allowing for lang-specific linking and formatting as needed. Benwing2 (talk) 20:38, 19 June 2023 (UTC)

@Wpi A few questions:

1. The code in Module:zh/link is a bit confusing but it appears to do special things with terms that have a circumflex (^) and/or an asterisk (*) in them. The circumflex seems to indicate that the translit should be capitalized, but there do not seem to be any occurrences of it in {{zh-der}} calls. The asterisk is less clear to me. There are 50 {{zh-der}} calls with an asterisk somewhere in them, any idea what to do with them?
2. For reference there are 87 {{zh-der}} calls that use hide_pron (out of 26,873 total; this does not include any aliases of {{zh-der}}, namely {{zh-list}}, {{zh-syn-list}} and {{zh-ant-list}}; I still need to investigate them), and 390 that include a semicolon anywhere in the {{zh-der}} call.
3. For lect names after semicolons, if you prefix the term with the corresponding langcode, it doesn't display the lect name. I'm thinking of adding some syntax to {{col}} for use in conjunction with langcode prefixes to display the corresponding language name as a qualifier; what do you think of this? If you think it's a good idea, any suggestions for the syntax?
4. The current code uses {{col3}} unless the pagetitle has more than one character, in which case it uses {{col2}}. Should we duplicate this functionality or always use {{col3}}?
5. I am planning on checking to see in case of TRAD/SIMP whether the SIMP is redundant, and removing it if so. Sound good?
6. When a note following a semicolon is not a lect name, should it use <q:...> (display to the left) or <qq:...> (display to the right)?
7. There are occurrences of items like 東城:] and 宜君:] in the {{zh-der}} calls. Are these supposed to be (badly specified) translits? They end up as glosses in the converted output. What should be done about them?
8. The following item 拍損:phah-sńg;Min Nan ends up as nan:拍損<t:phah-sńg> in the converted output. I assume this should be a translit and the regexps need to be tweaked to recognized things like ń?

Benwing2 (talk) 23:24, 21 June 2023 (UTC)

@Benwing2:

1. * suppresses simplification and pinyin (Sorry I forgot this one). For the warnings you got below, 乾 is usually simplified to 干 but for that particular etymology the simplified form is 乾 itself. Sometimes people use it simply to disable pinyin, which is a valid but not ideal use case; I think these will need to be done manually as well.

3. Maybe?

4. I think we discussed this before(?), and {{col3}} is still very readable on multiword items.

5. That makes sense, especially since on some pages the simplified form is specified to reduce memory usage, but that doesn't really help in {{col3}} according to TKW.

6. No strong preference, but putting the qualifier after the term aligns the characters.

7. These are place names, which are supposed to be glosses.

8. Correct. The regexes in Module:zh/link are probably written quite some time in the past and doesn't really do the job properly.

@justinrleung might have some insights on some of the issues. – Wpi (talk) 05:48, 22 June 2023 (UTC)

@Wpi: Thanks for the detailed replies. As for #1, do you mean all occurrences of asterisks should be handled manually? This seems doable as there are only around 50 template calls using them. For #4, I'll use {{col3}} everywhere. For #6 I'll put the qualifier after the chars. For #8 I'll fix up the regex for ń and maybe for other tone marks on n (can they also occur on m?). Benwing2 (talk) 07:15, 22 June 2023 (UTC)

@Wpi: One more question: what about cases like 潑雨;Hakka, Min Nan where two lects are listed after the semicolon? Currently it's unrecognized as a lect and ends up as a qq qualifier. Benwing2 (talk) 07:58, 22 June 2023 (UTC)

@Benwing2:

For #1, that should be doable. The correct procedure would be to convert *term to term// (i.e. an empty second form to supress simplified forms), but I don't think it's worth the hassle to bot the 50 or so instances.

For #8, yes. Other characters to look out for are: , combined with tone marks U+0300, U+0301, U+0302, U+0304, U+0306, U+030C, U+033F which occur in Kienning Romanised for Min Bei; , combined with U+0300, U+0301, U+0302, U+0304, U+0306, U+030D in Min Nan; combined with U+0300, U+0301, U+0302, U+030D in Hakka; and of course the combining forms. It might also be useful to modify the second regex to (+ )*+ to try and find the transliterations unaccounted for.

For the final question, I suppose we could list them twice, once for each lect, but it feels weird given we don't list the duplicate the ones with Mandarin in it. For now I reckon let's just leave them as is, i.e. <qq:Hakka, Min Nan>

– Wpi (talk) 09:34, 22 June 2023 (UTC)

@Wpi: What I ended up doing for the regexes is just decompose everything and look for any of the combining diacritics 0300 (GRAVE), 0301 (ACUTE), 0302 (CFLEX), 0304 (MACRON), 0306 (BREVE), 0307 (DOTOVER), 0308 (DIAER), 030B (DOUBLEACUTE), 030C (CARON), 030D (VERTLINEABOVE), 030F (DOUBLEGRAVE), 0323 (DOTUNDER), 0324 (DIAERUNDER), 033F (DOUBLEMACRON), 0358 (DOTABOVERIGHT) as well as superscript ⁿ. Benwing2 (talk) 10:27, 22 June 2023 (UTC)

OK, I have written the script. Some sample output:

-{{zh-der|fold=1|*乾乾|*乾元|乾圖/乾图|*乾坤|*乾宅|乾斷/乾断|*乾旦|*乾曜|乾清宮/乾清宫|乾紅/乾红|乾綱/乾纲|*乾象|*乾造|*乾道|*乾陵|*乾隆|*朝乾夕
惕|顛乾倒坤/颠乾倒坤|乾華/乾华}}
+{{col3|zh|*乾乾|*乾元|乾圖//乾图|*乾坤|*乾宅|乾斷//乾断|*乾旦|*乾曜|乾清宮//乾清宫|乾紅//乾红|乾綱//乾纲|*乾象|*乾造|*乾道|*乾陵|*乾隆|*朝乾夕惕|顛乾倒坤//颠乾倒坤|乾華//乾华}}

This output a bunch of warnings:

Page 137 乾: WARNING: Ignoring fold=1: {{zh-der|fold=1|*乾乾|*乾元|乾圖/乾图|*乾坤|*乾宅|乾斷/乾断|*乾旦|*乾曜|乾清宮/乾清宫|乾紅/乾红|乾綱/乾纲|*乾象|*乾造|*乾道|*乾陵|*乾隆|*朝乾夕惕|顛乾倒坤/颠乾倒坤|乾華/乾华}}
Page 137 乾: WARNING: For traditional 乾圖, explicit simplified 乾图 doesn't match auto-conversion 干图 in 3=乾圖/乾图, specifying explicitly: {{zh-der|fold=1|*乾乾|*乾元|乾圖/乾图|*乾坤|*乾宅|乾斷/乾断|*乾旦|*乾曜|乾清宮/乾清宫|乾紅/乾红|乾綱/乾纲|*乾象|*乾造|*乾道|*乾陵|*乾隆|*朝乾夕惕|
顛乾倒坤/颠乾倒坤|乾華/乾华}}
Page 137 乾: WARNING: For traditional 乾斷, explicit simplified 乾断 doesn't match auto-conversion 干断 in 6=乾斷/乾断, specifying explicitly: {{zh-der|fold=1|*乾乾|*乾元|乾圖/乾图|*乾坤|*乾宅|乾斷/乾断|*乾旦|*乾曜|乾清宮/乾清宫|乾紅/乾红|乾綱/乾纲|*乾象|*乾造|*乾道|*乾陵|*乾隆|*朝乾夕惕|
顛乾倒坤/颠乾倒坤|乾華/乾华}}
Page 137 乾: WARNING: For traditional 乾清宮, explicit simplified 乾清宫 doesn't match auto-conversion 干清宫 in 9=乾清宮/乾清宫, specifying explicitly: {{zh-der|fold=1|*乾乾|*乾元|乾圖/乾图|*乾坤|*乾宅|乾斷/乾断|*乾旦|*乾曜|乾清宮/乾清宫|乾紅/乾红|乾綱/乾纲|*乾象|*乾造|*乾道|*乾陵|*乾隆|*朝乾夕惕|顛乾倒坤/颠乾倒坤|乾華/乾华}}
Page 137 乾: WARNING: For traditional 乾紅, explicit simplified 乾红 doesn't match auto-conversion 干红 in 10=乾紅/乾红, specifying explicitly: {{zh-der|fold=1|*乾乾|*乾元|乾圖/乾图|*乾坤|*乾宅|乾斷/乾断|*乾旦|*乾曜|乾清宮/乾清宫|乾紅/乾红|乾綱/乾纲|*乾象|*乾造|*乾道|*乾陵|*乾隆|*朝乾夕惕|顛乾倒坤/颠乾倒坤|乾華/乾华}}
Page 137 乾: WARNING: For traditional 乾綱, explicit simplified 乾纲 doesn't match auto-conversion 干纲 in 11=乾綱/乾纲, specifying explicitly: {{zh-der|fold=1|*乾乾|*乾元|乾圖/乾图|*乾坤|*乾宅|乾斷/乾断|*乾旦|*乾曜|乾清宮/乾清宫|乾紅/乾红|乾綱/乾纲|*乾象|*乾造|*乾道|*乾陵|*乾隆|*朝乾夕惕|顛乾倒坤/颠乾倒坤|乾華/乾华}}
Page 137 乾: WARNING: For traditional 顛乾倒坤, explicit simplified 颠乾倒坤 doesn't match auto-conversion 颠干倒坤 in 18=顛乾倒坤/颠乾倒坤, specifying explicitly: {{zh-der|fold=1|*乾乾|*乾元|乾圖/乾图|*乾坤|*乾宅|乾斷/乾断|*乾旦|*乾曜|乾清宮/乾清宫|乾紅/乾红|乾綱/乾纲|*乾象|*乾造|*乾道|*
乾陵|*乾隆|*朝乾夕惕|顛乾倒坤/颠乾倒坤|乾華/乾华}}
Page 137 乾: WARNING: For traditional 乾華, explicit simplified 乾华 doesn't match auto-conversion 干华 in 19=乾華/乾华, specifying explicitly: {{zh-der|fold=1|*乾乾|*乾元|乾圖/乾图|*乾坤|*乾宅|乾斷/乾断|*乾旦|*乾曜|乾清宮/乾清宫|乾紅/乾红|乾綱/乾纲|*乾象|*乾造|*乾道|*乾陵|*乾隆|*朝乾夕惕|顛乾倒坤/颠乾倒坤|乾華/乾华}}

I'm not sure what's going on with the non-matching simplified stuff, let me know if it makes sense to you. Note also the asterisks in the output, which are getting carried over literally; I don't think this is correct. Benwing2 (talk) 04:55, 22 June 2023 (UTC)

@Wpi, Theknightwho I am going to run the script on {{zh-der}} and {{zh-list}}. I am planning on implementing syntax in {{colN}} whereby writing e.g. nan-hok::冷吱吱 with two colons causes the language name to be displayed as a right qualifier, equivalent to nan-hok:冷吱吱<qq:Hokkien>. Thoughts? Benwing2 (talk) 19:43, 22 June 2023 (UTC)

@Benwing2 Thanks - that sounds good. In terms of the double colon, I'd prefer if the language name always showed as a qualifier if there's a language override in a column template, as otherwise there's nothing visible to indicate it. Maybe we should exclude mul from that, though. Theknightwho (talk) 19:46, 22 June 2023 (UTC)

Yup, that makes sense. I was thinking originally of doing exactly what you proposed but then realized it would be problematic with mul links; but excluding them from the qualifiers should work fine. Benwing2 (talk) 19:49, 22 June 2023 (UTC)

Handling Template:zh-syn-list and Template:zh-ant-list

@Wpi, Justinrleung These two templates are redirects to {{zh-der}} but the code that implements {{zh-syn-saurus}} looks specifically for {{zh-syn-list}} (and likewise {{zh-ant-saurus}} looks specifically for {{zh-ant-list}}). I am thinking they can be replaced with {{col3}} where the title can be an abbreviation for Synonyms or Antonyms, expanded appropriately (I've been meaning to implement such abbreviations for awhile now, for various reasons), and {{zh-syn-saurus}} can look for a {{col3|zh}} whose title is "Synonyms" or the appropriate abbreviation. More specifically, instead of writing {{col3|zh|title=Synonyms}} you could write something like {{col3|zh|ti=@syn}} or maybe just {{col3|zh|ti=@s}}, where |ti= is a shorter alias for |title= and things beginning with at-signs are abbreviations. Similarly, @der or @d means "Derived terms", @rel or @r means "Related terms", etc. How does this sound? An alternative is to make the first numbered term after the language be the title with abbreviations if preceded by the appropriate char, e.g. {{col3|zh|@syn|*乾乾|*乾元|乾圖//乾图|*乾坤|*乾宅|乾斷//乾断|*乾旦|*乾曜|乾清宮//乾清宫|乾紅//乾红|乾綱//乾纲|*乾象|*乾造|*乾道|*乾陵|*乾隆|*朝乾夕惕|顛乾倒坤//颠乾倒坤|乾華//乾华}} for synonyms. Benwing2 (talk) 07:26, 22 June 2023 (UTC)

@Theknightwho, Wpi What do you think of my proposal here? We should generalize {{zh-syn-saurus}} to something like {{syn-saurus|zh}}; all it does it look up synonyms or antonyms in the specified thesaurus page and display them. BTW I am also thinking of changing the bare {{col}} template so the number of columns is specified using a named param |n= instead of a numbered param; that way, if |n= is omitted, you get some sort of auto behavior, similar to {{col-auto}} but with fewer chars. Benwing2 (talk) 19:53, 22 June 2023 (UTC)

@Wpi Conversion of {{zh-der}} and {{zh-list}} is in progress; conversion of the syn/ant stuff will come next. See User:Benwing2/zh-der-misc-warnings for warnings related to {{zh-der}} conversion other than those related to |hide_pron= (80 in total) and User:Benwing2/zh-der-hide-pron-warnings for automatic warnings triggered whenever |hide_pron= is seen (87 in total). Hopefully this isn't too many to process by hand but let me know if there's a way of automatically dealing with the |hide_pron= stuff. Benwing2 (talk) 03:55, 23 June 2023 (UTC)

@Benwing2: For {{zh-syn-saurus}}, I think that makes sense if we want to extend the format to other languages.

For the warnings some of them could be automated: more-than-one-slash can be safely converted from a/b/c to a//b//c, more-than-one-colon is just coexistence of both t and tr. The {{{hide_pron}}} stuff are probably automatable, if they are just plain list of terms with no special formatting stuff, but I figure it's not worth the hassle to write the checks for the bot, plus they are usually used on pages with memory errors so it might trigger that. – Wpi (talk) 05:46, 23 June 2023 (UTC)

@Wpi: Thanks. See User:Benwing2/zh-syn-list-zh-ant-list-misc-warnings; I converted all the 2000 or so Thesaurus pages using {{zh-syn-list}} and {{zh-ant-list}}, producing 49 more warnings, most about asterisks. You might want to get to these first because I am also doing a run to convert {{zh-syn-saurus}} and {{zh-ant-saurus}} to {{syn-saurus}} and {{ant-saurus}}, and for such pages that are converted, there will be a red warning message displayed in place of the synonyms/antonyms if it finds the old {{zh-syn-list}}/{{zh-ant-list}} templates instead of {{col3|zh}}. Benwing2 (talk) 06:20, 23 June 2023 (UTC)

@Wpi: BTW the number of cases with more than one slash or colon are small, so it might be easier to handle them manually; let me know if you want me to write the bot code to handle them automatically. Benwing2 (talk) 06:22, 23 June 2023 (UTC)

@Benwing2: Thanks. I'll do those manually, probably today or this weekend. – Wpi (talk) 07:39, 23 June 2023 (UTC)

@Wpi: Great, thank you! Benwing2 (talk) 07:52, 23 June 2023 (UTC)

@Benwing2, theknightwho

So here's a progress update, I've went through all the warnings for the thesaurus pages (as well as a bunch that the bot missed like an asterisk at the end of a term) and cleaned them all up.

During this I find that:

1. Some of the lect qualifiers were not recognized by the bot (or sometimes missed by myself), this seems to be mostly the etym codes under Hokkien, and Sichuanese (the etym code is new so we've decided on calling it Sichuanese, though it used to be split between Sichuanese and Sichuan, so <qq:Sichuan> is the same as <qq:Sichuanese> and should be converted to cmn-sic:)

   @TKW It would be helpful to have the Hokkien etym codes autogenerate pronunciation for the subdialect.

   Speaking of new etym codes, there's a number of qualifiers that would make sense for a new etym code, e.g. Taiwanese Hokkien, Hong Kong Cantonese, Beijing. I'll likely propose them on BP shortly, and after that there's probably a need to rerun the bot conversion for the qualifiers again.

2. The bot did not convert qualifiers like <qq:Cantonese, slang> or <qq:Wu, dated> to the ones with prefixes. I've changed them on sight, but there's a lot of them. We might want to use a bot to clean these up. Note that literary refers to the entire Chinese language and not the lect, so <qq:literary, Cantonese is not yue:<qq:literary>. EDIT: I've cleaned up all the ones I could find.

   There's also weird qualifiers like <qq:chiefly Cantonese>, which I think would benefit from some syntax that allows writing custom qualifiers like {{place}}, e.g. <qq:chiefly <<yue>>>. This might also solve the problem with multiple lects, so <qq:Hakka, Min Nan> could be <qq:<<hak>>, <<nan>>> (not sure how the transliteration would be handled?). However, I feel like this might be adding too much bloat to the columns module.

3. There's some stuff that should be in <t:gloss> were put into <qq:gloss>, due to how the people who added them used the wrong syntax; there's not much that we could do with it currently.
4. When zhx-min: (a family code) is used as a prefix, there's some unexpected behaviour. There are some words that are shared across the entire Min family, so it makes some sense to use it.

– Wpi (talk) 17:42, 23 June 2023 (UTC)

@Wpi Thanks. I'll take a look. I think the issue with the lect qualifiers missing is that the bot was only looking for the canonical name of etym languages. I'll redo the code to look at all names and see if I can fix the issues. Benwing2 (talk) 19:06, 23 June 2023 (UTC)

@Wpi Can you give me some examples of lect qualifiers that the bot didn't correctly convert? Benwing2 (talk) 22:57, 23 June 2023 (UTC)

@Benwing2: diff, diff, diff. I have most of them fixed already. – Wpi (talk) 06:11, 24 June 2023 (UTC)

@Benwing2: (Sorry for the pings) I think the problem with the lect names is simply because the name is not listed in the module data, not even as an alias. There's not a lot so I think I'll do them by hand. – Wpi (talk) 06:18, 24 June 2023 (UTC)

@Wpi Those pings you gave me are mostly of cases of a lect name + comma + qualifier. If there are a lot of these (too many to do by hand) I can look into writing a script to fix them. Benwing2 (talk) 06:30, 24 June 2023 (UTC)

You've promised several times to fix the documentation for Module:languages. Documentation is hella important, can you please please please actually do this? In this case I want to use lang:transliterate() and it now returns three values (which I've told you before is a bad idea, but that's another discussion). I don't know what those three values are, and it's not documented. Benwing2 (talk) 03:04, 19 June 2023 (UTC)

@Benwing2 Yep, I'll get started on this. Theknightwho (talk) 13:33, 19 June 2023 (UTC)

Thank you! Benwing2 (talk) 18:50, 19 June 2023 (UTC)

Some or all auto-cat categories are in this hidden category. Is this intentional? If so and if it's not just a temporary hack, it needs to be moved out of user space. Benwing2 (talk) 06:09, 19 June 2023 (UTC)

@Benwing2 This is intentional, though the work I was doing has stalled so I'll create a proper maintenance category. They're all using tables of contents that don't have a language code specified, which means they can't take advantage of proper sortkeys very easily. Theknightwho (talk) 13:33, 19 June 2023 (UTC)

Got it, thanks. Benwing2 (talk) 18:51, 19 June 2023 (UTC)

See CAT:E, they may be your doing. Benwing2 (talk) 23:19, 22 June 2023 (UTC)

@Benwing2 Thanks - fixed. Theknightwho (talk) 23:28, 22 June 2023 (UTC)

Hi, what is the purpose of passing in the calling function and template in Module:parameters? It seems like a lot of extra work. I see Module:parameters/data but I'm not quite sure the gist of this. Benwing2 (talk) 02:43, 23 June 2023 (UTC)

@Benwing2 It means the parameter table can be stored in Module:parameters/data, which means it doesn't get duplicated every time the module is called. If the new preparsing method works we can probably do away with this, as the saving is pretty small. Theknightwho (talk) 02:45, 23 June 2023 (UTC)

@Theknightwho Right, I see, and yes I expect the savings would not be much. Benwing2 (talk) 03:16, 23 June 2023 (UTC)

Or when you wake up, check out wolf and subséquent (and probably a few others), which have some errors related to stuff you just implemented. Benwing2 (talk) 06:25, 23 June 2023 (UTC)

@Benwing2 Thanks - I think it’s to do with stuff inside nowiki tags being processed instead of ignored, and possibly something else relating to strip markers in general. Theknightwho (talk) 12:51, 23 June 2023 (UTC)

Your code needs to be more robust in dealing with tags in general: nonvanishing is due to expressions inside <math></math> tags, for instance, and I don't think it's a coincidence that there are html comments in proximity to other false positives. Then there are the timeouts: there has to be some way to limit the time spent by a single transclusion. Wiktionary:Todo/long usage examples goes to a server timeout, so I can't even look at it without disabling my edit-mode-preview setting, but I viewed that page within the last week and remember it having a column for examples, which, if memory serves, are cut off at a certain number of bytes, leaving incomplete expressions at the end. It would be nice to not have whole sections or the whole page broken by individual errors.

Finally, the exclamation mark in the "Invalid syntax detected!" error message is just silly: it already says "Lua error" in screaming red. You might as well add "OHHH NOOO!!!!" or "Dear me, I think I'm going to faint!"... Chuck Entz (talk) 18:11, 23 June 2023 (UTC)

@Chuck Entz So there were some infinite loops going on that were causing problems, because it's (almost) impossible to tell something loaded via frame:preprocess that it shouldn't try to load the cached data, because it's being called while generating the cached data in the first place. I've managed to find an awkward way around it, though. I'll have a look at the others now. Theknightwho (talk) 18:22, 23 June 2023 (UTC)

It might also help to think about why the same thing can work fine when you preview the section, but has a false positive when the whole page is viewed, as with the Synonyms section at volition. By commenting things out there, I found a parameter in a quote template elsewhere in the Adjective section that had mismatched square brackets. I think you need to work on the limits of the parsing: the error message makes it look like the problem is in the first template, while it's actually somewhere else entirely- the error actual makes it harder for the average user to find the error. Chuck Entz (talk) 19:16, 23 June 2023 (UTC)

@Chuck Entz Sorry - did you mean a different entry? volition doesn't have a synonyms section. Theknightwho (talk) 19:24, 23 June 2023 (UTC)

Sorry! It was volitional. Chuck Entz (talk) 19:26, 23 June 2023 (UTC)

@Chuck Entz Got it - thanks. I've put the template parser inside a pcall function, which detects that an error has been thrown without actually throwing it. That way, it can fail a particular template parse without bringing down the whole thing. If the template is actually supposed to throw an error, that'll happen once the "real" template call is done later on. Theknightwho (talk) 19:38, 23 June 2023 (UTC)

@Chuck Entz I’ve been doing some work on a more general-purpose parser, which is based on a Python wikitext parser that’s commonly used by bots (mwparserfromhell). Parsing gets really complicated when you get into nested templates - especially when dealing with {{{template arguments}}}, or when {{{{templates are used}} to | generate template names}}. Fortunately, it’s not particularly resource-heavy, and should be able to prevent the issues we’re seeing here. Theknightwho (talk) 16:31, 24 June 2023 (UTC)

Let's disable the new stuff you added until everything is ready so we don't have so many errors. Benwing2 (talk) 23:06, 24 June 2023 (UTC)

@Benwing2 I think it's stabilised at about the position we were at - I'm not seeing any parser errors at the moment, as they're all memory-related.

The full parser is gradually getting there (Module:User:Theknightwho/parser). There's a bug in the tokenizer relating to wikitables that's proving difficult to isolate, but other than that the tokenizer's complete - and it's by far the most complex component. I'm pretty pleased with the performance so far, but coming in under 10 seconds on extremely large pages may prove to be a challenge: tokenising a 100KB page takes about 3 seconds at the moment. Theknightwho (talk) 19:26, 26 June 2023 (UTC)

By errors I mean memory errors. There are 59 entries in CAT:E and most of them are related to memory errors or timeouts. There weren't that many before. Benwing2 (talk) 20:43, 26 June 2023 (UTC)

As a general rule, you shouldn't launch code before it's ready. "It's gradually getting there" isn't a good solution for memory errors IMO. Benwing2 (talk) 20:44, 26 June 2023 (UTC)

BTW maybe there's a way to avoid parsing all parts of a complex page, e.g. skip some of the tables or other complexities and let the built-in PHP parser worry about that. Benwing2 (talk) 20:46, 26 June 2023 (UTC)

@Benwing2 The general-purpose parser is a completely different module, and isn't deployed yet. Aren't most of the new memory errors down to the recent changes to the Chinese templates? Theknightwho (talk) 20:46, 26 June 2023 (UTC)

The changes you made to Module:headword/data or similar. These are live and AFAIK added several non-Chinese entries to CAT:E. Benwing2 (talk) 21:33, 26 June 2023 (UTC)

@Benwing2 Just as an update on this, the general template parser is now in an alpha-ish state: at User:Theknightwho/sandbox10 it's able to render a copy of the page teacher (saved at User:Theknightwho/sandbox9) in a single module call, reading directly from the raw wikitext. I've had to comment-out a couple of templates that nest piped wikilinks in templates (as I forgot to account for them), but arbitrary template/argument nesting, noinclude/includeonly tags, most parser functions and most other things it needs are now working (e.g. T:langname-lite, T:quote-meta and T:q are no bother).

In terms of performance, it's adding about 0.2 seconds to the rendering time (about 2 seconds instead of 1.8), while reducing the memory consumption from 49MB to 31MB. I still need to add a lot of memoization, so I'm sure both of those will come down even further. Theknightwho (talk) 18:55, 6 July 2023 (UTC)

This is great, maybe you could write up a bit on how it works and esp. what the general principles are, for someone who isn't so familiar with how MediaWiki works? We'll need that documentation before this goes live so that other people can fix the parser if something goes wrong. Benwing2 (talk) 19:43, 6 July 2023 (UTC)

@Benwing2 Definitely - I want to get it into a semi-reliable state first, in case things need refactoring. I've just solved the issue with piped links, by the way, so the sandbox now works with those templates included as well.

One of the biggest challenges has been minimising the amount of parsing that needs to be done, which means things like {{#if:}} and {{#switch:}} need to be calculated dynamically as they're being parsed, instead of being passed through frame:callParserFunction() (which would necessitate evaluating every parameter in advance). Theknightwho (talk) 23:49, 6 July 2023 (UTC)

@Benwing2 Just FYI, adding the line mw.loadData = require in the parser tester reduced memory usage down to 22.4MB, and loading time is now equivalent with the standard page. Theknightwho (talk) 14:33, 8 July 2023 (UTC)

This is great! You should test on some other high-memory pages to see if the gains are similar across-the-board. Benwing2 (talk) 18:45, 8 July 2023 (UTC)

You should also test it on time hogs like Coptic entries with humongous inflections and frequency list appendices. Chuck Entz (talk) 21:08, 8 July 2023 (UTC)

@Chuck Entz Good call. (Another way to bring the Coptic entries within time and memory limits I think is by rewriting in Lua, but that would take some effort.) Benwing2 (talk) 21:12, 8 July 2023 (UTC)

@Benwing2 @Chuck Entz So it still runs into trouble on huge pages - partly because there are some bits I've not implemented yet - but also because it does seem to run into time-out issues.

I'll see if I can redesign it to convert templates into functions (instead of re-building them from the cached tokens every time), as that has the potential to massively speed things up. It's a bit tricky, since the parameter values affect which bits get parsed by the builder (which is why it's redone each time), and I imagine it will be a little less memory-efficient, but I think that's fine. Theknightwho (talk) 21:16, 8 July 2023 (UTC)

Sounds good. As a last resort we could add something to the page text of specific pages to indicate that the parser shouldn't be used but we should avoid the need for this if possible. Benwing2 (talk) 21:19, 8 July 2023 (UTC)

@Benwing2 Curiously, it's actually faster for ⲱϣ (ōš) (about 8 seconds), which I suspect is because the Coptic templates aren't that complex for the parser, but there are tons of links all being processed by separate invocations.

I'll let you know how I get on with this new idea, as it might take a little while to get right. Theknightwho (talk) 21:28, 8 July 2023 (UTC)

@Benwing2 @Chuck Entz As an example of a huge page, I've copied about 3/4 of a into User:Theknightwho/sandbox11 and converted all the lite templates into regular ones, which is about the point where memory errors start happening under normal circumstances. The parser manages it in about 8.5-9 seconds using 35MB, versus 5 seconds using 52MB (i.e. right on the cusp of memory errors).

If I'm able to get a little more speed out of it, I'd say we might even be able to remove all the lite templates from a, as memory increases seem to scale logarithmically with page length. Theknightwho (talk) 05:41, 9 July 2023 (UTC)

All good, thanks for the update. Benwing2 (talk) 05:46, 9 July 2023 (UTC)

@Benwing2 Hmm - the memory use on that page goes down to 23MB(!) if you make package.loaded a weak table, but I'm unsure if that might have weird side effects. By comparison, the original test page ("teacher") goes down to 13.8MB (a reduction of 72%). Even a page using multitrans like "wolf" sees a 50% reduction (38.5 to 19.5MB). Theknightwho (talk) 07:04, 9 July 2023 (UTC)

Can you check to see if there are weird side effects? Maybe copy the output and see if it differs? Benwing2 (talk) 07:19, 9 July 2023 (UTC)

@Benwing2 Seems to be the same, and the speed doesn't seem to be impacted either way. Theknightwho (talk) 07:29, 9 July 2023 (UTC)

Oh - one other thing I forgot to mention is that the post-expand include size is halved, too, which helps pages like 不 etc. Theknightwho (talk) 07:32, 9 July 2023 (UTC)

That I don't understand. If you make package.loaded a weak table, does this effectively require() each module once per page? I'm not sure what you are referring to by post-expand include size and why it would be halved. Benwing2 (talk) 07:37, 9 July 2023 (UTC)

┌────────────────────────────────────────────────────────────────────────────────────────────────────┘ BTW heading to bed now ... it's 2:39am local time. Please feel free to post further responses and updates, though, and I'll see them in the (late) morning. Benwing2 (talk) 07:39, 9 July 2023 (UTC)

@Benwing2 In theory it means that require has to load things from scratch every time, since no external modules are in use whenever the parser's not actively loading a template. In practice, though, heavily-used modules like MOD:languages will rarely be out of use (meaning that most garbage collect cycles won't affect them), while language-specific modules that are never used again will be getting periodically cleared.

The post-expand include size is sometimes called the template include size, I think - I mentioned it on the Grease Pit a little while ago, as it was affecting some of the very large Chinese entries. Certain text can have a multiplier applied to it (e.g. if it's used in certain parser functions), and I think I've jumped the gun as it looks like simple transclusion doubles the weight, so the reason it's halved is probably because I've been using a bare {{#invoke:}} for testing.

By the way, we're actually dangeorusly close to the include size limit on a (2,086,521 / 2,097,152), since the lite templates have tons of multipliers applied to their outputs due to loads of parser functions and several layers of transclusion. Once it's exceeded, templates just stop loading and show bare links to the template page instead. I know there are a couple of other pages in a similar position, but I can't remember which right now.

Get some rest! Speak soon. Theknightwho (talk) 08:31, 9 July 2023 (UTC)

@Benwing2 Just as a general update - I've done a major redesign of the parser in order to eke out some more speed, which is nearing completion. I thought this was going to be at the expense of memory, but it seems memory use has dropped even further anyway, but crossfingers it should be able to match or even beat the native parser in terms of speed. I'll let you know once it's in a position to go through proper testing. Theknightwho (talk) 22:39, 23 July 2023 (UTC)

OK great! Let me know when you've finished the updates. Benwing2 (talk) 22:41, 23 July 2023 (UTC)

Just FYI, the otherNames column in the list of etymology languages isn't properly populated in the list, e.g. 'Bohairic Coptic' should have 'Bohairic' and some other things listed but none are. I checked the code in Module:list of languages and it's directly accessing data.otherNames. Either this is the wrong field or something is wrong with that field. Benwing2 (talk) 22:39, 23 June 2023 (UTC)

I fixed this but there is still a lot of breakage in WT:LOL/S, e.g. the parents column of the etymology languages. In general this code needs some rewriting given all the changes you and others have made to Module:languages and the data format. Benwing2 (talk) 02:15, 24 June 2023 (UTC)

Hi, I am proposing deprecating or deleting a bunch more form-of and lang-specific headword templates that are trivial wrappers; see WT:RFDO#June 2023. You have commented in the past about these; feel free to take a look when you have a chance. Thanks! Benwing2 (talk) 08:05, 27 June 2023 (UTC)

Hi. You added a bunch of Mongolian-specific tags in Module:form of/data2. I added support for language-specific tags to Module:form of so I'd like to move all the lang-specific tags out of Module:form of/data2 and into the appropriate lang-specific modules. Can you help me do this, either by moving the tags (which is not hard) or by identifying which tags are Mongolian-specific, so I can do it? See Module:form of/lang-data/sw for an example. All you need to do is create Module:form of/lang-data/mn along the same lines and move the tag data. This should help reduce memory pressure. Thanks! Benwing2 (talk) 02:34, 28 June 2023 (UTC)

OK, you also need to add the language code of the module with lang-specific tags to the table at the top of Module:form of. Without this table, pcall() on a non-existent module seems to waste a ton of memory. Benwing2 (talk) 02:49, 28 June 2023 (UTC)

@Benwing2 Hiya - sorry, I missed this earlier. I'll have a look. Theknightwho (talk) 15:00, 3 July 2023 (UTC)

@Benwing2 Have you tried using package.loaders to grab modules instead? It's the primary loader used by require, and returns a function if the module exists or nil if not. The returned function takes no arguments, and will return the contents of the module if called. Doing it that way means we can conditionally load modules without any risk of error, so pcall is never necessary. Not sure how much it'd help with the memory issues, but it might be worth a shot? Alternatively, we could write a new require function (safe_require maybe?), which has this stuff built-in. Theknightwho (talk) 12:13, 9 September 2023 (UTC)

I haven't yet but this seems like a good idea. I think it might also be possible to just check if the module page exists using the regular page-exists functions. Maybe that's considered an "expensive" call but it doesn't take up memory. Benwing2 (talk) 18:14, 9 September 2023 (UTC)

@Benwing2 Yeah, it's pretty cheap in my experience. Theknightwho (talk) 18:17, 9 September 2023 (UTC)

Hi. I got sick of the undiscussed piecemeal changes you're making to the core modules so I reverted some of them, which significantly brought down the contents of CAT:E (from ~ 50 to now 31, some of which are not related to memory or timeout issues). You've promised multiple times not to make core changes without prior discussion, and to create design documents outlining what you were going to do before doing them, and you've never actually done these things. From now on I'm going to be more aggressive reverting such changes; you need to test your changes in your userspace sandbox and then push them as a whole once your parser is finished and you've verified that the result actually does reduce memory and increase speed. I know you keep saying that isn't possible, but I've never had an issue with this approach even with 10+ modules needing to be copied into user space and their links modified. I think there's also another method that User:Erutuon uses that avoids the need to do the copy-and-modify-links approach; you might want to ask them. Benwing2 (talk) 05:39, 28 June 2023 (UTC)

Hi again,

Actually, "high front unrounded vowel" is not necessarily accurate, and the transcription for that would be , not */i/. But the lexical transcription "ǣ" is specifically the FLEECE vowel, regardless of its phonetic value, so that is the more accurate description. It's not a "Wikipedia approximation", it's the standard lexical set -- it's the IPA that's a WP approximation in this case. Also, an example can be useful, and that example is the one given by the dictionary that was my source. kwami (talk) 19:59, 11 July 2023 (UTC)

@Kwamikagami That makes sense. It would possibly be good to deal with the lexical sets in a more standardised fashion (which includes a link to that Wikipedia page), as it would prevent misunderstandings like mine. Theknightwho (talk) 14:20, 12 July 2023 (UTC)

That's a good idea. I'll start adding links, as soon as I get done with all the fake "translingual" entries that are specifically Burmese. kwami (talk) 23:10, 12 July 2023 (UTC)

I see the 'strawberry' example with ㇲ゙ has an alt form in all hiragana. That makes me suspect that katakana is only used electronically because it's available from Ainu (perhaps by Jlect's source). As does examples like っざら ~ ｽﾞざら. If hiragana is preferred in print, or even just common, that would be an argument for proposing them to Unicode, which I'd be happy to do. Are you familiar with the lit? kwami (talk) 20:58, 15 July 2023 (UTC)

@Kwamikagami I'm not sure as I'm not all that familiar with Ryukyuan languages, but I think you're probably right: compare this entry for むとぅびㇲ゙ (mutubz) (with a syllabic む), which is given as an synonym of ㇺとぅびㇲ゙ (mtubz). To me, that strongly suggests the small ㇺ and ㇲ゙ are being used as substitutes for the hiragana forms. I suspect that Miyako follows the same rules as Japanese with the use of katakana and hiragana, but it would be useful to get evidence for that.

I have a feeling that only Ainu and Taiwanese kana have been used as sources for the consonantal small letters (both of which exclusively use katakana), so any use in Miyako (or the other Ryukyuan languages) is likely just coincidental. Theknightwho (talk) 21:36, 15 July 2023 (UTC)

Your sandboxes User:Theknightwho/sandbox11 and User:Theknightwho/sandbox12 are in various language categories such as Category:English one-letter words, Category:en:Latin letter names, etc. J3133 (talk) 17:29, 16 July 2023 (UTC)

@J3133 Thanks - it's because they're tests for the new wikitext parser I've been building, and the most realistic way to test it is to trick Lua into thinking it's actually on the page in question (and not in a sandbox). I've disabled them for now. Theknightwho (talk) 17:34, 16 July 2023 (UTC)

I don't think you noticed, but with this edit you stepped in something. As you may be aware, {{descendants tree}} is a complex and temperamental kludge that never quite does what you think it does. By changing {{desc|fr|cow-boy|cowboy|bor=1}} to {{desctree|fr|cow-boy|cowboy|bor=1}}, you told the module that there were Descendants sections in both French cow-boy and French cowboy. In the old days, that would have caused a module error, but I got tired of entries ending up in CAT:E every time someone moved the Descendants section from one descendant to the other (especially since the cause is often not in the same place as the error, which means the perpetrator is unaware), so I got consensus to change it. Instead of throwing a module error it displays an error message only visible in preview while adding a maintenance category so someone familiar with the languages in question can fix it later. @Fytcha was kind enough to implement it for me. It's not a perfect solution, because the fact that it pretends to be {{desc}} means that those who do things like you just did have to be informed that it only looks like the template works that way. Personally, I think we need to rethink the way {{descendants tree}} works. It should be possible to optionally have some parameters act like the parameters in {{desc}}- that is, just displayed as descendants on their own without having the module scan the targets for further descendants.

Also, the module only recognizes alternative forms that use {{alter}} or {{alt}} when it displays alternative forms. In a few cases, such as Middle English descendants with a gazillion alternative forms, I've exploited this by using {{l}} for the alternative forms that I didn't want to be displayed by {{desctree}}, and {{alt}} for the rest. Really, though, we should have a better way to control what displays. Another problem is that having multiple alternative forms that each have their own descendants results in a list of alternative forms followed by a list of descendants without a clear way to show which descendant is from which alternative form.

Then there are situations where the main entry that has the alternative forms section isn't the one that has the Descendants section: a clear case of lexicography (which form is or was the most common and correct according to contemporary usage?) conflicting with etymology (which one is the actual ancestor of the descendant in question?). There's no way for {{descendants tree}} to choose between the two, even when the default is wrong for the purposes of the template.

Anyway, I thought I'd make you aware of this so there will be one more person thinking about it. I have Special:WhatLinksHere/Template:tracking/descendants_tree/desctree-no-descendants bookmarked, and I do my best to fix what shows up there. There are occasions, like currently with Old Javanese kamantryan, where I'm not sure what to do- so I just leave it alone and hope that someone will preview the entry or see that it's in a maintenance category and discover there's a problem. So far, I've never had one of these remain in the maintenance category for more than a week or two. Thanks for reading all the way through this, Chuck Entz (talk) 00:28, 17 July 2023 (UTC)

@Chuck Entz I didn't write the {{desctree}} code but I've done some hacking on it. If you can make a concrete proposal as to new params/options to add, I may be able to implement it. Benwing2 (talk) 04:27, 17 July 2023 (UTC)

Thanks @Chuck Entz - that's useful to know. I would like to do a bit of an overhaul of descendants at some point (e.g. doing them in a single template call). It would allow for standardising the layout, and would hopefully prevent situations like these. Theknightwho (talk) 04:35, 17 July 2023 (UTC)

Yeah I think something like that makes sense. Benwing2 (talk) 04:37, 17 July 2023 (UTC)

Hi, I pinged you a few days ago in WT:Beer parlour/2023/April but didn't get a response. I am proposing renaming the qfa-sub-* codes to just sub-*, any objections? Also any suggestions you have about implementing script code aliases are welcome (see the BP entry for more info). Benwing2 (talk) 04:29, 17 July 2023 (UTC)

@Benwing2 Fine by me! Theknightwho (talk) 04:33, 17 July 2023 (UTC)

Hello again, I am starting to rename the anomalous script codes. This is significantly trickier than for the anomalous etymology lang codes, at least for 'polytonic' and 'Latinx', because they are used in so many places. I am following User:-sche's suggestion of five-letter codes with a capital initial letter; hence polytonic -> Polyt, Latinx -> Latnx, musical -> Music, Ruminumerals -> Rumin, IPAchar -> Ipach, while Morse and Semap stay as is. Let me know if you have any objections. I have set up tracking for use of the anomalous codes and added support for the new aliases in Module:scripts, but I haven't yet added the ability to use the new codes in places where script lookup happens (e.g. in per-script entries in the translit, entry_name, etc. fields), so those have to stay as-is until I implement that support. Once everything is converted I will rip out all the alias support to make Module:scripts and related modules leaner. Benwing2 (talk) 22:45, 17 July 2023 (UTC)

@Benwing2 That sounds good, as it'll make handling them more straightforward. I'm not fully convinced that we even have a need for a separate polytonic script anymore, since it's now widely supported, but that's a separate conversation. Theknightwho (talk) 22:52, 17 July 2023 (UTC)

why is this necessary? I don't see any functions in that module and it's loaded by all the language modules, which certainly makes it a memory suck. At least can you document in Module:languages/data why it needs to be required? Benwing2 (talk) 09:20, 17 July 2023 (UTC)

@Benwing2 It's Module:languages/data/patterns, and the reason is because the table has to be mutable as some things are conditionally added to it. As it's required by multiple functions, it's cheaper to put it in a separate module that can be required when needed, as opposed to just dumping it in the module (which would mean it got loaded every time no matter what).

The other thing to note is that all the mw.loadData calls are wrapped in a function called conditionalRequire, which uses mw.loadData unless a flag is specifically set to use require instead. This is used by Module:family tree, because mw.loadData is slow and really memory-inefficient for situations involving one massive invoke like category pages. We should probably update Module:category tree to use it, actually, as it might prevent the issue with certain language pages occasionally having memory errors. Theknightwho (talk) 09:26, 17 July 2023 (UTC)

OK given that this is rather subtle, can you *PLEASE* *PLEASE* document this in Module:languages/data? This should take you all of two minutes, just copy what you wrote above and expand on it slightly. I shouldn't have to ping you every time I need to understand something non-obvious about code you've written. Benwing2 (talk) 09:37, 17 July 2023 (UTC)

In addition, there's no mention of Module:languages/data/patterns anywhere in Module:languages/data, so am I confused. Benwing2 (talk) 09:39, 17 July 2023 (UTC)

@Benwing2 It's used in Module:languages. I'm wondering whether this is actually necessary, since the table needs to be cloned anyway to stop the table in packages.loaded from having things added to it. I vageuly remember there was a massive unexplained slowdown that forced me to use require over mw.loadData (as in, pages would go from 2 to 9 seconds to load), but that I have a feeling that was when the function was still using mw.ustring.gmatch which has a bug like that. However, there's still a weird memory spike if I test it with conditionalRequire (or mw.loadData), which makes no sense to me. In any event, I'll document that it seems to be necessary to keep memory down. Hopefully this will all be immaterial soon, once the parser is ready.

The reason it's saved there is because it's pattern data for Module:languages, and I wanted it to be in a separate module to minimise overhead. It's got nothing to do with Module:languages/data, but the general convention seems to be that subpages with other names are for functions that have been hived off, while data is always under the data subheading. I don't mind if you want to move it. Theknightwho (talk) 09:49, 17 July 2023 (UTC)

Thanks. Please do add this documentation. It's very important for maintainability purposes, to reduce the dependency on "tribal knowledge". As an example of what I'm looking for, see the comment at the top of Module:affix about the different types of hyphens and affixes. This stuff is key to understanding Module:affix, and it's not super obvious, so what I did is essentially a "brain dump" at the time I was rewriting the module, which is when all the information is freshest in your head. This entire comment took about an hour to write, which is peanuts time-wise compared to the amount of time you've spent working on the code itself. You don't have to write something so detailed, but a paragraph or so would be very nice explaining your learnings about keeping the memory down and what is going on with useRequire, conditionalRequire and such (none of which is currently documented). Benwing2 (talk) 22:51, 17 July 2023 (UTC)

See this recent discussion: Wikipedia:Talk:White supremacy#Nazi_Germany. 81.102.123.104 23:21, 18 July 2023 (UTC)

I'm not convinced by that discussion at all. The statement that the Nazis "never recognized the concept of the white race" is simply incorrect, because they were acutely aware of the parallels (and differences) between themselves and the American South at the time. Although they were focused on "Aryans", we are under no obligation to use their terminology. Theknightwho (talk) 00:04, 19 July 2023 (UTC)

It's still more precise and accurate to describe Nazism as an ideology of Aryan supremacy. The Nazis also regarded other groups of white people as inferior, such as Slavs, hence the OP's reference to it being like calling Hutu power a black supremacist movement. 81.102.123.104 09:17, 19 July 2023 (UTC)

Right, but many Americans didn’t perceive Irish or Italian people as white during the latter part of the 19th century, because racial categories are woolly, subject to cultural biases and prejudices etc. To use your example: it’s pretty common for neo-Nazists in Western Europe and North America to say Slavic people aren’t white, and it’s a pretty universal belief of white supremacists that Jews aren’t white, when obviously the real answer is “it depends on the person”. It’s hard to ignore that both of those beliefs come straight from the Nazis.

There’s also the fact that the meaning of the word “Caucasian” has shifted over time: today it’s usually used to refer to white people, but the Nazis used it in a wider sense that covered many people from Central and Southern Asia. They used “Aryan” to refer to a subgroup of Caucasians, and there had been a lot of debate as to whether eastern or southern Europeans should be counted as Aryan in the decades prior (e.g. look at this early map); it was political factors that ultimately decided the answers. To me, it seems like a specific form of white supremacy.

There’s also the fact that white supremacy is a movement, as well as an ideology, and it’s important to recognise the role the Nazis played in that movement. We can’t really say the same for “Aryan supremacy”, and there’s no reason we have to use their terminology either. Theknightwho (talk) 09:53, 19 July 2023 (UTC)

Your latest two changes caused a bunch of errors in CAT:E. Benwing2 (talk) 06:44, 19 July 2023 (UTC)

I think it's becoming clear that your discussion with Koavf is not coming to a resolution. If I may be so bold, may I ask that you refrain from replying to him for a week, to give both of you some time to cool off? I will make the same request on his talk page. I think it will be better for everyone if we take a step back from all the drama in the Beer Parlour and elsewhere for the time being and maybe come back to it in a different way later, if necessary. Thank you, and all the best. Andrew Sheedy (talk) 14:44, 20 July 2023 (UTC)

Yeah, I agree. Would you be so kind as to step in to ask him to stop? I cannot see how his actions are anything other than an egotistical power-trip at this point. Theknightwho (talk) 14:50, 20 July 2023 (UTC)

@Andrew Sheedy - whoops, missed the ping. To clarify: I meant stop with the WF business until the vote has been resolved, and let other admins handle WF as-and-when. Theknightwho (talk) 14:54, 20 July 2023 (UTC)

I'm happy to mediate between you, but I think as far as WF goes, you two are the only ones still discussing the issue actively. Although I disagree with him, Koavf is perfectly justified in his approach to dealing with WF, so I think it's best to let the vote determine the outcome. Given what Koavf has expressed, I think he will abide by the results of the vote. I hope that the vote can remain civil, without any direct arguing, because I think everything that needs to be said has been said already. Andrew Sheedy (talk) 14:57, 20 July 2023 (UTC)

@Andrew Sheedy Thanks. I think it would be a reasonable return to the status quo (and would heavily reduce the potential for conflict) if other admins were to agree to treat WF as they had been before (i.e. blocking on occasion), but for Koavf to refrain from doing so. After all, that is precisely what caused all of this fuss in the first place.

It is very clear that Koavf does not have consensus (either among the community or among the admin team) to keep acting in this way, so I think it's a reasonable compromise. Theknightwho (talk) 15:01, 20 July 2023 (UTC)

Perhaps it would be better, but I suspect it would be a bit hard to legislate. I think the reality is that while there may have been consensus before, there is no longer consensus. Koavf has a number of supporters and the Beer Parlour discussion seemed to indicate a wide range of opinions all over the map. So while there isn't a consensus supporting Koavf, I'm not convinced there is a consensus against him either. Anyway, let's wait to see the outcome of the vote before trying to find alternative solutions. Andrew Sheedy (talk) 15:58, 20 July 2023 (UTC)

@Andrew Sheedy I see only one person who supports his current approach, though, and quite a number of people have expressed serious concerns about his attitude (including you). Theknightwho (talk) 16:06, 20 July 2023 (UTC)

To be clear - I'm not trying to be defensive about this, but I'm just concerned about the community being run roughshod over. Theknightwho (talk) 16:11, 20 July 2023 (UTC)

Even if there is a community consensus, if the community consensus runs counter to the rules, there isn't really a good way to justify preventing someone from following the rules if they want to. I'm not a fan of Koavf's approach, but he's right in that if there is in fact a consensus against the rules, the best way of avoiding conflict is to just change the rules. I don't want to get caught up in yet another discussion about the issue at hand, so this will be my last reply. See you in the vote. Andrew Sheedy (talk) 16:18, 20 July 2023 (UTC)

Hi Theknightwho,

I am the author of virtually all of the content on the pages 𛄠 and 𛀆. You recently reverted my edit on 𛄠 redirecting most of the usage notes to 𛀆. There appears to be some confusion on kana equivalency, especially when dealing with historical kana.

First of all, your claim that "They aren’t even fully applicable." is not correct. Everything in the usage notes of 𛀆 is applicable to 𛄠. The only difference between them is that one is hiragana and one is katakana, which I made sure to account for by leaving {{ja-Kana-usage notes}} in. Conversely, the information in {{ja-Kana-usage notes}} itself is not very applicable to 𛄠, as it explains the modern usage of katakana, where 𛄠 has been obsolete for a very long time. During the period where 𛄠 was used, katakana and hiragana were used interchangeably between documents. Full-length documents where not a single hiragana was present - only katakana - were a common thing, an example being the script of the Japanese WWII surrender broadcast on 15 August 1945. Katakana was actually considered more official than hiragana not a very long time ago, a situation that has reversed today. In fact, the sources I used (1891 仮名遣 and 1897 日本大文典. 第1編) when creating the page 𛄠 uses it in ways that demonstrate the perfect applicability of the usage notes on 𛀆 to 𛄠. All the information I wrote pertaining to the evolution and scope of 𛀆 is entirely valid for 𛄠.

Furthermore, the information currently on 𛄠 is wrong. Quoting User:Eirikr,

There is no historical evidence that /ji/ existed in Japanese. There is circumstantial evidence for its existence prior to recorded history, but in the entirety of the Japanese written corpus, as best I understand it, there is no /ji/.

Consequently, the statement that "In modern Japanese, old /ji/ evolved into /i/..." is just plain wrong: there was no "old /ji/".

—— User:Eirikr, 00:58, 18 September 2020

His complaint on my talk page is what led me to reform the page 𛀆 into what it is now. 𛄠 has not been changed since then, and it needs to be changed to remove the incorrect information. It is more unhelpful to leave the page as it is, than to replace it with a link to the usage notes of 𛀆, just like what was done on 𛄡 for 𛀁. LittleWhole (talk) 21:59, 22 July 2023 (UTC)

It looks like the underlying concern has been addressed. Good on you both for getting things sorted. 😄 ‑‑ Eiríkr Útlendi │^{Tala við mig} 17:43, 24 July 2023 (UTC)

First of all, I took a look at the "Disallowed nesting of Malayic translations" abuse filter. The regex involved is awfully complicated and could really do with a good explanation of how it works. It uses some syntax that I'm not even familiar with like ?(1). I assume this is a PHP regex; if PHP supports Perl-style (?x...) regexes, you can use it to insert comments into the regex itself to explain how it works.

I wrote a script to unindent languages under Malay (especially) and Javanese. In the process it sorts all the translations; indented and unrecognized lines are grouped with the nearest language above them and kept in order if there are multiple such lines together. Some issues that maybe you could help with:

1. Should the script sort all translation sections or only those where an indented line needs to be unindented? There are an awful lot of mis-sorted translation lines. (The script does not currently sort lines under {{checktrans-top}} because of junk that occasionally appears there.)
2. The script doesn't unindent recognized scripts listed in place of languages; these are Carakan, Roman, Jawi, Rumi, Arabic and Latin. Not sure if this is correct.
3. The script indents the following languages, which include (almost) all the languages that appear under Malay and Indonesian: Acehnese, Ambonese Malay, Baba Malay, Balinese, Banda, Banjarese, Batavian, Buginese, Brunei, Brunei Malay, Ende, Indonesian, Jambi Malay, Javanese, Kelantan-Pattani Malay, Madurese, Makasar, Minangkabau, Nias, Sarawak Malay, Sarawakian, Sikule, Simeulue, Singkil, Sundanese and Terengganu Malay. Some of these aren't recognized Wiktionary languages, so I don't know if they should be unindented and sorted among other languages. Some should probably be renamed in the process (e.g. Brunei -> Brunei Malay). Note that neither Sarawak Malay nor Sarawakian (presumably the same thing) are found as Wiktionary languages, nor is Batavian.
4. The script doesn't unindent any other language variants under Javanese, which includes Central Javanese, Western Javanese, Kaili, Krama, Ngoko and Old Javanese. Not sure if this is correct.
5. When it comes to sorting, the script currently converts to NFD format before sorting by codepoint, so that e.g. Yámana comes before Yiddish, Mòcheno comes between Montagnais and Muong, Tày comes between Tatar and Telugu, Réunion Creole French comes before Romani, Māori comes between Maltese and Meru, Meänkieli comes before Megleno-Romanian, etc. Not sure if this is correct; the current sorting is inconsistent but often sorts these using NFC codepoint sorting.
6. By the same token, Oki-No-Erabu ends up before Okinawan when it seems it usually is currently sorted after. Maybe I should ignore hyphens when sorting (and apostrophes? Cf. S'gaw Karen, O'odham, 'Are'are).
7. Potentially the script could normalize lines beginning with ** (especially common with Chinese variants) to instead begin with *:. Potentially it could also correct issues like * Assyrian Neo-Aramaic {{t|aii|ܩܵܛܵܪ|tr=qaṭar}} (missing colon). Neither is currently done.

Benwing2 (talk) 03:31, 23 July 2023 (UTC)

OK, I made it fix both issues under (7), as well as replace Unicode +00A0 with regular space, replace uses of {{ttbc}} and remove blank lines. Also the script could potentially fix places that put Ancient Greek or Modern Greek under a Greek header, if there's consensus to do that. Benwing2 (talk) 04:17, 23 July 2023 (UTC)

@Benwing2 Hi Ben - I’ll get back to you properly once I’ve had some rest, but just to explain ?(1): it’s a conditional based on whether the first capture group matched, and in this regex capture groups 1 and 2 are inside different sections of an OR non-capture group towards the beginning. What it’s basically doing is saying “if the language is the one in capture group 1, then match…”. It’s a way of carving out exceptions for allowed indentations in certain languages, while keeping it specific to those languages, which it does by using negative lookaheads if the condition is met. Theknightwho (talk) 04:25, 23 July 2023 (UTC)

OK, Thanks. I wonder if things need to be this complex; my checks for indented langs are pretty simple. I notice you only check for a few langs that could be indented when there are many more, as mentioned above (or maybe you check for all but those few named langs? I haven't yet figured out the regex). BTW my script is ready to go, pending resolution of the issues above. It would change 3,716 pages, of which 148 of them have indented Malayic translations (often several per page), and issues 594 warnings, mostly for unparsable lines in translation sections (and most of those are genuine, typically by a single language spilling the translations onto multiple lines, sometimes as in would by a sentence being translated into multiple languages, and occasionally by HTML comments inserted into the table giving hints as to what types of entries are acceptable). Benwing2 (talk) 06:39, 23 July 2023 (UTC)

@Benwing2 It’s the other way around - it checks to make sure nothing is nested under 4 languages, except the handful of script exceptions.

I’ll do some comments for it, which should make it clearer what’s going on, as the overall explanation of what it does is actually not that complicated. Theknightwho (talk) 11:14, 23 July 2023 (UTC)

Thanks. Any comments on the above issues? If not I'll go ahead and run the script. BTW User:Benwing2/unhandled-indented-malayic contains warnings (only 17) related to the indented lines that couldn't be unindented. I'm not sure how to handle them, you might want to take a look. Also you should probably make script exceptions for Arabic and Latin, which show up occasionally (unless you think they ought to have different names). Benwing2 (talk) 19:03, 23 July 2023 (UTC)

@Benwing2 Hiya:

1. In theory it should sort all the translation sections, but I was mainly concerned with making sure there weren't any incorrect nestings.
2. "Arabic" corresponds to Jawi and "Latin" & "Roman" correspond to Rumi. For the sake of consistency, I went with the ones which were most common (which are Jawi and Rumi), so it's probably best to go with that.
3. In terms of nesting, I think the best approach would be to separate out any Malayic languages which we treat as full languages (e.g. list Brunei Malay under "B"), but if there are any that are not Wiktionary languages then it'd be best to flag them up for manual review. Another one to look out for is Indonesian, where some editors have just grouped any languages spoken in Indonesia under "Indonesian" as a heading, such as Javanese and Sundanese. I've sorted some of them out manually, but I'm sure there are more. We're also a bit inconsistent with our treatment of "Old X" and "Middle X" languages in general, so it'd be best to see what the general trend is before committing to indenting Old Malay under Malay.
4. Is it possible to use the UCA for sorting? That would bypass any of the NFD/NFC concerns. If you need a Lua implemention, Module:User:Theknightwho/sort has a function (sort) which takes a table of terms as an argument, and outputs a UCA-sorted table. I haven't implemented variable weighting yet (which applies to -), so you may have to manually disregard it. My inclination is to use English sorting rules in that regard, so ignore the apostrophe and dash (or more accurately, only give them a quaternary weighting so that they're a last-resort tie-breaker, which won't affect anything here).
5. I think normalising ** to *: is a good idea.

I'll do the comments on the regex now. Theknightwho (talk) 19:56, 23 July 2023 (UTC)

Thanks. The script is simply leaving alone any indented languages it doesn't recognize, and unindenting/sorting the remainder, whether found under Malay, Javanese or Indonesian (there are none indented under Sundanese). I'll have it normalize Arabic -> Jawi and Latin/Roman -> Rumi. As for UCA, I'd need a Python implementation since that's what the script is written in. There seems to be one here: However I'm a bit leery of doing this because the sorting should ideally match what is done by the translation adder gadget. Potentially the gadget could call into your Lua implementation but we'd need to make it production-ready and put it in the production space, which I'm wary of doing until we have a clear consensus to do this. Benwing2 (talk) 20:11, 23 July 2023 (UTC)

@Benwing2 Great - thanks. That makes sense. In terms of sorting, the UCA can take what are called "tailorings", which are contextual modifications. These are often for language-specific needs (e.g. Azerbaijani "Q" goes after "K"), or more general ones (e.g. whether to restrict punctuation to quaternary weights because we don't really care about it). If we did implement this for the translation-adder, it would be a good idea to make any tailorings easily accessible to bots using other languages, which we could probably do with JSON. Theknightwho (talk) 20:18, 23 July 2023 (UTC)

OK I discovered that I have an existing script to sort language sections that uses the following code, so I am sorting language translations the same way:

def normalize_lang(text):
  return re.sub("", "", unicodedata.normalize("NFD", text)).lower()

Benwing2 (talk) 21:25, 23 July 2023 (UTC)

@Benwing2 Sounds good! Theknightwho (talk) 21:28, 23 July 2023 (UTC)

@Benwing2 Check the notes in the abuse filter to see an explanation of how the regex works. I don't want to explain it publicly as there's a reasonable chance some users may try to subvert it if they know how it works (e.g. we already had one problem IP from Malaysia trip the filter, who then decided to nest Indonesian under a random language instead, so it wasn't just an innocent misunderstanding). Note also where they placed Malagasy, too. I blocked them for a week (as they've been doing this kind of thing for a while), but they'll no doubt be back. Theknightwho (talk) 22:11, 23 July 2023 (UTC)

OK, thank you! I took a look and it looks good. I am running my script now; it is still tripping the abuse filter on some terms because it doesn't fix everything, for reasons I've already mentioned. In some cases the page actually looks OK, e.g. many has two scripts (Carakan and Rumi) under Javanese; maybe the filter needs updating. Benwing2 (talk) 22:17, 23 July 2023 (UTC)

@Benwing2 Thanks. The filter allows "Carakan" and "Roman" for Javanese, but maybe "Rumi" is a better choice than "Roman" to be honest. Theknightwho (talk) 22:28, 23 July 2023 (UTC)

Hmm, I'm not sure. Currently the Roman -> Rumi change is not sensitive to Javanese vs. Malay, but I can fix this if you prefer. Benwing2 (talk) 22:30, 23 July 2023 (UTC)

So if you look up in Wikipedia, Javanese is said to use three scripts: "Javanese" (Carakan), "Latin" and "Pegon" (Arabic). Malay similarly has "Latin", "Arabic"/"Jawi", "Arabic"/"Pegon", "Thai", "Malay Braille" and some others used historically. This makes me wonder if we shouldn't use Latin in place of Rumi. Benwing2 (talk) 22:34, 23 July 2023 (UTC)

OK, I think Rumi is a misnomer for Javanese and Indonesian, which should use "Latin", while either Latin or Rumi is possible for Malay. Benwing2 (talk) 22:38, 23 July 2023 (UTC)

Hello, you mentioned that the sortkey was unnecessary here. Without the sortkey, in , the two words pronounced mendori, 雌鳥 and 雌鶏, are sorted under と and め respectively. I have two questions: how is 雌鶏 sorted under め in , and how can the two entries be sorted in as they should be? Mcph2 (talk) 11:59, 24 July 2023 (UTC)

@Mcph2 Hi - are you sure that's right? If I test the automatic sortkeys, {{sortkey|ja|雌鳥}} gives めんとり' and {{sortkey|ja|雌鶏}} also gives めんとり'.

That being said, some Japanese modules use other methods to sort terms (e.g. the katakana categories use katakana, and the kanji ones use radical sort). There might be something like that going on here, as they should both be sorting the same. Theknightwho (talk) 12:07, 24 July 2023 (UTC)

@Theknightwho: What I’m referring to is that Template:cl shows the following:

と

• 雌鳥

め

• 雌鶏

I suspect the {{pre}} in 雌鳥 somehow automatically sorts it as とり, ignoring the prefix, while {{cln}} in 雌鶏 uses the automatic sortkey of めんとり'.

Should we add the sort= parameter back to {{pre}} in 雌鳥? Mcph2 (talk) 14:05, 24 July 2023 (UTC)

@Mcph2 Thanks - a much better solution is to fix {{pre}}, because it's unhelpful for it to sort like that. Theknightwho (talk) 14:07, 24 July 2023 (UTC)

@Mcph2 This seems to be something unique to {{pre}}, where it sorts "X terms prefixed with Y" categories by ignoring the prefix, presumably because the assumption is that it's unnecessary to sort with the prefix. I actually think this method of sorting isn't a great idea, because it ignores situations like Japanese where sorting isn't necessarily based on the orthography, as well as situations where there might be two or more variants of a prefix (e.g. English anthrop- and anthropo-) where we want to use the same category for both, and ignoring the prefix just leads to confusion. Theknightwho (talk) 15:01, 24 July 2023 (UTC)

@Theknightwho: Thanks for investigating. I’ll just add the sort=

parameter until you can fix {{pre}}. Mcph2 (talk) 02:21, 25 July 2023 (UTC)

Hi, I know you've been dealing with sortkeys lately. There are some entries in which some editors added |sort= when categorising using non-templates, which of course sorts the page at s rather than the intended one. See this search for example, and it appears that Japanese is the main culprit. I've been removing them when I was dealing with the out-of-memory pages. There might still be a few of them not shown in this search, and also it might be a good idea to abuse-filter them? – Wpi (talk) 16:07, 24 July 2023 (UTC)

@Wpi Thanks for that. An abusefilter makes sense, yes. Theknightwho (talk) 16:14, 24 July 2023 (UTC)

Memory is increasing. Is it possibly related to your change to the Han sort keys? Several pages are hitting or getting close to hitting the memory limit that didn't use to be anywhere close. Benwing2 (talk) 03:26, 26 July 2023 (UTC)

@Benwing2 I think it's unlikely to be related to that: the changes were minor, and strings are stored centrally in Lua memory (meaning they don't get duplicated for each invoke), which is why serialisation is an effective memory-saver in the first place. Theknightwho (talk) 03:33, 26 July 2023 (UTC)

Hmm. Can you explain what you changed in the latest day or so related to core functionality? What were these minor changes?

Also, I am trying to move {{quote-meta}} into Lua but it seems to be drastically increasing memory. Do you mind taking a look at the quote_t function in Module:quote, which is an approximate port of {{quote-meta}}, and comparing it to {{quote-meta}}, and letting me know if you have any idea why memory is increasing so much when I use quote_t? You can see in Template:quote-book the switch back and forth between using {{quote-meta}} and quote_t; using quote_t causes at least 7 or 8 pages, e.g. ban and min, to increase their memory by 2MB or more. In ban, the single call to {{RQ:Shakespeare Hamlet}} uses around 4MB of memory when quote_t is in place, and at most 2 MB otherwise. What confuses me is that the old Template:quote-book code also calls into Module:quote (the source_t function) to do most of the work, and all that quote_t and {{quote-meta}} do is paste together the source line constructed using source_t with the usex lines constructed using Module:usex. Both methods do essentially the same thing and I'm not adding any new parser frames. Benwing2 (talk) 05:56, 26 July 2023 (UTC)

@Benwing2 I’ve only really been working on the parser, so other than Module:Hani-sortkey 2 days ago I’ve not changed anything likely to affect a lot of pages.

I’ll have a look at Module:quote, as that does seem weird. I suspect it’s the random effects of Scribunto memory issues again, and we just don’t notice the pages where memory’s fallen by 2MB. I’m starting to think there’s something seriously wrong with Scribunto’s design which isn’t merely down to the Lua 5.1 garbage collect issue, because I never see any wild fluctuations like this when everything’s contained in a single invoke. Theknightwho (talk) 06:59, 26 July 2023 (UTC)