User talk:Theknightwho/2023

Why did you revert my edits? The words are written in different scripts (Latin, Cyrillic, and one Inuit). At least be consistent and add each of them on every single page. And as for "aza", I only removed duplicates. What you just did makes no sense at all. Shumkichi (talk) 08:46, 2 January 2023 (UTC)

@Shumkichi Because that is irrelevant. We put things that look similar there, regardless of whether they’re the same script. I then reverted all the ones where you removed anything. Theknightwho (talk) 12:00, 2 January 2023 (UTC)

Hello Theknightwho! Many words in the Macedonian entries, in the headword lines, in declension tables, in derived and related terms etc. are messed up, they link to wrong or nonexistent words. I assume this is related to the problem pointed out above, about the Serbo-Croatian "ć".
In Macedonian Ќ ќ /c/ and Ѓ ѓ /ɟ/ are separate letters, different than К к /k/ and Г г /g/, and the diacritic shouldn't be stripped away. It shouldn't be stripped away from Ѐ ѐ (сѐ, нѐ vs се, не) and Ѝ ѝ (ѝ vs. и; See ◌̀#Macedonian) too. The diacritic mark should be stripped away only in the accented letters: А́ а́ Е́ е́ И́ и́ О́ о́ У́ у́ Л́ л́ Р́ р́; See ◌́#Macedonian. Gorec (talk) 11:55, 6 January 2023 (UTC)

@Theknightwho It seems the problem is in Module:mk-sortkey!? --Gorec (talk) 14:36, 6 January 2023 (UTC)

@Горец This is fixed! Theknightwho (talk) 19:56, 6 January 2023 (UTC)

👍 Thanks. Gorec (talk) 14:19, 7 January 2023 (UTC)

Hello @Theknightwho. I want to report another problem, which I think is related to the above. When we add Macedonian translations that contain accented letters (А́ а́ Е́ е́ И́ и́ О́ о́ У́ у́ Л́ л́ Р́ р́), the autogenerated edit descriptions that appear in Revision history or in Contributions history always show redlinks, even if those entries already exist. For instance, there is an entry забележителен, but in the autogenerated edit description "забележи́телен" is shown as redlink (t+mk:забележи́телен (Assisted)). Can this be fixed somehow? Thank you. --Gorec (talk) 20:49, 23 January 2023 (UTC)

@Горец Thanks for this. It's definitely a related issue, but I think it's a bug in the translation plugin (which is something I don't know anything about). I would put something on the WT:Grease Pit. Theknightwho (talk) 20:52, 23 January 2023 (UTC)

Could you please take a look at 일곱째 and the other similar pages in CAT:E? The only other person who has recently edited a module invoked on these entries is Benwing2, but their edits were to a function related to Korean usage examples, whereas yours were to entry name normalization code, which seems much more likely to be relevant. Maybe the code is trying to compare NFC-normalized Korean to NFD-normalized Korean, or similar. 70.172.194.25 18:58, 6 January 2023 (UTC)

Actually, it's not even limited to Korean. See Arabic أَرْبَعُونَ (ʔarbaʕūna), South Levantine Arabic مية ألف, Serbo-Croatian tisuća (which also isn't linkable), etc. 70.172.194.25 19:01, 6 January 2023 (UTC)

Correct. After a false start, I realised it's because the NFD form is being pushed to custom entry name templates (which is desirable), but that means they need to convert it back to NFC again at the end. I've updated Module:Kore-entryname and Module:ar-entryname to do just that. Theknightwho (talk) 19:37, 6 January 2023 (UTC)

And what about the Serbo-Croatian term being unlinkable? 70.172.194.25 19:38, 6 January 2023 (UTC)

I'm getting there. Theknightwho (talk) 19:39, 6 January 2023 (UTC)

This is fixed. I will do a review for any others. Theknightwho (talk) 19:55, 6 January 2023 (UTC)

Thank you! 70.172.194.25 19:56, 6 January 2023 (UTC)

As I saw you have recently been editing the expensive translation pages, I decided to run the script I wrote to check for a very specific kind of error: the use of {{tt}} or {{tt+}} outside of {{multitrans}}, thus leaving ⦃⦃unreadable code like this¦¦¦¦¦¦¦¦¦⦄⦄ in the visible page output. The script detected problems on the following pages:

• go
• love/translations

Not a big deal, just thought you should be aware of this failure mode. 70.172.194.25 23:16, 12 January 2023 (UTC)

Good catch! Not sure what happened with go, but love was down to me forgetting to include the first checktrans section. The savings with these are immense - just look at water/translations. Theoretically, we could put one invoke at the top of the page (taking no arguments), put the whole rest of the page inside <nowiki>/</nowiki>, and then process the whole page from a single call. There would be downsides to that (e.g. one error breaking everything), but it might be something to consider for very large pages. The current lite templates are a huge faff for tiny gain. Theknightwho (talk) 23:34, 12 January 2023 (UTC)

Would that break section edit links? 70.172.194.25 23:43, 12 January 2023 (UTC)

I'm not sure. Given {{head}} is probably the major culprit (meaning there's little use in doing this for each language separately), it would be a problem for sure. Theknightwho (talk) 23:47, 12 January 2023 (UTC)

In this minimal example, it seems like wrapping ==Headers== in a template that just spits out its input makes the links go away. I think that's probably a particularly bad thing on pages that are really large, because those are the ones where the links are most useful, vs. having to deal with the full page's wikicode when you just want to make a minor correction. It could still be worth it overall as a last resort if the alternative is having visible output break. 70.172.194.25 23:59, 12 January 2023 (UTC)

Yes, you're right. I've realised it's possible to use Erutuon's {{multitrans-nowiki}} this way, and have changed the sandbox to a (somewhat mangled) version of ⽉. It's removed all the edit links there as well. The way around this might be a Javascript gadget, though. Theknightwho (talk) 00:03, 13 January 2023 (UTC)

Such a gadget would have to not only add the links when action=view, but also figure out when action=edit which section is supposed to be the one being edited (using some ad-hoc URL parameter from the links), remove everything before and after that section from the edit window, and then upon saving add everything before and after back to the saved page content. And deal with "show changes" / "preview" too. It would be doable but it seems likely to be inelegant/brittle. 70.172.194.25 00:13, 13 January 2023 (UTC)

One way to do it might be to use the Labeled Section Transclusion extension (which we should have installed, as it's the way Wikisource works). The model would be:

1. Put the sourcecode for the very large page on a subpage. Everything must be nowiki-fied, except for the <section> tags and the headings we want to have edit buttons.
2. Use <section> tags to divide up the subpage as appropriate. A section for each language, and one for each etymology (if more than one) is a reasonable compromise.
3. A single invoke on the main page processes all of the transcluded sections in one go.

I don't know how the extension would react to its output being shoved through Lua, though. Theknightwho (talk) 00:33, 13 January 2023 (UTC)

Theknightwho (talk) 00:29, 13 January 2023 (UTC)

That's creative. It might solve the link problem, but from how you described it to me, a pretty technical user, I anticipate a significant cost of extra confusion for average editors, possibly to the extent that just losing the links would be better (and there may be other complications I can't think of, as I'm not intricately familiar with that extension). IDK. 70.172.194.25 00:39, 13 January 2023 (UTC)

Yes, I agree. It's inelegant at best. I did try forcing the parser to process a template before it processed the argument (i.e. wikitext) being passed to that template, and a small part of me still thinks it might be possible with some very careful manoeuvering. What I tried was:

1. Passing the wikitext we want to preprocess as a sole parameter to a template (as with {{multitrans}}).
2. Setting the main invoke as the sole parameter name of the template (i.e. {{template|{{#invoke:}}=...}}). This results in the parameter name being the desired output.
3. Have a second invoke within the template itself which outputs the only key in frame:getParent().args.

The reason for doing it this convoluted way was to see if the parser would process parameter names before parameters themselves, which seemed like it had an outside chance of letting me dynamically deploy strip markers before the argument itself got processed, but I couldnt seem to get it to work. Ideally, this would be a way to let us use nowiki tags without placing them on the page (thereby reducing confusion), but would of course be less efficient than doing the whole page all in one go. Theknightwho (talk) 01:00, 13 January 2023 (UTC)

For future reference: it doesn't do much good to hide a vandal's user name if it's still there in the rollback message. You can't hide the edit if it's the current one- but you only have to hide the edit summary. I prefer to hide as much as possible of that kind of vandalism so they have nothing to show for their efforts- it's like painting over grafitti. Chuck Entz (talk) 05:49, 16 January 2023 (UTC)

Good point - I'll remember that for the future. Thanks. Theknightwho (talk) 05:52, 16 January 2023 (UTC)

Hi there - I was taking a look at this edit https://en.wiktionary.orghttps://en.wiktionary.org/w/index.php?title=Module%3ASyrc-entryname&diff=70993671&oldid=70990886. The way it's currently written is definitely shorter but is really hard for me to read. Using uppercase, underscore delimited for constants with their proper utf-8 names is a widely used naming convention. Also naming the range of characters which are syriac diacritics, as syriac diacritics https://en.wikipedia.orghttps://dictious.com/en/Syriac_(Unicode_block), improves the readability since it's not clear what that range means otherwise.

I partially piggybacked off @Erutuon's style, here's a simple example: https://en.wiktionary.orghttps://dictious.com/en/Module:zlw-slv-entry_name

I know you characterized my preferred style as "pointless" but I thought I would offer an explanation for the reasoning behind it without protesting the changes you made. I'm open to better understanding how I can adhere to a style that's consistent with the expectations of the wiktionary community. Just let me know.

ColumbaBush (talk) 20:36, 18 January 2023 (UTC)

You're a right foul git. Why are you reverting my edits without explanation? I gave a quotation from the Oxford Dictionary, whilst you gave no reason or evidence. Qiu Ennan (talk) 10:52, 25 January 2023 (UTC)

Because per cent is also dated in the UK, and as a native speaker of British English who lives in the UK, I feel like I'm in a reasonably good position to say that. I also cannot find that quote in the OED. Are you sure that's where you got it? Theknightwho (talk) 10:59, 25 January 2023 (UTC)

Per cent is not dated in the UK: it's used in the BBC amongst many other sources. Despite being a native British English speaker (purportedly), you are also prone to mistakes and so you should still provide a source instead of using anecdote, especially when removing other people's contributions. Also, look at the dictionary: per cent is the first form whilst percent is labelled US. https://www.google.com/search?client=firefox-b-d&q=per+cent

Also, my quote is from the New Oxford American Dictionary – https://english.stackexchange.com/questions/105295/percent-or-per-cent Qiu Ennan (talk) 02:56, 26 January 2023 (UTC)

I have looked at “the dictionary”. Have you seen this one? Note that it’s also a British dictionary. And no - your quote is from StackExchange, as the link on that page doesn’t actually work.

At the end of the day, you can be as rude as you like, but it won’t change anything. Theknightwho (talk) 12:58, 26 January 2023 (UTC)

(Historical note: I blocked "Qiu Ennan" because he seems to speak some made-up Anglish and has fought previously against real everyday native English speakers like myself and Theknightwho, and against the truth of how the language is spoken. This harms the project and is just really fricking annoying.) Equinox ◑ 02:34, 28 January 2023 (UTC)

{{tlb}} is supposed to be placed on the same line as the head. ―Biolongvistul (talk) 18:31, 27 January 2023 (UTC)

@Biolongvistul It's clearer the way it is. Theknightwho (talk) 18:37, 27 January 2023 (UTC)

If you say so… ―Biolongvistul (talk) 18:38, 27 January 2023 (UTC)

In this case I agree with User:Biolongvistul, I've never seen {{tlb}} labels placed on their own line and it looks weird to me that way. Benwing2 (talk) 19:50, 1 February 2023 (UTC)

The South African and West African usage have the same meaning . Igbo/Nigerian scammers operate around the world including in South Africa however Mugu is an Igbo word. Explain at once why you have separated obviously same definitions without providing an edit summary. Beaneater00 (talk) 22:09, 27 January 2023 (UTC)

Because in South Africa, it's an alternative form of moegoe, which comes from Afrikaans (and before that, the etymology is uncertain). It's not at all clear that we can simply merge them together, especially given that moegoe doesn't only mean "fool", as the definition is a little more complex than that. There's no harm in keeping them as they are, as it shows additional nuance. Theknightwho (talk) 22:15, 27 January 2023 (UTC)

Who says that moegoe comes from afrikaans ? The other language listed is an urban cant. The Afrikaans usage could just as well reflect Cape Coloured or urban Black African use. This 'moegoe' does not exist on the af.wiktionary, I searched long and hard for an Afrikaans dictionary online, Do you know anyone who speaks Afrikaans ? The only things I found on the web for this << moegoe >> were verbatim copies of your Wiktionary entry. w:André Brink who you cite was a linguistic 'reformer' and anti-Apartheid writer . Perhaps he injected Bantu vocabulary into Afrikaans and it's not an indigenous Afrikaans word that evolved separately , with a separate range of meaning. The definitions which you have cited are apparently garnered from his book quote Why do you have so many comments on your page from today and yesterday about your contentious edits ? It is not apparent to me that you have done anything more than read the entry that already exists and take it as divine word .Beaneater00 (talk) 23:20, 27 January 2023 (UTC)

If you're disputing the etymology, then please take it to the Etymology Scriptorium, which is where we discuss these things. Theknightwho (talk) 23:29, 27 January 2023 (UTC)

I'm going to restore the existing version unless you can provide someone to dispute it with. Beaneater00 (talk) 04:21, 28 January 2023 (UTC)

I've already told you where to go if you want to dispute the etymology. If you do that without some kind of consensus, I'll just revert you and stop you from editing the page, because you'll be edit warring. Theknightwho (talk) 04:28, 28 January 2023 (UTC)

Hi. I see yet another error, 'Lua error in Module:data_consistency_check at line 807: attempt to index local 'frame' (a nil value)'. I've asked you before to use sandbox modules, you really need to use them even though they take more work than modifying the production modules directly. Benwing2 (talk) 19:44, 1 February 2023 (UTC)

@Benwing2 I naively assumed the data consistency check could only be called in one way. In any event, we don't want the new check to show on most pages, as it's really verbose. Theknightwho (talk) 19:56, 1 February 2023 (UTC)

OK but you should still be using sandbox modules, which you seem resistant to doing. Essentially you need to copy the module itself to a userspace module, along with any calling modules, then test in the userspace module, and then push all relevant modules to production at the same time. User:Erutuon has a different way of doing this that may be more clever. Benwing2 (talk) 20:20, 1 February 2023 (UTC)

@Benwing2 What I'm struggling with is predicting where errors will crop up. I spent about an hour writing that new function, and it worked on the page I expected it to (and which I wrongly assumed used the only way of calling it). In other cases, I've previewed changes on several pages where I expected any errors would manifest before pushing something, before realising the error happens on (say) 1% of pages. It's frustrating, because I'm not sure that a sandbox would solve that as I don't just push things blindly, but I'm not sure of the best way to test things like this short of having a functioning mirror, as often it's just not possible to know where the errors will happen. Theknightwho (talk) 20:28, 1 February 2023 (UTC)

I see. Some suggestions: (1) Program defensively if possible, e.g. in this case, it's possible to fetch the current frame using mw.getCurrentFrame() so you don't need to pass it in. (2) You can often work out all the places that call a module using Special:WhatLinksHere and being selective with the 'Namespace' dropdown. (3) If all else fails, you can always search through the dump file; I've done that several times when I need to change a function I know is called from various places. The dump file is about 1G compressed and it takes about 13 minutes to search through it entirely using a Python script on my (rather old) Mac Book pro (if you're interested in my scripts, let me know; they are in but this repository is huge and needs clean up, which I can do). You can also extract out just the Module space code, which is a lot smaller, and search through that. Benwing2 (talk) 20:44, 1 February 2023 (UTC)

@Benwing2 Thanks. I had the same idea re the frame, as all I really needed was to make sure preprocessing happened. That's a very good idea re the dump file - I'll check that out.

I've considered setting up a testpage which would ideally throw errors if any other page could throw one. That might be quite tricky to do, but it shouldn't be too hard to set up something that covers all the usual bases in one place. Theknightwho (talk) 20:57, 1 February 2023 (UTC)

Hi. I am not active on the English Wiktionary, I am active mostly on the German Wiktionary and very little on the Czech Wiktionary, where I had recently a short discussion with Dan Polansky. I'm not his friend, and I don't want to be his advocate or stand up for him. First of all, I would like to clarify which side is right and whether the blocks are correct and reasonable and Dan Polansky deserves them or whether they are exaggerated and unfair.

As a reason for the last block you stated: Continued engaging in personal attacks, despite numerous prior warnings and blocks. This sounds rather general, so I would like to politely ask you if you could provide diffs that show his personal attacks leading to the current block and also at least some of the numerous prior warnings he was given. Thank you very much. Amsavatar (talk) 22:34, 1 February 2023 (UTC)

@Amsavatar Hello. Just to give you an idea of the problem we’ve had with Dan, I think it would be good for you to look at his talk page here. Note the numerous sections where he makes personal attacks against other users (, , , , , ), including the one which he made today () which I blocked him for. Calling unwitting users “semi-intelligent aliens”, albeit amusing, is not acceptable, and is likely to drive contributors away given how frequently he does this sort of thing. Some of the other personal attacks, however, are considerably more serious, and display a level of hostility that is fundamentally at odds with a collaborative project.

Please also note his extensive block log, which includes a (later reduced) indefinite block which refers to this discussion, during which Dan receives the block and which gives a little background. The discussion was continued here. The numerous attacks on his talk page have all been made since he returned, and I have given him blocks of doubling length (1 week, 2 weeks etc) on the basis that he knows very well what the problem is, and simply refuses to acknowledge that his behaviour is a problem. He has been told hundreds (if not thousands) of times to cut it out.

If you insist, I will be happy to get you as many diffs as you want, but I’m going to tag @Chuck Entz, @Surjection, @Benwing2, @Vininn126, @-sche who can all attest to his obstructionism, rudeness, inability to participate in good faith and general detriment to the health of the project. Like me, they are all administrators (with Chuck and Surjection also being bureaucrats). Theknightwho (talk) 00:10, 2 February 2023 (UTC)

IMO Dan is in a class of his own. Along with what is mentioned above, Dan refuses both to cooperate with others and acknowledge even slightly the problematic nature of his edits. He also has a lot of energy and will engage in endless arguments, swamping pages like WT:Beer Parlour. I believe blocking him is warranted and I don't see any likeliness of him improving over time, esp. as he has been a very-long-time contributor and has been problematic for the entire time. Benwing2 (talk) 00:26, 2 February 2023 (UTC)

Not much more can be added. Dan is problematic. Vininn126 (talk) 07:31, 2 February 2023 (UTC)

"including the one which he made today () which I blocked him for"

That does not contain anything that rises to a personal attack however. His sputtering on his talk page is many things, but bannable it is not. A ban of one month for that is evidently excessive. ←₰-→ Lingo ^Bingo _Dingo (talk) 18:52, 2 February 2023 (UTC)

@Lingo Bingo Dingo In the wider context of everything else, it's a continuation of exactly the same thing. I have merely been doubling the length of the block, as recommended by Equinox in the discussion on -sche's talkpage. Every single time Dan has returned, he has immediately started engaging in exactly the same behaviour as before. Since his return on 30 January, he:

1. Made this attempt to publicly shame me (which I therefore discounted). The same comment also shows him openly making up policy in a disruptive manner.
2. Engaged in yet more rules lawyering.
3. Continued to push to discount people's votes when he doesn't like them: .

It's all just a continuation of the exact same behaviour he was engaging in before. In addition to those, there are plenty of new edits which, when taken together, show that his attitude has not changed one bit. Before his return, I also came across this thread on the Czech Wiktionary which had some quite revealing comments (which I've used Google Translate on, so they're a bit squiffy in places):

• In the English Wiktionary, there are countless liars and liars, as well as bullies; it's no honey. They are a disgrace to Anglo-Saxon culture.
• If it seems insane, it's probably because it is insane, and the administrators of English Wiktionary appear to be a bunch of incompetent morons. In addition, they appear to be a bunch of liars, liars, people unfit to keep correct corporate or official records, fraudsters, etc. Well, again, it's human, all too human. Better than Hitler, Stalin, Mao, Hegel, and other allegorical vehicles.
• That the number of powerful people displaying grossly objectionable behavior has increased on the English Wiktionary in recent months and years is another matter; there were always quite a few bad people, but they didn't have that much power.

Can you seriously argue that those seem like the statements of someone who is going to work positively and productively with other users here? Theknightwho (talk) 19:35, 2 February 2023 (UTC)

So you are unwilling to reflect and reconsider your actions. Fine, enjoy. ←₰-→ Lingo ^Bingo _Dingo (talk) 20:50, 2 February 2023 (UTC)

Hey, I noticed that the {{sic}} template is not displayed properly in entries (see the quote in up the creek for example). I suspect it is caused by one of your recent edits in important modules. Could you please take a look at it? (Sorry if I'm wrong and there is another reason for the error.) Thanks, Einstein2 (talk) 23:54, 2 February 2023 (UTC)

@Einstein2 Should be fixed. Was a bug that wasn't throwing errors, but was giving the wrong result. Theknightwho (talk) 23:58, 2 February 2023 (UTC)

There are cases where using {{inh}} without specifying a term does not add the and corresponding request category, e.g. {{inh|en|enm||*ar}}. I'm not sure if it's just when {{{4}}} / {{{alt}}} is given but {{{3}}} isn't, or if there are more complicated rules. If we were talking about {{der}} / {{bor}} then there would be things like {{der|ro|sla}} (specifying a language family avoids the term request), but that wouldn't apply to {{inh}} AFAICT. 70.172.194.25 23:15, 3 February 2023 (UTC)

Good point. I'm currently about to publish a big memory-saving edit on dar, and this came up re the Middle Persian borrowing of Tat dar. It seems as though that if {{{tr}}} is given that that also changes things, as it requests the native script instead (which is something I should have double-checked). I think this should all be possible to account for with the following logic:

1. If {{{3}}} is not given, then check if {{{4}}} has been.
   1. If yes, do nothing.
   2. If not, check if {{{tr}}} is also given.
      1. If yes, display , demand {{{parentlangname}}} and categorise as a native script request.
      2. If not, display , demand {{{parentlangname}}} and categorise as a term request.

There may be yet more rules (which I will check), but that should at least cover the current use-case and the most common one. We'll cross the {{der}} and {{bor}} issue when we get to them, I think, as dealing with language families obviously makes it a little trickier.

I'll publish dar before I add this, as I just want to get it out of the way (even if it will be wrongly categorised for a brief period). Theknightwho (talk) 23:33, 3 February 2023 (UTC)

Thanks. Also compare the current display of Template:inh-lite/sandbox vs. what it would have looked like before today's edits. 70.172.194.25 00:31, 4 February 2023 (UTC)

I've incorporated the new change into {{m-lite}} instead, which also covers various other edge cases. It gives the wrong output if a transliteration is given for an empty term in a language that uses the Latin script, but that's an extreme edge case. Plus, {{m}} and {{inh}} behave differently under those circumstances as well, which doesn't seem quite right.

Slightly confusingly, the argument parentlangname works differently in {{inh-lite}} and {{m-lite}}: in {{inh-lite}}, it works as mentioned above. In {{m-lite}}, langname fills that purpose already. As such, parentlangname is a boolean that changes the error message accordingly. Theknightwho (talk) 01:33, 4 February 2023 (UTC)

As you’ve probably spotted, I refactored this in a way that avoids duplicating the lists of lang codes in each module. I only encountered one language family in use - sla - but I’m not satisfied with the bodge I did to get it working properly. I think I’m going to have to add the family codes as the second-last branch of the main switch, which unfortunately means duplicating those ones specifically. However, this should make the code easier to read. Plus, I’ll update the consistency checker to account for it.

On that note, I’ll also add the checker to Module:data consistency check. Theknightwho (talk) 16:11, 4 February 2023 (UTC)

Oh, and another thing: I’ll see if it’s feasible to use a similar method to incorporate proto-languages. That way, we can drop the normal modules altogether from some of the large pages, which should help. Theknightwho (talk) 16:14, 4 February 2023 (UTC)

Thanks for implementing that. I had no idea there was a way to check the first character of a string using ParserFunctions. 70.172.194.25 11:11, 6 February 2023 (UTC)

It was a faff, but much cleverer people than me managed to work out how to do it before we had Lua, so I took some old revisions of the Wikipedia string functions and used those as a starting point. The fact that memory usage scales logarithmically with each subsequent call made me suspect that removing all calls into Lua link templates would cause a big drop in memory usage, and thankfully it seems that was correct! The really tricky bit was actually removing the asterisk for the link.

You will very likely see an error about the penultimate character of a checked string not being supported by Template:str index-lite/logic. When that happens, just add the character to the switch table. Theknightwho (talk) 11:27, 6 February 2023 (UTC)

Could you update Module:data consistency check to handle the restructured format? 70.172.194.25 23:47, 19 February 2023 (UTC)

I've decided against using it, as the effect on performance is intolerable for very little benefit, unfortunately. I'll delete the pages shortly. Theknightwho (talk) 23:49, 19 February 2023 (UTC)

I didn't check this edit thoroughly, but the Slovene link to eden should have the diacritic stripped, and if there's one error like that then there could be more. 70.172.194.25 10:47, 6 February 2023 (UTC)

What about making a template that we could subst to generate the proper invocation to m-lite? I'm thinking something like {{subst:m-lite-generator|ru|ино́й}} => {{m-lite|ru|sc=Cyrl|иной|ино́й|tr=inój}}. Might not be worth it. Just a potential idea. 70.172.194.25 10:52, 6 February 2023 (UTC)

Thanks. That subst idea sounds like a good time-saver.

I've been thinking of setting up templates similar to langname-lite which will let us do this stuff semi-automatically. It'll be clunky (as each term will need its own entry in the switch table), but it would mean we could take advantage of a data consistency check. That way, if the primary transliteration/entryname/sortkey function changes, any that need changing will get flagged up as well. Plus, they'll only need to be changed in one place. Theknightwho (talk) 11:00, 6 February 2023 (UTC)

I've thought of the possibility of transliterations (etc.) getting out of sync too, but I assumed it would be a minor issue, assuming the old transliteration was still "good enough". For entry names that would be a much more significant problem as the link wouldn't work at all. Checking these things would be a good idea. That said, IDK about putting everything in another big lookup table. I probably would've gone for a solution like scanning every page using {{m-lite}} and checking them using an external script. The table would have the benefits of reducing wikitext clutter and duplication, however, so I'm not opposed to it. It just seems like it might increase the effort needed to convert a page to using lite templates. 70.172.194.25 11:20, 6 February 2023 (UTC)

Another consideration is that so far we've been replacing things like {{m|faciō}} by {{m-lite|la|facio|faciō}}, but also replacing {{m|facio|facere}} by {{m-lite|facio|facere}}, so the entry name is not always a normalization of the display name. And similarly, some transliterations could potentially be manual overrides over the defaults, etc. 70.172.194.25 11:31, 6 February 2023 (UTC)

I think with sortkeys it's probably a must, because I don't think we want weird stuff like private use characters to be in "public facing" markup. You're right about it potentially being more faff, though. I think with overrides, that could be dealt with by having some kind of override exit point that allows manual specification without causing a data consistency issue. I'll have a think about how to do it.

I have a feeling that scanning every page is likely to cause Module:data consistency check to start throwing memory errors, as apparently that's one of the main reasons the Chinese modules are such memory hogs (though I don't know quite how many pages are involved at any given time). Theknightwho (talk) 11:33, 6 February 2023 (UTC)

It was trickier than I thought it would be, but I've developed template generators. e.g. {{subst:m-lite/new}} will generate the correct form of {{m-lite}} without the need to manually enter most of the info. Theknightwho (talk) 00:43, 8 February 2023 (UTC)

It seems great! Thanks! 70.172.194.25 17:37, 8 February 2023 (UTC)

. Maybe it would help to make m-lite/new just return the original m template call when the argument contains a link. It would be possible to make a smarter implementation but not sure it's worth it. 70.172.194.25 01:57, 20 February 2023 (UTC)

Thanks. It's probably possible to just subdivide the string with a gmatch or something, though the wikitext would start to get very messy. It's also "destructive", in that it's non-trivial to convert it back again if we don't need the lite templates anymore. Theknightwho (talk) 02:04, 20 February 2023 (UTC)

Special:Diff/71338532: Input: "*w(a/u)". Output: "*w(a&#47;u)". It doesn't display right in the HTML output either. 70.172.194.25 03:11, 20 February 2023 (UTC)

This is an unfortunate side-effect of the changes I made to Module:languages to escape formatting characters. I'll update Module:lite-new to resolve them into their normal forms. Not sure why it wasn't displaying correctly, though - possibly some kind of double-escape effect going on. Theknightwho (talk) 03:13, 20 February 2023 (UTC)

On 巴, {{subst:bor-lite/new|zh|en|bar}} should expand to {{bor-lite|zh|en|bar|sort=己01}}. 70.172.194.25 03:43, 20 February 2023 (UTC)

"I have a feeling that scanning every page is likely to cause Module:data consistency check to start throwing memory errors, as apparently that's one of the main reasons the Chinese modules are such memory hogs (though I don't know quite how many pages are involved at any given time)."

My idea was more along the lines of implementing a Lua module to check one page for correctness, and then using an off-wiki Python (or $FAVORITE_LANGUAGE) script to check the output for every page that uses these templates. Or maybe the whole thing could even be done in $FAVORITE_LANGUAGE. That seems in some ways a much easier solution than giving every single term "its own entry in switch table", but the switch table does have other benefits. And either way we'd have to handle cases of intentional entry name/transliteration overrides. 70.172.194.25 02:13, 20 February 2023 (UTC)

Hi, regarding this change.

I wonder if you accidentally made a mistake when removing :- from the {{unsupportedpage}}. This has resulted in that the page is now wrongfully sorted under U, for Unsupported. For example; Category:Swedish terms spelled with :. --Christoffre (talk) 22:38, 13 February 2023 (UTC)

@Christoffre Hiya - I amended {{unsupportedpage}} so that it should be sorting automatically, but it seems that some of the titles are still having issues. I'll do a better fix for it tomorrow. Theknightwho (talk) 22:53, 13 February 2023 (UTC)

OK, didn't know you where working on the template. I was about to change it back, but I'll just leave it for now as it will eventually be correct. --Christoffre (talk) 07:50, 14 February 2023 (UTC)

There are still entries flooding in to Cat:E from an absentminded error you made earlier, but there's also something else at work: entries that use {{ja-r}} either directly or through list templates are throwing an error with the message " Lua error in Module:ja-ruby at line 508: Separator "%" in the kanji and kana strings do not match" (see 々#Usage_notes_2 for a typical example). When I checked the transclusion list of one of them, I saw that a couple of your edits to the Module:links family had to do with "risky characters, and it looks like "Separator '%'" is one of those risky characters. I haven't figured out whether this is the cause, but I figure you can connect the dots a lot faster than I can, so here it is, for whatever it's worth. Chuck Entz (talk) 02:24, 18 February 2023 (UTC)

@Chuck Entz I'm dealing with the Japanese issue now. It's a simple fix. Theknightwho (talk) 02:25, 18 February 2023 (UTC)

@Chuck Entz That should now be fixed. These sorts of errors are (sadly) to be expected with the changes I'm making, as they're generally caused by nonstandard/problematic uses of these core module functions. I do try to screen for them first, but it can be very difficult to know what horrors will come out of the woodwork.

In this case, % probably shouldn't have been chosen to be used like this, because it's used in URL codes (e.g. wiki/%26 will take you to the page for &). That causes problems if rubytext ever needs to be used with numbers, for instance. Theknightwho (talk) 03:10, 18 February 2023 (UTC)

I don’t know what you are doing, but it has created bare errors: “Lua error in Module:languages at line 159: bad argument #2 to 'gsub' (string/function/table expected)” e.g. دوسر and אריסא Fay Freak (talk) 21:28, 18 February 2023 (UTC)

@Fay Freak This is dealt with. Theknightwho (talk) 21:42, 18 February 2023 (UTC)

e.g.:

• {{R:du Cange|verbum}}--> verbum in Charles du Fresne du Cange’s Glossarium Mediæ et Infimæ Latinitatis (augmented edition with additions by D. P. Carpenterius, Adelungius and others, edited by Léopold Favre, 1883–1887)

Chuck Entz (talk) 05:20, 20 February 2023 (UTC)

@Chuck Entz Thanks - will take a look. Theknightwho (talk) 05:21, 20 February 2023 (UTC)

@Chuck Entz Fixed. Theknightwho (talk) 05:48, 20 February 2023 (UTC)

Thanks all of your hard work recently with the language data. Do you know if the module error on Wiktionary:List_of_languages,_csv_format a side-effect of the recent changes to Module:languages/data2? I don't know exactly when that page stopped working but it was sometime after February 1st. It seems that Module:JSON_data is also broken, possibly from the same cause. JeffDoozan (talk) 20:49, 20 February 2023 (UTC)

@JeffDoozan Hiya. Yes it is - it's because it assumes the scripts for each language will be stored as a table, but I amended Module:languages to allow them to be stored as strings if a language only has one (for performance reasons). I judged Wiktionary:List_of_languages,_csv_format to be low priority, but I will get to it! Theknightwho (talk) 20:52, 20 February 2023 (UTC)

Okay, don't rush to fix it on my account. JeffDoozan (talk) 20:55, 20 February 2023 (UTC)

Hi. I wonder if it's anything to do with your edits on Module:scripts. If you try to add a translation to e.g. Ukrainian, the script automatically added is "C", not "Cyrl". Anatoli T. ^{(обсудить}/^вклад) 05:46, 23 February 2023 (UTC)

@Atitarev It's because I updated Module:languages so that we can store the list of scripts for a language as a string instead of a table if it only has one; Ukranian only has Cyrl listed, where (e.g.) Russian has Cyrl and Brai, so wasn't affected. I did this because modules shouldn't be reading from Module:languages/data2 etc. directly, but instead should be getting the list from Module:languages, which will take this issue into account. The translation adder seems to be a special case, but I'm looking into it.

In terms of practical impact, though, this shouldn't really cause any problems: the script isn't actually specified in the translation template that gets added unless the language doesn't have it listed (which means it wouldn't be affected by this issue anyway). Theknightwho (talk) 16:04, 23 February 2023 (UTC)

It's because Module:languages/javascript-interface uses Module:languages/alldata, right? 70.172.194.25 16:27, 23 February 2023 (UTC)

I didn't realise - thanks. I had assumed everything was being routed via Module:languages, due to the big warning at the top that nothing else should be accessing the data directly. Theknightwho (talk) 16:28, 23 February 2023 (UTC)

Thanks, it seems to work now. I didn't express myself clear before. It wasn't "weird", it was sort of broken. I had to manually remove "C" from "Script code:", otherwise it was giving "Please use a valid script code" error. Anatoli T. ^{(обсудить}/^вклад) 21:50, 23 February 2023 (UTC)

Ah of course - that's true. @Erutuon fixed it, in any event. Theknightwho (talk) 21:52, 23 February 2023 (UTC)

Hi. I use getLanguageData() in Module:User:MewBot to fetch data on all languages. Between yesterday and today this broke; now it hits the 10-second max running time after it's processed between 1100 and 1150 languages when before it got through all 8173 languages. This breaks various bot scripts such as the one I just wrote to remove redundant Chinese translations (which needs to fetch info on languages so it can convert lect translations from 'zh' to the appropriate code). Do you know why this might have happened? Did you add significant amounts of info to each language that would have resulted in this? Thanks! Benwing2 (talk) 20:02, 25 February 2023 (UTC)

@Benwing2 I’ll take a look when I get home. I certainly haven’t added lots of info to each language, but I did make the change that mul and und have every script - which does get pulled through if you access Module:languages/data/all as MewBot does. Does the issue still happen if you exclude those two? If that solves the problem, we can work out a way to handle scripts with those two langcodes more efficiently (e.g. I’ve already put special logic into findBestScript to avoid this issue). Theknightwho (talk) 20:21, 25 February 2023 (UTC)

Unfortunately it's not that; excluding those two doesn't help. Benwing2 (talk) 20:49, 25 February 2023 (UTC)

@Benwing2 Should be fixed now. Theknightwho (talk) 21:16, 25 February 2023 (UTC)

Hey. Re: "го́йда-хуёйда". I couldn't think of anything similar in English then. Now I recall. It's like someone childishly says "coffee- fuckoffee" or something, LOL. Anatoli T. ^{(обсудить}/^вклад) 01:12, 1 March 2023 (UTC)

Lmao okay, that makes sense. Theknightwho (talk) 16:29, 1 March 2023 (UTC)

Hi. Please include the missing language codes for major Chinese varieties: cjy (Jin) and cdo (Min Dong), e.g. Min Dong 國家 / 国家 (guók-gă). Thank you! Anatoli T. ^{(обсудить}/^вклад) 06:17, 1 March 2023 (UTC)

@Atitarev I've done this. I accidentally missed out the c codes for some reason. Theknightwho (talk) 16:30, 1 March 2023 (UTC)

Hi. Does it matter if we use e.g. mul-symbol or symbol|mul? Neither is marked as deprecated. kwami (talk) 21:18, 2 March 2023 (UTC)

Do you mean {{mul-symbol}} or {{head|mul|symbol}}? Either is fine, but I prefer the latter, as {{mul-symbol}} is just acting as a wrapper for the head template (but in a way that loses a lot of the functionality). Theknightwho (talk) 00:58, 3 March 2023 (UTC)

Okay, I'll try to go with the more general 'head' template. kwami (talk) 06:44, 14 March 2023 (UTC)

Hello, it seems that Nôm characters listed within headwords do not show up anymore in the entries. I'm not familar enough with these so I'm not sure if it's due to your edits at {{vi-noun}} (and other Vietnamese headword templates) or something else entirely, so pardon me if my assumption missed the mark. PhanAnh123 (talk) 11:23, 4 March 2023 (UTC)

@PhanAnh123 I have a feeling that I know what the cause is. I'm just finishing up something else, and then I'll deal. Theknightwho (talk) 11:24, 4 March 2023 (UTC)

Thanks for doing something to fix the tone markings in the Wu romanization. Hope other dialects like Wenzhounese can get added soon.

Right now I don't like how Shanghainese is the only supported dialect since it's not even considered the "purest" Wu (and that would be Suzhounese). WiktionariThrowaway (talk) 19:30, 6 March 2023 (UTC)

There is! The info's at WT:Discord server.

It would be good to add more lects - I agree. Right now, I've been focusing on getting automatic transliterations working for other lects, which was the main impetus for improving the way tones are handled with Wu: e.g. 吳 / 吴. More generally, I've been trying to better integrate the modules for the Chinese lects with Wiktionary's core modules, which should make adding new lects more straightforward. Theknightwho (talk) 19:41, 6 March 2023 (UTC)

Thanks for fixing this up. I don't know what to do with the transcription, though. Looks like we'll need to enter it manually, but I don't see an option for that. kwami (talk) 06:46, 14 March 2023 (UTC)

@Kwamikagami There is no current support in the Ukrainian modules for manual translit. This could potentially be added but it might be a lot of work. Probably better is to modify the Ukrainian transliteration code to recognize the ₚ char (which just displays as a box on my Mac in Chrome; we probably need a font addition to the CSS to make sure this shows up correctly). Benwing2 (talk) 02:34, 15 March 2023 (UTC)

Hello, Theknightwho. I see you edited Module:links recently. I wonder if it would be possible to make the {{l}} and {{m}} link displayed as nowrap, as it looks rather clumsy (in the case of suffixes) when a hyphen remains at the end of a line on its own and the rest of the suffix goes to the next line. (An example, see near the bottom right corner.) I've been thinking about replacing hyphens with non-breaking hyphens for the display, but they cannot be copied as real hyphens. Also I'm aware that a nowrap would affect spaces as well (and word-internal hyphens should be handled too) so it's not ideal either. Maybe an uncopiable nonbreaking space could be inserted after a leading hyphen? Do you have some idea? Adam78 (talk) 15:05, 14 March 2023 (UTC)

@Adam78: See Wiktionary:Grease pit/2022/August § Wrapping of hyphen in affixes. J3133 (talk) 16:20, 14 March 2023 (UTC)

Thank you! Adam78 (talk) 16:33, 14 March 2023 (UTC)

Hi, I see a substantial and increasing number of one-char Chinese entries in CAT:E again, just curious if you've made changes that result in this? Benwing2 (talk) 02:36, 15 March 2023 (UTC)

@Benwing2: Wiktionary:Beer parlour/2023/March#Retiring derivative subpages. Chuck Entz (talk) 05:44, 15 March 2023 (UTC)

The memory errors in CAT:E increased by ~ 12 in the last 30-60 minutes. I suspect this: I think we need to think more carefully about how to make changes without ever-increasing memory usage. Benwing2 (talk) 08:14, 16 March 2023 (UTC)

@Benwing2 I suspect you're right. I added those as a precautionary measure, as I don't know if anyone is pushing raw category/file links through any of the substitution methods (I'd hope not, but it's possible). It's not necessary if things are done via Module:links, though, as it's already set to ignore these four at a much earlier stage if it detects them as embedded links. See Module:links#L-209: they just get returned as-is. By extension, this means everything from {{also}} to {{lang}} is covered as well, as they all work via the link module at some stage. Do you think I can probably remove these? Theknightwho (talk) 08:21, 16 March 2023 (UTC)

I suspect they can be just removed; but if you're adding those just as a precautionary measure, you might want to remove them but in their place add some temporary template tracking code to see if they're actually doing anything. Within a few hours the Special:WhatLinksHere tracking "category" will have entries in it if the code is actually doing something, and then you can correct the callers as appropriate. Benwing2 (talk) 08:35, 16 March 2023 (UTC)

@Benwing2 Seems like something’s pushing raw category links through, though I’m unsure what as of yet: . That shouldn’t be happening, really, as it’s obviously pointless. Theknightwho (talk) 16:02, 16 March 2023 (UTC)

Add a call to error() to the code (without saving) and preview one of the pages. Benwing2 (talk) 16:47, 16 March 2023 (UTC)

@Benwing2 Yep - found the issues:

1. Each substitution method in Module:languages does temporary substitutions to protect formatting twice: once before preprocessing, and once straight after the substitutions/just before postprocessing, in case any formatting/weird stuff gets added by a module. Everything then gets put back at the end. A couple of transliteration modules were adding categories by concatenating them, which meant these were being picked up in the second round. I’ve adjusted those modules so their categories get dealt with properly.
2. When there are embedded links, Module:links still runs the unlinked text through makeDisplayText, which is done by iterating over all the gaps between the links. This is to pick up things like false palochkas, character escapes and whatever. However, I was only checking for piped links, as I didn’t think there was any way for an unpiped link to reach that stage. What I’d missed was that - very occasionally - there will be unpiped category links that get fed directly into the link template, which basically just get ignored for the reason I explained earlier. However, it meant they were incorrectly being treated as unlinked text at this stage. I’ve updated Module:links accordingly. I’ve not been able to figure out a pattern that perfectly matches piped and unpiped links in all contexts, but I’m confident the two separate patterns I came up with are 100% accurate to the parser (and are an improvement over what we had before). Theknightwho (talk) 17:07, 16 March 2023 (UTC)

Great, sounds good. When matching links I've always used separate patterns for piped and unpiped links; User:Erutuon sometimes uses a single pattern maybe using %f but it may not be 100% accurate. Benwing2 (talk) 17:12, 16 March 2023 (UTC)

@Benwing2 @Erutuon Given we were already being more permissive with our embedded links than standard wikitext anyway, I decided to make it as permissive as possible (since I noticed we do actually use "illegal" links like ] in headword templates already). As an extreme example, it can now cope with {{l|mul|] ]]}} to output ]. The only thing you can't do is use ] (though you can get round the display text problem with nowiki tags). Theknightwho (talk) 20:31, 16 March 2023 (UTC)

Cool. I wonder if we shouldn't fold Module:languages/data/patterns into Module:languages; small modules can cause big memory issues so we might get better luck with fewer modules. Benwing2 (talk) 20:38, 16 March 2023 (UTC)

@Benwing2 I did actually do that originally, but there are massive performance issues unless it's either declared in the function itself or cloned, which is a pain as three functions use it. The reason I opted for a separate module was to save it being loaded unnecessarily. Theknightwho (talk) 20:43, 16 March 2023 (UTC)

Hum. We still have a bunch of new entries in CAT:E since yesterday. Not sure what caused it. Benwing2 (talk) 20:44, 16 March 2023 (UTC)

@Benwing2 In all honesty, I think we should probably look at sorting out the Chinese modules. They're seriously bloated.

I did develop a bunch of string functions which are in Module:string utilities, which are designed to use the string library wherever possible - only using ustring as a last resort. It varies, but they definitely do help, so we could start using those in more places. Theknightwho (talk) 20:49, 16 March 2023 (UTC)

┌────────────────────────────────────────────────────────────────────────────────────────────────────┘ I agree and I think we should seriously look into my earlier suggestion of encoding the T<->S tables more efficiently as they have to be a big part of it. Benwing2 (talk) 20:51, 16 March 2023 (UTC)

@Benwing2 They are definitely a contributing factor, but there are quite literally thousands of Chinese data modules in the subpages of Module:zh. I've noticed T:zh-pron, T:zh-dial and similar using 8-15MB each in some cases, and I'm convinced it's due to this. By comparison, the traditional/simplified tables are small fry.

A major concern I have is that the Chinese modules make very little use of the main infrastructure, or at best it's piecemeal, which means there's a lot of duplication going on. I think we should probably tackle Module:zh-pron and its submodules Module:cmn-pron, Module:yue-pron etc. first, as it's the main Chinese module. I've done a few things, but they probably need a total rewrite. Theknightwho (talk) 20:58, 16 March 2023 (UTC)

@Benwing2 I think one cause of the spike yesterday was this diff, which was fair enough as there was a minor bug in a local function of Module:string utilities that tries to convert ustring-only patterns into string-compatible ones: ? after a multibyte character wasn’t being converted properly, which affected a few gsubs. Since patching that and restoring it, the number of entries in CAT:E has decreased. Theknightwho (talk) 14:16, 17 March 2023 (UTC)

Great, thanks for fixing the underlying issue. Benwing2 (talk) 15:26, 17 March 2023 (UTC)

The Hebrew-specific modules haven't been changed recently enough to cause this. I suspect it may have something to do with delimiters in the parameters being messed with by another module. Chuck Entz (talk) 15:14, 15 March 2023 (UTC)

@Chuck Entz Odd that it only affects these three entries. I’ll have a look. Theknightwho (talk) 16:19, 15 March 2023 (UTC)

Those are fixed, but there's a separate issue that seems to be correlated somehow with |pausalwv= (though there are a couple of exceptions with the same error but without that parameter and a few with the parameter but no error).It happened after @Erutuon edited Module:he-common, but I have no idea if that caused it. Chuck Entz (talk) 15:17, 16 March 2023 (UTC)

@Chuck Entz User:Erutuon accidentally deleted a line when making that change, leading to the issue. Benwing2 (talk) 16:58, 16 March 2023 (UTC)

@Benwing2: Thanks for fixing it. Embarrassing mistake. It happened because I had logged around that statement and then deleted the whole block of text. — Eru·tuon 18:33, 16 March 2023 (UTC)

@Erutuon No problem at all. Benwing2 (talk) 20:39, 16 March 2023 (UTC)

See for example Special:Diff/69225452/71670296 which tries to fix it. – Wpi31 (talk) 14:47, 16 March 2023 (UTC)

@Wpi31 This is fixed. It was a strange issue with certain redundant wikilinks in links. Theknightwho (talk) 14:50, 16 March 2023 (UTC)

Hey: I saw this diff and I have been trying to copy it elsewhere. I wanted to say that there is some functionality of zh-l that is lost in bor en cmn: look at Diaoyutai & Fengqiu (manually adding character forms) and also the * functionality in zh-l (where no traditional or simplified are displayed when you put the other into zh-l). I don't know if you're working in this area or not, but I assume you are/were. If I make no sense, nevermind. --Geographyinitiative (talk) 22:06, 17 March 2023 (UTC)

@Geographyinitiative These can both be solved by the same thing, actually. You can use // to give manual forms: {{l|cmn|釣魚臺//釣魚台//钓鱼台}} gives 釣魚臺 / 釣魚台 / 钓鱼台 (Diàoyútái). Because (1) manual forms override automatic ones and (2) empty forms aren't shown, you can put // at the end to stop automatic simplification: {{l|cmn|詞典//}} gives 詞典 (cídiǎn), whereas {{l|cmn|詞典}} gives 詞典 / 词典 (cídiǎn). On the other hand, * at the start is used for reconstructions, as with other languages. Theknightwho (talk) 22:14, 17 March 2023 (UTC)

Hi. Should we use IPA for this? I'm speaking of tone specifically. There was general consensus a couple years ago that Chinese should be in IPA just like any other languages, but we had one major contributor (who was adding a lot of valuable material) who threatened to quit Wiktionary if we switched to IPA. kwami (talk) 02:31, 18 March 2023 (UTC)

@Kwamikagami I have no opinion on this, to be honest. @Justinrleung? Theknightwho (talk) 02:33, 18 March 2023 (UTC)

@Kwamikagami: You mean with the tone letters, like ˥ ˦ ˧ ˨ ˩ ? I think this should be discussed more widely, not just for this particular module. — justin(r)leung _{{ (t...) | c=› }} 04:19, 18 March 2023 (UTC)

Agreed. kwami (talk) 04:34, 18 March 2023 (UTC)

Hi, I see that you're updating ryu-r to be in line with ja-r. Would you mind creating {{ryu-r/multi}} and {{ryu-r/args}}, similar to {{ja-r/multi}} and {{ja-r/args}}? This would be immensely helpful for some of the pages that are very close to the lua memory limit and has quite a number of transclutions of ryu-r, such as 人 and 月. – Wpi31 (talk) 09:21, 23 March 2023 (UTC)

@Wpi31 Good idea. I wonder if it's possible to do this without forking the code even more, as they'll inevitably get out of sync again. Theknightwho (talk) 09:23, 23 March 2023 (UTC)

@Wpi31 This is done. Theknightwho (talk) 10:19, 23 March 2023 (UTC)

Hi, I see CAT:E has ballooned again with memory errors. We really need to optimize Module:languages (especially) and Module:links to avoid doing unnecessary work. I think this needs to be your top priority currently. You should add checks in various places to see if there are any chars that could potentially cause issues, and do nothing if not. Currently it seems we're doing a whole lot of unnecessary splitting, processing and rejoining. I also see you're calling mw.clone() on Module:languages/data/patterns in various places; that uses extra memory, is it really necessary? Benwing2 (talk) 22:04, 23 March 2023 (UTC)

@Benwing2 This happened after I implemented the fix for the Korean transliteration issue, unfortunately, but I've been too tired to deal.

I remember that the pattern cloning was necessary because there was a massive, unexplained slowdown without it, but I can't seem to replicate it now. Looking back, I have a feeling it was probably down to the mw.ustring.gmatch bug.

I'll have a look at further optimisation tomorrow, as you're right that it's not ideal at the moment. Theknightwho (talk) 22:16, 23 March 2023 (UTC)

OK thanks, please do take care of yourself and get some sleep! (You mentioned working in law, and I know at least in the US some law firms are notorious for working their employees to death.) Benwing2 (talk) 22:31, 23 March 2023 (UTC)

@Benwing2 Thanks - appreciate it! Theknightwho (talk) 23:04, 23 March 2023 (UTC)

Hi, there are still 28 or so terms in CAT:E. We really need to bring them down. I can help you work on optimization, but I can't do it alone as I don't understand some of the specifics of the code you've written. Benwing2 (talk) 04:29, 26 March 2023 (UTC)

@Benwing2 Hiya - sorry, I forgot about this yesterday. I've dealt with most of them, but the two Han characters are proving quite tricky. I'll see what I can do. Theknightwho (talk) 20:20, 26 March 2023 (UTC)

Thanks. How did you bring them down? What I'm suggesting is looking for opportunities to avoid unnecessary work in Module:languages, not simply using ever more *-lite templates. IMO doing this is not hard and very important. I would do it myself but I don't know which sorts of optimizations of this nature are safe. Benwing2 (talk) 20:22, 26 March 2023 (UTC)

@Benwing2 Mostly with {{tt}}, but ~5 needed lite templates. I agree that we need a longer term solution, but I was trying to get these out of the category in the short-term, as it may take a bit of work to touch up Module:languages (as there's quite a lot to go through, and I'm not happy with the overall structure right now). In the immediate term, I think it's okay to keep adding more lite templates as and where necessary, as we can always remove them once the memory issues are a bit less pressing. Theknightwho (talk) 20:27, 26 March 2023 (UTC)

The optimizations I'm suggesting will not require major rewriting of Module:languages and will need to be done even once rewritten so IMO you may as well look into them now as they will save a lot of time worrying about adding lite templates and such. What I'm suggesting is similar to what I already do when parsing <...> inline modifiers; first check to see if there's a less-than sign anywhere, and if not, avoid loading Module:parse utilities and fall back to simpler code. In your case, check e.g. for brackets, apostrophes and other things that might require you to split the string into parts and process each part individually, and fall back to simple code that doesn't do any splitting. Essentially you optimize for the most common/simple case and avoid invoking (and ideally even loading) the more complex code. Benwing2 (talk) 20:33, 26 March 2023 (UTC)

@Benwing2 I've made a few changes along these lines, which seems to have helped. I've had to keep one instance of mw.clone in doTempSubstitutions: depending on the params, certain additional patterns get inserted into the table of patterns that gets iterated over. If you don't clone the pattern table, these seem to get inserted into the version of the table sitting in packages.loaded, which means they're still there the next time the pattern table gets loaded. On the other hand, using mw.loadData isn't an option as we need to insert the extra patterns, plus it seems to cause a memory increase anyway. Theknightwho (talk) 00:35, 27 March 2023 (UTC)

Also just as an FYI, I've used the U+100000-1FFFFD range for the PUA substitution characters, because we can take advantage of that in capture patterns. e.g. "\244*" will always match a character in that range, but it also means that patterns like "^*" are usable as well (for a string of PUA chars at the start of a string). The only false positives would be non-UTF-8 compliant, and they get caught earlier in the process. Theknightwho (talk) 00:49, 27 March 2023 (UTC)

Over the years, we've had several cases of people erroneously creating entries with PUA characters in the entry names. I hope that kind of thing won't lead to weird and hard-to-diagnose errors before we spot such entries and delete them. Chuck Entz (talk) 01:05, 27 March 2023 (UTC)

@Chuck Entz They won't cause errors - those characters simply won't go through a lot of the text processing that other characters will. Given they're PUA characters, it doesn't really make much sense for them to be processed anyway, so it doesn't really matter. It still works, though, if you really want to do it: e.g. {{l|en|􀀀􀀀}}, {{l|en|'''􀀀􀀀'''}}, {{l|en|w:'''􀀀􀀀'''}} becomes 􀀀􀀀, 􀀀􀀀, 􀀀􀀀 and so on. I can put a pre-check on it to stop people doing this, though, as there's a chance the text might get mangled (but again, as they're PUA characters, people shouldn't be making these in the first place, so what's getting "mangled" is actually totally meaningless). Still no errors, though. Theknightwho (talk) 01:15, 27 March 2023 (UTC)

Thanks. Yeah sometimes mw.loadData() increases memory usage; I ran into that with the {{place}} data, where separating out the functions and tables and using mw.loadData() on the remainder increased memory on a test page with about 60 invocations of {{place}} from I think 25 to 29M. I think the issue is the wrapping of tables, which adds a bunch of overhead, which is only made up for if you load it a whole bunch of times (apparently 60 wasn't enough). As for PUA chars in entries, I wouldn't worry too much about them esp. if the pre-check for them will add memory. Benwing2 (talk) 01:26, 27 March 2023 (UTC)

I'm not sure it's the number of uses, as doTempSubstitutions gets used (tens of) thousands of times on large pages. Who knows.

Re the precheck, if text:match("\244") then error(...) end at the beginning should be enough, I think. Theknightwho (talk) 01:30, 27 March 2023 (UTC)

If it helps any, the error at 二 is due to:

ɨ + ̱

U+0268 + U+0331 (Combining macron below)

I'm not sure how to enter that into {{str index-lite/logic}}, or I would have fixed it myself. Chuck Entz (talk) 02:09, 27 March 2023 (UTC)

@Chuck Entz Thanks - that happened after I updated the lite templates to escape * at the start of translits, as it was sometimes wrongly causing a new list to start. I just got rid of the lite template here, as the page is well below the limit. Tangut translit is intensive (as it's all in a character database), but the page is still only at 46MB.

Generally, it's good to avoid adding characters to {{str index-lite/logic}} wherever possible, as the longer it gets, the more likely other pages are to hit some of the other page limits. Theknightwho (talk) 02:17, 27 March 2023 (UTC)

For the record, I have been trying to bring down 一 and 茶 for the past few days but to no success – 一 is now not in CAT:E presumably due to changes with the Korean templates as mentioned, but there is too much stuff in the descendants of 茶, which I think a multidesc template would help. – Wpi31 (talk) 05:51, 27 March 2023 (UTC)

Look ma, no memory errors! Looks like you've done a lot of work on this, thank you! Benwing2 (talk) 16:54, 29 March 2023 (UTC)

Hi, there are a bunch of errors now in CAT:E. Looks like they are due to a malformed regex coming from one of the serialization modules. Benwing2 (talk) 04:41, 29 March 2023 (UTC)

@Benwing2 Turns out you still need to escape [ even if it's the whole pattern. Really dumb. Theknightwho (talk) 04:47, 29 March 2023 (UTC)

I am purging CAT:E but I still see some errors related to malformed regexes, maybe other chars need escaping? See also Module:pattern utilities. Benwing2 (talk) 04:55, 29 March 2023 (UTC)

@Benwing2 Sorted (though it was a pain, for some reason). Theknightwho (talk) 06:02, 29 March 2023 (UTC)

Hey. You have to block me, I'm about to go fucking crazy Van Man Fan (talk) 22:36, 30 March 2023 (UTC)

I assume this is related to your etymology-language changes, but categories like CAT:German terms derived from Classical Latin are now empty (see CAT:Empty categories), and the category text for a category like Category:German terms derived from Late Latin redlinks the word 'Late Latin' to Category:Late Latin language when it should link to Category:Late Latin. Benwing2 (talk) 01:55, 1 April 2023 (UTC)

@Benwing2 Good catch - fixed. Theknightwho (talk) 02:04, 1 April 2023 (UTC)

I think you need to rethink your edits relative to this code. For instance, Philistine is a real language of unknown affinities, and has 11 lemmas. Many of them are questionable, but we can't rfv them- because your edits have messed up the processing of "und-phi", which is an exception code based on "und", not "und" itself.

Also, "und" has been used as a placeholder when the correct language code is unknown. Most of the time this is a bad idea, but there are some cases where it's at least arguable. For instance, while inheritance by "und" is clearly wrong, borrowing doesn't require any relationship between the two languages, so borrowing into a language with an unknown code can certainly happen. On a practical level, these "und" descendants have been added by very knowledgable people to deal with problems they couldn't solve. They aren't going to be easy to clean up. Chuck Entz (talk) 16:00, 1 April 2023 (UTC)

After spending a good part of my day on module errors, it looks like the problem with Philistine has nothing to do with the "und" part. Instead, it's due to it having nil instead of a family code. There are a number of languages like this, mostly in the Americas but also in Africa or the ancient Near East. What they have in common is extremely limited attestation due to having died out before a decent corpus could be collected or produced. With such a limited corpus, it's often impossible to determine whether a given language is related to others, or even whether it's an isolate. I've spot-checked a number of these, and they all throw the same error in the {{rfv}} template, due to the module's checking the family in order to decide which rfv forum to send it to. The simplest solution would be to just assume that anything with a null family code goes to RFVN, and skip the rest. We really do need to have {{rfv}} working for these, since unknowns tend to attract crackpots or overconfident idiots who are only too happy to fill in the blanks. See for instance the trouble we've been having with Illyrian (never mind that we're pretty sure it's Indo-European), though Philistine itself is a pretty good illustration, too. Most of the Philistine entries were added and/or edited either by BedrockPerson and socks or by ShlomoKatsav, who probably knows better by now. Interestingly, most of ShlomoKatsav's entries were deleted after an RFV initiated by what seems in retrospect to be a BedrockPerson IP sock. But I digress...

Also, the two Proto-Turkic entries in CAT:E seem to be the same thing going in the other direction: they're only attested in one very old manuscript covering a multitude of languages- so it's apparently impossible to tell exactly what language they are- but there's enough data to figure out which branch they belong to. Chuck Entz (talk) 02:43, 2 April 2023 (UTC)

@Chuck Entz You make some very good points, which I hadn't considered - I've removed the ban.

This is something that came up due to the problem of substrates, where (at least internally) they're treated as etym-only variants of und, but with the language family of whatever their parent is; as what distinguishes them is that they have families for parents, not regular languages. Before the major update to etym-only languages that just happened, they were being caught by the ban on using family codes in descendant templates, as the template would grab the parent, see that it's a family, and throw an error. However, after the update, they'd be getting processed as und instead - and therefore working.

To be honest, I suspect the old logic was simply an oversight, as substrates are hardly used. If we allow und for descendants, then there's no reason we shouldn't allow substrate descendants either. Theknightwho (talk) 16:05, 2 April 2023 (UTC)

@Chuck Entz I've fixed the family bug. The underlying issue was that the language's inFamily first checks if the language is in family X. If not, it then checks if the language's family is in family X, (and if not checks that family's family etc). It's that second step which was throwing the error, because you can't check what family nil is in. I've changed it so that it immediately returns false if a language doesn't have a family; the practical result being that Module:request-forum will see that these terms aren't Italic, so sends them to WT:RFVN. Theknightwho (talk) 23:05, 2 April 2023 (UTC)

I saw you had recent edits to Module:ltc-pron/data so I wonder if you know where the following bug is coming from.

The final in the Middle Chinese table (4th row) for certain rimes loads the wrong rime character but the right number.

Examples where you can see include: 溢 (showing 眞 instead of 質) and 沃 (showing 冬 instead of 沃)

It seems the items in the "data.fin_conv" variable in the module are the ones affected.

Zywxn (talk) 16:32, 2 April 2023 (UTC)

@Zywxn: these are in fact correct and not a bug: 質 is the entering tone equivalent of 眞, likewise for 冬/沃 and the other ones in data.fin_conv. The entering tone variant is by all means phonologically the same as the non-entering tone equivalent except that it ends in a stop and being an entering tone.

For the purpose of the module, saying that a character has 冬 rime and entering tone is sufficient for telling the reader that it has 沃 rime. – Wpi31 (talk) 19:25, 2 April 2023 (UTC)

See CAT:E. Benwing2 (talk) 03:34, 3 April 2023 (UTC)

@Benwing2 Thanks - was a metatable issue, where the parent now caches _type during the creation of an etym-only language due to running :hasType(), but that was interfering with the etym-only language's own :hasType method by giving a false positive when it checked for a cached _type table: Theknightwho (talk) 04:12, 3 April 2023 (UTC)

This is definitely a non-obvious interaction; can you add a comment by your change in ? Otherwise it won't be at all obvious why this is being done. Benwing2 (talk) 04:17, 3 April 2023 (UTC)

@Benwing2 Done. Theknightwho (talk) 04:21, 3 April 2023 (UTC)

My bot created a bunch of these, which is an indication that the category code isn't working quite right. I've stopped it from creating more but the category code needs fixing here. Benwing2 (talk) 05:50, 4 April 2023 (UTC)

BTW I can delete these once you fix the category code. Benwing2 (talk) 05:51, 4 April 2023 (UTC)

@Benwing2 Thanks - all of these have been dealt with. Theknightwho (talk) 10:54, 5 April 2023 (UTC)

Hi again, see CAT:E. I purged the category, which eliminated the Old Iranian errors, but the Middle Iranian ones remain. I assume these categories shouldn't exist at all; "Middle Iranian" and "Old Iranian" are not languages at all, but families (and not even clades at that). However, this is a sore subject for User:Vahagn Petrosyan, who insists that these "languages" should exist for ease in porting over Armenian etymologies, where it's apparently difficult to figure out which of several candidate Old and Middle Iranian languages are the right donors for various borrowed terms, and so the linguists working in this area lazily put "Middle Iranian" and "Old Iranian". I have difficulty understanding how this can possibly work, as the phonologies of different Old and Middle Iranian languages are radically different, but go figure. Benwing2 (talk) 06:31, 5 April 2023 (UTC)

You can make both Old Iranian and Middle Iranian codes categorize into "Iranian languages" if that will make things easier. For example, the etymology of սրահ (srah) would display "From Middle Iranian *srāh" but categorize into Category:Old Armenian terms borrowed from Iranian languages. Vahag (talk) 07:18, 5 April 2023 (UTC)

@Benwing2 @Vahagn Petrosyan So Old and Middle Iranian are the only two etymology-only families that we have. Unlike with substrates, where they behave like variants of und, that doesn’t seem appropriate here. I can fix them up, but the question still remains whether we should even have something like this at all. Theknightwho (talk) 10:54, 5 April 2023 (UTC)

I have come to the conclusion that despite some claims, Old Iranian borrowings in Armenian probably do not exist. The main reason for having separate "Middle Iranian" and "Old Iranian" has disappeared for me. I don't mind if the codes are deleted. You can bot-replace {{der|xcl|MIr.}}, {{der|xcl|OIr.}} with "Middle {{der|xcl|ira}}", "Old {{der|xcl|ira}}". Vahag (talk) 11:58, 5 April 2023 (UTC)

I find it okay too, even though it is certain that both Old and Middle Iranian borrowings exist in Aramaic. Not every language conceptualization with a name needs to have a code, if it is not even a language per se and etymological storytelling does not require it either. It is perfectly okay if Sarri.Greek has written “Hellenistic Koine Greek” on various places because that’s what she got to know while language codes may not do anything here. I used “Middle Iranian” because of analogy and Systemzwang more than inherent reason. Fay Freak (talk) 14:29, 5 April 2023 (UTC)

@Vahagn Petrosyan @Fay Freak From a technical perspective this is easy to fix - I agree it isn’t a true family, but it feels a bit crap to remove it entirely. This feels like the sort of thing we could keep for the purpose of categorisation, but nothing more.

That being said, I don’t know much about Indo-Iranian languages - I’m just commenting on the fact that there’s no technical limitation. Theknightwho (talk) 14:34, 5 April 2023 (UTC)

Middle Iranian and Old Iranian complicate the categorization. Now we have Category:Old Armenian terms borrowed from Middle Iranian languages whose members do not directly appear in Category:Old Armenian terms borrowed from Iranian languages, which is not optimal. Let's remove them. Vahag (talk) 16:06, 5 April 2023 (UTC)

See for example 南#Coordinate terms, where ] appears after 北 and 南. – Wpi31 (talk) 19:48, 6 April 2023 (UTC)

@Wpi31 Fixed. I forgot that Module:zh-translit needed to be tweaked after a change to Module:links: . Theknightwho (talk) 20:11, 6 April 2023 (UTC)

This doesn't look right. There are a lot of such categories with 'Unspecified' for the script, e.g. Category:Requests for Unspecified script for Old Uyghur terms. I suspect you should avoid generating such categories for 'Unspecified' script as well as probably for the 'Undetermined' language. Benwing2 (talk) 06:41, 7 April 2023 (UTC)

@Benwing2 Thanks - I’m on mobile right now, but I suspect this is down to substrates being handled as und. Will have a look when I get home. Theknightwho (talk) 20:23, 7 April 2023 (UTC)

If it helps any, these are completely unrelated to the Middle Persian issue. There seems to be something about Module:ka-form of in those entries that's not setting up the parameters for Module:links the way the latter is expecting since you changed the code. I checked the transclusion list for one of the entries: none of the Georgian-specific modules or Georgian-specific data in data-modules has been changed recently except for some changes to Module:Geor-translit on March 30. Chuck Entz (talk) 21:43, 7 April 2023 (UTC)

@Chuck Entz I know what’s causing the problem, as it’s down to Module:links destructively modifying the input. Normally this doesn’t matter, but it does sometimes matter in loops when the same data container gets used over and over. This came up with a Pali module as well.

It’s possible to solve by doing what’s called a “shallow copy” (where you create a copy of the container that also contains the originals, versus a deep copy that copies everything inside as well). That way, any modifications to to the container don’t affect the original one. However, this has a (very minor) memory impact, and also seems to break modules that are accidentally passing the input data as a global variable. I need to catch all of those first (such as the one I just fixed in Module:interproject). The memory impact is negligible, but will probably require some additional saving measures on extremely vulnerable pages like go. Unfortunately, there’s no real way around that impact, as the destructive modification is the most efficient way to implement multiple forms for links.

Incidentally, Module:headword has suffered from this same problem for a while, so I’ll look into fixing that, too. Theknightwho (talk) 21:54, 7 April 2023 (UTC)

I'm not sure exactly how to fix these, but I think the problem stems from assuming that a |tr= parameter is always an implicit request for a native-script equivalent. After weeding out a couple of incorrect language codes and fixing cases where the |tr= was an improvised substitute for using an empty first parameter with a display text in the second parameter to display without linking (I haven't figured out yet how to fix some of these in {{desc}}), there are a few cases where the |tr= is some kind of phonetic transcription meant to display alongside the term specified by the usual parameters. There might be a need for a new parameter for this function, but I'm not sure what to call it and it should be discussed first. At the very least I think you should avoid generating a native-script request category for Latin-script languages, since {{auto cat}} will throw a module error if you create it. It may also be desirable to display Latin-script |tr= text for Latin-script terms as if it were a transliteration, even though it technically can't be one and there's no translit module for it. Chuck Entz (talk) 21:01, 8 April 2023 (UTC)

@Chuck Entz I think this is what ts= is for, though using that still produces the request for native script terms. To be honest, I actually think it's worth lifting the ban on having that for the Latin script, as it's still an accurate description for something like {{cog|lus||ts=puŋᴴ}}, which produces Mizo (/⁠puŋᴴ⁠/). Theknightwho (talk) 01:17, 9 April 2023 (UTC)

I think Template:trans-see currently generates wrong markup.

See, for example, amenity, then Translations, then "convenience — see convenience".

Currently generated makup:

<div class="NavFrame"><div class="pseudo NavFrame">...</div></div>

Should be

<div class="NavFrame pseudo">...</div>

So it should be sinlge .NavFrame element and I believe "NavFrame" should also come before "pseudo". Disfated (talk) 12:41, 9 April 2023 (UTC)

@Disfated Thanks - I’ll take a look. Theknightwho (talk) 13:09, 9 April 2023 (UTC)

@Disfated Are you sure that's what you're seeing? I see this:

<div class="pseudo NavFrame"><div class="NavHead" ... >...</div></div>

Which matches the pre-Lua version: . Theknightwho (talk) 15:34, 9 April 2023 (UTC)

Actually, I've just spotted there's a third div outside of those two which shouldn't be there. My bad. Theknightwho (talk) 15:35, 9 April 2023 (UTC)

We need to be more selective with where we put this. Using it on something like Module:zh/data/st is like throwing a device in a bucket of water so we can be 100% certain we know what the problem is...

It was working fine for over a month, and I spent a long time working on that check. Something new has thrown it off, but it’s important that we keep it. Theknightwho (talk) 22:08, 9 April 2023 (UTC)

@Chuck Entz - ping failed the first time. Theknightwho (talk) 22:10, 9 April 2023 (UTC)

That's okay, I have you watchlisted. While you're at it, take a look at Category:Proto-Indo-European language- this is the first time we've ever had an out-of-memory error in {{auto cat}}. I suspect it has something to do with family trees, since PIE has the biggest one. Chuck Entz (talk) 22:15, 9 April 2023 (UTC)

@Chuck Entz If you check Module:zh/data/ts or Module:zh/data/st now, you’ll see there are tons of results, but it runs quickly with a relatively low overhead, all things considered. Seems the serialisation checks are causing problems, which is not surprising given how massive Module:Hani-sortkey/data/serialized is; but using it means we can sort 1.5k Chinese terms with a 1.5MB overhead instead of a 15-20MB one.

The {{auto cat}} issue I’m aware of. It’s because I just consolidated Module:etymology languages into Module:languages, as the modular separation between them was becoming untenable. On pages with very few calls (e.g. 1), require is actually way more memory efficient than mw.loadData - using around 50% less in some cases. Currently, we’re in a brief transitional stage where we’re doing both, so once that’s over these issues should go away. Should hopefully be tomorrow. On a related point, I also rewrote Module:family tree/nested data to be more comprehensive and flexible, but it’s now an even bigger memory hog than it was before. It’s probably why we’re hitting the limit for the first time, as the PIE tree uses 37MB alone. That should drop down to about 25MB with the changes, bringing us well within the buffer. Theknightwho (talk) 22:33, 9 April 2023 (UTC)

@Chuck Entz The memory footprint of family trees has now dropped significantly, with Category:Proto-Indo-European language now using 42MB; the tree on its own uses 27MB. Incidentally, I think Category:Proto-Niger-Congo language has the largest tree - it's nearly twice the size. Theknightwho (talk) 18:09, 10 April 2023 (UTC)

Just curious what the solution is for Middle Iranian and Old Iranian errors. We also have a ton of categories in Special:WantedCategories that reference 'Middle Iranian languages' and for which {{auto cat}} doesn't work. Benwing2 (talk) 07:34, 10 April 2023 (UTC)

@Benwing2 I keep meaning to get around to this. It's just me being slow, that's all. The solution is probably to just delete the codes, but I'll put a bar against the category modules creating categories like these if the object is etymology-only. Theknightwho (talk) 18:09, 10 April 2023 (UTC)

There are also a lot of 'Requests for native script for ETYMLANG terms' as well as 'Requests for Unspecified script for LANG terms' in Special:WantedCategories. The latter should definitely not be generated at all; for the former, either we need to avoid generating them or modify the Requests category code to allow for etym langs (which would be easy but it's not clear to me what the right thing to do is). Benwing2 (talk) 23:06, 10 April 2023 (UTC)

@Benwing2 I've dealt with the Middle and Old Iranian issue, which involved splitting them out into Module:families/data/etymology (as both families and etym-only languages need to be allowed for them to work). This means terms are now getting categorised in (e.g.) Category:Old Armenian terms derived from Middle Iranian languages. This touches on a wider point about etym-only languages/families, which is that they need to categorise terms into any parent language/family categories as well. That means all those terms would go in Category:Old Armenian terms derived from Iranian languages, as currently that doesn't happen.

This could also be how we handle request categories, but since they're a maintenance issue, we might want to limit it to the non-etym language only. Theknightwho (talk) 23:37, 10 April 2023 (UTC)

Thank you for dealing with this. I'm a bit confused though about your assertion that we need to categorize e.g. all terms in Category:Old Armenian terms derived from Middle Iranian languages into Category:Old Armenian terms derived from Iranian languages as well. I don't think this is currently how these categories work; in general if something is categorized as being derived from Late Latin, it isn't also placed into the corresponding derived-from-Latin category. Instead the category itself has the derived-from-Latin category as its parent. This doesn't quite seem to be happening with Category:Old Armenian terms derived from Middle Iranian languages, which instead has Category:Old Armenian terms derived from Indo-Iranian languages as its parent; I'm not sure why. Benwing2 (talk) 00:12, 11 April 2023 (UTC)

@Benwing2 It's because Vahag expressed dissatisfaction at the terms being divided up between the categories, and I'm inclined to agree; it's much more useful to have the terms in one place, in a similar way that "derived from" is a catch-all category.

The reason Category:Old Armenian terms derived from Middle Iranian languages is being put in the Indo-Iranian category is because Middle Iranian has Iranian as its parent, which means that its family is Indo-Iranian (not Iranian). Currently, the boiler is only looking for the family, but we need to make sure it handles the parent as well - which is how etym-only languages are handled. Presuambly this is because etym-only families haven't really been dealt with properly until now.

I also suggest we formally disentagle substrates from families, and just make their parent und. If they need to have a specific family, we can just set it in the data. This is now easier, because the etym language data has some of its keys automatically moved around before being returned, to make sure that it's fully compatible with the normal language data.

One last thing that occurs to me is that there's actually no reason why we couldn't use class inheritance during the creation of full language objects, too. For example, it might make sense to use it for Bokmål and Nynorsk, since we also have the code no for Norwegian. Theknightwho (talk) 00:32, 11 April 2023 (UTC)

Hmm. If we are to add all terms derived from an etym language to the category corresponding to the etym language's parent, I think we should bring this up in the BP first as some people could conceivably object. As for substrates, I have no strong opinions here; go ahead if you want to make changes. As for the catboiler code, the relevant code is around line 1080 of Module:category tree/poscatboiler/data/terms by etymology; I don't think it will be hard to fix it. For regular-language inheritance, sure, I guess if we are stuck with all three of 'no', 'nb' and 'nn' as regular languages, we can make the latter two inherit from the former (although in practical terms, what would this get us?). Benwing2 (talk) 01:46, 11 April 2023 (UTC)

@Benwing2 I see your point - it could get out of hand. I’ve fixed the issue in Module:category tree/poscatboiler/data/terms by etymology, so (e.g.) Category:Old Armenian terms derived from Middle Iranian languages is now getting categorised correctly. On a related note, I also changed it so that anything with und as a parent gets categorised in its family instead, which catches substrates, as well as a few others like Category:Terms derived from Xiongnu. Otherwise, they’d go in Category:Terms derived from Undetermined (which I don’t think should exist).

This could also come in handy with Norwegian, too: Category:Terms derived from Norwegian Bokmål doesn’t go in Category:Terms derived from Norwegian at the moment, and it probably should. Inheritance would solve that. Plus, it would simplify the data, as it’s a simple way of coordinating languages. Perhaps this is a way to solve the problems we have with langs like Chinese or Tibetan, where multiple L2s sometimes get placed under the same header. I appreciate that this starts to blur the boundary between language and family, but in practical terms we already do that. Theknightwho (talk) 18:51, 13 April 2023 (UTC)

Hello, your edits broke stuff again. {{cog-lite|en|test}} now shows as English test, likewise for the other etymology lite templates. – Wpi31 (talk) 04:31, 15 April 2023 (UTC)

Your edits to Module:zh-translit seem to be the cause. If it helps any, the error in Appendix:Sinitic Swadesh lists goes away if you remove the language code "hsn" from the template, and the error in water/translations goes away if you remove the line with the Xiang translation. I'm not sure why this is only surfacing on these two pages and Appendix:Xiang Swadesh list. A search for Xiang translations finds 99 pages, some of which are also translation subpages that use {{multitrans}} like water/translations does. Chuck Entz (talk) 06:00, 20 April 2023 (UTC)

@Chuck Entz Thanks. I introduced a change so that Module:zh-translit will check the term linked by {{zh-see}} if it can't find any instances of {{zh-pron}} on the page, which means it now works for simplified or variant terms as well. Occasionally, there'll be a chain (e.g. simplified forms of variants), so it iterates until it finds {{zh-pron}} (or neither of them). Theoretically, that means it can get into an infinite loop if you point a group of pages at each other with {{zh-see}} without ever using {{zh-pron}}, but of course that should never happen...

It's relevant to note that you will get different results with different lects of Chinese, as it depends on whether the lect is referred to in {{zh-pron}} or not. If it isn't, it moves on to {{zh-see}}. I suspect there isn't a real mistake on the entries here, but rather an instance of two pages missing a Xiang reading, which causes an infinite loop due to missing data. It's not unusual for very common characters to be variants for certain readings, so a pair of these would cause this. Theknightwho (talk) 06:08, 20 April 2023 (UTC)

I was right: 水 can be a variant of 媠 in Min Nan, which can be a variant of 嫷, which in turn can be a variant of 媠 for a different reading (ad infinitum). I'll put in something to stop that happening, as it's an edge case I hadn't considered. Theknightwho (talk) 06:23, 20 April 2023 (UTC)

In Module:languages/data/2, you wrote "FIXME: Not clear what's going on with the PUA characters here" apropos U+F700 and U+F70F. The basic issue is that ญ YO YING and ฐ THO THAN drop their bottom parts when combined with a mark below. As basic rendering of Thai is fairly old, predating OpenType, there arose the convention of using those two PUA characters for the glyphs thus modified. There's also what I think is the barbarous practice (@Octahedron80 for comment) of using those truncated glyphs for YO YING and THO THAN in all positions when writing Pali. (It's certainly not a universal convention for Pali.) Standards-oriented font creators tend to reject this glyph encoding; the glyphs are highly unlikely to be encoded in Unicode as separate characters on the basis of Pali.

My view is this glyph encoding should be expunged from Wiktionary whenever it crops up. --11:27, 21 April 2023 (UTC) RichardW57m (talk) 11:27, 21 April 2023 (UTC)

An example of unmodified and modified yo ying in the same word is วิญญู (viññū); an example of the instrumental plural suffixed with the quotative particle, วิญญูหีติ (viññūhīti), can be seen on line at the bottom right of the sample of p5 at https://www.google.co.uk/books/edition/%E0%B8%AB%E0%B8%99%E0%B8%B1%E0%B8%87%E0%B8%AA%E0%B8%B7%E0%B8%AD_%E0%B8%AA%E0%B8%A7%E0%B8%94%E0%B8%A1%E0%B8%99%E0%B8%95/Ap3wBQAAQBAJ?hl=en&gbpv=1 . --RichardW57m (talk) 12:03, 21 April 2023 (UTC)

At thwikt, these two dated letters will only show in headwords, as Thai fonts can natively show them. Traditionally, ญ & ฐ should be "unfooted" even without marks under them. Why? Because old people didn't write the "foots". Not only Pali, but also Sanskrit. When naming title, we just use normal letters. The dated letters are added in entry_name and change them to normal, in case if they are linked from somewhere.

Adding VS01 is the new method that could be used as variation sequences, but currently no standard for Thai is released and no font supports it yet. (The fonts would be updated after the standard.) PS: VS01 is used in other scripts. --Octahedron80 (talk) 14:20, 21 April 2023 (UTC)

First off, thanks for your recent work on mod:cmn-pron. While it is possible to improve the other modules in a similar vein, I think ultimately mod:zh-pron will need some major revampping, which I already have some basic idea what to do – this involves moving most of the repetitive lect-specific stuff into a data module or their respective modules (plus, the mixture of the labels and the if else conditionals is difficult to maintain), and standardizing the interface etc. This will be a time-consuming job, which hopefully I could start working on it in late May. In the meantime, I don't think there is much point in changing the modules, since they'll probably have to be redone to suit the new interface anyways. – Wpi31 (talk) 17:09, 26 April 2023 (UTC)

@Wpi31 Yes, you're right! My main concern was to develop a more sensible baseline transcription which all the other conversions can work from, which is essentially Pinyin with some of the quirks removed (iu --> iou etc). That's now done, which should make the task of simplifying other parts of the module much simpler.

I've been putting off working on Module:zh-pron as it's so huge, but I'll follow your lead. Theknightwho (talk) 17:13, 26 April 2023 (UTC)

Hi again,

I created 韶神星 for the asteroid (6) Hebe, but am having trouble with zh-forms. The derivation is not 韶 + 神 + 星, but an abbreviation of 韶華 + 神 + 星. Is there not an option for that in zh-forms, or am I just missing it? kwami (talk) 00:05, 6 May 2023 (UTC)

@Kwamikagami I don't think there's a dedicated way to do it, but I've changed the gloss for the first character in a way that hopefully makes it clear what's going on. Theknightwho (talk) 00:27, 6 May 2023 (UTC)

That works! Thanks. kwami (talk) 00:52, 6 May 2023 (UTC)

Hi,

Thanks for working on the Russian transliteration scraper. I am more keen about this possibility - multiword Thai, Khmer, etc. transliterations, be it with spaces (or ]][[ between words) and respellings (I am sure it can't be done otherwise - no tool will transliterate Thai with 100% accuracy). Also about using |tr= to respell words with multiple readings. Don't mind negative comments you received. Using Japanese kana to transliterate everything is also desirable. Feel free to experiment. It's up to you but the Russian transliteration is working, even if adding manual translits is a bit of a burden. Also @Benwing2. Anatoli T. ^{(обсудить}/^вклад) 03:12, 8 May 2023 (UTC)

Hello, hope you are well. I wanted to ask you if you would take a look at Template:zh-wikipedia. You can see it in use at 中國／中国 (Zhōngguó). I believe that the order of Wikipedias on the Zhongguo entry (that I just linked) is not academically or ethically justified. I believe that Wiktionary should follow the order of Wikipedias as at , and that Template:zh-wikipedia should "auto-sort" (automatically sort) the words in the order of the Wikipedias that Wikipedia uses, no matter what order the Wikipedias appear in the editor's view in any given Wiktionary entry's Template:zh-wikipedia. (1) Even if you don't agree, would it be technically feasible to code in an auto-sort or something like that on Template:zh-wikipedia? Maybe for a different order? (2) Is the Wikipedia order justified within Wikipedia but yet not justified for sorting the Wikipedias when linked from Wiktionary? (3) Is there perhaps another order that you can think of? Even ones you don't think would be good? I've always felt there was something wrong with what I was doing, and I'd like to see if that resonates at all with you (or anybody reading this). Thanks for any feedback. --Geographyinitiative (talk) 12:46, 12 May 2023 (UTC) (Modified)

Hi.

1. Proto-North Caucasian. I think we should have removed everything related to Proto-North Caucasian, everything that it contains in itself and Proto-North Caucasian itself. Simply on the grounds that this is not a family, but a superfamily, which, moreover, is not yet a proven superfamily.
2. Proto-Northeast Caucasian. There are still no good reconstructions about the Proto-Northeast Caucasian. Perhaps the only revision of the reconstruction of Starostin and Nikolaev (1994) was the work of Nichols (2003). However, she uses the # sign, which stands for pseudo-reconstructions, which was introduced by Williams (1989). Here everything (Appendix:Proto-Nakh-Daghestanian reconstructions) is incorrectly indicated by an asterisk, this can be misleading. This family has not been proven.
3. Proto-Northeast Caucasian splits into two branches Proto-Nakh and Proto-Daghestanian. Proto-Nakh is quite well reconstructed, which is not to say, of course, about the Proto-Daghestanian. For some reasons, the Proto-Daghestanian can be left and not deleted, although it has also not been proven. Although recently, for example, in the work Schrijver (2021), the Proto-Nakh are compared with the Proto-Tsezian and Proto-Avaro-Andian. However, the Proto-Avaro-Andian is not worked out properly. And besides, Proto-Tsezian forms is adjusted to the data of Proto-Avaro-Andian.
4. In accordance with paragraph 3, the modern comparison is not with the Proto-Nakh and Proto-Daghestanian, but directly with the Daghestanian groups (Proto-Tsezian and Proto-Avaro-Andian). Which makes one doubt the existence of Proto-Daghestanian.
5. Proposed trees:

• Daghestanian ngn 🌳 ngn-pro
  • Avaro-Andian ngn-ava 🌳 ngn-ava-pro
  • Dargic ngn-drg 🌳 ngn-drg-pro
  • Khinalug kjj
  • Lak lbe
  • Lezgic ngn-lzg 🌳 ngn-lzg-pro
  • Tsezic ngn-tsz 🌳 ngn-tsz-pro
• Nakh nkh 🌳 nkh-pro
  • Bats bbl
  • Vainakh nkh-vay 🌳 nkh-vay-pro

In a sense, such a division will go into contradiction for macrocomparative comparisons. Gnosandes ❀ (talk) 10:33, 13 May 2023 (UTC)

When you excluded those languages from diacritic categorization, did you also intend to cause upwards of 150 module errors? When I first saw the module errors, I thought they would only be there for an hour or two while you were working on the changes to switch things over, so I didn't say anything. Chuck Entz (talk) 21:37, 13 May 2023 (UTC)

The problem is that char_category() is local to a portion of the main function when it needs to be pulled out and moved up. Benwing2 (talk) 21:41, 13 May 2023 (UTC)

@Chuck Entz @Benwing2 Fixed. I've been a bit off my game today, and didn't check CAT:E. Theknightwho (talk) 21:41, 13 May 2023 (UTC)

Hi. I'm not sure if this concerns you, but I noticed it first after you added the function of automatically adding categories.

Under the entries 1:a, 1:e, and 1:o there was a red Category:Swedish terms spelled with 1. I added {autocat}; but it only show one term, 180, and none of them above.

I assume it is those colons that mess it up, it usually is. --Christoffre (talk) 17:04, 14 May 2023 (UTC)

@Christoffre I don't think so - numbers were excluded from all automatic categorisation (which was something that's been the case for a long time), but I've now changed that as I think they're rare enough to warrant categorising: people were manually categorising them anyway, so it's better to just do it properly.

Changes to modules take time to filter through to individual entries, as (literally) millions of pages can be affected. If you purge the cache on a page it'll do an immediate update, though. If you look at the category now, you'll see all three are in it. I'm sure others will filter through in time. 17:08, 14 May 2023 (UTC) Theknightwho (talk) 17:08, 14 May 2023 (UTC)

Ah, it seems like I was just impatient. Thanks for the help anyway! --Christoffre (talk) 17:12, 14 May 2023 (UTC)

Aside from the module errors, these look wrong. For some reason, they're displaying the vowels with iota subscripts as vowels followed by free-standing iotas. This is deceptive, since there are plenty of examples like Greek Αιγίνης (Aigínis), which aren't in Category:Greek terms spelled with ᾼ.

It must have something to do with fonts, since "Category:Greek terms spelled with ᾼ" displays with the iota subscript in the edit window for me, but not in the preview. What's more, if I use a Greek keyboard to type the Greek part in manually as separate characters, it displays as separate characters but doesn't link to the correct category: Category:Greek terms spelled with Αι even though the spelling looks identical to me in preview. For the record, I'm using a MacBook Pro with an old version of MacOS, and I get the same thing with both Firefox and Safari (not logged in on the latter). Chuck Entz (talk) 21:48, 14 May 2023 (UTC)

I just noticed that there's also Category:Greek terms spelled with ◌ͅ- why both? Chuck Entz (talk) 21:54, 14 May 2023 (UTC)

@Chuck Entz So I have a feeling I know what's behind this, but I need to dig through the guts of the category tree to work out exactly what's happening.

In terms of the display, it looks fine for me, but I think this is related to the fact that subscript iota (a combining character) capitalises as a non-combining iota. Something, somewhere is making the assumption that diacritics never change on capitalisation, but this is an instance where they do. However, if that's down to the font you've got, there's not a huge amount we can do.

The reason for having Category:Greek terms spelled with ◌ͅ is because I recently upgraded the standard characters function so that it's no longer restricted to atomic Unicode characters, as they're often pretty arbitrary and not linguistically interesting. Before, any terms in categories for diacritics were only there because the diacritic couldn't form an atomic character with the base character (e.g. there is no Unicode character B̊, so the function was seeing the individual character ̊ and dumping them there instead). Since I made the change, I realised that it's probably quite interesting to see the spread of diacritics, too, so basically did the inverse of the first change by making sure all the atomic characters got categorised by diacritic as well (e.g. Å). However, the diacritic categories only appear if the diacritic isn't otherwise used in the language: so French Ÿ gets the category Category:French terms spelled with Ÿ, but there is no category for French terms spelled with ◌̈, because ◌̈ is a standard diacritic of French. Theknightwho (talk) 22:30, 14 May 2023 (UTC)

Judging by iota_subscript#Computer_encoding, the matter of uppercasing iota subscripts is a big mess. Unicode says to change them to adscripts (what I see), but a lot of usage stays with the subscript (what you see). An adscript isn't the same as an independent letter, which is why my typing the Greek ended up with a redlink. I'm guessing the module errors are due to different parts of the system (module code?) disagreeing on what exact Unicode codepoint an uppercased iota subscript is supposed to be: the part that converts the lowercase iota-subscripted letter to uppercase is using a different codepoint than the part that checks for uppercase vs. lowercase. Chuck Entz (talk) 00:01, 15 May 2023 (UTC)

I'm sure you're already done for the day/evening, and I'll be gone most of the day tomorrow, but just to be thorough: the alpha with iota subscript copypasted from the displayed title on the pagen in σοφίᾳ (sophíāi) is the precomposed character U+1FB3 (see ᾳ for the unicode info that displays even without an entry), and the category page has the precomposed character U+1FBC (see ᾼ). The first one is described as "GREEK SMALL LETTER ALPHA WITH YPOGEGRAMMENI", where "ypogegrammeni" is the iota subscript, and the second one is "GREEK CAPITAL LETTER ALPHA WITH PROSGEGRAMMENI", where "prosegegrammeni" is the iota adscript. The "Composition" section, however, has "Composition: α + ◌ͅ " for the first and "Composition: Α + ◌ͅ " for the second. I have no idea how that translates to the codepoints the module sees, but I figure some extra data won't hurt. I would also note that {{uc:ᾳ}} > ΑΙ, which is the U+1FBC codepoint, and {{lc:ᾼ}} > ᾳ, which is the U+1FB3 codepoint. Chuck Entz (talk) 02:31, 15 May 2023 (UTC)

This is getting stale. I just added the precomposed letters to the standardChars of Grek under "el" at Module:languages/data/2, but apparently with no effect (I noticed there were other precomposed letters there already). That's about all I dare to do myself, and even that may need to be reverted if there are unanticipated side-effects. We need to do something, though. Chuck Entz (talk) 01:38, 20 May 2023 (UTC)

@Chuck Entz I've been putting this off, but I'll look at it now. Theknightwho (talk) 15:45, 20 May 2023 (UTC)

@Chuck Entz This is now solved. The issue was that Module:category tree/poscatboiler/data/characters checks if single-character inputs are upper or lowercase, and throws an error if they're lowercase (with one edge-case exception if the uppercase form is actually standard, as sometimes happens with ı and I). It does this by running mw.ustring.upper on the input and then checking it's the same, because usually if you run it on a capital letter then nothing happens.

This caused a problem with ᾼ, because the output from mw.ustring.upper is actually ΑΙ (i.e. it capitalises the subscript iota and splits it out as a separate full character). This isn't an error, as it matches the Unicode specification, but it isn't what we want in this context (because what we really want is title case, but there's no function for that). I've added special-case behaviour for subscript iota so that it remains unchanged.

I'll investigate if there are any other oddities like this: one other I've accounted for so far is ß, which capitalises as SS by default. Instead we use ẞ, which is the correct form for contexts like this. Theknightwho (talk) 14:07, 21 May 2023 (UTC)

Please take a look. There are a ton of categories here related to 'LANG terms spelled with X' that shouldn't or arguably shouldn't be there. Most blatantly, Cyrillic is entirely normal for Mongolian. I also don't think digits in terms is remarkable for English. Please review what's there and make appropriate fixes to remove these categories from being created, thanks! Benwing2 (talk) 09:04, 16 May 2023 (UTC)

@Benwing2 Macedonian's been dealt with (which was an error on my part). Regarding numerical entries, they're not that remarkable in English, but if we take the most frequent number (1) and assume everything in Category:English terms spelled with 1‏ is a lemma (it's not), that only comprises 0.03% of English lemmas. I'd say that's rare enough to be notable. That being said, we may want to consolidate the categories for numerals.

Other than those, I'd say the categories are all fine: Tagalog, Danish and Norwegian don't use "C" in native words, for example. I think there needs to be a special exclusion for entries with "..." from being categorised as having ".", so I will deal with that. Theknightwho (talk) 09:58, 16 May 2023 (UTC)

I think at most we want one 'terms spelled with numbers' category, although I still believe it's better not to have these; nothing terribly remarkable about a term with a number in it (probably English terms with Q and Z are fairly rare too but not remarkable). Do you mean both Mongolian and Macedonian? Benwing2 (talk) 10:27, 16 May 2023 (UTC)

Also, all the "terms spelled with ." should go; these are just abbreviations. Benwing2 (talk) 10:28, 16 May 2023 (UTC)

Also IMO Category:English terms spelled with &. Benwing2 (talk) 10:30, 16 May 2023 (UTC)

@Benwing2 The only Mongolian categories listed relate to the traditional script, and they're filtering out. We do have a few categories for Mongolian Cyrillic letters, but only for the rare ones used in borrowings: see Category:Mongolian terms by their individual characters.

My initial inclination was to exclude numbers and characters like "." as well, but people were adding them manually anyway. I judged it was better to just handle it automatically, as if we're going to have them we may as well do it properly, and I don't want want to waste time periodically deleting them. Ultimately, they're pretty harmless. Theknightwho (talk) 10:42, 16 May 2023 (UTC)

Not sure I agree on Category:English terms spelled with & - we don't usually have entries with it unless it's either obligatory or overwhelmingly common: R & B, &c., R&R and so on. Theknightwho (talk) 10:47, 16 May 2023 (UTC)

The problem is that every one of these extra categories adds spam to the category list at the bottom of the page. On a page with a period or number in it, there may be several such categories if we allow them. So they're not totally harmless. I have already done runs in the past removing the manually added number categories and can do it again. Benwing2 (talk) 18:20, 16 May 2023 (UTC)

Hi. The punctuation marks are deleted on links but they don't transliterate then. E.g. at how much does it cost#Translations the Chinese/Wu 多少鈔票 / 多少钞票 (¹tu-sau ⁵tshau-phiau) works but 多少鈔票？ / 多少钞票？ doesn't. Could you please have a look? Anatoli T. ^{(обсудить}/^вклад) 23:53, 17 May 2023 (UTC)

@Atitarev I'll take a look, yes. Theknightwho (talk) 16:27, 29 May 2023 (UTC)

See diff. Chuck Entz (talk) 19:19, 20 May 2023 (UTC)

Fixed. Theknightwho (talk) 19:59, 20 May 2023 (UTC)

Hi, could you add documentation on the changes you've made to Module:languages? They're extensive and not well-documented and I'm finding it hard to use. E.g. there used to be a function to return the type of a given language object, which seems to have been replaced with hasType(), but the usage isn't well documented and the interface itself may have issues (e.g. is there a simple way of telling whether a language object describes a regular language)? BTW something needs to be done about languages like Rudbari, which are both a regular language and an etymology language; this causes all sorts of problems. Benwing2 (talk) 05:30, 29 May 2023 (UTC)

@Benwing2 Yes, you're right that I need to do this. You can check if a language is regular with lang:hasType("regular"). In this case, I think the documentation points to Module:languages/data/2, which lists the three types, but it would be better to explain it more fully, as the regular type is only assigned if the language isn't one of the other types. Theknightwho (talk) 16:20, 29 May 2023 (UTC)

You're right about the languages which are both etym-only and regular - I don't really know what to do with them, as they're all languages I don't have any experience editing. Theknightwho (talk) 16:21, 29 May 2023 (UTC)

By "regular" does this exclude reconstructed and appendix-reconstructed languages? If so, this may be a mistaken choice. Traditionally "regular" languages are all languages that aren't etym-only. I would keep "regular" in this meaning and use "mainspace" to indicate languages that go in the mainspace (excluding those that go in Reconstruction: or the appendix). Benwing2 (talk) 22:43, 31 May 2023 (UTC)

I think these terms should never have the prefix reduced, with either a secondary stress or an unreduced marker, so the respelling "трехстру́нный" at трёхстру́нный (trjoxstrúnnyj) is quite rare and even unnatural, IMO. Anatoli T. ^{(обсудить}/^вклад) 01:27, 30 May 2023 (UTC)

@Atitarev What is this in reference to? The "fast/casual" pronunciation at трёхстру́нный (trjoxstrúnnyj) has been there a long time. Did User:Theknightwho recently push their translation-scraping code to production? If so and if the scraping is looking at pronunciations, maybe there needs to be a way of indicating a pronunciation should not be scraped (e.g. a boolean flag to {{ru-IPA}}). Benwing2 (talk) 22:47, 31 May 2023 (UTC)

@Benwing2: The fast/casual pronunciation was added in diff. I let it slide, although I don't fully agree. I thought for completeness, it's OK, then. I don't see why all possible incorrect rare pronunciations should be added, though, especially unsourced, so I might remove that.

I messaged Theknightwho, since he added a few words prefixed with трёх- (trjox-), all of them with a reduced pronunciation, he probably referred to that edit by KoreanQuoter as a model. In my opinion, the most accurate and common pronunciation of the prefix is or , respelled "трё̂х-" or "трё̀х" and (reduced) should go. Anatoli T. ^{(обсудить}/^вклад) 02:17, 2 June 2023 (UTC)

@Benwing2 @Atitarev Transliteration scraping isn’t live yet. Atitarev is right about what happened re the pronunciations, and of course him being a native speaker means I’ll defer to his expertise. Theknightwho (talk) 02:32, 2 June 2023 (UTC)

Hi, Theknightwho. I have a question and I've discovered I don't know where I can find out any information.

I've looked at my "User contributions" and discovered that there is an "edit count". When I looked at my edit count I discovered that there were 50 "Deleted edits". Wiktionary allows me to see what contributions I've made but I do not know how to access information on which edits have been deleted. This concerns me, because I don't know what it is referring to. Is it edits I've deleted myself? Edits that someone has been going through and deleting? How can I locate these "deleted edits"? Can you tell me anything about this? Any help would be greatly appreciated.

Bathrobe (talk) 20:26, 30 May 2023 (UTC)

@Bathrobe They're edits on pages which have been deleted. That may be because you created them, but it may also be if they're on someone's userpage which got deleted or something. Without knowing any of the details, it's not possible for me to say, but it's almost certainly nothing to worry about. Theknightwho (talk) 20:31, 30 May 2023 (UTC)

I had a look (as it seems admins can view a user's deleted contributions), and it was mostly pages like white/translations which have since been folded back into the main page. Theknightwho (talk) 20:43, 30 May 2023 (UTC)

Ok, thanks! As you may know, I'm not on very good terms with one of the editors, one of the grand old men of Wiktionary, an admin, and an extremely prolific contributor from way back, who added some poor translations in languages he knew very little about. I was concerned that such an editor might have been simply deleting my contributions. I'm relieved that it's nothing like this. Bathrobe (talk) 21:54, 30 May 2023 (UTC)

Hi. Go to User:Benwing2/test-ca-noun and click "Show preview" and you'll see a bunch of warnings for headwords that attempted to set displayTitle. This only applies to terms with accented characters in them for some reason. Any idea what's going on? Benwing2 (talk) 22:40, 31 May 2023 (UTC)

@Benwing2 Yes - it's Module:headword trying to set the display title. For some reason, it only does that if there are non-ASCII characters in the string, which seems a bit hacky to me. Theknightwho (talk) 22:53, 31 May 2023 (UTC)

Hmm. I see the code in question and I didn't add it. Where did it come from? Benwing2 (talk) 22:55, 31 May 2023 (UTC)

@Benwing2 Which bit do you mean? The module's been handling the display title since before I could edit the module, from the look of it.

I was considering moving it into Module:headword/data so that it only gets calculated once per page, but that ran into issues with determining the best script without a language, so that project is stalled for now. Theknightwho (talk) 22:58, 31 May 2023 (UTC)

The garbagey code in lines 732-770 of Module:headword. When I rewrote the module I added a FIXME and isolated the code in one place, but it may need a rewrite. The check for ASCII seems very hacky to me as well as some of the other stuff being done in that code. Benwing2 (talk) 23:04, 31 May 2023 (UTC)

@Benwing2 Yes, I agree. I did recently add a bit for unsupported titles, but it's a bit pointless unless the Javascript gadget can access it in some way. I remember doing the bit to keep the Han script region-neutral, too, but that was a while back. Theknightwho (talk) 23:07, 31 May 2023 (UTC)

Might I suggest that you check for lower(upper(x))=x before throwing an error? In template syntax that would be: {{lc:{{uc:ſ}}}}=ſ producing s=ſ. If false, don't throw the error. Chuck Entz (talk) 23:57, 2 June 2023 (UTC)

Can we do something to fix these? They've been in CAT:E literally for weeks, since you changed the category modules. They shouldn't be throwing errors at all, since the characters only exist in lowercase. Chuck Entz (talk) 13:27, 6 June 2023 (UTC)

Could 'digraph' or 'multigraph' be added as an option for Module:headword/data? ch for example might count as a 'letter' in Spanish orthography (at least it was considered a letter for a while, maybe not anymore), but it isn't one in English. Which means that technically we can have a Spanish entry for orthographic 'ch' but not an English one. Or is 'letter' already understood to include multigraphs?

This came up because of the (IMO) odd obsolete English digraph ée, which might deserve an entry, but I see that we don't have entries for ee or ea. kwami (talk) 06:59, 5 June 2023 (UTC)

It feels like every month somebody has to publicly and explicitly denounce you as a tyrant and the scourge of Wiktionary. I don’t get it. —(((Romanophile))) ♞ (contributions) 16:10, 8 June 2023 (UTC)

@Romanophile I think it's a manipulation tactic: accusing someone of abusing their power makes it much more difficult to hold you accountable, because it "proves" you right. Once one person does it, it doesn't take much for other people who don't like being held accountable to do the same. Theknightwho (talk) 16:16, 8 June 2023 (UTC)

Hi. Before things go on too much farther can we please use the type 'regular' to mean any non-etymology language and 'mainspace' to refer to what you have redefined 'regular' as? Formerly regular languages were any non-etymology languages but you redefined it to refer only to mainspace languages. There's not even any simple way any more of checking for regular languages. I can make this change myself but I'm not sure what code depends on the current definition of 'regular'. Thanks. Benwing2 (talk) 21:55, 10 June 2023 (UTC)

@Benwing2 I think the definition pre-dates any changes I made: compare line 58 of this version of Module:languages with the for-loop at the bottom of Module:languages/data/2. You can see it in use in line 41 of this version of Module:translations, for example, which is still there at line 32.

I'm in two minds as to whether we should change it to your suggestion: on the one hand, it makes explicit the distinction between full languages and etymology-only ones (i.e. variants), but on the other hand the current definition is a useful catch-all for any language with no special characterstics, and you can simply check for non-etym languages by doing if not lang:hasType("etymology-only") then .... Theknightwho (talk) 22:17, 10 June 2023 (UTC)

So you can't actually just check the way you mention; you need to also check for !family when calling getNonEtymological, which is why I make this suggestion. It is annoying and non-obvious to have to write two checks of the form "not A and not B" to get regular (full) languages, and if we introduce a new type of language-like entity in the future, it will break all such checks. Benwing2 (talk) 23:46, 10 June 2023 (UTC)

I see, yes I didn't realize the 'regular' language type in this defn goes back a way. If you'd rather we can call non-etymological languages "full" languages but IMO we do need a type for this. Benwing2 (talk) 23:49, 10 June 2023 (UTC)

@Benwing2 That's a good point, and I agree "full" is the best name here. Because types specified in the data are inheritable (i.e. etym-only children get all the types of their parents), the full and etymology-only types have had to be hard-coded into Module:languages instead, and get generated the first time :hasType gets called based on whether self:getNonEtymologicalCode() == self:getCode().

For the sake of completeness I've also implemented this for families, so if you only want languages you can do if lang:hasType("language", "full") then ....

I guess we could also modify the method so that you can check not-X as part of a multi-type check (e.g. if lang:hasType("language", "!etymology-only") then ...), but I'm not sure how useful this would be in practice. Theknightwho (talk) 14:11, 11 June 2023 (UTC)

Cool, thank you. Benwing2 (talk) 16:30, 11 June 2023 (UTC)

Hi. Many Spanish terms have origins in Lunfardo, which is a Buenos Aires argot derived by mixing Italian dialect words into Spanish grammar. It is given an etymological code es-lun, but that prevents deriving Spanish words from Lunfardo; as a result many existing etymologies wrongly use {{cog|Lunfardo|...}} in place of {{bor|es|es-lun|...}} or whatever. What is the correct solution here? Benwing2 (talk) 06:49, 14 June 2023 (UTC)

@Benwing2 Should we give it a new creole language code? Theknightwho (talk) 15:28, 14 June 2023 (UTC)

I don't think it's a creole, as creoles have their own grammar. Here the grammar is still Spanish. Wikipedia defines it as an "argot", which is its own class of thing. Benwing2 (talk) 15:34, 14 June 2023 (UTC)

Maybe it's similar to if Classical Latin borrowed terms from Vulgar Latin, which is entirely possible; it suggests we need a type of etymology language that runs concurrent with the non-etymology parent, so borrowing can go in both directions. Benwing2 (talk) 15:49, 14 June 2023 (UTC)

@Benwing2 Isn’t it only inheritance where there’s a restriction? Borrowing can go in both directions anyway, can’t it? Maybe I’m misunderstanding the issue. Theknightwho (talk) 17:25, 14 June 2023 (UTC)

Hmm, you are right. Benwing2 (talk) 17:30, 14 June 2023 (UTC)

I can see that you've been very busy with fixing sortkeys, but you really need to deal with the 500 (and counting) entries in CAT:E as a result of your adding error() to the scrape_page function in Module:Jpan-sortkey. The number has increased by almost a hundred in the time it took me to compose this message. Chuck Entz (talk) 19:24, 17 June 2023 (UTC)

I'm trying to clear the "...terms spelled with...read as..." cats. If I add "kun" in the first parameter of {{auto cat}} as the error message suggests, it displays "terms spelled with " ++" with its kun reading of", where should be the kanji in the name of the category, but is instead "Template:-l". I'm guessing the module is assuming that all of these language codes have counterparts to Japanese and its {{ja-l}}, since the display is what one would expect from putting a parameter in a non-existent template. Chuck Entz (talk) 21:08, 17 June 2023 (UTC)

@Chuck Entz Thanks. That bit of the function is little-used, and only there as a last resort. Thankfully that meant it was ~500 entries, not 100k.

The link template issue was very straightforward to solve by simply slotting in {{l}} instead, which means it'll work regardless of the language. They're just links to kanji with no transliterations, so the output is exactly the same as it would be. Theknightwho (talk) 21:20, 17 June 2023 (UTC)

Hi. Memory errors seem to be increasing again. Category:Proto-Niger-Congo language is now throwing a memory error. Can you let me know what you've been doing lately and if it is increasing memory load (and if so what we're gonna do about it)? Benwing2 (talk) 02:58, 19 June 2023 (UTC)

@Benwing2 I've not done anything major for a while, but Category:Proto-Niger-Congo language and Category:Proto-Atlantic-Congo language occasionally seem to throw errors for no reason at all. Scribunto memory usage varies each time the page is refreshed, so occasionally a page will start throwing memory errors for no reason. I suspect it's exacerbated by the fact it's almost all down to the family tree, as opposed to most pages close to the limit where it's hundreds of separate template calls. Atlantic-Congo is the largest language family we have, so it's not massively surprising. Theknightwho (talk) 13:31, 19 June 2023 (UTC)

OK, thanks. Benwing2 (talk) 18:52, 19 June 2023 (UTC)

I think you asked me awhile ago to obsolete {{zh-der}}? I can do that but I forget exactly what needs to be done. Do you know where the discussion was? Benwing2 (talk) 03:00, 19 June 2023 (UTC)

@Benwing2: It's at Wiktionary:Grease pit/2023/March#Deprecating Template:zh-der. I believe the following needs to be done:

• Replacing {{zh-der}} with {{col3|zh}} suffices for most pages
• If a derived term contains foo/bar (which is used to manually specify simplified forms), it should be replaced by foo//bar
• If a derived term contains foo:bar, this should be converted to either foo<tr:bar> or foo<t:bar>. The distinction between tr and t is determined via the regexes at Module:zh/link#L-63.
• If a derived term contains foo;bar, this should be converted to foo<q:bar>; when bar is the name of a lect, it needs to be linked with the langcode prefix, e.g. nan:foo or foo<q:Min Nan> for foo;Min Nan. (I'm not sure what the practice should be for this one, TKW has been doing the former and me the latter)
• Note that the ; and : syntaxes can (and usually) occur simultaneously. The two are not that common (I estimated that it's probably around 1400 in total, out of the 38k invocations of {{zh-der}}), so I think they could be done manually since they require extra care in most cases, and also because there seems to be sometimes confusion between the two syntaxes.
• If {{zh-der}} has |hide_pron=1, or if {{zh-der/fast}} is used, these need to be dealt with manually.
• |fold= can be ignored in any case; |title= can be copied wholesale into {{col3}}

– Wpi (talk) 19:34, 19 June 2023 (UTC)

@Wpi Thank you. I think the langcode prefix form should be used if possible in place of using a qualifier, because it supplies the info in a machine-readable format, allowing for lang-specific linking and formatting as needed. Benwing2 (talk) 20:38, 19 June 2023 (UTC)

@Wpi A few questions:

1. The code in Module:zh/link is a bit confusing but it appears to do special things with terms that have a circumflex (^) and/or an asterisk (*) in them. The circumflex seems to indicate that the translit should be capitalized, but there do not seem to be any occurrences of it in {{zh-der}} calls. The asterisk is less clear to me. There are 50 {{zh-der}} calls with an asterisk somewhere in them, any idea what to do with them?
2. For reference there are 87 {{zh-der}} calls that use hide_pron (out of 26,873 total; this does not include any aliases of {{zh-der}}, namely {{zh-list}}, {{zh-syn-list}} and {{zh-ant-list}}; I still need to investigate them), and 390 that include a semicolon anywhere in the {{zh-der}} call.
3. For lect names after semicolons, if you prefix the term with the corresponding langcode, it doesn't display the lect name. I'm thinking of adding some syntax to {{col}} for use in conjunction with langcode prefixes to display the corresponding language name as a qualifier; what do you think of this? If you think it's a good idea, any suggestions for the syntax?
4. The current code uses {{col3}} unless the pagetitle has more than one character, in which case it uses {{col2}}. Should we duplicate this functionality or always use {{col3}}?
5. I am planning on checking to see in case of TRAD/SIMP whether the SIMP is redundant, and removing it if so. Sound good?
6. When a note following a semicolon is not a lect name, should it use <q:...> (display to the left) or <qq:...> (display to the right)?
7. There are occurrences of items like 東城:] and 宜君:] in the {{zh-der}} calls. Are these supposed to be (badly specified) translits? They end up as glosses in the converted output. What should be done about them?
8. The following item 拍損:phah-sńg;Min Nan ends up as nan:拍損<t:phah-sńg> in the converted output. I assume this should be a translit and the regexps need to be tweaked to recognized things like ń?

Benwing2 (talk) 23:24, 21 June 2023 (UTC)

@Benwing2:

1. * suppresses simplification and pinyin (Sorry I forgot this one). For the warnings you got below, 乾 is usually simplified to 干 but for that particular etymology the simplified form is 乾 itself. Sometimes people use it simply to disable pinyin, which is a valid but not ideal use case; I think these will need to be done manually as well.

3. Maybe?

4. I think we discussed this before(?), and {{col3}} is still very readable on multiword items.

5. That makes sense, especially since on some pages the simplified form is specified to reduce memory usage, but that doesn't really help in {{col3}} according to TKW.

6. No strong preference, but putting the qualifier after the term aligns the characters.

7. These are place names, which are supposed to be glosses.

8. Correct. The regexes in Module:zh/link are probably written quite some time in the past and doesn't really do the job properly.

@justinrleung might have some insights on some of the issues. – Wpi (talk) 05:48, 22 June 2023 (UTC)

@Wpi: Thanks for the detailed replies. As for #1, do you mean all occurrences of asterisks should be handled manually? This seems doable as there are only around 50 template calls using them. For #4, I'll use {{col3}} everywhere. For #6 I'll put the qualifier after the chars. For #8 I'll fix up the regex for ń and maybe for other tone marks on n (can they also occur on m?). Benwing2 (talk) 07:15, 22 June 2023 (UTC)

@Wpi: One more question: what about cases like 潑雨;Hakka, Min Nan where two lects are listed after the semicolon? Currently it's unrecognized as a lect and ends up as a qq qualifier. Benwing2 (talk) 07:58, 22 June 2023 (UTC)

@Benwing2:

For #1, that should be doable. The correct procedure would be to convert *term to term// (i.e. an empty second form to supress simplified forms), but I don't think it's worth the hassle to bot the 50 or so instances.

For #8, yes. Other characters to look out for are: , combined with tone marks U+0300, U+0301, U+0302, U+0304, U+0306, U+030C, U+033F which occur in Kienning Romanised for Min Bei; , combined with U+0300, U+0301, U+0302, U+0304, U+0306, U+030D in Min Nan; combined with U+0300, U+0301, U+0302, U+030D in Hakka; and of course the combining forms. It might also be useful to modify the second regex to (+ )*+ to try and find the transliterations unaccounted for.

For the final question, I suppose we could list them twice, once for each lect, but it feels weird given we don't list the duplicate the ones with Mandarin in it. For now I reckon let's just leave them as is, i.e. <qq:Hakka, Min Nan>

– Wpi (talk) 09:34, 22 June 2023 (UTC)

@Wpi: What I ended up doing for the regexes is just decompose everything and look for any of the combining diacritics 0300 (GRAVE), 0301 (ACUTE), 0302 (CFLEX), 0304 (MACRON), 0306 (BREVE), 0307 (DOTOVER), 0308 (DIAER), 030B (DOUBLEACUTE), 030C (CARON), 030D (VERTLINEABOVE), 030F (DOUBLEGRAVE), 0323 (DOTUNDER), 0324 (DIAERUNDER), 033F (DOUBLEMACRON), 0358 (DOTABOVERIGHT) as well as superscript ⁿ. Benwing2 (talk) 10:27, 22 June 2023 (UTC)

OK, I have written the script. Some sample output:

-{{zh-der|fold=1|*乾乾|*乾元|乾圖/乾图|*乾坤|*乾宅|乾斷/乾断|*乾旦|*乾曜|乾清宮/乾清宫|乾紅/乾红|乾綱/乾纲|*乾象|*乾造|*乾道|*乾陵|*乾隆|*朝乾夕
惕|顛乾倒坤/颠乾倒坤|乾華/乾华}}
+{{col3|zh|*乾乾|*乾元|乾圖//乾图|*乾坤|*乾宅|乾斷//乾断|*乾旦|*乾曜|乾清宮//乾清宫|乾紅//乾红|乾綱//乾纲|*乾象|*乾造|*乾道|*乾陵|*乾隆|*朝乾夕惕|顛乾倒坤//颠乾倒坤|乾華//乾华}}

This output a bunch of warnings:

Page 137 乾: WARNING: Ignoring fold=1: {{zh-der|fold=1|*乾乾|*乾元|乾圖/乾图|*乾坤|*乾宅|乾斷/乾断|*乾旦|*乾曜|乾清宮/乾清宫|乾紅/乾红|乾綱/乾纲|*乾象|*乾造|*乾道|*乾陵|*乾隆|*朝乾夕惕|顛乾倒坤/颠乾倒坤|乾華/乾华}}
Page 137 乾: WARNING: For traditional 乾圖, explicit simplified 乾图 doesn't match auto-conversion 干图 in 3=乾圖/乾图, specifying explicitly: {{zh-der|fold=1|*乾乾|*乾元|乾圖/乾图|*乾坤|*乾宅|乾斷/乾断|*乾旦|*乾曜|乾清宮/乾清宫|乾紅/乾红|乾綱/乾纲|*乾象|*乾造|*乾道|*乾陵|*乾隆|*朝乾夕惕|
顛乾倒坤/颠乾倒坤|乾華/乾华}}
Page 137 乾: WARNING: For traditional 乾斷, explicit simplified 乾断 doesn't match auto-conversion 干断 in 6=乾斷/乾断, specifying explicitly: {{zh-der|fold=1|*乾乾|*乾元|乾圖/乾图|*乾坤|*乾宅|乾斷/乾断|*乾旦|*乾曜|乾清宮/乾清宫|乾紅/乾红|乾綱/乾纲|*乾象|*乾造|*乾道|*乾陵|*乾隆|*朝乾夕惕|
顛乾倒坤/颠乾倒坤|乾華/乾华}}
Page 137 乾: WARNING: For traditional 乾清宮, explicit simplified 乾清宫 doesn't match auto-conversion 干清宫 in 9=乾清宮/乾清宫, specifying explicitly: {{zh-der|fold=1|*乾乾|*乾元|乾圖/乾图|*乾坤|*乾宅|乾斷/乾断|*乾旦|*乾曜|乾清宮/乾清宫|乾紅/乾红|乾綱/乾纲|*乾象|*乾造|*乾道|*乾陵|*乾隆|*朝乾夕惕|顛乾倒坤/颠乾倒坤|乾華/乾华}}
Page 137 乾: WARNING: For traditional 乾紅, explicit simplified 乾红 doesn't match auto-conversion 干红 in 10=乾紅/乾红, specifying explicitly: {{zh-der|fold=1|*乾乾|*乾元|乾圖/乾图|*乾坤|*乾宅|乾斷/乾断|*乾旦|*乾曜|乾清宮/乾清宫|乾紅/乾红|乾綱/乾纲|*乾象|*乾造|*乾道|*乾陵|*乾隆|*朝乾夕惕|顛乾倒坤/颠乾倒坤|乾華/乾华}}
Page 137 乾: WARNING: For traditional 乾綱, explicit simplified 乾纲 doesn't match auto-conversion 干纲 in 11=乾綱/乾纲, specifying explicitly: {{zh-der|fold=1|*乾乾|*乾元|乾圖/乾图|*乾坤|*乾宅|乾斷/乾断|*乾旦|*乾曜|乾清宮/乾清宫|乾紅/乾红|乾綱/乾纲|*乾象|*乾造|*乾道|*乾陵|*乾隆|*朝乾夕惕|顛乾倒坤/颠乾倒坤|乾華/乾华}}
Page 137 乾: WARNING: For traditional 顛乾倒坤, explicit simplified 颠乾倒坤 doesn't match auto-conversion 颠干倒坤 in 18=顛乾倒坤/颠乾倒坤, specifying explicitly: {{zh-der|fold=1|*乾乾|*乾元|乾圖/乾图|*乾坤|*乾宅|乾斷/乾断|*乾旦|*乾曜|乾清宮/乾清宫|乾紅/乾红|乾綱/乾纲|*乾象|*乾造|*乾道|*
乾陵|*乾隆|*朝乾夕惕|顛乾倒坤/颠乾倒坤|乾華/乾华}}
Page 137 乾: WARNING: For traditional 乾華, explicit simplified 乾华 doesn't match auto-conversion 干华 in 19=乾華/乾华, specifying explicitly: {{zh-der|fold=1|*乾乾|*乾元|乾圖/乾图|*乾坤|*乾宅|乾斷/乾断|*乾旦|*乾曜|乾清宮/乾清宫|乾紅/乾红|乾綱/乾纲|*乾象|*乾造|*乾道|*乾陵|*乾隆|*朝乾夕惕|顛乾倒坤/颠乾倒坤|乾華/乾华}}

I'm not sure what's going on with the non-matching simplified stuff, let me know if it makes sense to you. Note also the asterisks in the output, which are getting carried over literally; I don't think this is correct. Benwing2 (talk) 04:55, 22 June 2023 (UTC)

@Wpi, Theknightwho I am going to run the script on {{zh-der}} and {{zh-list}}. I am planning on implementing syntax in {{colN}} whereby writing e.g. nan-hok::冷吱吱 with two colons causes the language name to be displayed as a right qualifier, equivalent to nan-hok:冷吱吱<qq:Hokkien>. Thoughts? Benwing2 (talk) 19:43, 22 June 2023 (UTC)

@Benwing2 Thanks - that sounds good. In terms of the double colon, I'd prefer if the language name always showed as a qualifier if there's a language override in a column template, as otherwise there's nothing visible to indicate it. Maybe we should exclude mul from that, though. Theknightwho (talk) 19:46, 22 June 2023 (UTC)

Yup, that makes sense. I was thinking originally of doing exactly what you proposed but then realized it would be problematic with mul links; but excluding them from the qualifiers should work fine. Benwing2 (talk) 19:49, 22 June 2023 (UTC)

@Wpi, Justinrleung These two templates are redirects to {{zh-der}} but the code that implements {{zh-syn-saurus}} looks specifically for {{zh-syn-list}} (and likewise {{zh-ant-saurus}} looks specifically for {{zh-ant-list}}). I am thinking they can be replaced with {{col3}} where the title can be an abbreviation for Synonyms or Antonyms, expanded appropriately (I've been meaning to implement such abbreviations for awhile now, for various reasons), and {{zh-syn-saurus}} can look for a {{col3|zh}} whose title is "Synonyms" or the appropriate abbreviation. More specifically, instead of writing {{col3|zh|title=Synonyms}} you could write something like {{col3|zh|ti=@syn}} or maybe just {{col3|zh|ti=@s}}, where |ti= is a shorter alias for |title= and things beginning with at-signs are abbreviations. Similarly, @der or @d means "Derived terms", @rel or @r means "Related terms", etc. How does this sound? An alternative is to make the first numbered term after the language be the title with abbreviations if preceded by the appropriate char, e.g. {{col3|zh|@syn|*乾乾|*乾元|乾圖//乾图|*乾坤|*乾宅|乾斷//乾断|*乾旦|*乾曜|乾清宮//乾清宫|乾紅//乾红|乾綱//乾纲|*乾象|*乾造|*乾道|*乾陵|*乾隆|*朝乾夕惕|顛乾倒坤//颠乾倒坤|乾華//乾华}} for synonyms. Benwing2 (talk) 07:26, 22 June 2023 (UTC)

@Theknightwho, Wpi What do you think of my proposal here? We should generalize {{zh-syn-saurus}} to something like {{syn-saurus|zh}}; all it does it look up synonyms or antonyms in the specified thesaurus page and display them. BTW I am also thinking of changing the bare {{col}} template so the number of columns is specified using a named param |n= instead of a numbered param; that way, if |n= is omitted, you get some sort of auto behavior, similar to {{col-auto}} but with fewer chars. Benwing2 (talk) 19:53, 22 June 2023 (UTC)

@Wpi Conversion of {{zh-der}} and {{zh-list}} is in progress; conversion of the syn/ant stuff will come next. See User:Benwing2/zh-der-misc-warnings for warnings related to {{zh-der}} conversion other than those related to |hide_pron= (80 in total) and User:Benwing2/zh-der-hide-pron-warnings for automatic warnings triggered whenever |hide_pron= is seen (87 in total). Hopefully this isn't too many to process by hand but let me know if there's a way of automatically dealing with the |hide_pron= stuff. Benwing2 (talk) 03:55, 23 June 2023 (UTC)

@Benwing2: For {{zh-syn-saurus}}, I think that makes sense if we want to extend the format to other languages.

For the warnings some of them could be automated: more-than-one-slash can be safely converted from a/b/c to a//b//c, more-than-one-colon is just coexistence of both t and tr. The {{{hide_pron}}} stuff are probably automatable, if they are just plain list of terms with no special formatting stuff, but I figure it's not worth the hassle to write the checks for the bot, plus they are usually used on pages with memory errors so it might trigger that. – Wpi (talk) 05:46, 23 June 2023 (UTC)

@Wpi: Thanks. See User:Benwing2/zh-syn-list-zh-ant-list-misc-warnings; I converted all the 2000 or so Thesaurus pages using {{zh-syn-list}} and {{zh-ant-list}}, producing 49 more warnings, most about asterisks. You might want to get to these first because I am also doing a run to convert {{zh-syn-saurus}} and {{zh-ant-saurus}} to {{syn-saurus}} and {{ant-saurus}}, and for such pages that are converted, there will be a red warning message displayed in place of the synonyms/antonyms if it finds the old {{zh-syn-list}}/{{zh-ant-list}} templates instead of {{col3|zh}}. Benwing2 (talk) 06:20, 23 June 2023 (UTC)

@Wpi: BTW the number of cases with more than one slash or colon are small, so it might be easier to handle them manually; let me know if you want me to write the bot code to handle them automatically. Benwing2 (talk) 06:22, 23 June 2023 (UTC)

@Benwing2: Thanks. I'll do those manually, probably today or this weekend. – Wpi (talk) 07:39, 23 June 2023 (UTC)

@Wpi: Great, thank you! Benwing2 (talk) 07:52, 23 June 2023 (UTC)

@Benwing2, theknightwho

So here's a progress update, I've went through all the warnings for the thesaurus pages (as well as a bunch that the bot missed like an asterisk at the end of a term) and cleaned them all up.

During this I find that:

1. Some of the lect qualifiers were not recognized by the bot (or sometimes missed by myself), this seems to be mostly the etym codes under Hokkien, and Sichuanese (the etym code is new so we've decided on calling it Sichuanese, though it used to be split between Sichuanese and Sichuan, so <qq:Sichuan> is the same as <qq:Sichuanese> and should be converted to cmn-sic:)

   @TKW It would be helpful to have the Hokkien etym codes autogenerate pronunciation for the subdialect.

   Speaking of new etym codes, there's a number of qualifiers that would make sense for a new etym code, e.g. Taiwanese Hokkien, Hong Kong Cantonese, Beijing. I'll likely propose them on BP shortly, and after that there's probably a need to rerun the bot conversion for the qualifiers again.

2. The bot did not convert qualifiers like <qq:Cantonese, slang> or <qq:Wu, dated> to the ones with prefixes. I've changed them on sight, but there's a lot of them. We might want to use a bot to clean these up. Note that literary refers to the entire Chinese language and not the lect, so <qq:literary, Cantonese is not yue:<qq:literary>. EDIT: I've cleaned up all the ones I could find.

   There's also weird qualifiers like <qq:chiefly Cantonese>, which I think would benefit from some syntax that allows writing custom qualifiers like {{place}}, e.g. <qq:chiefly <<yue>>>. This might also solve the problem with multiple lects, so <qq:Hakka, Min Nan> could be <qq:<<hak>>, <<nan>>> (not sure how the transliteration would be handled?). However, I feel like this might be adding too much bloat to the columns module.

3. There's some stuff that should be in <t:gloss> were put into <qq:gloss>, due to how the people who added them used the wrong syntax; there's not much that we could do with it currently.
4. When zhx-min: (a family code) is used as a prefix, there's some unexpected behaviour. There are some words that are shared across the entire Min family, so it makes some sense to use it.

– Wpi (talk) 17:42, 23 June 2023 (UTC)

@Wpi Thanks. I'll take a look. I think the issue with the lect qualifiers missing is that the bot was only looking for the canonical name of etym languages. I'll redo the code to look at all names and see if I can fix the issues. Benwing2 (talk) 19:06, 23 June 2023 (UTC)

@Wpi Can you give me some examples of lect qualifiers that the bot didn't correctly convert? Benwing2 (talk) 22:57, 23 June 2023 (UTC)

@Benwing2: diff, diff, diff. I have most of them fixed already. – Wpi (talk) 06:11, 24 June 2023 (UTC)

@Benwing2: (Sorry for the pings) I think the problem with the lect names is simply because the name is not listed in the module data, not even as an alias. There's not a lot so I think I'll do them by hand. – Wpi (talk) 06:18, 24 June 2023 (UTC)

@Wpi Those pings you gave me are mostly of cases of a lect name + comma + qualifier. If there are a lot of these (too many to do by hand) I can look into writing a script to fix them. Benwing2 (talk) 06:30, 24 June 2023 (UTC)

You've promised several times to fix the documentation for Module:languages. Documentation is hella important, can you please please please actually do this? In this case I want to use lang:transliterate() and it now returns three values (which I've told you before is a bad idea, but that's another discussion). I don't know what those three values are, and it's not documented. Benwing2 (talk) 03:04, 19 June 2023 (UTC)

@Benwing2 Yep, I'll get started on this. Theknightwho (talk) 13:33, 19 June 2023 (UTC)

Thank you! Benwing2 (talk) 18:50, 19 June 2023 (UTC)

Some or all auto-cat categories are in this hidden category. Is this intentional? If so and if it's not just a temporary hack, it needs to be moved out of user space. Benwing2 (talk) 06:09, 19 June 2023 (UTC)

@Benwing2 This is intentional, though the work I was doing has stalled so I'll create a proper maintenance category. They're all using tables of contents that don't have a language code specified, which means they can't take advantage of proper sortkeys very easily. Theknightwho (talk) 13:33, 19 June 2023 (UTC)

Got it, thanks. Benwing2 (talk) 18:51, 19 June 2023 (UTC)

See CAT:E, they may be your doing. Benwing2 (talk) 23:19, 22 June 2023 (UTC)

@Benwing2 Thanks - fixed. Theknightwho (talk) 23:28, 22 June 2023 (UTC)

Hi, what is the purpose of passing in the calling function and template in Module:parameters? It seems like a lot of extra work. I see Module:parameters/data but I'm not quite sure the gist of this. Benwing2 (talk) 02:43, 23 June 2023 (UTC)

@Benwing2 It means the parameter table can be stored in Module:parameters/data, which means it doesn't get duplicated every time the module is called. If the new preparsing method works we can probably do away with this, as the saving is pretty small. Theknightwho (talk) 02:45, 23 June 2023 (UTC)

@Theknightwho Right, I see, and yes I expect the savings would not be much. Benwing2 (talk) 03:16, 23 June 2023 (UTC)

Or when you wake up, check out wolf and subséquent (and probably a few others), which have some errors related to stuff you just implemented. Benwing2 (talk) 06:25, 23 June 2023 (UTC)

@Benwing2 Thanks - I think it’s to do with stuff inside nowiki tags being processed instead of ignored, and possibly something else relating to strip markers in general. Theknightwho (talk) 12:51, 23 June 2023 (UTC)

Your code needs to be more robust in dealing with tags in general: nonvanishing is due to expressions inside <math></math> tags, for instance, and I don't think it's a coincidence that there are html comments in proximity to other false positives. Then there are the timeouts: there has to be some way to limit the time spent by a single transclusion. Wiktionary:Todo/long usage examples goes to a server timeout, so I can't even look at it without disabling my edit-mode-preview setting, but I viewed that page within the last week and remember it having a column for examples, which, if memory serves, are cut off at a certain number of bytes, leaving incomplete expressions at the end. It would be nice to not have whole sections or the whole page broken by individual errors.

Finally, the exclamation mark in the "Invalid syntax detected!" error message is just silly: it already says "Lua error" in screaming red. You might as well add "OHHH NOOO!!!!" or "Dear me, I think I'm going to faint!"... Chuck Entz (talk) 18:11, 23 June 2023 (UTC)

@Chuck Entz So there were some infinite loops going on that were causing problems, because it's (almost) impossible to tell something loaded via frame:preprocess that it shouldn't try to load the cached data, because it's being called while generating the cached data in the first place. I've managed to find an awkward way around it, though. I'll have a look at the others now. Theknightwho (talk) 18:22, 23 June 2023 (UTC)

It might also help to think about why the same thing can work fine when you preview the section, but has a false positive when the whole page is viewed, as with the Synonyms section at volition. By commenting things out there, I found a parameter in a quote template elsewhere in the Adjective section that had mismatched square brackets. I think you need to work on the limits of the parsing: the error message makes it look like the problem is in the first template, while it's actually somewhere else entirely- the error actual makes it harder for the average user to find the error. Chuck Entz (talk) 19:16, 23 June 2023 (UTC)

@Chuck Entz Sorry - did you mean a different entry? volition doesn't have a synonyms section. Theknightwho (talk) 19:24, 23 June 2023 (UTC)

Sorry! It was volitional. Chuck Entz (talk) 19:26, 23 June 2023 (UTC)

@Chuck Entz Got it - thanks. I've put the template parser inside a pcall function, which detects that an error has been thrown without actually throwing it. That way, it can fail a particular template parse without bringing down the whole thing. If the template is actually supposed to throw an error, that'll happen once the "real" template call is done later on. Theknightwho (talk) 19:38, 23 June 2023 (UTC)

@Chuck Entz I’ve been doing some work on a more general-purpose parser, which is based on a Python wikitext parser that’s commonly used by bots (mwparserfromhell). Parsing gets really complicated when you get into nested templates - especially when dealing with {{{template arguments}}}, or when {{{{templates are used}} to | generate template names}}. Fortunately, it’s not particularly resource-heavy, and should be able to prevent the issues we’re seeing here. Theknightwho (talk) 16:31, 24 June 2023 (UTC)

Let's disable the new stuff you added until everything is ready so we don't have so many errors. Benwing2 (talk) 23:06, 24 June 2023 (UTC)

@Benwing2 I think it's stabilised at about the position we were at - I'm not seeing any parser errors at the moment, as they're all memory-related.

The full parser is gradually getting there (Module:User:Theknightwho/parser). There's a bug in the tokenizer relating to wikitables that's proving difficult to isolate, but other than that the tokenizer's complete - and it's by far the most complex component. I'm pretty pleased with the performance so far, but coming in under 10 seconds on extremely large pages may prove to be a challenge: tokenising a 100KB page takes about 3 seconds at the moment. Theknightwho (talk) 19:26, 26 June 2023 (UTC)

By errors I mean memory errors. There are 59 entries in CAT:E and most of them are related to memory errors or timeouts. There weren't that many before. Benwing2 (talk) 20:43, 26 June 2023 (UTC)

As a general rule, you shouldn't launch code before it's ready. "It's gradually getting there" isn't a good solution for memory errors IMO. Benwing2 (talk) 20:44, 26 June 2023 (UTC)

BTW maybe there's a way to avoid parsing all parts of a complex page, e.g. skip some of the tables or other complexities and let the built-in PHP parser worry about that. Benwing2 (talk) 20:46, 26 June 2023 (UTC)

@Benwing2 The general-purpose parser is a completely different module, and isn't deployed yet. Aren't most of the new memory errors down to the recent changes to the Chinese templates? Theknightwho (talk) 20:46, 26 June 2023 (UTC)

The changes you made to Module:headword/data or similar. These are live and AFAIK added several non-Chinese entries to CAT:E. Benwing2 (talk) 21:33, 26 June 2023 (UTC)

@Benwing2 Just as an update on this, the general template parser is now in an alpha-ish state: at User:Theknightwho/sandbox10 it's able to render a copy of the page teacher (saved at User:Theknightwho/sandbox9) in a single module call, reading directly from the raw wikitext. I've had to comment-out a couple of templates that nest piped wikilinks in templates (as I forgot to account for them), but arbitrary template/argument nesting, noinclude/includeonly tags, most parser functions and most other things it needs are now working (e.g. T:langname-lite, T:quote-meta and T:q are no bother).

In terms of performance, it's adding about 0.2 seconds to the rendering time (about 2 seconds instead of 1.8), while reducing the memory consumption from 49MB to 31MB. I still need to add a lot of memoization, so I'm sure both of those will come down even further. Theknightwho (talk) 18:55, 6 July 2023 (UTC)

This is great, maybe you could write up a bit on how it works and esp. what the general principles are, for someone who isn't so familiar with how MediaWiki works? We'll need that documentation before this goes live so that other people can fix the parser if something goes wrong. Benwing2 (talk) 19:43, 6 July 2023 (UTC)

@Benwing2 Definitely - I want to get it into a semi-reliable state first, in case things need refactoring. I've just solved the issue with piped links, by the way, so the sandbox now works with those templates included as well.

One of the biggest challenges has been minimising the amount of parsing that needs to be done, which means things like {{#if:}} and {{#switch:}} need to be calculated dynamically as they're being parsed, instead of being passed through frame:callParserFunction() (which would necessitate evaluating every parameter in advance). Theknightwho (talk) 23:49, 6 July 2023 (UTC)

@Benwing2 Just FYI, adding the line mw.loadData = require in the parser tester reduced memory usage down to 22.4MB, and loading time is now equivalent with the standard page. Theknightwho (talk) 14:33, 8 July 2023 (UTC)

This is great! You should test on some other high-memory pages to see if the gains are similar across-the-board. Benwing2 (talk) 18:45, 8 July 2023 (UTC)

You should also test it on time hogs like Coptic entries with humongous inflections and frequency list appendices. Chuck Entz (talk) 21:08, 8 July 2023 (UTC)

@Chuck Entz Good call. (Another way to bring the Coptic entries within time and memory limits I think is by rewriting in Lua, but that would take some effort.) Benwing2 (talk) 21:12, 8 July 2023 (UTC)

@Benwing2 @Chuck Entz So it still runs into trouble on huge pages - partly because there are some bits I've not implemented yet - but also because it does seem to run into time-out issues.

I'll see if I can redesign it to convert templates into functions (instead of re-building them from the cached tokens every time), as that has the potential to massively speed things up. It's a bit tricky, since the parameter values affect which bits get parsed by the builder (which is why it's redone each time), and I imagine it will be a little less memory-efficient, but I think that's fine. Theknightwho (talk) 21:16, 8 July 2023 (UTC)

Sounds good. As a last resort we could add something to the page text of specific pages to indicate that the parser shouldn't be used but we should avoid the need for this if possible. Benwing2 (talk) 21:19, 8 July 2023 (UTC)

@Benwing2 Curiously, it's actually faster for ⲱϣ (ōš) (about 8 seconds), which I suspect is because the Coptic templates aren't that complex for the parser, but there are tons of links all being processed by separate invocations.

I'll let you know how I get on with this new idea, as it might take a little while to get right. Theknightwho (talk) 21:28, 8 July 2023 (UTC)

@Benwing2 @Chuck Entz As an example of a huge page, I've copied about 3/4 of a into User:Theknightwho/sandbox11 and converted all the lite templates into regular ones, which is about the point where memory errors start happening under normal circumstances. The parser manages it in about 8.5-9 seconds using 35MB, versus 5 seconds using 52MB (i.e. right on the cusp of memory errors).

If I'm able to get a little more speed out of it, I'd say we might even be able to remove all the lite templates from a, as memory increases seem to scale logarithmically with page length. Theknightwho (talk) 05:41, 9 July 2023 (UTC)

All good, thanks for the update. Benwing2 (talk) 05:46, 9 July 2023 (UTC)

@Benwing2 Hmm - the memory use on that page goes down to 23MB(!) if you make package.loaded a weak table, but I'm unsure if that might have weird side effects. By comparison, the original test page ("teacher") goes down to 13.8MB (a reduction of 72%). Even a page using multitrans like "wolf" sees a 50% reduction (38.5 to 19.5MB). Theknightwho (talk) 07:04, 9 July 2023 (UTC)

Can you check to see if there are weird side effects? Maybe copy the output and see if it differs? Benwing2 (talk) 07:19, 9 July 2023 (UTC)

@Benwing2 Seems to be the same, and the speed doesn't seem to be impacted either way. Theknightwho (talk) 07:29, 9 July 2023 (UTC)

Oh - one other thing I forgot to mention is that the post-expand include size is halved, too, which helps pages like 不 etc. Theknightwho (talk) 07:32, 9 July 2023 (UTC)

That I don't understand. If you make package.loaded a weak table, does this effectively require() each module once per page? I'm not sure what you are referring to by post-expand include size and why it would be halved. Benwing2 (talk) 07:37, 9 July 2023 (UTC)

┌────────────────────────────────────────────────────────────────────────────────────────────────────┘ BTW heading to bed now ... it's 2:39am local time. Please feel free to post further responses and updates, though, and I'll see them in the (late) morning. Benwing2 (talk) 07:39, 9 July 2023 (UTC)

@Benwing2 In theory it means that require has to load things from scratch every time, since no external modules are in use whenever the parser's not actively loading a template. In practice, though, heavily-used modules like MOD:languages will rarely be out of use (meaning that most garbage collect cycles won't affect them), while language-specific modules that are never used again will be getting periodically cleared.

The post-expand include size is sometimes called the template include size, I think - I mentioned it on the Grease Pit a little while ago, as it was affecting some of the very large Chinese entries. Certain text can have a multiplier applied to it (e.g. if it's used in certain parser functions), and I think I've jumped the gun as it looks like simple transclusion doubles the weight, so the reason it's halved is probably because I've been using a bare {{#invoke:}} for testing.

By the way, we're actually dangeorusly close to the include size limit on a (2,086,521 / 2,097,152), since the lite templates have tons of multipliers applied to their outputs due to loads of parser functions and several layers of transclusion. Once it's exceeded, templates just stop loading and show bare links to the template page instead. I know there are a couple of other pages in a similar position, but I can't remember which right now.

Get some rest! Speak soon. Theknightwho (talk) 08:31, 9 July 2023 (UTC)

@Benwing2 Just as a general update - I've done a major redesign of the parser in order to eke out some more speed, which is nearing completion. I thought this was going to be at the expense of memory, but it seems memory use has dropped even further anyway, but crossfingers it should be able to match or even beat the native parser in terms of speed. I'll let you know once it's in a position to go through proper testing. Theknightwho (talk) 22:39, 23 July 2023 (UTC)

OK great! Let me know when you've finished the updates. Benwing2 (talk) 22:41, 23 July 2023 (UTC)

Just FYI, the otherNames column in the list of etymology languages isn't properly populated in the list, e.g. 'Bohairic Coptic' should have 'Bohairic' and some other things listed but none are. I checked the code in Module:list of languages and it's directly accessing data.otherNames. Either this is the wrong field or something is wrong with that field. Benwing2 (talk) 22:39, 23 June 2023 (UTC)

I fixed this but there is still a lot of breakage in WT:LOL/S, e.g. the parents column of the etymology languages. In general this code needs some rewriting given all the changes you and others have made to Module:languages and the data format. Benwing2 (talk) 02:15, 24 June 2023 (UTC)

Hi, I am proposing deprecating or deleting a bunch more form-of and lang-specific headword templates that are trivial wrappers; see WT:RFDO#June 2023. You have commented in the past about these; feel free to take a look when you have a chance. Thanks! Benwing2 (talk) 08:05, 27 June 2023 (UTC)

Hi. You added a bunch of Mongolian-specific tags in Module:form of/data2. I added support for language-specific tags to Module:form of so I'd like to move all the lang-specific tags out of Module:form of/data2 and into the appropriate lang-specific modules. Can you help me do this, either by moving the tags (which is not hard) or by identifying which tags are Mongolian-specific, so I can do it? See Module:form of/lang-data/sw for an example. All you need to do is create Module:form of/lang-data/mn along the same lines and move the tag data. This should help reduce memory pressure. Thanks! Benwing2 (talk) 02:34, 28 June 2023 (UTC)

OK, you also need to add the language code of the module with lang-specific tags to the table at the top of Module:form of. Without this table, pcall() on a non-existent module seems to waste a ton of memory. Benwing2 (talk) 02:49, 28 June 2023 (UTC)

@Benwing2 Hiya - sorry, I missed this earlier. I'll have a look. Theknightwho (talk) 15:00, 3 July 2023 (UTC)

@Benwing2 Have you tried using package.loaders to grab modules instead? It's the primary loader used by require, and returns a function if the module exists or nil if not. The returned function takes no arguments, and will return the contents of the module if called. Doing it that way means we can conditionally load modules without any risk of error, so pcall is never necessary. Not sure how much it'd help with the memory issues, but it might be worth a shot? Alternatively, we could write a new require function (safe_require maybe?), which has this stuff built-in. Theknightwho (talk) 12:13, 9 September 2023 (UTC)

I haven't yet but this seems like a good idea. I think it might also be possible to just check if the module page exists using the regular page-exists functions. Maybe that's considered an "expensive" call but it doesn't take up memory. Benwing2 (talk) 18:14, 9 September 2023 (UTC)

@Benwing2 Yeah, it's pretty cheap in my experience. Theknightwho (talk) 18:17, 9 September 2023 (UTC)

Hi. I got sick of the undiscussed piecemeal changes you're making to the core modules so I reverted some of them, which significantly brought down the contents of CAT:E (from ~ 50 to now 31, some of which are not related to memory or timeout issues). You've promised multiple times not to make core changes without prior discussion, and to create design documents outlining what you were going to do before doing them, and you've never actually done these things. From now on I'm going to be more aggressive reverting such changes; you need to test your changes in your userspace sandbox and then push them as a whole once your parser is finished and you've verified that the result actually does reduce memory and increase speed. I know you keep saying that isn't possible, but I've never had an issue with this approach even with 10+ modules needing to be copied into user space and their links modified. I think there's also another method that User:Erutuon uses that avoids the need to do the copy-and-modify-links approach; you might want to ask them. Benwing2 (talk) 05:39, 28 June 2023 (UTC)

Hi again,

Actually, "high front unrounded vowel" is not necessarily accurate, and the transcription for that would be , not */i/. But the lexical transcription "ǣ" is specifically the FLEECE vowel, regardless of its phonetic value, so that is the more accurate description. It's not a "Wikipedia approximation", it's the standard lexical set -- it's the IPA that's a WP approximation in this case. Also, an example can be useful, and that example is the one given by the dictionary that was my source. kwami (talk) 19:59, 11 July 2023 (UTC)

@Kwamikagami That makes sense. It would possibly be good to deal with the lexical sets in a more standardised fashion (which includes a link to that Wikipedia page), as it would prevent misunderstandings like mine. Theknightwho (talk) 14:20, 12 July 2023 (UTC)

That's a good idea. I'll start adding links, as soon as I get done with all the fake "translingual" entries that are specifically Burmese. kwami (talk) 23:10, 12 July 2023 (UTC)

I see the 'strawberry' example with ㇲ゙ has an alt form in all hiragana. That makes me suspect that katakana is only used electronically because it's available from Ainu (perhaps by Jlect's source). As does examples like っざら ~ ｽﾞざら. If hiragana is preferred in print, or even just common, that would be an argument for proposing them to Unicode, which I'd be happy to do. Are you familiar with the lit? kwami (talk) 20:58, 15 July 2023 (UTC)

@Kwamikagami I'm not sure as I'm not all that familiar with Ryukyuan languages, but I think you're probably right: compare this entry for むとぅびㇲ゙ (mutubz) (with a syllabic む), which is given as an synonym of ㇺとぅびㇲ゙ (mtubz). To me, that strongly suggests the small ㇺ and ㇲ゙ are being used as substitutes for the hiragana forms. I suspect that Miyako follows the same rules as Japanese with the use of katakana and hiragana, but it would be useful to get evidence for that.

I have a feeling that only Ainu and Taiwanese kana have been used as sources for the consonantal small letters (both of which exclusively use katakana), so any use in Miyako (or the other Ryukyuan languages) is likely just coincidental. Theknightwho (talk) 21:36, 15 July 2023 (UTC)

Your sandboxes User:Theknightwho/sandbox11 and User:Theknightwho/sandbox12 are in various language categories such as Category:English one-letter words, Category:en:Latin letter names, etc. J3133 (talk) 17:29, 16 July 2023 (UTC)

@J3133 Thanks - it's because they're tests for the new wikitext parser I've been building, and the most realistic way to test it is to trick Lua into thinking it's actually on the page in question (and not in a sandbox). I've disabled them for now. Theknightwho (talk) 17:34, 16 July 2023 (UTC)

I don't think you noticed, but with this edit you stepped in something. As you may be aware, {{descendants tree}} is a complex and temperamental kludge that never quite does what you think it does. By changing {{desc|fr|cow-boy|cowboy|bor=1}} to {{desctree|fr|cow-boy|cowboy|bor=1}}, you told the module that there were Descendants sections in both French cow-boy and French cowboy. In the old days, that would have caused a module error, but I got tired of entries ending up in CAT:E every time someone moved the Descendants section from one descendant to the other (especially since the cause is often not in the same place as the error, which means the perpetrator is unaware), so I got consensus to change it. Instead of throwing a module error it displays an error message only visible in preview while adding a maintenance category so someone familiar with the languages in question can fix it later. @Fytcha was kind enough to implement it for me. It's not a perfect solution, because the fact that it pretends to be {{desc}} means that those who do things like you just did have to be informed that it only looks like the template works that way. Personally, I think we need to rethink the way {{descendants tree}} works. It should be possible to optionally have some parameters act like the parameters in {{desc}}- that is, just displayed as descendants on their own without having the module scan the targets for further descendants.

Also, the module only recognizes alternative forms that use {{alter}} or {{alt}} when it displays alternative forms. In a few cases, such as Middle English descendants with a gazillion alternative forms, I've exploited this by using {{l}} for the alternative forms that I didn't want to be displayed by {{desctree}}, and {{alt}} for the rest. Really, though, we should have a better way to control what displays. Another problem is that having multiple alternative forms that each have their own descendants results in a list of alternative forms followed by a list of descendants without a clear way to show which descendant is from which alternative form.

Then there are situations where the main entry that has the alternative forms section isn't the one that has the Descendants section: a clear case of lexicography (which form is or was the most common and correct according to contemporary usage?) conflicting with etymology (which one is the actual ancestor of the descendant in question?). There's no way for {{descendants tree}} to choose between the two, even when the default is wrong for the purposes of the template.

Anyway, I thought I'd make you aware of this so there will be one more person thinking about it. I have Special:WhatLinksHere/Template:tracking/descendants_tree/desctree-no-descendants bookmarked, and I do my best to fix what shows up there. There are occasions, like currently with Old Javanese kamantryan, where I'm not sure what to do- so I just leave it alone and hope that someone will preview the entry or see that it's in a maintenance category and discover there's a problem. So far, I've never had one of these remain in the maintenance category for more than a week or two. Thanks for reading all the way through this, Chuck Entz (talk) 00:28, 17 July 2023 (UTC)

@Chuck Entz I didn't write the {{desctree}} code but I've done some hacking on it. If you can make a concrete proposal as to new params/options to add, I may be able to implement it. Benwing2 (talk) 04:27, 17 July 2023 (UTC)

Thanks @Chuck Entz - that's useful to know. I would like to do a bit of an overhaul of descendants at some point (e.g. doing them in a single template call). It would allow for standardising the layout, and would hopefully prevent situations like these. Theknightwho (talk) 04:35, 17 July 2023 (UTC)

Yeah I think something like that makes sense. Benwing2 (talk) 04:37, 17 July 2023 (UTC)

Hi, I pinged you a few days ago in WT:Beer parlour/2023/April but didn't get a response. I am proposing renaming the qfa-sub-* codes to just sub-*, any objections? Also any suggestions you have about implementing script code aliases are welcome (see the BP entry for more info). Benwing2 (talk) 04:29, 17 July 2023 (UTC)

@Benwing2 Fine by me! Theknightwho (talk) 04:33, 17 July 2023 (UTC)

Hello again, I am starting to rename the anomalous script codes. This is significantly trickier than for the anomalous etymology lang codes, at least for 'polytonic' and 'Latinx', because they are used in so many places. I am following User:-sche's suggestion of five-letter codes with a capital initial letter; hence polytonic -> Polyt, Latinx -> Latnx, musical -> Music, Ruminumerals -> Rumin, IPAchar -> Ipach, while Morse and Semap stay as is. Let me know if you have any objections. I have set up tracking for use of the anomalous codes and added support for the new aliases in Module:scripts, but I haven't yet added the ability to use the new codes in places where script lookup happens (e.g. in per-script entries in the translit, entry_name, etc. fields), so those have to stay as-is until I implement that support. Once everything is converted I will rip out all the alias support to make Module:scripts and related modules leaner. Benwing2 (talk) 22:45, 17 July 2023 (UTC)

@Benwing2 That sounds good, as it'll make handling them more straightforward. I'm not fully convinced that we even have a need for a separate polytonic script anymore, since it's now widely supported, but that's a separate conversation. Theknightwho (talk) 22:52, 17 July 2023 (UTC)

why is this necessary? I don't see any functions in that module and it's loaded by all the language modules, which certainly makes it a memory suck. At least can you document in Module:languages/data why it needs to be required? Benwing2 (talk) 09:20, 17 July 2023 (UTC)

@Benwing2 It's Module:languages/data/patterns, and the reason is because the table has to be mutable as some things are conditionally added to it. As it's required by multiple functions, it's cheaper to put it in a separate module that can be required when needed, as opposed to just dumping it in the module (which would mean it got loaded every time no matter what).

The other thing to note is that all the mw.loadData calls are wrapped in a function called conditionalRequire, which uses mw.loadData unless a flag is specifically set to use require instead. This is used by Module:family tree, because mw.loadData is slow and really memory-inefficient for situations involving one massive invoke like category pages. We should probably update Module:category tree to use it, actually, as it might prevent the issue with certain language pages occasionally having memory errors. Theknightwho (talk) 09:26, 17 July 2023 (UTC)

OK given that this is rather subtle, can you *PLEASE* *PLEASE* document this in Module:languages/data? This should take you all of two minutes, just copy what you wrote above and expand on it slightly. I shouldn't have to ping you every time I need to understand something non-obvious about code you've written. Benwing2 (talk) 09:37, 17 July 2023 (UTC)

In addition, there's no mention of Module:languages/data/patterns anywhere in Module:languages/data, so am I confused. Benwing2 (talk) 09:39, 17 July 2023 (UTC)

@Benwing2 It's used in Module:languages. I'm wondering whether this is actually necessary, since the table needs to be cloned anyway to stop the table in packages.loaded from having things added to it. I vageuly remember there was a massive unexplained slowdown that forced me to use require over mw.loadData (as in, pages would go from 2 to 9 seconds to load), but that I have a feeling that was when the function was still using mw.ustring.gmatch which has a bug like that. However, there's still a weird memory spike if I test it with conditionalRequire (or mw.loadData), which makes no sense to me. In any event, I'll document that it seems to be necessary to keep memory down. Hopefully this will all be immaterial soon, once the parser is ready.

The reason it's saved there is because it's pattern data for Module:languages, and I wanted it to be in a separate module to minimise overhead. It's got nothing to do with Module:languages/data, but the general convention seems to be that subpages with other names are for functions that have been hived off, while data is always under the data subheading. I don't mind if you want to move it. Theknightwho (talk) 09:49, 17 July 2023 (UTC)

Thanks. Please do add this documentation. It's very important for maintainability purposes, to reduce the dependency on "tribal knowledge". As an example of what I'm looking for, see the comment at the top of Module:affix about the different types of hyphens and affixes. This stuff is key to understanding Module:affix, and it's not super obvious, so what I did is essentially a "brain dump" at the time I was rewriting the module, which is when all the information is freshest in your head. This entire comment took about an hour to write, which is peanuts time-wise compared to the amount of time you've spent working on the code itself. You don't have to write something so detailed, but a paragraph or so would be very nice explaining your learnings about keeping the memory down and what is going on with useRequire, conditionalRequire and such (none of which is currently documented). Benwing2 (talk) 22:51, 17 July 2023 (UTC)

See this recent discussion: Wikipedia:Talk:White supremacy#Nazi_Germany. 81.102.123.104 23:21, 18 July 2023 (UTC)

I'm not convinced by that discussion at all. The statement that the Nazis "never recognized the concept of the white race" is simply incorrect, because they were acutely aware of the parallels (and differences) between themselves and the American South at the time. Although they were focused on "Aryans", we are under no obligation to use their terminology. Theknightwho (talk) 00:04, 19 July 2023 (UTC)

It's still more precise and accurate to describe Nazism as an ideology of Aryan supremacy. The Nazis also regarded other groups of white people as inferior, such as Slavs, hence the OP's reference to it being like calling Hutu power a black supremacist movement. 81.102.123.104 09:17, 19 July 2023 (UTC)

Right, but many Americans didn’t perceive Irish or Italian people as white during the latter part of the 19th century, because racial categories are woolly, subject to cultural biases and prejudices etc. To use your example: it’s pretty common for neo-Nazists in Western Europe and North America to say Slavic people aren’t white, and it’s a pretty universal belief of white supremacists that Jews aren’t white, when obviously the real answer is “it depends on the person”. It’s hard to ignore that both of those beliefs come straight from the Nazis.

There’s also the fact that the meaning of the word “Caucasian” has shifted over time: today it’s usually used to refer to white people, but the Nazis used it in a wider sense that covered many people from Central and Southern Asia. They used “Aryan” to refer to a subgroup of Caucasians, and there had been a lot of debate as to whether eastern or southern Europeans should be counted as Aryan in the decades prior (e.g. look at this early map); it was political factors that ultimately decided the answers. To me, it seems like a specific form of white supremacy.

There’s also the fact that white supremacy is a movement, as well as an ideology, and it’s important to recognise the role the Nazis played in that movement. We can’t really say the same for “Aryan supremacy”, and there’s no reason we have to use their terminology either. Theknightwho (talk) 09:53, 19 July 2023 (UTC)

Your latest two changes caused a bunch of errors in CAT:E. Benwing2 (talk) 06:44, 19 July 2023 (UTC)

I think it's becoming clear that your discussion with Koavf is not coming to a resolution. If I may be so bold, may I ask that you refrain from replying to him for a week, to give both of you some time to cool off? I will make the same request on his talk page. I think it will be better for everyone if we take a step back from all the drama in the Beer Parlour and elsewhere for the time being and maybe come back to it in a different way later, if necessary. Thank you, and all the best. Andrew Sheedy (talk) 14:44, 20 July 2023 (UTC)

Yeah, I agree. Would you be so kind as to step in to ask him to stop? I cannot see how his actions are anything other than an egotistical power-trip at this point. Theknightwho (talk) 14:50, 20 July 2023 (UTC)

@Andrew Sheedy - whoops, missed the ping. To clarify: I meant stop with the WF business until the vote has been resolved, and let other admins handle WF as-and-when. Theknightwho (talk) 14:54, 20 July 2023 (UTC)

I'm happy to mediate between you, but I think as far as WF goes, you two are the only ones still discussing the issue actively. Although I disagree with him, Koavf is perfectly justified in his approach to dealing with WF, so I think it's best to let the vote determine the outcome. Given what Koavf has expressed, I think he will abide by the results of the vote. I hope that the vote can remain civil, without any direct arguing, because I think everything that needs to be said has been said already. Andrew Sheedy (talk) 14:57, 20 July 2023 (UTC)

@Andrew Sheedy Thanks. I think it would be a reasonable return to the status quo (and would heavily reduce the potential for conflict) if other admins were to agree to treat WF as they had been before (i.e. blocking on occasion), but for Koavf to refrain from doing so. After all, that is precisely what caused all of this fuss in the first place.

It is very clear that Koavf does not have consensus (either among the community or among the admin team) to keep acting in this way, so I think it's a reasonable compromise. Theknightwho (talk) 15:01, 20 July 2023 (UTC)

Perhaps it would be better, but I suspect it would be a bit hard to legislate. I think the reality is that while there may have been consensus before, there is no longer consensus. Koavf has a number of supporters and the Beer Parlour discussion seemed to indicate a wide range of opinions all over the map. So while there isn't a consensus supporting Koavf, I'm not convinced there is a consensus against him either. Anyway, let's wait to see the outcome of the vote before trying to find alternative solutions. Andrew Sheedy (talk) 15:58, 20 July 2023 (UTC)

@Andrew Sheedy I see only one person who supports his current approach, though, and quite a number of people have expressed serious concerns about his attitude (including you). Theknightwho (talk) 16:06, 20 July 2023 (UTC)

To be clear - I'm not trying to be defensive about this, but I'm just concerned about the community being run roughshod over. Theknightwho (talk) 16:11, 20 July 2023 (UTC)

Even if there is a community consensus, if the community consensus runs counter to the rules, there isn't really a good way to justify preventing someone from following the rules if they want to. I'm not a fan of Koavf's approach, but he's right in that if there is in fact a consensus against the rules, the best way of avoiding conflict is to just change the rules. I don't want to get caught up in yet another discussion about the issue at hand, so this will be my last reply. See you in the vote. Andrew Sheedy (talk) 16:18, 20 July 2023 (UTC)

Hi Theknightwho,

I am the author of virtually all of the content on the pages 𛄠 and 𛀆. You recently reverted my edit on 𛄠 redirecting most of the usage notes to 𛀆. There appears to be some confusion on kana equivalency, especially when dealing with historical kana.

First of all, your claim that "They aren’t even fully applicable." is not correct. Everything in the usage notes of 𛀆 is applicable to 𛄠. The only difference between them is that one is hiragana and one is katakana, which I made sure to account for by leaving {{ja-Kana-usage notes}} in. Conversely, the information in {{ja-Kana-usage notes}} itself is not very applicable to 𛄠, as it explains the modern usage of katakana, where 𛄠 has been obsolete for a very long time. During the period where 𛄠 was used, katakana and hiragana were used interchangeably between documents. Full-length documents where not a single hiragana was present - only katakana - were a common thing, an example being the script of the Japanese WWII surrender broadcast on 15 August 1945. Katakana was actually considered more official than hiragana not a very long time ago, a situation that has reversed today. In fact, the sources I used (1891 仮名遣 and 1897 日本大文典. 第1編) when creating the page 𛄠 uses it in ways that demonstrate the perfect applicability of the usage notes on 𛀆 to 𛄠. All the information I wrote pertaining to the evolution and scope of 𛀆 is entirely valid for 𛄠.

Furthermore, the information currently on 𛄠 is wrong. Quoting User:Eirikr,

There is no historical evidence that /ji/ existed in Japanese. There is circumstantial evidence for its existence prior to recorded history, but in the entirety of the Japanese written corpus, as best I understand it, there is no /ji/.

Consequently, the statement that "In modern Japanese, old /ji/ evolved into /i/..." is just plain wrong: there was no "old /ji/".

—— User:Eirikr, 00:58, 18 September 2020

His complaint on my talk page is what led me to reform the page 𛀆 into what it is now. 𛄠 has not been changed since then, and it needs to be changed to remove the incorrect information. It is more unhelpful to leave the page as it is, than to replace it with a link to the usage notes of 𛀆, just like what was done on 𛄡 for 𛀁. LittleWhole (talk) 21:59, 22 July 2023 (UTC)

It looks like the underlying concern has been addressed. Good on you both for getting things sorted. 😄 ‑‑ Eiríkr Útlendi │^{Tala við mig} 17:43, 24 July 2023 (UTC)

First of all, I took a look at the "Disallowed nesting of Malayic translations" abuse filter. The regex involved is awfully complicated and could really do with a good explanation of how it works. It uses some syntax that I'm not even familiar with like ?(1). I assume this is a PHP regex; if PHP supports Perl-style (?x...) regexes, you can use it to insert comments into the regex itself to explain how it works.

I wrote a script to unindent languages under Malay (especially) and Javanese. In the process it sorts all the translations; indented and unrecognized lines are grouped with the nearest language above them and kept in order if there are multiple such lines together. Some issues that maybe you could help with:

1. Should the script sort all translation sections or only those where an indented line needs to be unindented? There are an awful lot of mis-sorted translation lines. (The script does not currently sort lines under {{checktrans-top}} because of junk that occasionally appears there.)
2. The script doesn't unindent recognized scripts listed in place of languages; these are Carakan, Roman, Jawi, Rumi, Arabic and Latin. Not sure if this is correct.
3. The script indents the following languages, which include (almost) all the languages that appear under Malay and Indonesian: Acehnese, Ambonese Malay, Baba Malay, Balinese, Banda, Banjarese, Batavian, Buginese, Brunei, Brunei Malay, Ende, Indonesian, Jambi Malay, Javanese, Kelantan-Pattani Malay, Madurese, Makasar, Minangkabau, Nias, Sarawak Malay, Sarawakian, Sikule, Simeulue, Singkil, Sundanese and Terengganu Malay. Some of these aren't recognized Wiktionary languages, so I don't know if they should be unindented and sorted among other languages. Some should probably be renamed in the process (e.g. Brunei -> Brunei Malay). Note that neither Sarawak Malay nor Sarawakian (presumably the same thing) are found as Wiktionary languages, nor is Batavian.
4. The script doesn't unindent any other language variants under Javanese, which includes Central Javanese, Western Javanese, Kaili, Krama, Ngoko and Old Javanese. Not sure if this is correct.
5. When it comes to sorting, the script currently converts to NFD format before sorting by codepoint, so that e.g. Yámana comes before Yiddish, Mòcheno comes between Montagnais and Muong, Tày comes between Tatar and Telugu, Réunion Creole French comes before Romani, Māori comes between Maltese and Meru, Meänkieli comes before Megleno-Romanian, etc. Not sure if this is correct; the current sorting is inconsistent but often sorts these using NFC codepoint sorting.
6. By the same token, Oki-No-Erabu ends up before Okinawan when it seems it usually is currently sorted after. Maybe I should ignore hyphens when sorting (and apostrophes? Cf. S'gaw Karen, O'odham, 'Are'are).
7. Potentially the script could normalize lines beginning with ** (especially common with Chinese variants) to instead begin with *:. Potentially it could also correct issues like * Assyrian Neo-Aramaic {{t|aii|ܩܵܛܵܪ|tr=qaṭar}} (missing colon). Neither is currently done.

Benwing2 (talk) 03:31, 23 July 2023 (UTC)

OK, I made it fix both issues under (7), as well as replace Unicode +00A0 with regular space, replace uses of {{ttbc}} and remove blank lines. Also the script could potentially fix places that put Ancient Greek or Modern Greek under a Greek header, if there's consensus to do that. Benwing2 (talk) 04:17, 23 July 2023 (UTC)

@Benwing2 Hi Ben - I’ll get back to you properly once I’ve had some rest, but just to explain ?(1): it’s a conditional based on whether the first capture group matched, and in this regex capture groups 1 and 2 are inside different sections of an OR non-capture group towards the beginning. What it’s basically doing is saying “if the language is the one in capture group 1, then match…”. It’s a way of carving out exceptions for allowed indentations in certain languages, while keeping it specific to those languages, which it does by using negative lookaheads if the condition is met. Theknightwho (talk) 04:25, 23 July 2023 (UTC)

OK, Thanks. I wonder if things need to be this complex; my checks for indented langs are pretty simple. I notice you only check for a few langs that could be indented when there are many more, as mentioned above (or maybe you check for all but those few named langs? I haven't yet figured out the regex). BTW my script is ready to go, pending resolution of the issues above. It would change 3,716 pages, of which 148 of them have indented Malayic translations (often several per page), and issues 594 warnings, mostly for unparsable lines in translation sections (and most of those are genuine, typically by a single language spilling the translations onto multiple lines, sometimes as in would by a sentence being translated into multiple languages, and occasionally by HTML comments inserted into the table giving hints as to what types of entries are acceptable). Benwing2 (talk) 06:39, 23 July 2023 (UTC)

@Benwing2 It’s the other way around - it checks to make sure nothing is nested under 4 languages, except the handful of script exceptions.

I’ll do some comments for it, which should make it clearer what’s going on, as the overall explanation of what it does is actually not that complicated. Theknightwho (talk) 11:14, 23 July 2023 (UTC)

Thanks. Any comments on the above issues? If not I'll go ahead and run the script. BTW User:Benwing2/unhandled-indented-malayic contains warnings (only 17) related to the indented lines that couldn't be unindented. I'm not sure how to handle them, you might want to take a look. Also you should probably make script exceptions for Arabic and Latin, which show up occasionally (unless you think they ought to have different names). Benwing2 (talk) 19:03, 23 July 2023 (UTC)

@Benwing2 Hiya:

1. In theory it should sort all the translation sections, but I was mainly concerned with making sure there weren't any incorrect nestings.
2. "Arabic" corresponds to Jawi and "Latin" & "Roman" correspond to Rumi. For the sake of consistency, I went with the ones which were most common (which are Jawi and Rumi), so it's probably best to go with that.
3. In terms of nesting, I think the best approach would be to separate out any Malayic languages which we treat as full languages (e.g. list Brunei Malay under "B"), but if there are any that are not Wiktionary languages then it'd be best to flag them up for manual review. Another one to look out for is Indonesian, where some editors have just grouped any languages spoken in Indonesia under "Indonesian" as a heading, such as Javanese and Sundanese. I've sorted some of them out manually, but I'm sure there are more. We're also a bit inconsistent with our treatment of "Old X" and "Middle X" languages in general, so it'd be best to see what the general trend is before committing to indenting Old Malay under Malay.
4. Is it possible to use the UCA for sorting? That would bypass any of the NFD/NFC concerns. If you need a Lua implemention, Module:User:Theknightwho/sort has a function (sort) which takes a table of terms as an argument, and outputs a UCA-sorted table. I haven't implemented variable weighting yet (which applies to -), so you may have to manually disregard it. My inclination is to use English sorting rules in that regard, so ignore the apostrophe and dash (or more accurately, only give them a quaternary weighting so that they're a last-resort tie-breaker, which won't affect anything here).
5. I think normalising ** to *: is a good idea.

I'll do the comments on the regex now. Theknightwho (talk) 19:56, 23 July 2023 (UTC)

Thanks. The script is simply leaving alone any indented languages it doesn't recognize, and unindenting/sorting the remainder, whether found under Malay, Javanese or Indonesian (there are none indented under Sundanese). I'll have it normalize Arabic -> Jawi and Latin/Roman -> Rumi. As for UCA, I'd need a Python implementation since that's what the script is written in. There seems to be one here: However I'm a bit leery of doing this because the sorting should ideally match what is done by the translation adder gadget. Potentially the gadget could call into your Lua implementation but we'd need to make it production-ready and put it in the production space, which I'm wary of doing until we have a clear consensus to do this. Benwing2 (talk) 20:11, 23 July 2023 (UTC)

@Benwing2 Great - thanks. That makes sense. In terms of sorting, the UCA can take what are called "tailorings", which are contextual modifications. These are often for language-specific needs (e.g. Azerbaijani "Q" goes after "K"), or more general ones (e.g. whether to restrict punctuation to quaternary weights because we don't really care about it). If we did implement this for the translation-adder, it would be a good idea to make any tailorings easily accessible to bots using other languages, which we could probably do with JSON. Theknightwho (talk) 20:18, 23 July 2023 (UTC)

OK I discovered that I have an existing script to sort language sections that uses the following code, so I am sorting language translations the same way:

def normalize_lang(text):
  return re.sub("", "", unicodedata.normalize("NFD", text)).lower()

Benwing2 (talk) 21:25, 23 July 2023 (UTC)

@Benwing2 Sounds good! Theknightwho (talk) 21:28, 23 July 2023 (UTC)

@Benwing2 Check the notes in the abuse filter to see an explanation of how the regex works. I don't want to explain it publicly as there's a reasonable chance some users may try to subvert it if they know how it works (e.g. we already had one problem IP from Malaysia trip the filter, who then decided to nest Indonesian under a random language instead, so it wasn't just an innocent misunderstanding). Note also where they placed Malagasy, too. I blocked them for a week (as they've been doing this kind of thing for a while), but they'll no doubt be back. Theknightwho (talk) 22:11, 23 July 2023 (UTC)

OK, thank you! I took a look and it looks good. I am running my script now; it is still tripping the abuse filter on some terms because it doesn't fix everything, for reasons I've already mentioned. In some cases the page actually looks OK, e.g. many has two scripts (Carakan and Rumi) under Javanese; maybe the filter needs updating. Benwing2 (talk) 22:17, 23 July 2023 (UTC)

@Benwing2 Thanks. The filter allows "Carakan" and "Roman" for Javanese, but maybe "Rumi" is a better choice than "Roman" to be honest. Theknightwho (talk) 22:28, 23 July 2023 (UTC)

Hmm, I'm not sure. Currently the Roman -> Rumi change is not sensitive to Javanese vs. Malay, but I can fix this if you prefer. Benwing2 (talk) 22:30, 23 July 2023 (UTC)

So if you look up in Wikipedia, Javanese is said to use three scripts: "Javanese" (Carakan), "Latin" and "Pegon" (Arabic). Malay similarly has "Latin", "Arabic"/"Jawi", "Arabic"/"Pegon", "Thai", "Malay Braille" and some others used historically. This makes me wonder if we shouldn't use Latin in place of Rumi. Benwing2 (talk) 22:34, 23 July 2023 (UTC)

OK, I think Rumi is a misnomer for Javanese and Indonesian, which should use "Latin", while either Latin or Rumi is possible for Malay. Benwing2 (talk) 22:38, 23 July 2023 (UTC)

Hello, you mentioned that the sortkey was unnecessary here. Without the sortkey, in , the two words pronounced mendori, 雌鳥 and 雌鶏, are sorted under と and め respectively. I have two questions: how is 雌鶏 sorted under め in , and how can the two entries be sorted in as they should be? Mcph2 (talk) 11:59, 24 July 2023 (UTC)

@Mcph2 Hi - are you sure that's right? If I test the automatic sortkeys, {{sortkey|ja|雌鳥}} gives めんとり' and {{sortkey|ja|雌鶏}} also gives めんとり'.

That being said, some Japanese modules use other methods to sort terms (e.g. the katakana categories use katakana, and the kanji ones use radical sort). There might be something like that going on here, as they should both be sorting the same. Theknightwho (talk) 12:07, 24 July 2023 (UTC)

@Theknightwho: What I’m referring to is that ] shows the following:

と

• 雌鳥

め

• 雌鶏

I suspect the {{pre}} in 雌鳥 somehow automatically sorts it as とり, ignoring the prefix, while {{cln}} in 雌鶏 uses the automatic sortkey of めんとり'.

Should we add the sort= parameter back to {{pre}} in 雌鳥? Mcph2 (talk) 14:05, 24 July 2023 (UTC)

@Mcph2 Thanks - a much better solution is to fix {{pre}}, because it's unhelpful for it to sort like that. Theknightwho (talk) 14:07, 24 July 2023 (UTC)

@Mcph2 This seems to be something unique to {{pre}}, where it sorts "X terms prefixed with Y" categories by ignoring the prefix, presumably because the assumption is that it's unnecessary to sort with the prefix. I actually think this method of sorting isn't a great idea, because it ignores situations like Japanese where sorting isn't necessarily based on the orthography, as well as situations where there might be two or more variants of a prefix (e.g. English anthrop- and anthropo-) where we want to use the same category for both, and ignoring the prefix just leads to confusion. Theknightwho (talk) 15:01, 24 July 2023 (UTC)

@Theknightwho: Thanks for investigating. I’ll just add the sort=

parameter until you can fix {{pre}}. Mcph2 (talk) 02:21, 25 July 2023 (UTC)

Hi, I know you've been dealing with sortkeys lately. There are some entries in which some editors added |sort= when categorising using non-templates, which of course sorts the page at s rather than the intended one. See this search for example, and it appears that Japanese is the main culprit. I've been removing them when I was dealing with the out-of-memory pages. There might still be a few of them not shown in this search, and also it might be a good idea to abuse-filter them? – Wpi (talk) 16:07, 24 July 2023 (UTC)

@Wpi Thanks for that. An abusefilter makes sense, yes. Theknightwho (talk) 16:14, 24 July 2023 (UTC)

Memory is increasing. Is it possibly related to your change to the Han sort keys? Several pages are hitting or getting close to hitting the memory limit that didn't use to be anywhere close. Benwing2 (talk) 03:26, 26 July 2023 (UTC)

@Benwing2 I think it's unlikely to be related to that: the changes were minor, and strings are stored centrally in Lua memory (meaning they don't get duplicated for each invoke), which is why serialisation is an effective memory-saver in the first place. Theknightwho (talk) 03:33, 26 July 2023 (UTC)

Hmm. Can you explain what you changed in the latest day or so related to core functionality? What were these minor changes?

Also, I am trying to move {{quote-meta}} into Lua but it seems to be drastically increasing memory. Do you mind taking a look at the quote_t function in Module:quote, which is an approximate port of {{quote-meta}}, and comparing it to {{quote-meta}}, and letting me know if you have any idea why memory is increasing so much when I use quote_t? You can see in Template:quote-book the switch back and forth between using {{quote-meta}} and quote_t; using quote_t causes at least 7 or 8 pages, e.g. ban and min, to increase their memory by 2MB or more. In ban, the single call to {{RQ:Shakespeare Hamlet}} uses around 4MB of memory when quote_t is in place, and at most 2 MB otherwise. What confuses me is that the old Template:quote-book code also calls into Module:quote (the source_t function) to do most of the work, and all that quote_t and {{quote-meta}} do is paste together the source line constructed using source_t with the usex lines constructed using Module:usex. Both methods do essentially the same thing and I'm not adding any new parser frames. Benwing2 (talk) 05:56, 26 July 2023 (UTC)

@Benwing2 I’ve only really been working on the parser, so other than Module:Hani-sortkey 2 days ago I’ve not changed anything likely to affect a lot of pages.

I’ll have a look at Module:quote, as that does seem weird. I suspect it’s the random effects of Scribunto memory issues again, and we just don’t notice the pages where memory’s fallen by 2MB. I’m starting to think there’s something seriously wrong with Scribunto’s design which isn’t merely down to the Lua 5.1 garbage collect issue, because I never see any wild fluctuations like this when everything’s contained in a single invoke. Theknightwho (talk) 06:59, 26 July 2023 (UTC)

Hi. I can now click on the Edit button from WT:GP and it works, but the Reply gadget still gives an error that doesn't happen from WT:Grease pit/2023/July. Benwing2 (talk) 21:12, 30 July 2023 (UTC)

@Benwing2 Hiya - I’ve been using it with no issues today, but I think there’s a problem with the reply gadget itself as I’ve had that problem before at random intervals.

To be honest, the savings aren’t that significant, as the only way to get the edit links to work was to preprocess the entire pair of pages together, which has knocked out most of the savings we were seeing before. It’s the exact same issue with edit links that I raised a few months ago when discussing the idea of page parsing, and unfortunately there’s no native workaround.

This is also an issue that we’re probably going to have to deal with once the Wikitext parser is ready, to allow us to parse pages in one invoke. I did read something somewhere about generating edit links manually with JavaScript, which sounds promising. Theknightwho (talk) 21:46, 30 July 2023 (UTC)

Yes, we'll need something like that. Benwing2 (talk) 22:43, 30 July 2023 (UTC)

Hi. I was looking at embedded_language_links in Module:links and I noticed that I don't see any of what I call "common-case optimizations". For example, whenever I implement inline modifiers, I make sure to check for < before loading Module:parse utilities and doing the full inline modifier processing. Similarly in my new code for Module:quote, which allows for all sorts of parameters to be in foreign scripts and potentially have inline modifiers (including page numbers, since they may use Eastern Arabic, Thai, etc. numerals), I have various checks to avoid most of the work in the common cases, e.g. no script checking if the text is all-ASCII, no inline modifier checking unless < is present, no comma-splitting/multi-arg processing (when such a thing is possible) unless a comma is present, etc. I think adding some of these could reduce memory significantly. Although it's true that there is a lot of statistical variance in Scribunto's memory implementation, the fact remains that we have 46 entries in CAT:E even after a ton of {{multitrans}}, {{*-lite}} templates, etc., when formerly there were no lite templates and only a few over-memory items, and this is likely to be due more to all the changes you made to the core modules than to additions to the pages themselves. Benwing2 (talk) 19:18, 31 July 2023 (UTC)

@Benwing2 Yes, I agree with you (and incidentally I’ve been doing a lot of work on that today, in an effort to get the parser timings down).

One thing that I’ve found is that sometimes these optimisations can be counter-productive, because anything that fails the tests will always be doing more work. For example, there’s not much point in avoiding loading things that use mw.loadData, because it only takes one failure on the entire page for the benefit to be cancelled out - and that’s extremely likely on a very large page. On the other hand, anything that avoids string functions is more likely to be beneficial. Theknightwho (talk) 04:21, 1 August 2023 (UTC)

Yes, I think there's no need to do this with loadData() but avoiding require()ing a large module can definitely be helpful, likewise avoiding string manipulation as you mention. Benwing2 (talk) 18:25, 1 August 2023 (UTC)

Can you take a look? When embedded_language_links() is called on certain pieces of text, an error results deep in some of the code you added. The text appears to be message-ID <AUTISM%[email protected]> (NOTE, it displays with less-than/greater-than signs but the actual wikitext contains HTML entities; see the source code of this message). Benwing2 (talk) 04:22, 1 August 2023 (UTC)

FYI I count 6 pages currently with this error (xe, xyr, kokoro, e-publisher, Citations:Garfield and Citations:Junior Woodchucks' Guidebook), all coming from errors related to the section= param in Module:quote, coming from {{quote-newsgroup}}. Benwing2 (talk) 04:49, 1 August 2023 (UTC)

Will do. Those functions need a total rewrite anyway. I’ve been considering doing it by tokenising the strings, which allows us to do arbitrary lookaheads (among other regex-like things that we can’t do with ordinary patterns). Judicious use of repeat loops can make this extremely fast (e.g. the tokenisation of a now takes less than 0.2 seconds, down from 3 a month ago). Theknightwho (talk) 05:03, 1 August 2023 (UTC)

@Benwing2 Forgot to ping. Theknightwho (talk) 05:12, 1 August 2023 (UTC)

@Benwing2 I just did a test with ZeroBrane (a Lua IDE), comparing string.find's character set () with an iterator that checks a table lookup. The table lookup is over 100 times faster: it completes 1 billion iterations in under 2.5 seconds, versus 10 million in 2.9 seconds. I have a feeling we could take advantage of things like this if we developed our own library similar to Python regex library, where regex objects get compiled in advance. Theknightwho (talk) 20:40, 3 August 2023 (UTC)

Sounds like a project for a rainy day :) ... BTW were you able to figure out the bug on xyr etc.? There are still several pages in CAT:E due to this. Benwing2 (talk) 00:17, 4 August 2023 (UTC)

@Benwing2 I’ve got a hunch that it’s going to be a pain to solve, so I’ve been procrastinating on it, but I’ll make some time tomorrow to take a look. If it’s due to a design flaw it might be worth taking the opportunity to do a rewrite tbh. Theknightwho (talk) 02:18, 4 August 2023 (UTC)

I think it's a matter of unescaped percent-signs in the ID parameters. Chuck Entz (talk) 04:00, 4 August 2023 (UTC)

@Chuck Entz I think you're right, because when you delete the % sign, the error goes away. User:Theknightwho does that help you figure out a fix? Benwing2 (talk) 06:16, 4 August 2023 (UTC)

Sorry to bug you but can you take a look at this? The errors are still present in CAT:E, and it looks like the fix shouldn't be hard as we know what is triggering the error. Benwing2 (talk) 04:55, 7 August 2023 (UTC)

@Benwing2 @Chuck Entz It's an issue with mw.uri.decode, which treats %93 as a percent-encoding (for byte 147, in this case, which made the text invalid UTF-8). We should only be applying mw.uri.decode to text in links, which will require a proper overhaul of the code. As a temporary solution, I've set Module:links to escape any percent signs in non-linked text, which cancels out the issue.

This will be much easier once the parser is finished, as we can simply use that to analyse anything put into the link templates, which should greatly simplify a lot of the code in Module:links. Theknightwho (talk) 21:34, 7 August 2023 (UTC)

Awesome, thank you for looking into this! Where is mw.uri.decode being called incorrectly? Benwing2 (talk) 21:53, 7 August 2023 (UTC)

@Benwing2 It's in processDisplayText in Module:languages. It's probably worth moving it into the link module entirely, but that goes for a lot of the code currently in Module:languages anyway, so it's probably better to do it as a proper redesign. Theknightwho (talk) 21:57, 7 August 2023 (UTC)

Hey Theknightwho.

I disagree with the obsolete kana as being romanized as if they are in the a-row. It has been inconsistently romanized one way or another, but I beliveve the w-row romanization is more accurate. The three extinct/reconstructed kana are not exceptional in their theoretical Hepburn romanization.

--MULLIGANACEOUS-- (talk) 04:19, 9 August 2023 (UTC)

@Eirikr This is probably one for you to answer. Theknightwho (talk) 14:40, 10 August 2023 (UTC)

@Theknightwho, cheers. 😄

@--MULLIGANACEOUS--, I'm looking at Module:ja-translit/data right now, and I don't understand what you mean.

• Which obsolete kana are you concerned with?
• I see obsolete kana on lines 27 (the や行), 28 (an additional one-off for an alternative ye kana), 32 (the わ行), and 33 (the small-ゎ行). None of these are on the あ行...?
• There are additional mystery glyphs on lines 75 and 82, but I don't have the right font installed to see what these are, and I can't tell from context what they're supposed to be. I also cannot select these for further inspection, as apparently they're some kind of combining glyph -- attempting to select them always includes the preceding apostrophe.

Could you explain better what issue you see in the code? Line numbers and glyph descriptions would be helpful. ‑‑ Eiríkr Útlendi │^{Tala við mig} 16:29, 10 August 2023 (UTC)

@Eirikr They're referring to ゐ (i), ゑ (e) and を (o), which they want to change to "wi", "we" and "wo" respectively. I've not got a strong opinion, but I was under the impression that the current approach is what has consensus. Theknightwho (talk) 16:47, 10 August 2023 (UTC)

Aha, thank you!

Hmm, that's a tricky one. It really depends on context, in a way where I think the only reasonable approach is what we currently have -- automatically output the straight vowel values, and editors can override as needed to specify the /w-/ glides.

... Thinking it through some more, I can say that automatically outputting the glide values /wi, we, wo/ would definitely be a mistake.

• In pretty much all cases I can think of, the ゐ and ゑ kana are used as stylistic spellings that don't affect the pronunciation. The exception is the brand name for Yebisu Beer, which is spelled with the ヱ katakana. This is technically the we kana, but due to historical sound shifts, the pronunciation was actually ye during the Edo period, collapsing to just e within the past few centuries (at least, in mainstream Japanese; I'm unsure of possible dialectal use). And even though it's spelled ヱビス and romanized as Yebisu, nowadays it's pronounced as Ebisu.
• And を is nearly always pronounced as straight /o/, with basically the only exception being hyper-careful speech where the speaker is deliberately chewing the scenery to over-pronounce the particle.

@--MULLIGANACEOUS--, I hope the above makes sense to you. As things stand, I think the current approach is the correct one. That said, if you can make a strong case for changing the default behavior, I'm all ears.  :) ‑‑ Eiríkr Útlendi │^{Tala við mig} 16:59, 10 August 2023 (UTC)

@Eirikr I actually encountered this yesterday, when adding ウヰスキー (uisukī) and ヰスキー (isukī) (dated forms of ウイスキー (uisukī); the latter especially so). I've left it as the default, as I was unsure what the best approach to take was. Theknightwho (talk) 17:06, 10 August 2023 (UTC)

Hmm, hmm, ya. When we're recording a historical term where a /w/ glide was actually pronounced, as in these two entries, we probably want to override the default behavior to add in that ⟨w⟩. ‑‑ Eiríkr Útlendi │^{Tala við mig} 17:21, 10 August 2023 (UTC)

(たびたびの投稿ですみません)

Also, kana usage is changing, as is the "default" Japanese sound system. We might discover that these kana are seeing increased use for modern terms, in which case we may want to revisit our handling. ‑‑ Eiríkr Útlendi │^{Tala við mig} 17:22, 10 August 2023 (UTC)

@Eirikr I think it might be worth changing ゐ (i) to "wi", since it doesn't have the particle issue like を (o), and I find it extremely unlikely that it will have developed analogously with ゑ (e) due to the fact that "yi" is nonstandard. Although it's usually pronounced "i", it's comparatively rare, and would be more etymologically accurate. Theknightwho (talk) 17:36, 10 August 2023 (UTC)

I'm okay with that. Dunno about others, and I'm open to the idea that I'm missing something or otherwise off-base -- might be good to canvass others' opinions, perhaps at the WT:BP? ‑‑ Eiríkr Útlendi │^{Tala við mig} 18:44, 10 August 2023 (UTC)

My opinion is to stick with "wi" and "we", which are the ones I've changed.

The "o" pronunciation for "wo" is just a specific usage of the kana warranting the need of another pronunciation (for historical reasons), the others are "he" pronounced as "e", and "ha" pronounced as "wa".

Basically, just go with "wi" and "we". It is ambiguous to change them to i" and "e", even though the kana are made obsolete because they are pronounced like the a-row kana in modern Japanese.

Japanese materials printed before the reformation still uses "wi" and "we" katakana to transliterate. --MULLIGANACEOUS-- (talk) 23:30, 10 August 2023 (UTC)

Best approach is to just use the "wi" and "we" even though it is common to use these two as emphesis for their a-row counterparts.

Japanese input systems still use the "w" to allow input of these w-row kana. --MULLIGANACEOUS-- (talk) 23:31, 10 August 2023 (UTC)

Additionally, I saw a few artifacts regarding syllables similar to ヰャ. It should be "wya", not "ya". Should we state that "ゐ" should be "wi", and "ゑ" as "we"? It is honestly getting confusing. --MULLIGANACEOUS-- (talk) 04:00, 16 August 2023 (UTC)

Why do you replace the existing Thai language templates with nonspecific templates? Just want to ask about the reasons.

Wouldn't it be easier to directly modify the Thai language templates instead of replacing them with nonspecific templates? New Thai entries are created every day, and the Thai language templates will surely be used, and you will have to replace them again and again without end (not to mention that your replacements will likely be reverted).

-- Miwako Sato (talk) 08:17, 15 August 2023 (UTC)

@Miwako Sato Two reasons: they don’t do anything special except add labels that duplicate the heading, and they present a maintenance headache for those of us that maintain the core modules. The ultimate aim should be to delete the Thai link and list templates (but not the pronunciation one or headwords etc, as they clearly do need to exist). Also, please don’t revert these changes. It would be totally pointless.

The reason they exist is because they pre-date a time when the generic modules could handle Thai properly, but that’s no longer the case. They simply don’t have any reason to exist anymore. Plus, they use a slightly different syntax that’s now unnecessary, and there’s no good reason to have inconsistencies like that.

They were also added by Wyang, who had lots of good ideas, but wasn’t very good at coding and was absolutely terrible at integrating their modules into the core modules, which means the ones they wrote tend to end up stagnating in an unmaintained state, where they don’t get to take advantage of any updates to the main modules. That’s a problem for users who aren’t expecting that, and it creates a lot of extra work for people who do module maintenance. I’ve had a lot of problems with Wyang’s Chinese modules, but the Thai ones are thankfully not as bad as that.

Theknightwho (talk) 11:08, 15 August 2023 (UTC)

Hey Theknightwho,

It has been almost two weeks and this issue of "ゐ" being "i" hasn't been resolved. I noticed that "ヰャ" has been romanized as "ya" though it should really be romanized as "wya".

It is not consistent that its small counterparts are romanized as usual.

--MULLIGANACEOUS-- (talk) 02:51, 21 August 2023 (UTC)

@--MULLIGANACEOUS-- You should raise this at the WT:Beer parlour. I’m not just going to change things like this unilaterally. Theknightwho (talk) 11:16, 21 August 2023 (UTC)

Where was the decision to romanize "ゐ" as "i" in this wiktionary raised? It once indeed was "wi". --MULLIGANACEOUS-- (talk) 04:43, 22 August 2023 (UTC)

@--MULLIGANACEOUS-- I don’t know, because I didn’t decide it in the first place. The Beer Parlour is the best place to raise this. Theknightwho (talk) 17:38, 22 August 2023 (UTC)

See for example Category:Hungarian nouns suffixed with Unsupported titles/-t`num`Hungarian: noun-forming. These are getting generated somehow or other by {{affix}}; I suspect it's due to some code you wrote. There are a ton of such categories in Special:WantedCategories. My bot script created a few of them before I stopped it. Can you take a look? Benwing2 (talk) 08:15, 21 August 2023 (UTC)

@Benwing2 That looks like something’s going wrong with the fragment in the affix templates. I’ll take a look. Theknightwho (talk) 11:18, 21 August 2023 (UTC)

Did you have a chance to take a look? Nothing has changed recently in the affix template code and the pages in question haven't changed in a long time, either. Benwing2 (talk) 02:25, 22 August 2023 (UTC)

@Benwing2 Not yet, but it’s a priority. I recently standardised the names of most pages with unsupported titles as a way to simplify the code that deals with them, which shortened the list of hard-coded ones to 6, down from 123. However, I don’t think that’s the cause, because there’s a separate bug with the affix template not removing the fragment, because it happens regardless of the target. For example: {{af|en|fun|-y#English}} wrongly gives fun +‎ -y, whereas {{l|en|-y#English}} correctly gives -y. Theknightwho (talk) 14:13, 22 August 2023 (UTC)

OK, that suggests that maybe your change exposed an existing bug in the affix template code. Benwing2 (talk) 20:36, 22 August 2023 (UTC)

@Benwing2 This is not a particularly simple fix, unfortunately. The issue is down to Module:affix treating the fragment as part of the term when generating the display_term. At Module:affix#L-743 part.alt is set as equal to display_term if no alt text has been specified, and this is then put into the link template. The link template will only remove the fragment if no alt text has been specified, which is why it's leaving it in place in the display. The categorisation is handled wholly within Module:affix, but it essentially boils down to the same issue with display_term fragment being treated as part of the term.

While we could patch this by handling the fragment specially within Module:affix, that's not ideal because (a) it's duplication of work, and (b) it makes the handling of fragments more fragile, because any changes will need to be changed in both places.

The core issue is ultimately the fact that we want the fragment to be handled first, but that would require much stronger integration between Module:affix and Module:links. Theknightwho (talk) 05:20, 24 August 2023 (UTC)

Thank you for looking into this. It sounds to me like we need a way of calling into Module:links to remove the fragment before processing categories and such. I think the cleanest way of doing this is to have an additional entry point in Module:links that parses off the fragment and returns it, and then allow the fragment to be specified as an additional field in the data structure sent to full_link(). This is a bit similar to how you can specify alt text either in the link itself using ] or through a separate field. If you can add the entry point in Module:links, I can fix up Module:affix to use it. Benwing2 (talk) 05:40, 24 August 2023 (UTC)

@Benwing2 I've separated it out as get_fragment, which can be called externally. You can specify the fragment in the input data by using the fragment key. It's not ideal, but it'll do for now. Theknightwho (talk) 08:02, 24 August 2023 (UTC)

I reverted your addition of a bunch of combining character stuff to Module:utilities/data because it was causing ~ 25 more memory errors. This isn't the first or second (or third) time this has happened; can you *please* try to avoid this happening in the future? I don't understand what the purpose of the combining char table is, and why it needs to be in Module:utilities/data; you haven't documented it anywhere and it seems unnecessary. In general you seem very cavalier about memory errors, but they cause a lot of problems for users. Benwing2 (talk) 03:31, 23 August 2023 (UTC)

For the record, CAT:E now has 70 entries when yesterday it had ~ 45. Some example new entries: donkey, fax, film, shoe, rod, soft, which contain almost exclusively English entries; also some others like mal, mala, z that are non-English. Benwing2 (talk) 03:36, 23 August 2023 (UTC)

@Benwing2 It's a replacement for the table currently in Module:headword/data which will allow me to move some intensive stuff out of Module:headword, but it's a multi-stage process. It was never intended to be like that for any significant period of time. Theknightwho (talk) 03:37, 23 August 2023 (UTC)

You gotta push changes like this all at once if they're gonna cause a bunch of memory issues. Benwing2 (talk) 03:50, 23 August 2023 (UTC)

I noticed you created a bunch of categories like Category:Classical Syriac entries with topic categories using raw markup whose description says "use {{c}}". I have a script to templatize categories, and I've been using {{C}} instead of {{c}} when converting and canonicalizing topic categories. It doesn't make a lot of difference but maybe we should be consistent here. Benwing2 (talk) 19:39, 25 August 2023 (UTC)

@Benwing2 Thanks, yeah, I'll change that. It'd be good to get these converted en masse, but the categories should make keeping an eye on them straightforward. Theknightwho (talk) 19:43, 25 August 2023 (UTC)

full_link() does not appear to be respecting a fragment passed in. I think the problem is around line 812, which isn't setting the fragment in the split parts of a link. Benwing2 (talk) 02:50, 26 August 2023 (UTC)

BTW this is based off of my recent code change to Module:affix, which should fix the fragment processing but the result on pages like biztos is that the fragment is being ignored. Benwing2 (talk) 02:59, 26 August 2023 (UTC)

Did you have a chance to look at this? It is affecting all terms that use fragments, in addition to the other bug mentioned in the Grease Pit related to parsing fragments. Benwing2 (talk) 17:48, 27 August 2023 (UTC)

@Benwing2 I'm just fixing a tricky bug in Module:ja-translit and then I'll be on it. Theknightwho (talk) 18:15, 27 August 2023 (UTC)

Thanks! Benwing2 (talk) 18:22, 27 August 2023 (UTC)

@Benwing2 Hiya - this is done. I tried to look into what was causing the other bug with compounds (e.g. Percolozoa +‎ -an). It seems to occur when there's a fragment nested inside a raw link, which is why it always fails with nested templates: (i.e. {{af|en|{{l|mul|Percolozoa}}|-an}} is equivalent to {{af|en|]|-an}}). This seems to have been introduced with your recent changes to Module:affix, so it's possibly better for you to have a look at it. Theknightwho (talk) 16:49, 29 August 2023 (UTC)

Hi. Thanks for fixing the first bug. The second bug is definitely a bug in get_fragment() in Module:links. Module:affix simply calls get_fragment() on each term and it's clearly looking for # in all circumstances instead of ignoring fragments inside of links, which it should do. Benwing2 (talk) 18:16, 29 August 2023 (UTC)

@Benwing2 Thanks - will deal. I wasn't sure where it was originating from. Theknightwho (talk) 18:20, 29 August 2023 (UTC)

Can you help me understand why {{ja-usex}} exists? I would like to add |origtext= as a parameter similarly to what I did with {{usex}}/{{quote}}/{{quote-*}} but I don't really understand why there's a separate Japanese version at all, much less one that doesn't even use Module:usex as its underpinnings. Benwing2 (talk) 20:00, 27 August 2023 (UTC)

@Benwing2 I think it's plugged into Japanese transliteration (and possibly other Japanese-specific things), but I don't really know. We'll probably need to add Japanese transliteration to the main modules before we can make it redundant. Theknightwho (talk) 20:39, 27 August 2023 (UTC)

@Benwing2 There's also Module:ryu-usex, which is a shitty fork from a few years ago. I've been slowly deprecating/deleting the Okinawan forks as the process of converting the Japanese modules into Japonic modules advances, but haven't tackled that one yet. Theknightwho (talk) 00:20, 28 August 2023 (UTC)

Yes, so much shitty CJK code. There's also Module:Quotations, which is somewhat questionable and doesn't use Module:usex. The least we can do is use Module:usex for formatting quotations/usage examples everywheree; it should have all the necessary functionality now and if not we can add what's needed. Benwing2 (talk) 00:39, 28 August 2023 (UTC)

作, 割, 勝, 新 and 有 are all in CAT:E because of a Module:ja bug. Chuck Entz (talk) 00:34, 28 August 2023 (UTC)

+ 清 and 食 Chuck Entz (talk) 00:43, 28 August 2023 (UTC)

@Chuck Entz I'm working on it - the inputs seem strange, though. Theknightwho (talk) 00:52, 28 August 2023 (UTC)

@Chuck Entz This is fixed - it’s because text with tag formatting was being passed through. Theknightwho (talk) 01:36, 28 August 2023 (UTC)

Great! Now it's ゞ and ヾ... Chuck Entz (talk) 01:38, 28 August 2023 (UTC)

I would like to see what in your judgement merits banning me from the Wiktionary namespace. —Justin (koavf)❤T☮C☺M☯ 23:13, 28 August 2023 (UTC)

@Koavf I warned you that I would block you if you continued to be disruptive, and you decided to be disruptive anyway, which went over the line when you decided to derail the discussion by accusing @Vininn126 of personal attacks. He was absolutely correct that you'd changed the subject away from your misbehaviour as an admin, which you conveniently ignored, and it came off as textbook evasion. Given your relentless habit of sealioning in every discussion, I don't think there's much point in me trying to convince you it was justified. I will not be responding further. Theknightwho (talk) 23:21, 28 August 2023 (UTC)

lol, okay: so you're not answering the question again. Looking forward to you answering the questions I asked in that thread. —Justin (koavf)❤T☮C☺M☯ 23:49, 28 August 2023 (UTC)

@Koavf Alright, given you clearly do want a response: the next time you tell someone they've ignored your question in response to them answering it directly, I will block you again for a month. This is exactly the kind of disruptive rubbish that amounts to sealioning, because you consistently refuse to engage in a reasonable way if people don't agree with you. Theknightwho (talk) 23:58, 28 August 2023 (UTC)

lol, I didn't do that. I asked which diffs justify me being banned and you responded with a diff of you writing that you'll ban me. The question was: which diffs of edits I made justify me being banned. You're a funny guy. Looking forward to your answers to the questions I asked on the Beer Parlour. —Justin (koavf)❤T☮C☺M☯ 23:59, 28 August 2023 (UTC)

There were around 39 yesterday in CAT:E and now there are 49. The additions all appear to be single CJK chars. Could it be due to this? Benwing2 (talk) 23:50, 29 August 2023 (UTC)

@Benwing2 It’s more complex than that and unlikely to be down to just that change, because the Japanese modules were using a hodge-podge of functions with overlapping purposes, and I’ve been unifying them. Rolling it back in isolation will break some templates that rely on {{xlit}}, so I’ll see what I can do to reduce the load. Theknightwho (talk) 00:34, 30 August 2023 (UTC)

Hai there! I'm Peace, and I'm currently a moderator of mswikt, right now upgrading our really out-of-date modules to match the ones here in enwikt. I have some issues with Module:category tree (especially the poscatboiler part) that I can't seem to solve. One thing that bugs me is this error here; I think the topic cat part of the category tree is fine. Can you help me identify what's the issue? I assume it's either an issue of poscatboiler/data/languages or Module:families (and/or its sister modules). Thanks!

Perhaps later on, I can ask you more questions on my wiki module upgrades as I progress. PeaceSeekers (talk) 11:20, 30 August 2023 (UTC)

@PeaceSeekers Hi - thanks for the message. I can certainly look at this at some point, but there'll be a delay as I'm really busy with some other things at the moment. Theknightwho (talk) 08:41, 1 September 2023 (UTC)

Take your time! Anyway, I update some of the modules (plus, adding some missing modules), so the errors are a bit different now. Here are some issues I found, among others:

• Regarding template:inherited, it seems that the template is unable to pick up proto-language codes inserted (things are okay with sister templates like template:bor, i.e. non-reconstructed language codes), but I checked the language module datas and it seems fine.
• I found out that poscatboiler seems to can't pick up {{{langcat}}}.
• May I know where in the modules, is the code that set up the "lang + label" format of poscatboiler categories to suit our format at mswikt as I can't seem to find it?

Thanks! PeaceSeekers (talk) 10:01, 1 September 2023 (UTC)

Hi. There's a bug in Module:links; embedded_language_links() has data.sc param that's a script object, but it calls process_embedded_links(), which tries to access data.sc as if data.sc is an array of script objects. In general I'm trying to make sense of Module:links (which you've completely rewritten), and it needs some cleanup; I can't figure out what some of the functions do, they're not well documented, and it appears there isn't good separation of concerns. This suggests it need some major refactoring. BTW what about the still-extant bug mentioned above in "explicit fragment handling"? It's been almost 3 days since that bug has been reported and it doesn't seem it should be hard to fix. Benwing2 (talk) 13:32, 30 August 2023 (UTC)

@Benwing2 I was actually writing a comment to you about these very issues, but it essentially boils down to the same thing. Once I've cleared up the obvious outstanding bugs I'm going to do a major rewrite, because the current model's got a lot of problems. My experience with the wikitext parser means I'm in a much better position than I was when I originally wrote these.

I've been thinking that we probably want to use string mixins for the text input, which would allow us to avoid the limitations of the string library. This would need to be done carefully, to avoid making language-specific stuff too complicated for the average editor to specify, but it would massively simplify the workload. Lua is extremely capable for this sort of string manipulation, so it would be good to take proper advantage of that. Theknightwho (talk) 13:41, 30 August 2023 (UTC)

Thanks. What do you mean about "string mixins"? Benwing2 (talk) 13:44, 30 August 2023 (UTC)

@Benwing2 Character arrays that represent strings; there's probably a better term for them. You then iterate over the array and manipulate it with a lot more freedom than you can with the string libraries. If you really get under the hood, this is actually what the string libraries are already doing; literally in the case of ustring, while the native string library does it in C. However, they don't have things like lookaheads/lookbehinds, or-operators for strings of arbitrary length etc. Theknightwho (talk) 13:55, 30 August 2023 (UTC)

@Theknightwho Hmm, that is possible but IMO it's likely to be slow and hard to use in comparison with Lua patterns. The ustring libraries convert to PHP regexes, which use C under the hood. Nothing is being done in raw Lua; it would be really slow. Benwing2 (talk) 14:26, 30 August 2023 (UTC)

@Benwing2 It's how the parser works, and I've found it's extremely fast. With the newest optimisations, the parser now manages that stage of a in 0.15 seconds. It's the sort of thing Lua does best: it's about 10 times faster than ustring, and comparable to string (slower for raw characters, but faster for character classes). Theknightwho (talk) 14:28, 30 August 2023 (UTC)

Just FYI what you are describing are exploded strings. "Mixins" refer to something entirely different; see mixin. Benwing2 (talk) 08:53, 31 August 2023 (UTC)

@Benwing2 Thanks - I got my wires crossed. I've made some decent progress on this, using mwparserfromhell as a base. Performance still needs to be optimised, but it it can currently parse internal + external links plus HTML tags, which are the trickiest parts. Doing it this way makes a ton of the old code redundant, and also means wikitext parsing can be dealt with in a self-contained way. Should be ready in a few days, but it'll probably take a couple of weeks to rewrite Module:links and Module:languages to take advantage of it. Theknightwho (talk) 20:08, 31 August 2023 (UTC)

Sounds good. Benwing2 (talk) 21:07, 31 August 2023 (UTC)

@Benwing2 Currently needs clean-up and a load of debugging, but Module:User:Theknightwho/wikitext parser is the current prototype.

Currently handles:

• Internal links, including pipes and fragments.
• External links:
  • With brackets: or link to Google.
  • Without: https://google.com/, including proper handling of trailing punctuation; compare https://google.com/) and https://google.com/().
  • Including URIs with no slashes: tel:0
  • Even when placed inside double square brackets: ] is ].
• HTML tags, including arbitrary classes and styles. This makes nested templates trivial to handle.
  • Invalid or disallowed tags are correctly treated as raw text.
• Italics and bold.
• HTML entities, which are decoded at the end of the parse to allow onward processing.
• HTML comments.
• Strip markers.
• Lists (*, #, :, ;).
• Horizontal rules (----).
• Wikitables (of arbitrary size and complexity).

To do:

• Headings.
• Percent encoding inside links.
• Decoding <nowiki> strip markers at the end of the parse.
• Compressing multiple consecutive whitespace to a single space (i.e. what is actually displayed). This may need special consideration, as it might not always be desirable.
• Pipe trick and trailing link suffixes.
• Automatic detection of link prefixes.
• Files (]), which can take multiple pipes + parameters).
• Category and interwiki links, which need special handling.
• Colon trick (]).

Not in scope:

• Templates and parser functions.
• Arguments.
• Parser tags (e.g. <nowiki>). This means <pre> tags don’t have their nowiki property, which is the correct behaviour if they’re added via a module and not in raw wikitext; no other tags can be both.
• Special handling of HTML comments, where new lines that only contain comments are fully deleted.

Anything listed as out-of-scope is handled by the other parser I’ve been writing, which I used as the model for this one. In short, this handles everything that’s done after template expansion.

Although it can cope with very complex inputs (many of which will be rare), they don’t add unnecessary load because it’s all done in a single iteration over the string, with the relevant functions being accessed only when necessary. The current designs of Module:links and Module:languages iterate over the string (at least) once for every type of processing that they do, which is one of the reasons they’re so expensive.

In some cases, we may actually want to change the behaviour away from the default parser (e.g. treating interwiki links as normal ones, or not treating ] as ], but simply a link to sms:a instead). The current design matches the default parser exactly, since that made sense as a starting point, but some of those assumptions may not make sense for the type of text that’s going to be input to templates. Theknightwho (talk) 15:53, 2 September 2023 (UTC)

@Theknightwho All sounds good, thanks for the posting! What is the status of the other parser? Benwing2 (talk) 20:07, 2 September 2023 (UTC)

BTW your code looks pretty clean. It could do with some comments but that can come later. Benwing2 (talk) 20:08, 2 September 2023 (UTC)

@Benwing2 It’s on hold until I can work out how best to handle the template tree, which is built from the tokenised string. At the moment, it’s about 15-25% slower than than the native parser on very large pages, but I think it should be possible to reduce that.

The overall design is that it does a minimal parse when constructing the template tree of each new template that it encounters, which means that it only parses so much as is absolutely necessary (e.g. at the first pass, neither outcome from {{#if:}} is expanded). That tree is then stored, and each time the template is called, it expands the parts of the tree that are relevant to the given arguments (so if the is true, it’ll expand only the first parameter). Anything that’s permanent from that expansion (e.g. {{PAGENAME}} will never change) is supposed to be expanded in the cached tree as well, which means that any other templates that use that branch can take advantage of it. This is useful with {{t}}, for example, which checks the namespace every time it’s called to see if it needs to be categorised. Ideally, that check would only be done once, and later calls would simply get the appropriate bit to parse, bypassing that check. That’s the part that’s proving tricky to do, because it requires decoupling the specific pass from the cached tree in a way that doesn’t trash the cache or force the pass to do more work than it needs to.

If I can get it right, it will greatly reduce the work required for templates that make heavy use of {{#switch:}} etc. Theknightwho (talk) 00:47, 3 September 2023 (UTC)

@Theknightwho Sounds something like judicious use of thunks or memoization is needed. Let me know if you need any help. Benwing2 (talk) 01:14, 3 September 2023 (UTC)

@Benwing2 Yes, thunks sound like exactly what I need.

In terms of the postprocessor, percent encoding, headings and nowiki tags are now working, but there are a ton of weird and wonderful edge cases that need testing across the board.

I've also done some experimenting with the overall design, because I was getting concerned with the number of conditions that each iteration was having to check against, to account for all the numerous possible context states. Instead, it now uses a simple loop that constantly iterates over self.handlers, with a bunch of different handler tables being swapped into that key as contextually appropriate. This means:

1. It's trivial to bypass checks which aren't relevant.
2. It keeps the code compartmentalised. This makes introducing new features much more straightforward, since there won't be a ton of obscure dependencies to be concerned about. This will be useful when it comes time to add support for things like ^ capitalisation, rubytext support, your angle-bracket syntax etc.
3. It's easy to avoid duplication, because the same handler function can be used in multiple handler tables. It's also possible to use object inheritance between handler tables, which is useful if you need to bolt on/swap out a handler temporarily, or to tell it something like "use X handlers as normal, but check Y once that's done".

One other thing to note is that Scribunto has a C stack size limit of 200, for some reason. Since the parser uses a ton of recursive xpcalls, it is actually possible to run up against it if you make the wikitext complex enough. This hasn't affected the postprocessing parser (i.e. this one) yet, but I did manage to find a real-world example which hit the limit for the preprocessor, which had 100+ nested uses of {{#if:}}. This is potentially a risk when it comes to HTML tags and/or tables, since they can be nested to arbitrary levels. Theknightwho (talk) 22:32, 9 September 2023 (UTC)

Hmm, 200 is not a big stack size. If this becomes an issue you might have to rewrite the code to use your own stack (which is always possible but sometimes a bit painful, depending on how your code is designed and how much branching there is). Benwing2 (talk) 22:59, 9 September 2023 (UTC)

BTW your use of handlers sounds great. Benwing2 (talk) 22:59, 9 September 2023 (UTC)

@Benwing2 Thanks. I got around the issue with the preprocessor by explicitly returning the BadRoute object instead of throwing/catching it, but it makes the code more complex since you need to account for every return path, and they aren’t always obvious. We’ve got a lot more moving parts here, and it was too much of a headache to keep track of. Theknightwho (talk) 23:18, 9 September 2023 (UTC)

@Benwing2 I’m making decent progress on this, though it’s a little slow as I’m trying to be thorough while also minimising overhead, which has entailed rewriting things a few times. Some design questions:

1. How closely do we want to imitate the native parser? I keep running into situations where I’m having to write special cases to account for bugs in the native parser, which is a bit of a pain. It’s always going to be necessary to some degree, but there are some really shitty hacks that feel silly to replicate, because all the functions have clearly been written by different people at different times. For example, ISBN 978-0000000000 will work (ISBN 978-0000000000), but ISBN 978-0000000000 will not (ISBN 978-0000000000), despite both entities being for the same character. There are tons more examples like this, as every function seems to roll its own regexes and iterators for everything, so things like whitespace, multibyte-characters and HTML entities get handled very inconsistently. Obviously most of them won’t matter, but there’s always a risk that something actually depends on a bug like this.
2. What would the most useful structure for the output be, and what methods should we have for processing it? At the moment, it tokenises a Wikitext string into an array, with special objects to represent specific formatting (for example, the beginning of a wikilink or the separator between an external link and its display text). However, it might be useful to have more of a tree structure, where a link, HTML tag or whatever is packaged up in its own object in the string, and can be accessed or skipped as necessary. The best structure will be dictated by the methods we want, so it’d be good to have an idea of what those should be.
3. We’ll need some way of converting the array into an output string, but do we want to be able to recreate the exact input? The native parser and mwparserfromhell both place a heavy emphasis on exact round-trip conversion, but I think we’d be better off just normalising the output, since I don’t think it matters for our purposes if <br> becomes <br/> or whatever.

Theknightwho (talk) 02:39, 17 September 2023 (UTC)

Thanks for all the work and sorry for the late reply. We should ask ourselves what the purpose of the parser is; AFAIK it's simply to reduce memory in page rendering and potentially speed it up, right? And do it less painfully than other possible solutions. That suggests that (a) we should not try to be bug-compatible with the native parser, and (b) it doesn't matter if we can recreate the input with the parsed structure since we're never going to do that (I can't imagine why they care in the native parser either; mwparserfromhell is a different story since it's specifically intended for people manipulating the wikitext and putting it back). I would say in general, if some edge cases are handled slightly differently, that's fine; people don't create pages by reading the (nonexistent) parser spec and coding to that, instead they try to see what happens when they code things a particular way, and it's probably unlikely there are any pages depending on these really edgy buggy edge cases. (And if there are, well, there are a ton of pages that are currently broken, so it's unlikely to matter if one extra page gets slightly broken. Furthermore, we should operate by enabling the parser at first only on an allow-list (whitelist) of memory-intensive pages, and then maybe only on pages with short names, and maybe never on all pages, so the "blast radius" will be less. The more memory-intensive pages get more eyes on them, so any behavioral differences are more likely to get caught.)

As for the output structure, I'm not sure; we should definitely enumerate all the potential use cases of the parser up front, to avoid having to do major redesign work if it turns out there's a significant use case we missed. Benwing2 (talk) 18:53, 18 September 2023 (UTC)

@Benwing2 It's partly about memory reduction, but it's also about removing the need to account for wikitext while manipulating input strings, and having a consistent, reliable way to know what a given string is for. That greatly simplifies the amount of work that needs to be done in Module:links, Module:headword and Module:languages, and also allows us to do more sophisticated processing in substitution modules (translits etc).

A major side-effect of that is memory reduction, which is the reason I started doing this in the first place, but it's definitely about more than that at this point: there's a lot of potential here, if we can get it right. Theknightwho (talk) 19:09, 18 September 2023 (UTC)

Edit: just to point out that this isn't the same as the preprocessor parser, which is solely about memory reduction. The savings with this one will be relatively modest by comparison. It's a totally different approach, as it's about building a modular structure for parsing strings which we can build features into. It's about code cleanup and reducing duplicated work, as it strips away a lot of the spaghetti code we have at the moment. Theknightwho (talk) 19:14, 18 September 2023 (UTC)

I see. In that case yes your output structure should probably be hierarchical, but I think my other points still stand. Benwing2 (talk) 19:25, 18 September 2023 (UTC)

@Benwing2 I'm not sure it'd be possible to do a partial implementation, as it would act as a replacement for what's there at the moment. I'll let you know once it's in an alpha state so you can have a play around with it, but at that point we can get an idea of how the core modules would need to be modified. Currently, the output is in this format:

• Text
• Text
• Text

Where text represents any number of characters as strings. Obviously it can get a lot more complex than this, and the special tokens are tables, so can store any amount of data (e.g. for representing complex HTML tags). Theknightwho (talk) 19:34, 18 September 2023 (UTC)

You definitely need to implement this in a way that we can transition bit-by-bit, even if that requires extra effort. There's no way something this major can be cut over to in one fell swoop. Benwing2 (talk) 19:43, 18 September 2023 (UTC)

@Benwing2 Absolutely, yeah. It'll have a __tostring method, so it'd just be a case of doing that before running code that expects conventional string inputs. Theknightwho (talk) 19:50, 18 September 2023 (UTC)

@Benwing2 I’ve had some real breakthroughs with the template parser in the last few days, and it’s finally starting to outperform the native parser in terms of speed; the memory savings are still hovering around a ~75% reduction, which I suspect is a hard floor.