PO and RTL support

A while ago I added RTL text support to my wiki:

http://ikiwiki.info/tips/Right-to-left___40__RTL__41___page_text

But this support does not work with PO files. When I write a page in English, I need the Hebrew/Arabic translation to have additional text (in my case, using the template directive) which causes the direction of the text to be RTL.

I saw a recent patch which claims to solve the problem by exposing the language code and direction to the templates (which would help a lot), but when I go to the original website from which it came, it looks like the Arabic text is still aligned LTR just like English:

http://addons.nvda-project.org/index.ar.html

Another issue is that I use Debian stable, and I'm not sure it's safe to use some unstable ikiwiki (currently I use the version from backports) - advice welcome :-)

It's still important to have the ability to change direction inside the page, but the default direction specified either in CSS on in the page.tmpl file should be dynamic. I didn't check how the PO plugin works, but it may be necessary to update there, because if all it does is copy the HTML page and switch strings with translations, it must be modified to also edit the LTR/RTL directives so that different translations of the same page can have different directions.

I hope I'll have some time to look into it myself, I'm just a bit behind now with non-recent ikiwiki version (maybe it's time for me to try sid or from source).

--fr33domlover

RSS

next steps

HTML5 says:

Authors are strongly encouraged to use the dir attribute to indicate text direction rather than using CSS, since that way their documents will continue to render correctly even in the absence of CSS (e.g. as interpreted by search engines).

Could you test whether your tip works with <div dir="rtl"> or something, please? If it does, please change the tip, if not, we'll have to look at whether the htmlscrubber is getting in the way.

After that, I think the next step towards good RTL support would be to put together some test-cases for things that are meant to work, in the form of:

self-contained source code and setup file for a very simple wiki
the pages in that wiki making it clear what their intended text direction is (e.g. "this paragraph should be right to left")

As far as I know, none of the IkiWiki committers can read any RTL languages, so if you use Arabic or Hebrew or whatever in those test-cases, we'll need a screenshot/image of what it's meant to look like. Using Latin text marked as RTL (so it should come out backwards if everything is working correctly) might be easier.

The obvious cases that I can think of are:

the wiki is "mostly" in a RTL language
the master language is LTR but the po plugin provides a translation into a RTL language

and possibly

the master language is RTL but the po plugin provides a translation into a LTR language

It might be necessary to add support for a per-wiki, per-page or (for po) per-translation-language direction override that would set the <html dir> attribute, but we should find test-cases first, then we can work out solutions.

Comment by smcv — Wed Sep 17 04:09:50 2014

comment 2

If I'm interpreting that Arabic website correctly, it is RTL, but left-justified (which is a somewhat confusing CSS glitch, but hopefully not a barrier to understanding by people who can read Arabic). English words embedded in the Arabic are LTR, but my understanding of the bidi algorithm is that that's meant to happen.

For instance, in the English version, the last paragraph before the inline says:

Please feel free to subscribe to the rss or atom feeds to be informed on when new addons or a new version of an addon is made available. The following community supported addons are available:

and in the Arabic version, the last paragraph looks like this in my browser (where ***** represents Arabic that I don't know how to read):

: ***** (... lots more ....) ***** atom feeds * rss **** ****

So that looks right for RTL: the colon is at the end (left), and the mentions of rss feeds and atom feeds are at the beginning (right). When I "view source", it's the other way round.

Also, the page source says:

<html xmlns="http://www.w3.org/1999/xhtml" lang="ar" xml:lang="ar" dir="rtl">

which looks right?

Comment by smcv — Wed Sep 17 04:19:38 2014

comment 3

I saw a recent patch which claims to solve the problem by exposing the language code and direction to the templates

It looks as though you mean mhameed's change from expose html language and direction, which exposed them to the templates, but did not modify the default page.tmpl to make use of them. Perhaps you or mhameed could provide a page.tmpl patch?

Comment by smcv — Wed Sep 17 04:29:48 2014

comment 4

Could you test whether your tip works with \
or something, please?

Sure, I will check that soon. I think it does, I just tried here in ikiwiki. Just curious, why is div preferred? IIRC I use "class" there after looking at some existing templates. But I'm not an expert, especially not in CSS. Would that be used as an HTML4 parallel of the dir attribute?

As to that website with the patch, the problem is that the text is aligned to the left. When I type Hebrew in an LTR page, it already shows more or less correctly - English words are shown in correct letter order thanks to the bidi algorithm. The issue seems to be aligning to the right - that is what my tip does. Maybe the direction setting in the CSS also has other effects - I just know it works :-)

I'll happily help with the tests. I also have a test page on my wiki which uses many ikiwiki features, to demonstrate how they all look in RTL. Test case ideas:

Page in RTL (e.g. Arabic) with an LTR paragraph (e.g. English)
Page in RTL with LTR paragraph in the same language (e.g. fancy way to write a poem)
Page in LTR (e.g. English) with an RTL paragraph (e.g. Hebrew)
Page in LTR with RTL paragraph in the same language (poem again)
Translated page - master language is LTR, slave is RTL
Translated page - master language is RTL, slave is LTR
Master LTR page has RTL paragraph, all slaves have it RTL too regardless of their global direction
Master RTL page has LTR paragraph, all slaves have it LTR too regardless of their global direction

An example for the last 2 tests is an English master page about linguistics which has a paragraph in some RTL language that is being studied, and all slave pages must keep that paragraph intact - both the text itself and its RTL direction. But the rest of the page can be translated and correctly made RTL when translated to RTL languages.

This gives me another idea - most of the time what you actually mean is to reverse the direction: RTL becomes LTR and vice versa. When writing some fancy poem, that's what you probably want. But in the previous example, the direction should not be reversed - so there should maybe be two kinds of direction modifiers:

Dynamic (the default) - You write e.g. a master page in LTR and some RTL paragraphs. an RTL translation automatically reverses directions, RTL <=> LTR.
Fixed - this is like my tip, e.g. An RTL paragraph in an LTR page has a fixed direction set, which is kept even in translations for RTL languages - the page in general is reversed, but that paragraph is not.

Another very useful thing (at least to me) would be an option to have different wiki pages/section with different master languages. I have sections in English and sections in Hebrew, which makes the PO plugin a problem to use, unless I keep one of these sections untranslated.

Comment by fr33domlover — Wed Sep 17 07:22:57 2014

comment 5

<div> is not specifically preferred, any block-level element will do (e.g. <p>); but <div> is something you can wrap around any block, so it's good for a generic \[[!template]].

The difference between the use of a dir attribute and the use of a class attribute is that dir has a spec-defined semantic meaning in HTML4 and HTML5: search engines can look at <div dir="rtl"> and know that it is definitely right-to-left.

<div class="rtl"> might mean right-to-left, but it could equally well mean (for instance) documentation about a run-time library, or something; classes have no built-in semantic meaning that generic user-agents like browsers and search engines can rely on.

Comment by smcv — Wed Sep 17 07:35:07 2014

comment 6

smcv wrote:

As far as I know, none of the IkiWiki committers can read any RTL languages

I read Hebrew well enough to detect chirality errors (e.g., L-Hebrew in an R-Hebrew universe). --schmonz

Comment by Amitai — Wed Sep 17 10:24:38 2014

comment 7

LTR with embedded RTL, or vice versa, sounds like a job for the ?em>page text tip or something very similar.

Maybe the direction setting in the CSS also has other effects

https://html.spec.whatwg.org/#the-dir-attribute suggests that the dir attribute is meant to be sufficient, but perhaps it's overridden by an explict text-align: left?

most of the time what you actually mean is to reverse the direction: RTL becomes LTR and vice versa

I don't think "I know I am switching between English and Arabic, but I don't know which one I'm currently writing" is a major use-case :-)

an option to have different wiki pages/section with different master languages

It sounds as though the po plugin is not really what you want, and you'd be better off with being able to write \[[!meta lang=ar dir=rtl]] or something.

Comment by smcv — Wed Sep 17 11:52:36 2014

comment 8

I don't think "I know I am switching between English and Arabic, but I don't know which one I'm currently writing" is a major use-case

Sure you need to know, but there's a difference between "make the whole text RTL" or "make the LTR text RTL and make the RTL text LTR". It depends on what the user means.

It sounds as though the po plugin is not really what you want, and you'd be better off with being able to write [[!meta lang=ar dir=rtl]] or something.

Yes, that would help. But I'd need the PO plugin to respect this lang tag. Is this already possible?

Could you test whether your tip works with
or something, please?

I did and it works, but there's an issue: In both ways - my CSS and the dir attribute - some things don't work right, for example, right-aligned floating boxes need to be made left-aligned. How do I handle this with dir? I don't know CSS, but I think it may be possible to define CSS for various page elements differently when in the ".rtl" class - am I right? Another problem is that the horizontal bars of polls (poll plugin) still go LTR. Is there a way to fix these things without CSS class? With the notebox template, the right-alignment uses CSS so if we want to use 'dir' it needs to be changed to something non-CSS with correct semantics. Maybe if you could set dir to "flip" instead of "rtl" and "ltr", to mean "the direction opposite to the page's direction". And that new 'dir' would also need to control alignment, since right now notebox is not affected by the dir like I said. The text inside does become RTL but the box is still on the right like in LTR.

I don't see other issues but there are plugins I didn't try, e.g. does the box generated by pagetstats directive align to the left in RTL?

If CSS isn't needed for this I'll update the tip, otherwise I should probably fix these issues (notebox-rtl and poll) and add this to the CSS in the tip, until there is a better solution (and even then, people will be running previous ikiwiki versions without the solution).

Comment by fr33domlover — Sat Sep 20 08:50:13 2014

comment 9

Authoring HTML: Handling Right-to-left Scripts might be useful reading.

In modern browsers (i.e. not IE6) it should be possible to make CSS conditional on any attribute, not just class, so you could maybe do something like this:

/* ikiwiki's existing CSS */
.sidebar {
    float: right;
    margin-left: 4px;
}

/* new */
html[dir="rtl"] .sidebar {
    float: left;
    margin-left: 0px;
    margin-right: 4px;
}

If you contributed a patch for style.css to make markup like this "just work", I'd be happy to review it. (notebox could use the same technique).

Comment by smcv — Sun Sep 21 08:41:56 2014

comment 11

Yes, [a RTL meta tag] would help. But I'd need the PO plugin to respect this lang tag. Is this already possible?

Do you need the po plugin at all? The po plugin is specifically for sites that are written in a master language (usually English) and then translated into a bunch of other languages - the same general approach as https://www.debian.org/ (that site does not use IkiWiki but the idea is the same).

If you're selecting languages in some other way - e.g. all your content is in Arabic except that the /programming/ subtree is in English, or something like that - then the po plugin is not designed for what you're doing, and adding support for a new meta tag to the meta plugin would be a better way to get the language code into the header. It could use the same TMPL_VAR hooks in page.tmpl that po does? I'd review a patch.

Comment by smcv — Sun Sep 21 08:50:11 2014

comment 11

Maybe if you could set dir to "flip" instead of "rtl" and "ltr"

This feature does not exist in HTML. The allowed values for dir are rtl, ltr, and auto (where auto means "please guess based on the content" - see the HTML5 spec for the exact algorithm used).

Comment by smcv — Sun Sep 21 09:06:24 2014

comment 12

As to exposing the language tag, I was told here that there is a patch already:

Right-to-left support

The CSS should requires that I modify my local.css to use the conditional instead of an "rtl" class. For that I need to understand on which items it affects (and just insert direction=rtl there, like I'm doing now with the class).

When I make the changes in my wiki and test them, I'll send you a patch.

Comment by fr33domlover — Wed Oct 22 12:46:01 2014

comment 13

I was told here that there is a patch already

That patch is for the po plugin, which is specifically designed for a wiki in which every page foo is written in a "master language" (often English) in a file like /foo.mdwn, and then translated into secondary languages via translation files like /foo.ar.po.

If that doesn't describe your wiki, then the po plugin is not intended for you, and you would be better off with a change to the meta plugin to make it possible to emit the same language and/or direction attributes in the HTML, but triggered by different source code.

Comment by smcv — Thu Oct 23 04:06:48 2014

Add a comment