recently fixed bugs

This might not be the same cause or solution as git test receive wrapper fails so filing it separately.

Anon git push is broken again

remote: error: cannot lock ref 'HEAD': Unable to create '/home/b-ikiwiki/source.git/./HEAD.lock': Permission denied
To git://git.ikiwiki.info/
! [remote rejected]     master -> master (failed to update ref)


Jon, 2020-10-06

It does not seem related to the other problem. I don't understand how it could have broken in a new way since I fixed it before. git has not been upgraded in the meantime. This is affecting more than one site, and the permissions do not seem obviously broken.

Since this is now seeming so fragile -- after working for about a decade, it's broken twice in a matter of months -- I'm questioning whether it's worth trying to keep the feature working. --Joey

Of course, that's got to be your call. I haven't made a great deal of use of it, but it does seem more convenient if one is working on an IkiWiki patch, as I can write the website notes about it in the same tree (although I then have to cherry-pick to push that to the live site, of course.). If you decide to drop it from ikiwiki.info, would you leave the code as-is, or drop it as broken? Any further clues what went wrong this time? Jon, 2021-01-13

The HEAD.lock permissions error does not come from the post-receive hook or from ikiwiki when it runs it. Instead, stracing git-daemon shows that it happens after ikiwiki has checked the push and accepted it, and exited successfully.

So the problem is that git-daemon is unable to write to the git repo.

drwxr-sr-x+ 8 b-git-annex b-git-annex 4096 Mar 29 17:07 /home/b-git-annex/source.git/


ikiwiki-hosting's ikisite is supposed to arrange for git-daemon to be able to write there by using an ACL, so something about that must be what is broken now. --Joey

Ok, found the problem. ikisite runs setfacl, and that does the right thing. Then ikisite runs chmod on the source.git directory (to make it suid as seen above). That seems to mess up the ACLs that were set earlier.

The diff from good to bad ACLs, as shown by getfacl is:

-group::rwx
-group:ikiwiki-anon:rwx
-group:b-ikiwiki:rwx
-group:b-git-annex:rwx
+group::rwx #effective:r-x
+group:ikiwiki-anon:rwx #effective:r-x
+group:b-ikiwiki:rwx    #effective:r-x
+group:b-git-annex:rwx  #effective:r-x


I don't understand ACLs or how a chmod could clear them but ok, ikisite needs to chmod before setting the ACLS.

This is not an ikiwiki bug and I'll fix it in ikisite-hosting then. done --Joey

Posted Tue Oct 6 08:46:43 2020

Hi,

XML error:

Created <time datetime="2009-03-24T18:02:14Z" pubdate class="relativedate" title="Tue, 24 Mar 2009 14:02:14 -0400">2009-03-24</time>


The pubdate REQUIRES a date, so e.g. pubdate="2009-03-24T18:02:14Z"

No, pubdate="pubdate". It's a boolean attribute. applied && done --Joey

awesome, thanks for fixing my fix --simonraven

This seems to be happening either still or again with version 3.20200202.3-1. I'm getting strings generated like

Posted <time datetime="2007-12-06T05:00:00Z" pubdate="pubdate">Thu 06 Dec 2007 12:00:00 AM EST</time>

which shows up as an error on https://validator.w3.org/ --Luke Schierer

My reading of Joey's response, above, was that (according to the spec at the time), pubdate="pubdate" is what should be generated, not pubdate="timestamp", and so what you are seeing is expected. However, looking at the current Spec (linked elsewhere in this page), pubdate is not actually a valid attribute any more at all. And indeed, running my own blog through the Validator, I see:

Error: Attribute pubdate not allowed on element time at this point. Jon, 2020-10-05

I've filed a separate bug page for this, since this one is already marked done: pubdate not valid for html5. I've filed a patch. Jon, 2020-10-06

Otherwise the XML parser chokes.

http://www.whatwg.org/specs/web-apps/current-work/multipage/text-level-semantics.html#attr-time-pubdate

(indented exactly 4 spaces)

    diff --git a/IkiWiki.pm b/IkiWiki.pm
index 1f2ab07..6ab5b56 100644
--- a/IkiWiki.pm
+++ b/IkiWiki.pm
@@ -1004,7 +1004,7 @@ sub displaytime ($;$$) { my time=formattime(_[0], _[1]); if (config{html5}) { return '<time datetime="'.date_3339(_[0]).'"'. - (_[2] ? ' pubdate' : ''). + (_[2] ? ' pubdate="'.date_3339(_[0]).'"' : ''). '>'.time.'</time>'; } else { diff --git a/IkiWiki/Plugin/relativedate.pm b/IkiWiki/Plugin/relativedate.pm index fe8ef09..8c4a1b4 100644 --- a/IkiWiki/Plugin/relativedate.pm +++ b/IkiWiki/Plugin/relativedate.pm @@ -59,7 +59,7 @@ sub mydisplaytime (;$$) { if ($config{html5}) {
return '<time datetime="'.IkiWiki::date_3339($time).'"'. - ($pubdate ? ' pubdate' : '').$mid.'</time>'; + ($pubdate ? ' pubdate="'.IkiWiki::date_3339($time).'"' : '').$mid.'</time>';
}
else {
return '<span'.$mid.'</span>';  Posted Sat May 8 19:42:20 2010 Table directive can not deal with Chinese, when format csv [[!table format=csv data=""" a,b,c 1,2,你好 """ ]]  But the below example works well. [[!table format=csv data=""" a,b,c 1,2,3 """ ]]  The below example works well too [[!table format=dsv delimiter=, data=""" a,b,c 1,2,你好 """ ]]  You don't say what actually happens when you try this, but I hit something similar trying unicode symbols in a CSV-based table. (I wasn't aware of the DSV work-around. Thanks!) The specific error I get trying is [\[!table Error: Cannot decode string with wide characters at /usr/lib/x86_64-linux-gnu/perl/5.24/Encode.pm line 243.]]  That file is owned by the libperl5 package, but I think I've seen an error mentioning Text::CSV i.e. libtext-csv-perl when I've encountered this before. -- Jon A related problem, also fixed by using DSV, is messing up the encoding of non-ASCII, non-wide characters, e.g. £ (workaround was to use &pound; instead) -- Jon Sorry, I have faced the same error: [[!table Error: Cannot decode string with wide characters at /usr/lib/x86_64-linux-gnu/perl/5.24/Encode.pm line 243.]] -- ?tumashu1 The below patch seem to deal with this problem: From d6ed90331b31e4669222c6831f7a0f40f0677fe1 Mon Sep 17 00:00:00 2001 From: Feng Shu <tumashu@163.com> Date: Sun, 2 Dec 2018 08:41:39 +0800 Subject: [PATCH 2/2] Fix table plugin can handle UTF-8 csv format --- IkiWiki/Plugin/table.pm | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) From ad1a92c796d907ad293e572a168b6e9a8219623f Mon Sep 17 00:00:00 2001 From: Feng Shu <tumashu@163.com> Date: Sun, 2 Dec 2018 08:41:39 +0800 Subject: [PATCH 2/2] Fix table plugin can handle UTF-8 csv format --- IkiWiki/Plugin/table.pm | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/IkiWiki/Plugin/table.pm b/IkiWiki/Plugin/table.pm index f3c425a37..7fea8ab1c 100644 --- a/IkiWiki/Plugin/table.pm +++ b/IkiWiki/Plugin/table.pm @@ -135,6 +135,7 @@ sub split_csv ($$) { my csv = Text::CSV->new({ sep_char => delimiter, binary => 1, + decode_utf8 => 1, allow_loose_quotes => 1, }) || error("could not create a Text::CSV object"); @@ -143,7 +144,7 @@ sub split_csv ($$) { foreach my$line (@text_lines) {
$l++; if ($csv->parse($line)) { - push(@data, [ map { decode_utf8$_ } $csv->fields() ]); + push(@data, [$csv->fields() ]);
}
else {
debug(sprintf(gettext('parse fail at line %d: %s'),
--
2.19.0


Thanks, I've applied that patch and added test coverage. done --smcv

I can confirm that the above patch fixes the issue for me. Thanks! I'm not an ikiwiki committer, but I would encourage them to consider the above. Whilst I'm at it, I would be really grateful for some input on support multi-row table headers which relates to the same plugin. Jon

I've hit this bug with an inline-table and 3.20190228-1 (so: patch applied), with the following definition

[[\!table class=fullwidth_table delimiter="      " data="""

I (HB1), 70 (PB1), 5 (PB50)     Dune    O       ✓"""]]


I'm going to attempt to work around it by moving to an external CSV. ­— Jon

What version of Text::CSV (Debian: libtext-csv-perl) are you using? What version of Text::CSV::XS (Debian: libtext-csv-xs-perl) are you using, if any?

I could't reproduce this with libtext-csv-perl_2.00-1 and libtext-csv-xs-perl_1.39-1, assuming that the whitespace in delimiter="..." was meant to be a literal tab character, and that the data row has literal tabs before Dune, before O and before ✓.

It would be great if you could modify t/table.t to include a failing test-case, and push it to your github fork or something, so I can apply it without having to guess precisely what the whitespace should be. --smcv

Sorry, I appreciate as bug reports go my last post was not that useful. It's serving as a sort-of personal placeholder to investigate further. The issue can be seen live here, the source is here. The web servers versions are ikiwiki 3.20190228-1, libtext-csv-perl 1.33-2 and libtext-csv-xs-perl is not installed. I'll do some futher diagnosis and poking around. ­— Jon

OK: issue exists with oldstable/Stretch, and is seemingly fixed in stable/Buster. libcsv-text-xs-perl doesn't seem to matter (presence or absence doesn't change the bug). Upgrading just libtext-csv-perl on a Stretch host with Buster's 1.99-1 is not sufficent to fix it. As per the error, I think libperl5 might be relevant, i.e. bug present in 5.24.1-3+deb9u5 and fixed by 5.28.1-6.

EDIT: yes indeed merely upgrading to libperl5.28=5.28.1-6 in stretch fixes the issue. — Jon

Post Buster-upgrade, and it's still broken on my webhost, which shows [\[!table Error: Wide character at /usr/lib/x86_64-linux-gnu/perl/5.28/Encode.pm line 296.]] with libperl5.28:amd64 5.28.1-6, libtext-csv-perl 1.99-1 and libtext-csv-xs-perl 1.38-1. Further fiddling will commence. (removing libtext-csv-xs-perl does not help.) — Jon

Still hitting this table bug, using ikiwiki=3.20200202.3, buster system, no libtext-csv-perl installed, libperl5.28:amd64 5.28.1-6, no libtext-csv-xs-perl. Workaround is to set format=tsv, even though I'm overriding the delimiter to \t. This reproduces easily in the quay.io/jdowland/opinionated-ikiwiki container FWIW. Jon, 2020-06-26

Posted Mon Nov 26 04:44:34 2018

Getting this when a git push to git:// runs the pre-receive hook which is set up by the git_test_receive_wrapper:

remote: fatal: Not a git repository (or any of the parent directories): .git
remote: 'git log --pretty=raw --raw --abbrev=40 --always -c -r 21161ba01a093534ef97188eae098d83554dbcc6..73820a1d7e76318d8b1ac23e1c6d47e50a3e8ca2 --no-renames -- .' failed:
To git://git-annex.branchable.com/
! [remote rejected]     master -> master (pre-receive hook declined)
error: failed to push some refs to 'git://git-annex.branchable.com/'


Relevant code:

            # Avoid chdir when running git here, because the changes
# are in the master git repo, not the srcdir repo.
# (Also, if a subdir is involved, we don't want to chdir to
# it and only see changes in it.)
push @rets, git_parse_changes('.', 0, git_commit_info('.', $oldrev."..".$newrev));


This is with git 1:2.11.0-3+deb9u2 on debian stable, ikiwiki 3.20171002.

Tossing a call to pwd in there, it's at the top of the master (bare) git repository, which seems right. I can do a similar git log at that location manually (using different revs). Looking at the environment at that point (in another wiki that has the same problem), I found only these git env vars:

remote: GIT_ALTERNATE_OBJECT_DIRECTORIES=/home/b-joeyh/source.git/./objects
remote: GIT_OBJECT_DIRECTORY=/home/b-joeyh/source.git/./objects/incoming-hVfXvD
remote: GIT_QUARANTINE_PATH=/home/b-joeyh/source.git/./objects/incoming-hVfXvD


[[!commit 6fb43c29f63b85c3424520819427903e5a204426]] is relevant to that, and I guess it didn't fully solve the problem. --Joey

Stracing git-daemon -f I noticed this:

[pid 22616] lstat64("/home/b-ikiwiki/source.git/HEAD", {st_mode=S_IFREG|0664, st_size=23, ...}) = 0
[pid 22616] openat(AT_FDCWD, "/home/b-ikiwiki/source.git/HEAD", O_RDONLY|O_LARGEFILE) = 3
[pid 22616] read(3, "", 232)            = 0
[pid 22616] close(3)                    = 0
[pid 22616] lstat64("/home/b-ikiwiki/source.git/commondir", 0xbf83896c) = -1 ENOENT (No such file or directory)
[pid 22616] access("/home/b-ikiwiki/source.git/./objects/incoming-gXNPXm", X_OK) = -1 EACCES (Permission denied)
[pid 22616] stat64("/home/b-ikiwiki", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0


So the git diff is in the right cwd, it gets as far as reading HEAD. But then this permissions error on this incoming directory happens, and it then seems to give up and search for a different git repo to use in the parent directory (and all the way up to root).

The directory is created by git earlier in the strace:

[pid 22559] mkdir("./objects/incoming-gXNPXm", 0700) = 0


And here's how it looks:

drwx------+ 7 ikiwiki-anon ikiwiki-anon 4096 Jun 14 00:22 incoming-y6a8pe/


And I think that's the problem, by the time ikiwiki runs it's switched away from the ikiwiki-anon user that git-daemon uses, and over to the site user. Which can't read that.

source.git has an ACL set to let ikiwiki-anon write to it.

ikisite:        eval { shell("setfacl", "-R", "-m", "d:g:$config{gitdaemonuser}:rwX,d:g:$user:rwX,g:$config{gitdaemonuser}:rwX,g:$user:rwX", "$home/source.git") };  Can this ACL be adjusted so that all directories created under it will be readable by the site user (b-ikiwiki)? I don't know ACLs very well. Alternatively, GIT_QUARANTINE_PATH is set to the directory, so the C wrapper could fix up its permissions. The wrapper is suid, so either would need to switch user ID back to ikiwiki-anon, if that's allowed, or there would need to be an outer wrapper that's not suid (just a shell script would work) that then runs the regular suid wrapper. This was not a bug in ikiwiki, but ikiwiki-hosting. Fixed there (using the wrapper wrapper approach). done --Joey Posted Mon Jul 2 13:24:34 2018 I edited some pages on the ikiwiki ikiwiki (shortcuts and ikiwikiusers). The edits show up in RecentChanges and History, but not in the compiled pages. --JoshTriplett Well, I seem to have fixed this now (crossed fingers) --Joey Looks fixed. Out of curiosity, what caused the problem? --JoshTriplett Looks like a build died halfway through, so it was stumbling over rendered html pages that it didn't have record of. I don't know what build failed exactly. --Joey Has this just happened again? datearchives-plugin is now exhibiting the same symptoms -- it's in the repository and RecentChanges, but the actual page is 404. --Ben Yes, it seems to have happened again. Added debugging to track it down next time it occurs. It seems to be happening when you add things to patchqueue. --Joey Got it, it seems that htperestradier was dying and this was killing ikiwiki before it could save state filed && done, for real this time. --Joey Posted Mon Feb 19 01:40:38 2007 Hello, I have just updated an ikiwiki installation of mine from the master branch in 2014 to the current master (430f69034f7c9f64325ef48da3b3eaf0d685dcc5), and found out that the the footnote support with MultiMarkdown is broken. After some investigation, I believe that the breakage was introduced in 4db4e589e4c73f076b666a77b86743695454a3ce, with Discount support for footnotes. The conditional for the footnote support in MultiMarkdown is inverted. I've been running with this patch to fix it: From 8c624b3cc67bf41b3987a27b15a8dee5fe1087f7 Mon Sep 17 00:00:00 2001 From: Giuseppe Bilotta <giuseppe.bilotta@gmail.com> Date: Thu, 29 Aug 2019 16:51:45 +0200 Subject: [PATCH] Fix inverted conditional for multimarkdown footnote support --- IkiWiki/Plugin/mdwn.pm | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/IkiWiki/Plugin/mdwn.pm b/IkiWiki/Plugin/mdwn.pm index eefa29a97..cff4204ef 100644 --- a/IkiWiki/Plugin/mdwn.pm +++ b/IkiWiki/Plugin/mdwn.pm @@ -77,9 +77,7 @@ sub htmlize (@) {$markdown_sub=sub {
my %flags=( use_metadata => 0 );

-                   if ($config{mdwn_footnotes}) { -$flags{disable_footnotes}=1;
-                   }
+                   $flags{disable_footnotes}= not$config{mdwn_footnotes};

Text::MultiMarkdown::markdown(shift, \%flags);
}
--
2.21.0.595.g6364b05174


done

Applied (along with some accompanying tests) in e642784. Thanks for finding and fixing the bug. --schmonz

Posted Fri Aug 30 01:32:33 2019 Tags:

Please consider this patch for merging in.

From e697ba4ef7952ce549d449c4e4daea2e3f0a1aa7 Mon Sep 17 00:00:00 2001
From: Nikolay Orlyuk <virkony@gmail.com>
Date: Sun, 19 Oct 2014 18:46:34 +0300
Subject: [PATCH] fix shebang paths manipulations

Small enhancements for 67e778f4 to avoid erroneous she-bangs
---
Makefile.PL | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/Makefile.PL b/Makefile.PL
index 61fe336..2d54658 100755
--- a/Makefile.PL
+++ b/Makefile.PL
@@ -63,7 +63,7 @@ docwiki:
perl_shebangs:
ifneq "$(PERL)" "/usr/bin/perl" for file in$(shebang_scripts); do \
-      $(SED) -e "1s|^#!/usr/bin/perl|#!$(PERL)|" < $$file > "$$file.new"; \
+      $(SED) -e "1s|^#!/usr/bin/perl\>|#!$(PERL)|" < $$file > "$$file.new"; \
[ -x $$file ] && chmod +x "$$file.new"; \
mv -f "$$file.new"$$file; \
done
@@ -72,7 +72,7 @@ endif
perl_shebangs_clean:
ifneq "$(PERL)" "/usr/bin/perl" for file in$(shebang_scripts); do \
-      $(SED) -e "1s|^#!$(PERL)|#!/usr/bin/perl|" < $$file > "$$file.new"; \
+      $(SED) -e "1s|^#!$(PERL)\>|#!/usr/bin/perl|" < $$file > "$$file.new"; \
[ -x $$file ] && chmod +x "$$file.new"; \
mv -f "$$file.new"$$file; \
done
--
2.1.2


Done, but this word-boundary construct didn't work on at least one of my systems, so now we're using $(PERL) to do the job portably. --schmonz Posted Sun Oct 19 12:04:01 2014 When uploading a PNG file on the wiki, through the webinterface or anonymous git, i get: icon.png prohibited by allowed_attachments (file MIME type is application/octet-stream, not application/vnd.oasis.opendocument.*)  attachment_allowed_attachments is set to: virusfree() and (mimetype(image/*) or mimetype(text/*) or mimetype(application/x-gzip) or mimetype(application/vnd.oasis.opendocument.*)) and maxsize(2048kb)  Maybe a bug in the filecheck plugin? This is ikiwiki 3.20130904.1~bpo70+1 on Debian wheezy, with some patches applied, namely: Weird... --anarcat Well, the pagespec seems to be matching correctly, given that it thinks the mime type is application/octet-stream. If File::MimeInfo::Magic is installed, ikiwiki uses it. If not, or if it fails to find any mime type, it falls back to using file -bi, and if that fails, it falls back to a default of application/octet-stream. --Joey File::MimeInfo::Magic is installed: ii libfile-mimeinfo-perl 0.16-1 all Perl module to determine file types  it turns out there's (still) a problem with the way we use the module. This test code: #!/usr/bin/perl -w my$file='icon.png';
use File::MimeInfo::Magic;
print "mime::magic: " . File::MimeInfo::Magic::magic($file) . "\n"; print "mime::default: " . File::MimeInfo::Magic::default($file) . "\n";


...returns:

mime::magic: image/png
mime::default: application/octet-stream


file -ib returns the right thing (image/png; charset=binary).

So it should work: it seems that the ::default code kicks in even if the ::magic one actually works.

I have traced down the problem to this block of code:

    if (! defined $mimetype ||$mimetype !~s /;.*//) {
# Fall back to default value.
$mimetype=File::MimeInfo::Magic::default($file)


If you take a look deeply, this will fire up the default if there's no semicolon in the mimetype, which is expected for file calls, but not for ::magic() calls. So ::magic() works, but then the ::default kicks in anyways.

Available in a git repository branch.
Branch: anarcat/dev/magic-fails
Author: anarcat

I have a stupid patch in my git repo which just appends a semicolon to the ::magic() output, but maybe this should be done in another way...

--anarcat

Available in a git repository branch.
Author: smcv

If the regex match isn't necessary and it's just about deleting the parameters, I think I'd prefer

if (! defined $mimetype) { ... }$mimetype =~ s/;.*//;


as done in my ready/more-magic branch.

I'm a little hesitant to do that without knowing why Joey implemented it the way it is, but as far as I can tell it's just an oversight.

Or, if the result of the s/// is checked for a reason, and it's about catching a result from file(1) that is not, in fact, a MIME type at all (empty string or error message or something), maybe something more like this?

if (! defined $mimetype ||$mimetype !~ s{[-\w]+/[-\w]+(?:;.*)?}{})


(or whatever the allowed characters in MIME types are). --smcv

I don't mind either way, but i feel this should be fixed for the next release, as I need to reapply this patch at every upgrade now. -- anarcat

This is still a problem in 3.20140831. -- anarcat

I still don't think appending a semicolon is the right answer: at best it's equivalent to what I suggested, and at worst it's disabling a check that does have some reason behind it. I've turned the version I suggested above into a proper branch. Review by someone who can commit to ikiwiki.git would be appreciated. --smcv

Turns out "someone who can commit" includes me. Merged this version, we can revert or alter it if Joey remembers a reason to require ; --smcv

Posted Tue Dec 3 14:47:06 2013 Tags:

the preprocessing hook makes sure that no infinite loops occur by restricting the depth of nested directives to 3.

this is insufficient in some situations in which sidebars are conditionally assembled from templates.

given there are no limits on the number of directives per page and the number of edits a user can do in a particular time frame, i assume that raising that limit slightly won't make the DoS attacks that can be done against ikiwiki too much worse.

i'd like to suggest 8 as a new value for recursion depth limit. most people can wrap their minds around a depth 3 nested directive setup, but when you reach a depth of 8, it's likely to be easier to write a dedicated plugin.

diff --git a/IkiWiki.pm b/IkiWiki.pm
--- a/IkiWiki.pm
+++ b/IkiWiki.pm
@@ -1487 +1487 @@ sub preprocess ($$;$$) {
-                       if ($preprocessing{$page}++ > 3) {
+                       if ($preprocessing{$page}++ > 8) {


Seems reasonable --smcv

done --Joey

Posted Sat Aug 10 08:48:49 2013 Tags:

Now that the rst plugin uses Python3, the test should test docutils existence also with Python3:

--- rst.t.orig  2018-02-28 10:41:06.000000000 +0000
+++ rst.t   2018-03-03 17:17:23.862702468 +0000
@@ -3,7 +3,7 @@
use strict;

BEGIN {
-   if (system("python -c 'import docutils.core'") != 0) {
+   if (system("python3 -c 'import docutils.core'") != 0) {
eval 'use Test::More skip_all => "docutils not available"';
}
}


Applied, thanks. --smcv

Posted Sat Mar 3 13:21:45 2018