Hi,

I'm trying to set up a personal wiki and I'm having trouble getting search to work. All my searches give zero results. I eventually figured out that the Xapian database that's being created has an empty postlist table. The position and termlist tables are all fine, and when I add new content to the wiki I can see the database is updated and the search terms are in the position table in plaintext. But I can't query for them, even using Xapian's command line tools.

mexon:~/Test/.ikiwiki/xapian/default$ ls -l
total 76
-rw-rw-r-- 1 mexon mexon     0 Dec 16 15:56 flintlock
-rw-rw-r-- 1 mexon mexon    28 Dec 16 15:55 iamchert
-rw-rw-r-- 1 mexon mexon    13 Dec 16 15:55 position.baseA
-rw-rw-r-- 1 mexon mexon 49152 Dec 16 15:55 position.DB
-rw-rw-r-- 1 mexon mexon    13 Dec 16 15:55 postlist.baseA
-rw-rw-r-- 1 mexon mexon     0 Dec 16 15:55 postlist.DB
-rw-rw-r-- 1 mexon mexon    13 Dec 16 15:55 record.baseA
-rw-rw-r-- 1 mexon mexon     0 Dec 16 15:55 record.DB
-rw-rw-r-- 1 mexon mexon    13 Dec 16 15:55 termlist.baseA
-rw-rw-r-- 1 mexon mexon 16384 Dec 16 15:55 termlist.DB
mexon:~/Test/.ikiwiki/xapian/default$ delve -a .
All terms in database:
mexon:~/Test/.ikiwiki/xapian/default$ 

I don't know how to debug from here. Clearly ikiwiki is doing something right when it's building the database, but one of the tables is missing. Can anyone guess what's wrong, or tell me where to start troubleshooting?

I'm using Centos 5. Xapian is version 1.2.5. Ikiwiki version 3.20111107.

Perhaps you should try a current verison of xapian, 1.2.7 works here. You don't say what version of the xapian perl module you have; 1.2.7.0 is working here. The "postlist" is an internal part of xapian AFAICS, not something that has to be explicitly set up, and it gets populated here.
Comment by joey Fri Dec 16 14:51:16 2011

I'm using RPMs to install Xapian packages, xapian-omega and xapian-bindings-perl, and they're all 1.2.5. I originally tried building and installing Xapian 1.2.7 from source, but found that ikiwiki failed like this:

Use of inherited AUTOLOAD for non-method Search::Xapian::DB_CREATE_OR_OPEN() is deprecated at /home/mexon/system/Linux//lib/perl5/site_perl/5.8.8/IkiWiki/Plugin/search.pm line 220.
Can't locate auto/Search/Xapian/DB_CREATE_O.al in @INC (@INC contains: /home/mat/.ikiwiki /home/mexon/system/Linux//lib/perl5/site_perl/5.8.8/i386-linux-thread-multi /home/mexon/system/Linux//lib/perl5/site_perl/5.8.8 /home/mexon/system/perl5lib /usr/lib/perl5/site_perl/5.8.8/i386-linux-thread-multi /usr/lib/perl5/site_perl/5.8.8 /usr/lib/perl5/site_perl /usr/lib/perl5/vendor_perl/5.8.8/i386-linux-thread-multi /usr/lib/perl5/vendor_perl/5.8.8 /usr/lib/perl5/vendor_perl /usr/lib/perl5/5.8.8/i386-linux-thread-multi /usr/lib/perl5/5.8.8 .) at /home/mexon/system/Linux//lib/perl5/site_perl/5.8.8/IkiWiki/Plugin/search.pm line 220

Much fruitless googling later I found that there were these 1.2.5 RPMs lying around so I switched to those. If you know a solution to the DB_CREATE_O problem I could give 1.2.7 another go.

Comment by Matthew Sat Dec 17 05:10:38 2011

I had another go, this time with Xapian 1.2.8.0, and I finally got it working. The errors I was seeing earlier were because Xapian installs itself in /usr/local, but the default CentOS environment doesn't have /usr/local/lib in the LD_LIBRARY_PATH. As usual, the problem and solution are very duh, it's the error messages that make everything hard. It's a lot more clear if you do a "make test" while building Search::Xapian:

mexon:~/ikiwiki-temp/Search-Xapian-1.2.8.0$ make test
PERL_DL_NONLAZY=1 /usr/bin/perl "-MExtUtils::Command::MM" "-e" "test_harness(0, 'blib/lib', 'blib/arch')" t/*.t
t/01use...............
#   Failed test 'use Search::Xapian;'
#   in t/01use.t at line 3.
#     Tried to use 'Search::Xapian'.
#     Error:  Can't load '/home/mexon/ikiwiki-temp/Search-Xapian-1.2.8.0/blib/arch/auto/Search/Xapian/Xapian.so' for module Search::Xapian: libxapian.so.22: cannot open shared object file: No such file or directory at /usr/lib/perl5/5.8.8/i386-linux-thread-multi/DynaLoader.pm line 230.
#  at (eval 3) line 2
# Compilation failed in require at (eval 3) line 2.
# BEGIN failed--compilation aborted at t/01use.t line 3.
# Looks like you failed 1 test of 3.

So yeah. Worth noting that Xapian 1.2.5 is apparently broken with ikiwiki. Maybe some kind of warning?

Comment by Matthew Mon Dec 19 02:18:56 2011
I use ikiwiki version 3.20130212 and tried xapian versions 1.2.5, 1.2.8, 1.2.13. All to no avail. The postlist.DB file is empty for me. I think that is the crux of the problem. Does anybody know why this could be?
Comment by Ramsey Wed Mar 6 09:10:15 2013
Tried version xapian 1.2.3 and it did not work either. Can someone please help me debug this?
Comment by Ramsey Wed Mar 6 18:42:54 2013

As a Xapian database is updated, changes are written out straight away to most of the tables, but the postlist changes are batched up in memory and only written to disk just before they are committed (or to free up memory during a large transaction). So the empty postlist table you're seeing means that some documents were indexed, but the indexer stopped running before anything was committed. By default, there's an auto-commit every 10000 documents added, removed, or changed, so it presumably managed to process less than 10000 documents.

The issue with 1.2.5 RPMs may be down to there being two versions of the Xapian perl bindings, both of which claim to be Search::Xapian and both of which have been packaged up as RPMs. For Xapian 1.2.x, you probably want to use the XS bindings (perl-Search-Xapian) not those generated with SWIG (xapian-bindings-perl). The SWIG-generated ones are aimed to replace the XS ones, but in 1.2.x they're not really ready for prime time. If your perl bindings are described with a 4 coponent version (e.g. 1.2.15.0) then you should be good; if it's 3 components (e.g. 1.2.15) then you probably want the other ones.

Comment by ojwb Sun Aug 25 23:20:05 2013