TL;DR

  1. An odyssey through lots of things that have to be right before OpenID works
    1. no_identity_server: Could not determine ID provider from URL.
      1. various possible causes ...
      2. make a luckier setting of useragent ?!
        1. culprit was an Atomicorp ModSecurity rule
    2. Error: OpenID failure: naive_verify_failed_network: Could not contact ID provider to verify response.
      1. set PERL_NET_HTTPS_SSL_SOCKET_CLASS appropriately
      2. undo change that broke PERL_NET_HTTPS_SSL_SOCKET_CLASS
    3. Still naive_verify_failed_network, new improved reason
      1. ensure OpenSSL, Net::SSLeay, IO::Socket::SSL new enough for SNI
      2. Local OpenSSL installation will need certs to trust
    4. Still certificate verify failed
      1. ensure that LWPx::ParanoidAgent passes server name to SSL layer for SNI
  2. Success!!

An odyssey through lots of things that have to be right before OpenID works

Having just (at last) made an ikiwiki installation accept my OpenID, I have learned many of the things that may have to be checked when getting the openid plugin to work. (These are probably the reasons why ikiwiki.info itself won't accept my OpenID!)

Just to describe my OpenID setup a bit (and why it makes a good stress-test for the OpenID plugin :).

I'm using my personal home page URL as my OpenID. My page lives at a shared-hosting service I have hired. It contains links that delegate my OpenID processing to indieauth.com.

IndieAuth, in turn, uses rel-me authentication to find an OAuth provider that can authenticate me. (At present, I am using github for that, which is an OAuth provider but not an OpenID provider, so the gatewaying provided by IndieAuth solves that problem.) As far as ikiwiki is concerned, IndieAuth is my OpenID provider; the details beyond that are transparent.

So, what were the various issues I had to sort out before my first successful login with the openid plugin?

no_identity_server: Could not determine ID provider from URL.

This is the message ikiwiki.info shows as soon as I enter my home URL as an OpenID. It is also the first one I got on my own ikiwiki installation.

various possible causes ...

There could be lots of causes. Maybe:

  • the offered OpenID is an https: URL and there is an issue in checking the certificate, so the page can't be retrieved?
  • the page can be retrieved, but it isn't well-formed HTML and the library can't parse it for the needed OpenID links?
  • ...?

make a luckier setting of useragent ?!

In my case, it was none of the above. It turns out my shared-hosting provider has a rule that refuses requests with User-Agent: libwww-perl/6.03 (!!). This is the sort of problem that's really hard to anticipate or plan around. I could fix it (for this case!) by changing useragent: in ikiwiki.setup to a different string that my goofy provider lets through.

Recommendation: set useragent: in ikiwiki.setup to some unlikely-to-be-blacklisted value. I can't guess what the best unlikely-to-be-blacklisted value is; if there is one, it's probably the next one all the rude bots will be using anyway, and some goofy provider like mine will blacklist it.

If your shared hosting provider is going to randomly break functionality, I would suggest "voting with your wallet" and taking your business to one that does not.

In principle we could set the default UA (if $config{useragent} is unspecified) to IkiWiki/3.20140915, or IkiWiki/3.20140915 libwww-perl/6.03 (which would be the "most correct" option AIUI), or some such. That might work, or might get randomly blacklisted too, depending on the whims of shared hosting providers. If you can't trust your provider to behave helpfully then there isn't much we can do about it.

Blocking requests according to UA seems fundamentally flawed, since I'm fairly sure no hosting provider can afford to blacklist UAs that claim to be, for instance, Firefox or Chrome. I wouldn't want to patch IkiWiki to claim to be an interactive browser by default, but malicious script authors will have no such qualms, so I would argue that your provider's strategy is already doomed... --smcv

I agree, and I'll ask them to fix it (and probably refer them to this page). One reason they still have my business is that their customer service has been notably good; I always get a response from a human on the first try, and on the first or second try from a human who understands what I'm saying and is able to fix it. With a few exceptions over the years. I've dealt with organizations not like that....

But I included the note here because I'm sure if they're doing it, there's probably some nonzero number of other hosting providers where it's also happening, so a person setting up OpenID and being baffled by this failure needs to know to check for it. Also, while the world of user-agent strings can't have anything but relatively luckier and unluckier choices, maybe libwww/perl is an especially unlucky one?

Yippee! My provider found their offending mod_security rule and took it out, so now ikiwiki.info accepts my OpenID. I'm still not sure it wouldn't be worthwhile to change the useragent default.... -- Chap

culprit was an Atomicorp ModSecurity rule

Further followup: my provider is using ModSecurity with a ruleset commercially supplied by Atomicorp, which seems to be where this rule came from. They've turned the rule off for my account. I followed up on my ticket with them, suggesting they at least think about turning it off more systemwide (without waiting for other customers to have bizarre problems that are hard to troubleshoot), or opening a conversation with Atomicorp about whether such a rule is really a good idea. Of course, while they were very responsive about turning it off for me, it's much iffier whether they'll take my advice any farther than that.

So, this may crop up for anybody with a provider that uses Atomicorp ModSecurity rules.

The ruleset produces a log message saying "turn this rule off if you use libwww-perl", which just goes to show whoever wrote that message wasn't thinking about what breaks what. It would have to be "turn this rule off if any of your customers might ever need to use or depend on an app or service hosted anywhere else that could have been implemented using libwww-perl, over which you and your customer have no knowledge or control."

Sigh. -- Chap

Thanks for the pointer. It seems the open-source ruleset blacklists libwww-perl by default too... this seems very misguided but whatever. I've changed our default User-Agent to ikiwiki/3.20141012 (or whatever the version is). If we get further UA-blacklisting problems I'm very tempted to go for Mozilla/5.0 (but not really) as the next try. --smcv

Error: OpenID failure: naive_verify_failed_network: Could not contact ID provider to verify response.

Again, this could have various causes. It was helpful to bump the debug level and get some logging, to see:

500 Can't connect to indieauth.com:443 (Net::SSL from Crypt-SSLeay can't
verify hostnames; either install IO::Socket::SSL or turn off verification
by setting the PERL_LWP_SSL_VERIFY_HOSTNAME environment variable to 0)

I don't belong to the camp that solves every verification problem by turning verification off, so this meant finding out how to get verification to be done. It turns out there are two different Perl modules that can be used for SSL:

  • IO::Socket::SSL (verifies hostnames)
  • Net::SSL (does not verify hostnames)

Both were installed on my hosted server. How was Perl deciding which one to use?

set PERL_NET_HTTPS_SSL_SOCKET_CLASS appropriately

It turns out there's an environment variable. So just set PERL_NET_HTTPS_SSL_SOCKET_CLASS to IO::Socket::SSL and the right module gets used, right?

Wrong. That change was made to ParanoidAgent.pm back in November 2013 because of an unrelated bug in IO::Socket::SSL. Essentially, hmm, something goes wrong in IO::Socket::SSL when reading certain large documents, so we'll fix it by forcing the use of Net::SSL instead (the one that never verifies hostnames!), no matter what the admin has set PERL_NET_HTTPS_SSL_SOCKET_CLASS to!

undo change that broke PERL_NET_HTTPS_SSL_SOCKET_CLASS

Plenty of comments quickly appeared about how good an idea that wasn't, and it was corrected in June 2014 with one commit to fix the original reading-long-documents issue in IO::Socket::SSL and another commit that reverts the forcing of Net::SSL no matter how the environment is set.

Unfortunately, there isn't a release in CPAN yet that includes those two commits, but they are only a few lines to edit into your own locally-installed module.

To be clear, these are patches to LWPx::ParanoidAgent. Debian's liblwpx-paranoidagent-perl (>= 1.10-3) appears to have those two patches. --smcv

Irrelevant to this ikiwiki instance, perhaps relevant to others: I've added these patches to pkgsrc's www/p5-LWPx-ParanoidAgent and they'll be included in the soon-to-be-cut 2014Q3 branch. --schmonz

Still naive_verify_failed_network, new improved reason

500 Can't connect to indieauth.com:443 (SSL connect attempt failed
with unknown error error:14090086:SSL
routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed)

Yay, at least it's trying to verify! Now why can't it verify IndieAuth's certificate?

Here's why. As it turns out, indieauth.com is itself a virtual host on a shared server. If you naively try

openssl s_client -connect indieauth.com:443

you get back a certificate for indieweb.org instead, so the hostname won't verify. If you explicitly indicate what server name you're connecting to:

openssl s_client -connect indieauth.com:443 -servername indieauth.com

then, magically, the correct certificate comes back.

ensure OpenSSL, Net::SSLeay, IO::Socket::SSL new enough for SNI

If your openssl doesn't recognize the -servername option, it is too old to do SNI, and a newer version needs to be built and installed. In fact, even though SNI support was reportedly backported into OpenSSL 0.9.8f, it will not be used by IO::Socket::SSL unless it is 1.0 or higher.

Then a recent Net::SSLeay perl module needs to be built and linked against it.

I would tend to be somewhat concerned about the update status and security of a shared hosting platform that is still on an OpenSSL major version from pre-2010 - it might be fine, because it might be RHEL or some similarly change-averse distribution backporting security fixes to ye olde branch, but equally it might be as bad as it seems at first glance. "Let the buyer beware", I think... --smcv

As far as I can tell, this particular provider is on Red Hat (EL 5). I can't conclusively tell because I'm in what appears to be a CloudLinux container when I'm in, and certain parts of the environment (like rpm) I can't see. But everything I can see is like several RHEL5 boxen I know and love.

Local OpenSSL installation will need certs to trust

Bear in mind that the OpenSSL distribution doesn't come with a collection of trusted issuer certs. If a newer version is built and installed locally (say, on a shared server where the system locations can't be written), it will need to be given a directory of trusted issuer certs, say by linking to the system-provided ones. However, a change to the certificate hash algorithm used for the symlinks in that directory was reportedly made with OpenSSL 1.0.0. So if the system-provided trusted certificate directory was set up for an earlier OpenSSL version, all the certificates in it will be fine but the hash symlinks will be wrong. That can be fixed by linking only the named certificate files from the system directory into the newly-installed one, and then running the new version of c_rehash there.

Still certificate verify failed

Using SNI-supporting versions of IO::Socket::SSL, Net::SSLeay, and OpenSSL doesn't do any good if an upper layer hasn't passed down the name of the host being connected to so the SSL layer can SNI for it.

ensure that LWPx::ParanoidAgent passes server name to SSL layer for SNI

That was fixed in LWPx::ParanoidAgent with this commit, which needs to be backported by hand if it hasn't made it into a CPAN release yet.

Also in Debian's liblwpx-paranoidagent-perl (>= 1.10-3), for the record. --smcv

And now in pkgsrc's www/p5-LWPx-ParanoidAgent, FWIW. --schmonz

Only that still doesn't end the story, because that hand didn't know what this hand was doing. What good is passing the name in PeerHost if the SSL code looks in PeerAddr first ... and then, if that doesn't match a regex for a hostname, decides you didn't supply one at all, without even looking at PeerHost?

Happily, is is possible to assign a key that explicitly supplies the server name for SNI:

--- LWPx/Protocol/http_paranoid.pm    2014-09-08 03:33:00.000000000 -0400
+++ LWPx/Protocol/http_paranoid.pm    2014-09-08 03:33:27.000000000 -0400
@@ -73,6 +73,7 @@
        close($el);
         $sock = $self->socket_class->new(PeerAddr => $addr,
                                          PeerHost => $host,
+                                         SSL_hostname => $host,
                                          PeerPort => $port,
                                          Proto    => 'tcp',
                                          Timeout  => $conn_timeout,

... not submitted upstream yet, so needs to be applied by hand.

I've reported this to Debian (which is where ikiwiki.info's supporting packages come from). Please report it upstream too, if the Debian maintainer doesn't get there first. --smcv

Applied in pkgsrc. I haven't attempted to conduct before-and-after test odysseys, but here's hoping your travails save others some time and effort. --schmonz

Reported upstream as LWPx-ParanoidAgent#14 and IO-Socket-SSL#16. -- Chap

Success!!

And with that, ladies and gents, I got my first successful OpenID login! I'm pretty sure that if the same fixes can be applied to ikiwiki.info itself, a wider range of OpenID logins (like mine, for example :) will work here too.

-- Chap