An assets directory for my wiki with git lfs or annex?

Is it somehow possible to have an assets directory powered by git-lfs or maybe git-annex that can handle pictures and other bigger data so that they won't clutter up my git history?

RSS

git-annex support

There are multiple pages on the wiki regarding git-annex support, also I found that by just enabling use of symlinks, git-annex works well [1].

1: http://git.cbaines.net/ikiwiki/commit/?h=git-annex&id=3957bbab17874d60f2597cebcc02cda5c212067a

Comment by mail — Thu May 25 10:43:18 2017

Remove comment

Please do not patch out the symlink check

The check for symbolic links avoids a security vulnerability. Please do not patch it out. We will not support versions of ikiwiki that have been modified in this way.

(In particular, if your wiki has more than one committer, then the other committers can use symbolic links to leak the contents of any file that is readable by the wiki.)

If you want to store a separate assets directory, I would recommend using an underlay directory. You can use git-annex for this if it is placed in direct mode.

I do want to support git-annex and some limited/safe subset of symlinks in ikiwiki, but not until we can do that without introducing a security flaw.

Comment by smcv — Fri May 26 02:20:22 2017

Remove comment

comment 3

Thanks for you comments, I'll try to document my journey to get this to work. Right now the situation is this:

I have a kinda big wiki which also includes a directory assets under which all pictures and anything else is found. this worked well in the past, however it makes for a pretty bloated git repository. I'm not completely sure how, but these are some tasks I have to accomplish:

Understand what actually is an underlay directory: As far as I can tell this is simply a directory which is 1. not in the git repo 2. added to the wiki before compilation.
Remove the assets directory from my ikiwki-git-repo on my local computer and on my server: In theory removing it locally should cause to be removed from the server as well, if we mean "remove" here in the sense of delete. However this still leaves us with a) a bloated git repo because it retains the repos history b) we need to put the assets directory somewhere safe locally before removing it from git
Retain the current directory structure so links won't be broken: This probably means we have to put the assets directory into the gitignore of the ikiwiki-git-repo, since we don't want to move the repo somewhere else.
Init git lfs or git annex in the assets directory, etc.: I need to start tracking my files in the assets directory with the help of one of these tools in order to ensure easy movement between my local wiki and my server wiki. I need to find out which of these tools is better suited for the job. RIght now it feels like that git annex is overkill for the simple job of having a slim git repo for the actual wiki and having one for the assets. Droping, decentralized backups, etc. etc. are not something I need, since my local wiki as well as my server wiki depends on the asset data being present. For the actual setup of this it might be easier to init it on the server first, make a local backup, clone the server repo into the correct directory locally at see of there are files missing.
Setting up the underlayer dir: Here I have to do it two times, one local one on the server since the absolute paths to the assets directories will be different.
Regenerate the local wiki and the server wiki: I'm not sure yet if regenerating the local wiki, then pushing to the server repo is the actual way to do this
Clean up bulky git repo: After confirming all of this works as intended, I will have to slim down my repo by removing the big files from my repo's history: the bfg-repo-cleaner looks like a promising solution for this problem.

If I have imagined this correctly then I should be able to have an independently run assets directory which works basically the same as before, but the difference will be, that it doesn't bloat my repo's history anymore.

Comment by openmedi — Sun Jun 25 12:12:18 2017

Remove comment

comment 4

Understand what actually is an underlay directory: As far as I can tell this is simply a directory which is 1. not in the git repo 2. added to the wiki before compilation.

Yes. Almost every ikiwiki site is configured to use the basewiki underlay, /usr/share/ikiwiki/basewiki or similar, which contains things like the default style.css and ikiwiki/formatting.mdwn. You can add more underlays, and they behave a bit like a union mount: every (non-excluded) page or attachment found in the srcdir or in any underlay is used, unless it is masked by a page or attachment of the same name in a "higher-level" underlay. The srcdir is the highest level, the basewiki is the lowest, and the other underlays are stacked in between. So you'd have something like this:

srcdir
- index.mdwn is rendered as /index.html
- photos.mdwn is rendered as /photos/index.html
your assets directory
- photos/1234.jpg is copied to /photos/1234.jpg
... other underlays like javascript ...
basewiki
- style.css is copied to style.css
- index.mdwn is ignored because it is masked by the one in the srcdir

Remove the assets directory from my ikiwki-git-repo on my local computer and on my server: In theory removing it locally should cause to be removed from the server as well, if we mean "remove" here in the sense of delete. However this still leaves us with a) a bloated git repo because it retains the repos history b) we need to put the assets directory somewhere safe locally before removing it from git

Yes, you'd need to rebase your git history with git filter-branch or similar, or discard the history completely and re-import.

Retain the current directory structure so links won't be broken: This probably means we have to put the assets directory into the gitignore of the ikiwiki-git-repo, since we don't want to move the repo somewhere else.

No, the underlay would normally be next to the srcdir rather than inside it, so there would be no need to .gitignore it.

Init git lfs or git annex in the assets directory, etc.: I need to start tracking my files in the assets directory with the help of one of these tools in order to ensure easy movement between my local wiki and my server wiki

ikiwiki does not currently have any automation for this. You can use git-annex (currently only in direct mode because of the symlink check) but ikiwiki will not help you to sync between the local copy and the server copy.

Comment by smcv — Sun Jul 23 14:02:29 2017

Remove comment

tried and tried and failed

I've tried again and again to make that work in git-annex support. I'm not sure what's going on anymore, because I've been looking at this for 5 years, wrote a patch 3 years ago, and gave up 2 years ago, so my memory is failing me a little.

The takeaway, for me, is that the recommended approach of using the underlay does not work without assistance. There are two main issues:

the "source" directory still have those broken symlinks, and those shadow the underlay. I wrote a patch to work around that issue, to make sure that security checks properly fallback to the underlay when there's a broken symlink. that's still awaiting review
we need a better workflow to move files between the client, the source directory and the underlay. It's unclear to me how we're supposed to handle this, and the UI is especially nasty with the hook I used because it gets called every time there's a push, which can happen multiple times if automatic tag creation is enabled.

It would be really nice to have a better setup for this. I wonder if it wouldn't be easier to do that with Git LFS, because it's a little better at bundling the files along. But it generally assumes there's a magic central location where we can find files from, which is not the model I'm looking for here.

Anyways, just a "me too", I guess, but it would be awesome to have the greatest ikiwiki minds lean over this problem and fix it. It's kind of weird to not have ikiwiki support here considering the history of the project, but I understand that Joey has mostly moved away from active ikiwiki maintenance at this point, so it's understandable.

Cheers! -- anarcat

Comment by anarcat — Mon Mar 19 11:53:01 2018

Remove comment

comment 6

the "source" directory still have those broken symlinks, and those shadow the underlay

Oh, you're using git-annex for the srcdir? The approach I'd vaguely had in mind was to have an ordinary git repository with the Markdown/smaller assets/etc. as the srcdir, and a parallel (no common commits) git-annex with larger assets (photos) as an underlay.

I feel as though broken symlinks in the srcdir probably should shadow the underlay, because otherwise there's nothing we can use as a "white-out" to suppress files from the underlay. (But perhaps the canonical white-out should be a symlink to /dev/null, as used in systemd.)

In an ideal world, symlinks in the srcdir would be treated as the content that they point to, if and only if the symlink is somehow "safe", with symlinks to non-pruned files in the srcdir and symlinks to non-pruned files in .git/annex/objects/ specifically being considered "safe". This is not yet that ideal world, because my to-do list for ikiwiki is a lot longer than the time I can justify spending on it.

I think this mechanism would need to be in terms of "for page/attachment X (a symlink), read file Y (the target of the symlink) instead of X" determined during scanning, rather than removing the -l check from readfile(), because that -l check is a good safety-catch against implementation mistakes that could lead to private file disclosure.

I wrote a patch to work around that issue, to make sure that security checks properly fallback to the underlay when there's a broken symlink

Sorry, I am not going to review patches that relax the symlink security check unless I can concentrate on them enough to be confident that I'm not introducing security vulnerabilities. I realise this means that review has taken too long, but delays (even long ones) seem better than CVEs.

Comment by smcv — Wed Mar 21 05:30:41 2018

Remove comment

some clarifications

Oh, you're using git-annex for the srcdir? The approach I'd vaguely had in mind was to have an ordinary git repository with the Markdown/smaller assets/etc. as the srcdir, and a parallel (no common commits) git-annex with larger assets (photos) as an underlay.

Yeah well, here I am using ikiwiki-hosting which sets up a standard set of repositories to push to. Here I push to ssh://user@host/ and everything is magic after. So adding another repo would be quite clunky.

In an ideal world, symlinks in the srcdir would be treated as the content that they point to, if and only if the symlink is somehow "safe", with symlinks to non-pruned files in the srcdir and symlinks to non-pruned files in .git/annex/objects/ specifically being considered "safe". This is not yet that ideal world, because my to-do list for ikiwiki is a lot longer than the time I can justify spending on it.

I understand I agree with that general design.

I think this mechanism would need to be in terms of "for page/attachment X (a symlink), read file Y (the target of the symlink) instead of X" determined during scanning, rather than removing the -l check from readfile(), because that -l check is a good safety-catch against implementation mistakes that could lead to private file disclosure.

I do not believe I'm removing the -l check from readfile() in the latest patch. It is removed from find_src_files, but only as a strong armed measure: some more clever checks should be implemented there to check the targets...

Sorry, I am not going to review patches that relax the symlink security check unless I can concentrate on them enough to be confident that I'm not introducing security vulnerabilities. I realise this means that review has taken too long, but delays (even long ones) seem better than CVEs.

Agreed. I don't mind the delay so much anymore, TBH... As I said, I've mostly given up and I assume something will either pop up by magic in the future or this will never happen. It's not like I have the time to "concentrate on them enough to be confident that I'm not introducing security vulnerabilities" as you so clearly put it: I probably don't have your grasp on the ikiwiki source so it makes my job even harder, but I should be able to figure out a better patch than 1 || -l at this point. :p

Anyways, I just wanted to provide a slight clarification on the workflow here...

Comment by anarcat — Wed Mar 21 09:41:18 2018

Remove comment

Add a comment

Last edited Fri Oct 25 19:27:48 2019