Support for epub

[Jugnu]

Git PR for issue : ​https://github.com/xapian/xapian/pull/235

EPUB ------

Skipping - unknown MIME type 'application/epub+zip'

Skipping - unknown MIME type 'application/zip'

Current situation is the omindex is able to index the epub files. However, there is work needed to perfectly parse the metadeta information correctly such as overall title, individual chapters, authors etc. There are many other parts within index_file.cc which does the basic indexing but lacks perfect metadata parsing. For instance, the formats can be found through searching for : FIXME: Implement support for metadata.

To do :

1. Also tests are needed to ensure that epub supports is generalized across different generation of epubs, their file directory structures etc..
2. Correct metadata parsing

[Jugnu]

For the courtesy, my omindex and omega both are 1.5.0

[Olly Betts]

Are these files you are attaching redistributable? The epub says "Copyright © 2009 by John C. Bogle. All rights reserved" on the title page, which suggests not.

If we don't have a licence to redistribute them we shouldn't be redistributing (and please don't attach files that don't have a licence allowing redistribution).

[Olly Betts]

The error in the description is rather strange:

Exception: DatabaseError: Modifications failed (DatabaseError: Error reading block 0 (Protocol error)), and couldn't open at the old revision: Error reading block 0.

Is that repeatable?

Is there anything unusual about the disk partition which you have the database on?

[Jugnu]

I guess this came because with my vagrant, sometimes it misbehaves like doing ls and it said " ls: cannot access shared: Protocol error ". I do not have the problem now as i did some googling about not able to access shared drives. So did vagrant reload and vagrant ssh.

Also in addition to this, I was also not able to use cgi-bin for query and it gave some error like "cannot access query template ; protocol error" when I performed search in search bar. But errors were gone after vagrant reload.

[Jugnu]

Added support for inner and outer level html files exploration.

[Olly Betts]

I've deleted the random polarities.pkl as it seems unrelated to EPUB so doesn't seem relevant here, and its copyright status was unclear.

[Olly Betts]

We now support indexing files using libe-book, and I just noticed libe-book has some "experimental" support for EPUB, which is disabled unless configured with --enable-experimental.

I've not tried it yet, but probably improving that to a suitable standard to no longer be deemed experimental would be a good way to resolve this. I'd expect that's much less work than implementing our own EPUB parsing, and less work to maintain going forwards. It also potentially benefits other projects using libe-book.

[Olly Betts]

I had a brief go at trying it, but it seems to require libcss which isn't packaged for Debian.

Looks like libgepub may be a plausible option and is packaged: ​https://github.com/danigm/libgepub/blob/master/libgepub/gepub-doc.h

[Olly Betts]

Implemented support using libgepub in b2109fcdc802565bb148b8d2be6d2b38920fa7e8. There are automated tests with EPUB 2 and EPUB 3 files exported from libreoffice.

Currently libgepub 0.6 or 0.7 should work, but CI only tests 0.6 due to that being what's available on the Ubuntu versions there.

Author and title are extracted successfully. Page count is set to the number of "chapters" which seems about as close as we can get since each chapter seems to be an HTML document. I wrote code for GEPUB_META_DESC but I don't see that actually appearing from the example files I tried with.

The EPUB format seems to support some other metadata types, but I didn't see how to get them.