HTML API docs for python bindings

[Richard Boulton]

Now that the python bindings have documentation strings, I've tried running epydoc on the xapian module to generate an HTML format version of the documentation. This does a pretty good job (good enough that we should include it on the xapian website for each released version). However, there are several issues that could do with being tidied up, so I'll list them here.

1. epydoc seems to consider some of the methods (eg Document.termlist) which are

added to the classes by "extra.i" as "private", and therefore doesn't display them by default. This should be changed so that they're visible by default.

2. Methods which aren't intended to be called externally should be hidden so

that epydoc considers them "private". This could be done by renaming them. For example, Document.termlist_begin() shouldn't be public; renaming it to Document._termlist_begin() would make epydoc consider it to be private, and would prevent users calling it and not knowing how to use the returned iterator. In particular, this would reduce the likelihood of confusion between classes like MSetIter and MSetIterator, since it would be impossible to get an instance of MSetIterator without accessing a private method or attribute.

3. "epydoc xapian" reports several errors, due to markup in the documentation

comments being invalid restructuredText. This should be fixed - in many cases the fix will lie in doxy2swig.py, but in some cases the documentation comments in the C++ headers could do with fixing up.

[Olly Betts]

This should be addressed in the 1.0.x series, if not in 1.0.0 itself (feel free to change blockage to bug#118 instead).

[Richard Boulton]

Changing this to block 1.0 because there's some chance that it will require changing the API of the bindings.

I hope to get this resolved by the end of Monday (30th), since it shouldn't take long.

[Richard Boulton]

Changing this to be aimed at the 1.0 series, not a 1.0.0 blocker.

Investigations have shown that it just requires fixes to the doxy2swig.py script, so runs no risk of requiring API changes, therefore isn't needed for 1.0.0.

I'll try and patch it up as and when I have time, but the documentation is generally in pretty good shape now - certainly compared to how it used to be.

[Richard Boulton]

After a quick look at flint_alltermslist.h, the FIXME comment at line 71 makes me think that maybe we should be storing the total number of terms in the database in the metainfo - so maybe applying the small patch would be a good idea after all... (Trying to implement this FIXME would be too likely to introduce bugs at this stage, but it would be nice to be able to implement it without breaking database compatibility.)

For posterity, the comment is: FIXME : this can badly overestimate (it could be several times too large) - perhaps keep track of the number of terms (or number of extra chunks) and store this along with the last_docid and total_doclen? (Mind you, not sure this value is ever used, but we should really make the exact number of terms available somewhere in the API).

[Richard Boulton]

Bahh! Wrong bug.

[Olly Betts]

The "here's your next bug" is probably my most hated bugzilla feature.

[Richard Boulton]

(In #120) Remove the unfixed dependencies so we can close this bug - they're all marked for the 1.1.0 milestone.

[Olly Betts]

Should this remain targetted at 1.1.0?

[Richard Boulton]

No need to block 1.1.0, and hopefully the SOC swig-doccomment work will go a fair way towards fixing this without work. Moving to 1.1.1.

[Olly Betts]

Not sure the SWIG-doxygen stuff will be ready for us to use any time soon, sadly.

[Olly Betts]

This could be done at any point, and we're not making very rapid progress with 1.1.x currently, so I'm going to mark it as addressable in the 1.2.x series since it clearly doesn't affect API or ABI. I've no objections if it gets fixed sooner though (but it's probably one for Richard or someone else who knows something about epydoc).

[Olly Betts]

Marking for 1.3.x. If this gets implemented and the patch isn't invasive, we could certainly consider backporting it if 1.2.x is still alive at that point.

[Olly Betts]

I notice epydoc was last released Jan 2008, and the last commit to their VCS was 2009, so it doesn't look like it's very alive as a tool.

We should however find something to turn the docstrings into HTML docs and put them on the website.

[Richard Boulton]

Sphinx is the usual python documentation tool for this kind of thing these days ​http://sphinx-doc.org/

No idea how much work it would be to make that work nicely.

[Olly Betts]

James also suggested sphinx on IRC (and pdoc, but that seems fairly new and to have already petered out). Retitling, since this isn't really about epydoc now.

Getting some docs with sphinx is very easy:

$ cd xapian-bindings/python
$ sphinx-apidoc -F -o sphinx-doc xapian
$ make -C sphinx-doc html
$ iceweasel sphinx-doc/_build/html/index.html 

Looking at the output it clearly needs more work to get something nice to actually use. Not sure how the output from the above compares to the epydoc output.

It would be good to have something online, even if it's not totally perfect, so setting for milestone:1.3.3. Even the output from the above probably beats not having anything.

[Olly Betts]

Fixed in trunk r18081, thanks to Assem.

I think this change is probably too complex to be worth backporting to 1.2.x, so just closing.