Deadlocks with apache mod_python and mod_wsgi

[richard]

Summary of current known status

mod_python

Calling any Xapian methods or functions is likely to cause dead-lock unless you set this option in the Apache configuration section for all mod_python scripts which use Xapian:

PythonInterpreter main_interpreter

You may also need to use Python >= 2.4 (due to [ http://issues.apache.org/jira/browse/MODPYTHON-217 problems in Python 2.3 with the APIs the code uses).

mod_wsgi

You'll need to set:

WSGIApplicationGroup %{GLOBAL}

For details see:  http://code.google.com/p/modwsgi/wiki/ConfigurationDirectives#WSGIApplicationGroup and  http://code.google.com/p/modwsgi/wiki/ApplicationIssues#Python_Simplified_GIL_State_API

The mod_wsgi developers say this should be sufficient, and you should be able to subclass Xapian objects in Python. If you encounter problems, please talk to us or the mod_wsgi developers so we can investigate.

---

Originally reported on the mailing list:  http://thread.gmane.org/gmane.comp.search.xapian.general/4486

[richard]

 http://www.modpython.org/pipermail/mod_python/2007-April/023445.html has a good explanation of what is going on here.

Basically, mod-python creates a separate python interpreter for each vhost. However, the SWIG generated xapian python bindings use the simplified python handling routines, which assume that the thread lock to use is that held in the python interpreter called "main_interpreter". This results in the wrong thread lock being requested, and this always seems to result in a deadlock.

Short term workaround: Set the option "PythonInterpreter? main_interpreter" in the apache configuration section for all mod_python scripts which use xapian.

Longer term fix: change SWIG to generate code which uses the full threading api.

[richard]

mod-python script testing xapian

[richard]

Site configuration which works.

[mhammond]

FYI: "Longer term fix: change SWIG to generate code which uses the full threading api." isn't as easy as it sounds (well - uglier than it sounds).

The full threading API *forces* you to specify an explicit PyInterpreter? object when using the GIL. For libraries which may spawn new threads which attempt to call back into Python, this is very tricky - that thread has no way to determine exactly what context it should use, so that library would be forced to remember an interpreter object in a global variable and use that. The simplified API just formalizes this process - an interpreter object is nominated the "main" one and the simplified API uses that.

In xapian's case, the threading consideration doesn't really apply - but it still would force xapian to track which interpreter is currently in use.

Eg:

python calls xapian xapian bindings release the GIL xapian winds up in code that needs to call back into Python xapian bindings restore the GIL

The problem is that in the last step, the "restore the GIL" must refer to the same PyInterpreter? object that was used in step 2, when the GIL was released. The problem is how to "remember" what should be used. It is likely to require thread-local-storage, for example. (Note that in the general case though, when new calls may come in on a thread never previously seen by Python, even this falls down - eg, imagine if xapian spawned new threads that wanted to call back into Python). The simplified API just papers over this - there is a single, nominated "main" interpreter, and that interpreter is *always* used, regardless of the thread making the call.

I'm surprised apache went this way, as multiple interpreter support in Python is patchy at best. For example, extension modules don't play this game - they often store Python objects in C global variables, for example. This means that extension modules are *not* isolated in such an environment. So although each interpreter is somewhat isolated, its not complete isolation.

 http://www.python.org/dev/peps/pep-0311/ has more info on this (I wrote it :)

[olly]

What was the fate of that PEP? Has the patch been included in a Python release?

Xapian doesn't thread internally, so the TLS approach should work for us. I imagine that it would be suitable for many C libraries wrapped by SWIG, so patching SWIG is worth considering.

[mhammond]

The PEP was accepted (although that fact was not recorded in the PEP), and those GIL state APIs referenced in the PEP are what SWIG uses. While it may be possible to patch SWIG such that it works with many C++ extensions, the GIL state APIs are more general purpose and will work with far more (ie, those that do thread internally as well as those that do not), and also ensures that multiple threaded extensions can cooperate (which is not possible using Apache's approach). It's also not clear that SWIG would always have a TLS implementation available to it. Apache using the explicit APIs is still widely considered a mistake (eg,  http://mail.python.org/pipermail/web-sig/2007-August/002757.html is the most recent message to Python's web-sig, and it makes that point - google will offer many more) and almost all modern code uses the saner API provided by that PEP.

[olly]

OK, have reread everything twice, I think I follow.

Richard has added a note about "PythonInterpreter? main_interpreter" to the documentation. Since there's a workaround, and we're not the only Python extension to have this issue, I think we have an acceptable resolution to this for the time being.

[deron.meranda]

The workaround is of course only acceptable for those mod_python users who do not make use of the multiple-interpreter feature it exposes. If you do use multiple interpreters then there is still no offered workaround.

I do use other extension modules which have no problems in a multiple interpreter environment, such as MySQLdb. However this is the first SWIG-generated module I've tried using in this way. Is this really a SWIG problem, and can it be worked around in the Xapian case?

I do have a couple questions. First does Xapian ever call back into Python as PEP-311 describes. If not, then is there any reason to use the simplified GIL API to begin with? And secondly, assuming one only uses a non-threaded Apache MPM, is there any reason to unlock/relock the GIL inside C code anyway? And if not, is there a way to get the module to compile in a manner which would work in a multiple-interpreter environment as long as the process was always single-threaded?

Also, I don't really agree that the multiple-interpreter support that mod_python provides is "widely" considered a mistake. The only misgivings really are that some extension modules are buggy, and that the simplified GIL API is now promoting "broken" modules (although this was a reasonable trade off between reducing bugginess for everybody and completely breaking one feature of Python that only a few users would notice). But that's all a matter of opinion.

[mhammond]

xapian does have a need to acquire the GIL, but only from a thread that previously released it. xapian does not currently create new threads that call back into Python; without the simplified API, this is impossible to do in a cross-platform way. Each app managing its own interpreter also makes it impossible for multiple libraries to share python thread-states etc, making certain interactions between such libraries impossible.

Sadly, it really is Python that is "buggy" here - the multi-interpreter support is only 1/2 baked and the shortcomings of the original API are acknowledged - see the python-dev discussions leading up to that PEP. I'm glad it works for Apache and its extensions and I didn't mean to suggest anything other than SWIG continue supporting the simplified API by default. I'm sure contributions of alternative strategies that are optimized for Apache would be most welcome.

[deron.meranda]

Yes, this seems to be the result of an improperly designed Python arichtecture where two internal parts (GIL/threading and multiple-interpreter) can be mutually-exclusive (or mutually-deadlocking). Too bad.

However, even if one is able to use the "PythonInterpreter? main_interpreter" workaround previously mentioned, it still doesn't prevent deadlock in all cases.

This appears to be a problem only when using Python 2.3.x (probably an issue with the simplified GIL code in that version). More detail can be found in mod_python issue 217 notes:

 http://issues.apache.org/jira/browse/MODPYTHON-217

Other useful links to related information for the curious:

 http://docs.python.org/api/threads.html  http://modpython.org/live/current/doc-html/pyapi-interps.html

I can reproduce this deadlock with Python 2.3.4 while calling nothing other than

xapian.xapian_version_string()

while in the main_interpreter interpreter. You don't need any of the complexities of all the virtual host stuff in the attached config to reproduce this deadlock, just a simple 4-line .htaccess is enough.

So there's still no reasonable workaround. I'm beginning to think that the best approach is to get the SWIG stuff completely out of the mod_python process: such as by writing a pure-python xapian proxy module which communicates with a fork-exec'd child python process which can then import the SWIG-generated xapian without conflicting GIL/thread issues. But I'm suspect there's challenging issues there too.

[olly]

Others understand most of this much better than I do, but it seems it would be a real headache to fork/exec and then (I'm assuming) have to do everything via IPC or similar. I'm not sure we really want to go there.

Are you saying this works correctly with Python 2.4 and later, or is it just that the problems can't be demonstrated with such a trivial example? If it's only Python 2.3, what's different there, and can we work around that difference?

If SWIG could implement the locking better, I'm happy to apply suitable patches there (unless the SWIG Python people disagree with the new approach).

[richard]

My understanding is that this is only a problem with Python 2.3. Therefore, I've updated the documentation of the workaround to suggest using Python 2.4 or later (I've tested that the main_interpreter workaround works with some version of python (can't remember whether it was 2.4 or 2.5), but haven't tested that it fails with 2.3 - I'm willing to believe the claim made earlier in this bug, though).

Another approach might be to add a configure option to remove the "-threads" option from the swig invocation when compiling for use with mod python. IIUC, without this option swig doesn't make any attempts to reduce the GIL, which is bad news for any normal multi-threaded program using xapian, but might be a worthwhile tradeoff for mod-python programs in setups which insist on using multiple interpreters. On the other hand, this would only work in maintainer mode, and would therefore require anyone building for such a setup to have all the necessary build tools installed (in particular, having the right version of swig), so this could be more trouble for the Xapian team than it's worth. Plus, it's liable to kill throughput on multi-core systems, which we don't want to get a reputation for. So, I don't propose trying this.

[michael.barton]

We've run into this problem with xapian 1.0.2 on python 2.4. Switching to main_interpreter hasn't helped.

If rebuilding the bindings without swig's threading support will work, I think we'd trade xapian performance for deadlocks. Especially since our current solution is to popen() a second script that makes all the xapian calls.

[olly]

I've just fixed a bug in the mutex code swig uses for python.

The fix was simply to change `PyThread?_free_lock(mutex_);' to `PyThread?_release_lock(mutex_);'. This occurs just once in python/modern/xapian_wrap.cc.

Can those of you who have experienced these problems try that and report back?

I suspect this will fix problems for some people but not everyone but it will be interesting to see.

[richard]

As far as I can see, in modern/xapian_wrap.cc, the only call to PyThread?_release_lock is in the destructor of "Guard", which in turn is only used by the SWIG_GUARD() macro, which in turn is used in 4 places in the code. However, these 4 places are in the Director base class, and none of them seem to be called by the Xapian code. So, unless I'm missing something, this won't change anything.

However, I've just tested the current Xapian HEAD with mod-python, using python 2.5 (on Ubuntu Dapper), and it works fine there. (I do need to use the main_interpreter workaround described earlier in this bug report, however.) So, we need more information from someone who is having problems to make any progress with this bug.

[kahless]

i am having the same problem - on debian with apache 2 + prefork, mod_python-3.3.1, python 2.5.1 and xapian+bindings 1.0.6

i have tried using: PythonInterpreter? main_interpreter in my apache config .. but it doesn't change anything .. - i have verified that the setting is working by outputting req.interpreter: main_interpreter ... so no idea what i missed ?

do i have to compile xapian-bindings with a special option or something ? any idea what else i could try ? (other than not using mod_python or xapian)

[michael.barton]

I'm still having this problem on ubuntu dapper (so I'm on 1.0.5) with python 2.4 and mod_python 3.1.4. It locks up every time, and running xapian code in the main interpreter has never helped any that I could tell. We have a semi-complex apache configuration with some legacy stuff that requires explicit separate interpreters, and that's probably a factor.

I've been using a build of the bindings minus swig -threads and that works, but maintainer mode is sort of harrowing.

[olly]

Replying to michael.barton:

I've been using a build of the bindings minus swig -threads and that works, but maintainer mode is sort of harrowing.

Does it work with -threads if you use SWIG SVN HEAD instead of the SWIG in the Xapian tree (or use Xapian SVN HEAD, which includes a newer SWIG snapshot)? There's a change in SWIG which might be relevant, but neither Richard nor I can reproduce this issue currently.

Specifying "main_interpreter" will still be necessary.

[olly]

It seems this probably affects mod_wsgi too. The equivalent of "main_interpreter" there appears to be setting WSGIApplicationGroup to %{GLOBAL} - for details see:

 http://code.google.com/p/modwsgi/wiki/ConfigurationDirectives#WSGIApplicationGroup

[olly]

Updating summary and description to reflect that this affects mod_wsgi too.

I'll update the python bindings documentation for this issue shortly.

[grahamd]

Note, Xapian definitely should work with mod_wsgi provided that you delegate web application code making use of Xapian to run in first interpreter. Thus try using:

WSGIApplicationGroup %{GLOBAL}

This should be the case even if Xapian performs callbacks into Python code whereby it needs to reacquire the Python GIL. It should all work because mod_wsgi specifically detects case where code is running in first interpreter and itself uses simplified Python GIL API in order to be compatible with other Python modules that use it.

Note that mod_python (up to 3.3.1, regardless of Python version) is however broken and does not use simplified Python GIL API even when code executing in first interpreter. Thus, the suggestion of setting interpreter as 'main_interpreter' when using mod_python would definitely not work and lock ups would be seen if callbacks into Python from C code are occurring. If people are suggesting that it is working, they can't be using that feature of Xapian or have not compiled the SWIG bindings with thread support.

In order for mod_python to work properly, as detailed in:

 http://issues.apache.org/jira/browse/MODPYTHON-217

it must be fixed to use simplified Python GIL API for first interpreter. Even with fix, 'main_interprerter' would still need to be used.

In summary, mod_wsgi works as long as first interpreter, use it if you are hosting a WSGI capable application. The mod_python package is however completely broken for the callbacks case, so avoid it if you can.

I am getting sufficiently annoyed about the misinformation I am seeing on some forums about this, that it may be time to fix mod_python finally. Although, still recommend you just use mod_wsgi instead. :-)

[olly]

I didn't add the note about mod_wsgi randomly - a user reported problems using WSGI (on IRC) which matched these symptoms, and apparently WSGIApplicationGroup %{GLOBAL} didn't help.

It's possible they failed to actually set it in the right config file or something I guess.

[grahamd]

Was that aa_ on #pocoo? Still waiting for them to get back with what the original issue was so can see if there is an issue or not. Right now though, I know of no technical reason why mod_wsgi shouldn't work as believe it does everything in a way such that use of simplified GIL API by a module should work. If some one suspects a problem, really need them to bring it over to mod_wsgi mailing list so can be properly investigated. At this point, no one has indicated directly to the mod_wsgi mailing list that things aren't working. So, need more information with an actual test case if possible.

[olly]

Just checked my IRC log and it was aa_ (on #xapian though).

I've updated the bug description to attempt to summarise the current state of play.

[grahamd]

Sorry, what is said about mod_python is I believe still wrong.

The MODPYTHON-217 is not saying only Python 2.3 is the problem, it relates to new Python APIs introduced in Python 2.3. Those APIs still exist and act the same. Thus it doesn't matter which version of Python is used at this point. Thus saying that Python 2.4 or later may help is I believe wrong. The code in mod_python needs to be fixed.

FWIW, I have a patched version of mod_python which fixes the problem so at least main interpreter can be used, but am trying to sort out issues around trying to load that modified version at same time as mod_wsgi. Problem is that mod_python did things with threads wrongly and mod_wsgi had some fiddles in it to cope with that. Now that mod_python code has been fixed, mod_wsgi fails becauses its fiddles no longer apply. Trying to work out best thing to do. Maybe case that if want to use modified mod_python with mod_wsgi, may have to just use mod_wsgi 2.2 or later, and will make a change in mod_wsgi 2.2 to allow it to also work with that modified mod_python.

[olly]

Well, MODPYTHON-217 clearly says At this stage it is believed this only occurs with Python 2.3 and shouldn't be a problem with later versions of Python and there are no comments added which contradict this. Here this refers to deadlock when calling PyGILState_Ensure() in the way in which SWIG generated wrappers do. Is that wrong?

And in comment:23 you say Thus, the suggestion of setting interpreter as 'main_interpreter' when using mod_python would definitely not work and lock ups would be seen if callbacks into Python from C code are occurring. If people are suggesting that it is working, they can't be using that feature of Xapian or have not compiled the SWIG bindings with thread support. I understood that to mean that provided you don't use Xapian features which call from C++ back to Python, you can use Xapian bindings with thread support (which they have unless you regenerate them yourself without -threads).

So either I've misunderstood here, or I've expressed this badly in my summary. I've tweaked the wording in a couple of places - notably problems in Python 2.3 with the APIs the code uses - is that better?

[grahamd]

I'll have to do some more research as my memory is hazy. That issue though really has two parts to it. The first is the bit that only affects Python 2.3 and the second is the general problem of using using simplified GIL state API. If is the latter part which affects SWIG bindings such as Xapian. The first part refers only to a particular scenario which I don't think can occur for SWIG bindings but is more an issue for mod_python internals.

When using SWIG bindings there are normally two use case scenarios. The first is:

1. Python code executing and GIL held.
2. SWIG binding leaves Python and releases GIL but remembers its original thread state.
3. C code is executed.
4. C code finishes and SWIG binding restores original thread state and returns back to calling Python code.

The second is:

1. Python code executing and GIL held.
2. SWIG binding leaves Python and releases GIL but remembers its original thread state.
3. C code is executed.
4. C code wants to make callback into Python and uses simplified GIL state API to reacquire GIL.
5. Python code executes and returns.
6. GIL released using simplified GIL state API.
7. C code finishes and SWIG binding restores original thread state and returns back to calling Python code.

It is the second of these which fails in mod_python even when main interpreter due to mod_python not using simplified GIL state API for all main interpreter code.

The first case is safe in all interpreters as not involving simplified GIL state API.

The Python 2.3 specific scenario is:

1. C code executing and active thread state still exists with GIL locked.
2. Simplified GIL state API called to try and acquire lock again.

This final case I don't believe can arise with SWIG bindings being used from Python code, thus why I think Python 2.3 exclusion is not relevant to Xapian.

BTW, not compiling SWIG bindings with thread support would preclude callbacks. Plus, if C code blocked for long time, would cause all of Python to block as GIL not released for case 1 above.

More later maybe, as have to go some where now.

[richard]

The Python 2.3 problem definitely did exist with Xapian, and I think it still does: last year, when this issue was first opened, I observed it failing as described, and the Xapian code is exactly as described in the  http://issues.apache.org/jira/browse/MODPYTHON-217. To pick a random example, the wrapper function for xapian.version_string() begins:

SWIGINTERN PyObject *_wrap_version_string(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
  PyObject *resultobj = 0;
  char *result = 0 ;

  SWIG_PYTHON_THREAD_BEGIN_BLOCK;

and SWIG_PYTHON_THREAD_BEGIN_BLOCK eventually expands to a call to PyGILState_Ensure(). This is called with the interpreter holding the GIL, and the situation is the same for pretty much every SWIG generated method, so any call to xapian results in deadlock.

I don't have a Python 2.3 installation to hand to test this now, but the Xapian code hasn't changed, and I assume the mod-python code hasn't either, since the MODPYTHON-217 issue in the apache tracker is still open. I therefore believe the summary in this bug about the situation with 2.3 is correct.

With python 2.4, it is still necessary to set the interpreter to main_interpreter, as described at length in  http://issues.apache.org/jira/browse/MODPYTHON-77 - this is because any use of PyGILState_Ensure() assumes that the interpreter running is the main interpreter. Also, as you describe in your comment, callbacks to Python from C still won't work. I think our "Summary of current known status" is therefore fully up-to-date and correct.

I don't understand your comment "BTW, not compiling SWIG bindings with thread support would preclude callbacks." If swig is told to generate the bindings without thread support, the resulting code doesn't touch the GIL at any point, so the callbacks should work fine (the lock is never released, or re-acquired, but the thread state isn't accessed either). I fully agree with "Plus, if C code blocked for long time, would cause all of Python to block as GIL not released for case 1 above", though!

[grahamd]

You are correct about -threads comments. I threw that in at last minute when trying to rush out door and didn't think about it properly. Yes, it will work. The only case where there would be a problem would be if the SWIG software was creating its own foreign threads and those threads were attempting callbacks, but then since it would become multithreaded at that point, when SWIG code isn't, there would be lots of problems anyway.

As for the Python 2.3 issue, I'll just have to take your word on it at this point. It has been a very long time since have looked at what code SWIG generates. To make things worse, there were some differences back then in how SWIG generated code did things for commonly used versions. So, my memory may be clouded over what more recent versions do. I don't right now have further time to investigate further as actually on an extended trip overseas. If I did have the time, I would have sat down and refreshed my memory properly over everything first. :-)

[olly]

Graham, in comment:27 you say:

FWIW, I have a patched version of mod_python which fixes the problem so at least main interpreter can be used, but am trying to sort out issues around trying to load that modified version at same time as mod_wsgi. Problem is that mod_python did things with threads wrongly and mod_wsgi had some fiddles in it to cope with that. Now that mod_python code has been fixed, mod_wsgi fails becauses its fiddles no longer apply. Trying to work out best thing to do. Maybe case that if want to use modified mod_python with mod_wsgi, may have to just use mod_wsgi 2.2 or later, and will make a change in mod_wsgi 2.2 to allow it to also work with that modified mod_python.

Did you get a chance finish off this approach?

[grahamd]

As far as I am concerned mod_python is pretty well a dead project. There has been no active development now for over two years and it is unlikely to be ported to Python 3.0. This may change if someone else comes along and wants to take up development of mod_python, but frankly, you would probably be better off getting your application to run on top of WSGI interface and use mod_wsgi for Apache instead. As such, not going to be pursuing trying to fix mod_python at this point.

[olly]

I've been looking into this issue.

If I patch SWIG to not generate all the calls to lock the GIL in situations where it is already locked, then this seems to fix the mod_python lock-ups. It's also knocks 11% of the stripped size of _xapian.so, and pythontest.py takes 2.7% less time to run (according to cachegrind's cycle estimate).

I'm going to see about applying this SWIG patch upstream as I think it will benefit other SWIG users too. Patch attached here for now, plus a version which checks that the GIL is locked when we assume it is (if not, we abort) and that it isn't locked when we assume it isn't (if not, we deadlock).

[olly]

Patch to remove redundant locking in SWIG generated wrappers

[olly]

Patch to remove redundant locking in SWIG generated wrappers (with checks that the GIL is/isn't locked as we expect)

[olly]

OK, patch applied to SWIG SVN; Xapian SVN trunk r12540 now uses this patched SWIG. Still need to test with mod_python to see if this avoids the deadlocks (and mod_wsgi to make sure we didn't break that).

[olly]

No deadlocks with mod_python or mod_wsgi, even when subclassing a matchdecider in python and calling it from the matcher. I used Python 2.6.

I guess we don't know if this fixes the issues with Python 2.3, but otherwise we're looking good here.

The main_interpreter stuff is still required currently. I think this restriction can be removed without hacking SWIG further - you can override what SWIG uses in the blocking/allowing thread begin/end boilerplate, so I think someone who understands the python threading stuff better than me "just" needs to write suitable replacements which store the interpreter in thread local storage and then we can plug them in.

[olly]

I've updated the documentation to cover the current status, and to document the fix for mod_wsgi - trunk r12635.

And I've spun off the "use the full threading API" as ticket #364, since I don't plan to fix address that issue right now.

The remaining issues in this ticket are now fixed in trunk. We should try to backport this for 1.0.x (though it isn't totally trivial to do so) so setting milestone:1.0.13 now.

Also updated the summary in the description.

[olly]

Backported just the fix for this to the snapshot of SWIG we are using for 1.0.x in branches/1.0 r12749, so this will be fixed in 1.0.13.

[olly]

Backported the documentation updates to branches/1.0 in r12819.