parallel processing - Python C Extension OpenMP -

i getting segmentation violation in python interpreter when trying access variable returned own openmp c++ extension.

all solutions have found either using ctypes or cython (which cannot use). http://snatverk.blogspot.de/2012/03/c-parallel-modules-for-python.html shows small example of openmp enabled python extension. although tried implement loops in example, still not work.

my code extension code function looks this:

static pyobject * matcher_match(pyobject *self, pyobject *args) {  if(pytuple_size(args) != 2) {     return null; }  pyobject *names = pytuple_getitem(args, 0); py_ssize_t namessize = pylist_size(names);  pyobject *namesb = pytuple_getitem(args, 1); py_ssize_t namesbsize = pylist_size(namesb);  pyobject *matchidcs = pylist_new(namessize);  py_begin_allow_threads;  int i, j; #pragma omp parallel private(i, j) for(i = 0; < namessize; i++) {     for(j = 0; j < namesbsize; j++)     {         // test_pair_ij pure c function without callbacks python         // uses c++ stl std::vector         float = pyfloat_asdouble(pylist_getitem(names, i));         float b = pyfloat_asdouble(pylist_getitem(namesb, j));         bool res = test_pair_ij(a, b)         pyobject *matchval;         if(res)         {             matchval = py_buildvalue("i", j);         }         else         {             matchval = py_buildvalue("i", -1);         }         pylist_setitem(matchidcs, i, matchval);     } } py_end_allow_threads;  return matchidcs; } 

the function matcher_match() receives 2 lists, names , namesb. check every combination of names , namesb (their float attributes) specific condition indicated function test_pair_ij(). function pure c(++) implementation not callback python.

the c extension called with:

from matcher import match # random lists example names = ['123', '231', ...] namesb = ['342', ...] matchresult = match(names, namesb) import pandas pd mr = pd.series(matchresult) mr.to_csv('matchresult.csv') 

when lists names , namesb rather small, code running ok. larger lists, cannot access matchresult anymore in python code. when try to, segmentation violation (which inside python interpreter guess). have recompiled c extension without openmp , ran ok again, larger lists.

i guess problem messup in memory of python variables access extension. may have gil, although releasing , acquiring it. need make more variables private in case? other ideas on this?

edit: fixed calling arguments of function test_pair_ij.

edit 2: fixed code of storing matchidcs

answer:

the code releasing gil , call pylist_setitem(matchidcs, i, matchval); modifying python structure, not allowed (see http://docs.cython.org/src/userguide/external_c_code.html#releasing-the-gil).