Make boost compatible with python limited API

[Raincloud7]

Currently boost python cannot be compiled with limited python API. To support many python versions it would be helpful to have this option.

https://docs.python.org/3/c-api/stable.html
http://code.activestate.com/lists/python-cplusplus-sig/17408/
https://www.python.org/dev/peps/pep-0384/

[cas--]

To help with searching, specifically ability to define Py_LIMITED_API

[cas--]

I have made a start on this and replace the methods in #270 which builds and passes tests.

However defining Py_LIMITED_API results in the build errors below:

--- include/boost/python/detail/wrap_python.hpp
+++ include/boost/python/detail/wrap_python.hpp
@@ -175,6 +175,7 @@ typedef int pid_t;
 #if PY_MAJOR_VERSION == 2 && PY_MINOR_VERSION == 2 && PY_MICRO_VERSION < 2
 # include <boost/python/detail/python22_fixed.h>
 #else
+#define Py_LIMITED_API
 # include <Python.h>
 #endif

Snippet of build error:

gxx.compile src.list.o
In file included from include/boost/python/object/make_ptr_instance.hpp:8:0,
                 from include/boost/python/to_python_indirect.hpp:11,
                 from include/boost/python/converter/arg_to_python.hpp:10,
                 from include/boost/python/call.hpp:15,
                 from include/boost/python/object_core.hpp:14,
                 from include/boost/python/object.hpp:9,
                 from include/boost/python/list.hpp:10,
                 from ./src/list.cpp:5:
include/boost/python/object/make_instance.hpp: In static member function ‘static PyObject* boost::python::objects::make_instance_impl<T, Holder, Derived>::execute(Arg&)’:
include/boost/python/object/make_instance.hpp:35:36: error: invalid use of incomplete type ‘PyTypeObject {aka struct _typeobject}’
         PyObject* raw_result = type->tp_alloc(

I'm a little out of my depth here so some expertise would be welcome

[earonesty]

That error is because tp_alloc is not available in the limited api. Follow https://www.python.org/dev/peps/pep-0384/ to fix. Basically, you have to walk through all the failures, change boost to using the limited api for as many as you can (a good practice anyway), and disable the feature if you can't. It's a big chore.

[AllSeeingEyeTolledEweSew]

This is really important.

Compilation times with boost are already huge, since boost is meant for large-scale C++ projects.

Lack of support for abi3 / Py_LIMITED_API means that users have to build and ship 5x as many artifacts, to cover all currently-supported python versions (3.6 through 3.10 as of writing).

[cas--]

I thought I would do some research on this to determine what effort is required to get to a stable abi. The only downsides I have read is that it might come at a small performance cost.

cPython Reference:

• https://www.python.org/dev/peps/pep-0384/ (Although quite outdated now)
• https://docs.python.org/3.11/c-api/stable.html
• https://github.com/python/cpython/blob/main/Tools/scripts/abitype.py

Push to maintain the stable abi:

• https://pythoncapi.readthedocs.io/stable_abi.html
• https://github.com/encukou/abi3 resulting in pep-0652

Other projects with in-progress work:

• Support Python 3 stable API swig/swig#1009 and wip: Support stable Python ABI swig/swig#1613
• https://sourceforge.net/p/cxx/patches/28/

Building the pip wheel with py_limited_api:

• https://stackoverflow.com/a/66055366/175584
• Add support for Py_LIMITED_API ? pypa/cibuildwheel#78

For the current code there is usage of fields that will need alternatives to just using the Py_ prefix:

• tp_name e.g. Py_TYPE(callable)->tp_name see: https://bugs.python.org/issue42035
• tp_dict ?
• tp_flags -> PyType_GetFlags
• tp_basicsize PyType_Spec?
• tp_as_number PyNumberMethods are only used in Python 2 blocks

[cas--]

@AllSeeingEyeTolledEweSew I am not sure whether discussing on the boost mailing lists might get more traction?