
Porting Extension Modules to 3.0
********************************

author:
   Benjamin Peterson


Abstract
^^^^^^^^

Although changing the C-API was not one of Python 3.0's objectives,
the many Python level changes made leaving 2.x's API intact
impossible.  In fact, some changes such as ``int()`` and ``long()``
unification are more obvious on the C level.  This document endeavors
to document incompatibilities and how they can be worked around.


Conditional compilation
=======================

The easiest way to compile only some code for 3.0 is to check if
``PY_MAJOR_VERSION`` is greater than or equal to 3.

   #if PY_MAJOR_VERSION >= 3
   #define IS_PY3K
   #endif

API functions that are not present can be aliased to their equivalents
within conditional blocks.


Changes to Object APIs
======================

Python 3.0 merged together some types with similar functions while
cleanly separating others.


str/unicode Unification
-----------------------

Python 3.0's ``str()`` (``PyString_*`` functions in C) type is
equivalent to 2.x's ``unicode()`` (``PyUnicode_*``).  The old 8-bit
string type has become ``bytes()``.  Python 2.6 and later provide a
compatibility header, ``bytesobject.h``, mapping ``PyBytes`` names to
``PyString`` ones.  For best compatibility with 3.0, ``PyUnicode``
should be used for textual data and ``PyBytes`` for binary data.  It's
also important to remember that ``PyBytes`` and ``PyUnicode`` in 3.0
are not interchangeable like ``PyString`` and ``PyUnicode`` are in
2.x.  The following example shows best practices with regards to
``PyUnicode``, ``PyString``, and ``PyBytes``.

   #include "stdlib.h"
   #include "Python.h"
   #include "bytesobject.h"

   /* text example */
   static PyObject *
   say_hello(PyObject *self, PyObject *args) {
       PyObject *name, *result;

       if (!PyArg_ParseTuple(args, "U:say_hello", &name))
           return NULL;

       result = PyUnicode_FromFormat("Hello, %S!", name);
       return result;
   }

   /* just a forward */
   static char * do_encode(PyObject *);

   /* bytes example */
   static PyObject *
   encode_object(PyObject *self, PyObject *args) {
       char *encoded;
       PyObject *result, *myobj;

       if (!PyArg_ParseTuple(args, "O:encode_object", &myobj))
           return NULL;

       encoded = do_encode(myobj);
       if (encoded == NULL)
           return NULL;
       result = PyBytes_FromString(encoded);
       free(encoded);
       return result;
   }


long/int Unification
--------------------

In Python 3.0, there is only one integer type.  It is called ``int()``
on the Python level, but actually corresponds to 2.x's ``long()``
type.  In the C-API, ``PyInt_*`` functions are replaced by their
``PyLong_*`` neighbors.  The best course of action here is using the
``PyInt_*`` functions aliased to ``PyLong_*`` found in
``intobject.h``.  The abstract ``PyNumber_*`` APIs can also be used in
some cases.

   #include "Python.h"
   #include "intobject.h"

   static PyObject *
   add_ints(PyObject *self, PyObject *args) {
       int one, two;
       PyObject *result;

       if (!PyArg_ParseTuple(args, "ii:add_ints", &one, &two))
           return NULL;

       return PyInt_FromLong(one + two);
   }


Module initialization and state
===============================

Python 3.0 has a revamped extension module initialization system.
(See **PEP 3121**.)  Instead of storing module state in globals, they
should be stored in an interpreter specific structure.  Creating
modules that act correctly in both 2.x and 3.0 is tricky.  The
following simple example demonstrates how.

   #include "Python.h"

   struct module_state {
       PyObject *error;
   };

   #if PY_MAJOR_VERSION >= 3
   #define GETSTATE(m) ((struct module_state*)PyModule_GetState(m))
   #else
   #define GETSTATE(m) (&_state)
   static struct module_state _state;
   #endif

   static PyObject *
   error_out(PyObject *m) {
       struct module_state *st = GETSTATE(m);
       PyErr_SetString(st->error, "something bad happened");
       return NULL;
   }

   static PyMethodDef myextension_methods[] = {
       {"error_out", (PyCFunction)error_out, METH_NOARGS, NULL},
       {NULL, NULL}
   };

   #if PY_MAJOR_VERSION >= 3

   static int myextension_traverse(PyObject *m, visitproc visit, void *arg) {
       Py_VISIT(GETSTATE(m)->error);
       return 0;
   }

   static int myextension_clear(PyObject *m) {
       Py_CLEAR(GETSTATE(m)->error);
       return 0;
   }


   static struct PyModuleDef moduledef = {
           PyModuleDef_HEAD_INIT,
           "myextension",
           NULL,
           sizeof(struct module_state),
           myextension_methods,
           NULL,
           myextension_traverse,
           myextension_clear,
           NULL
   };

   #define INITERROR return NULL

   PyObject *
   PyInit_myextension(void)

   #else
   #define INITERROR return

   void
   initmyextension(void)
   #endif
   {
   #if PY_MAJOR_VERSION >= 3
       PyObject *module = PyModule_Create(&moduledef);
   #else
       PyObject *module = Py_InitModule("myextension", myextension_methods);
   #endif

       if (module == NULL)
           INITERROR;
       struct module_state *st = GETSTATE(module);

       st->error = PyErr_NewException("myextension.Error", NULL, NULL);
       if (st->error == NULL) {
           Py_DECREF(module);
           INITERROR;
       }

   #if PY_MAJOR_VERSION >= 3
       return module;
   #endif
   }


CObject replaced with Capsule
=============================

The ``Capsule`` object was introduced in Python 3.1 and 2.7 to replace
``CObject``.  CObjects were useful, but the ``CObject`` API was
problematic: it didn't permit distinguishing between valid CObjects,
which allowed mismatched CObjects to crash the interpreter, and some
of its APIs relied on undefined behavior in C. (For further reading on
the rationale behind Capsules, please see issue 5630.)

If you're currently using CObjects, and you want to migrate to 3.1 or
newer, you'll need to switch to Capsules. ``CObject`` was deprecated
in 3.1 and 2.7 and completely removed in Python 3.2.  If you only
support 2.7, or 3.1 and above, you can simply switch to ``Capsule``.
If you need to support 3.0 or versions of Python earlier than 2.7
you'll have to support both CObjects and Capsules.

The following example header file ``capsulethunk.h`` may solve the
problem for you; simply write your code against the ``Capsule`` API,
include this header file after ``"Python.h"``, and you'll
automatically use CObjects in Python 3.0 or versions earlier than 2.7.

``capsulethunk.h`` simulates Capsules using CObjects.  However,
``CObject`` provides no place to store the capsule's "name".  As a
result the simulated ``Capsule`` objects created by ``capsulethunk.h``
behave slightly differently from real Capsules.  Specifically:

   * The name parameter passed in to ``PyCapsule_New()`` is ignored.

   * The name parameter passed in to ``PyCapsule_IsValid()`` and
     ``PyCapsule_GetPointer()`` is ignored, and no error checking of
     the name is performed.

   * ``PyCapsule_GetName()`` always returns NULL.

   * ``PyCapsule_SetName()`` always throws an exception and returns
     failure.  (Since there's no way to store a name in a CObject,
     noisy failure of ``PyCapsule_SetName()`` was deemed preferable to
     silent failure here.  If this is inconveient, feel free to modify
     your local copy as you see fit.)

You can find ``capsulethunk.h`` in the Python source distribution in
the ``Doc/includes`` directory.  We also include it here for your
reference; here is ``capsulethunk.h``:

   #ifndef __CAPSULETHUNK_H
   #define __CAPSULETHUNK_H

   #if (    (PY_VERSION_HEX <  0x02070000) \
        || ((PY_VERSION_HEX >= 0x03000000) \
         && (PY_VERSION_HEX <  0x03010000)) )

   #define __PyCapsule_GetField(capsule, field, default_value) \
       ( PyCapsule_CheckExact(capsule) \
           ? (((PyCObject *)capsule)->field) \
           : (default_value) \
       ) \

   #define __PyCapsule_SetField(capsule, field, value) \
       ( PyCapsule_CheckExact(capsule) \
           ? (((PyCObject *)capsule)->field = value), 1 \
           : 0 \
       ) \


   #define PyCapsule_Type PyCObject_Type

   #define PyCapsule_CheckExact(capsule) (PyCObject_Check(capsule))
   #define PyCapsule_IsValid(capsule, name) (PyCObject_Check(capsule))


   #define PyCapsule_New(pointer, name, destructor) \
       (PyCObject_FromVoidPtr(pointer, destructor))


   #define PyCapsule_GetPointer(capsule, name) \
       (PyCObject_AsVoidPtr(capsule))

   /* Don't call PyCObject_SetPointer here, it fails if there's a destructor */
   #define PyCapsule_SetPointer(capsule, pointer) \
       __PyCapsule_SetField(capsule, cobject, pointer)


   #define PyCapsule_GetDestructor(capsule) \
       __PyCapsule_GetField(capsule, destructor)

   #define PyCapsule_SetDestructor(capsule, dtor) \
       __PyCapsule_SetField(capsule, destructor, dtor)


   /*
    * Sorry, there's simply no place
    * to store a Capsule "name" in a CObject.
    */
   #define PyCapsule_GetName(capsule) NULL

   static int
   PyCapsule_SetName(PyObject *capsule, const char *unused)
   {
       unused = unused;
       PyErr_SetString(PyExc_NotImplementedError,
           "can't use PyCapsule_SetName with CObjects");
       return 1;
   }



   #define PyCapsule_GetContext(capsule) \
       __PyCapsule_GetField(capsule, descr)

   #define PyCapsule_SetContext(capsule, context) \
       __PyCapsule_SetField(capsule, descr, context)


   static void *
   PyCapsule_Import(const char *name, int no_block)
   {
       PyObject *object = NULL;
       void *return_value = NULL;
       char *trace;
       size_t name_length = (strlen(name) + 1) * sizeof(char);
       char *name_dup = (char *)PyMem_MALLOC(name_length);

       if (!name_dup) {
           return NULL;
       }

       memcpy(name_dup, name, name_length);

       trace = name_dup;
       while (trace) {
           char *dot = strchr(trace, '.');
           if (dot) {
               *dot++ = '\0';
           }

           if (object == NULL) {
               if (no_block) {
                   object = PyImport_ImportModuleNoBlock(trace);
               } else {
                   object = PyImport_ImportModule(trace);
                   if (!object) {
                       PyErr_Format(PyExc_ImportError,
                           "PyCapsule_Import could not "
                           "import module \"%s\"", trace);
                   }
               }
           } else {
               PyObject *object2 = PyObject_GetAttrString(object, trace);
               Py_DECREF(object);
               object = object2;
           }
           if (!object) {
               goto EXIT;
           }

           trace = dot;
       }

       if (PyCObject_Check(object)) {
           PyCObject *cobject = (PyCObject *)object;
           return_value = cobject->cobject;
       } else {
           PyErr_Format(PyExc_AttributeError,
               "PyCapsule_Import \"%s\" is not valid",
               name);
       }

   EXIT:
       Py_XDECREF(object);
       if (name_dup) {
           PyMem_FREE(name_dup);
       }
       return return_value;
   }

   #endif /* #if PY_VERSION_HEX < 0x02070000 */

   #endif /* __CAPSULETHUNK_H */


Other options
=============

If you are writing a new extension module, you might consider Cython.
It translates a Python-like language to C.  The extension modules it
creates are compatible with Python 3.x and 2.x.
