For CloudABI (https://mail.python.org/pipermail/python-dev/2016-July/145708.html) we're providing packages containing a precompiled copy of Python. As we had to make some changes to importlib (namely to deal with directory file descriptors), we have to do a rebuild of importlib.h during the cross compilation process, using Programs/_freeze_importlib.

We've run into a couple of issues that require us to patch up the build system to make this work:

- First of all, Programs/_freeze_importlib is built using the compiler for the target. There is no way to override this. The OpenWRT folks have also run into this issue and have patched up the Makefile to add a $FREEZE_IMPORTLIB that allows you to use a different executable:

https://github.com/openwrt/packages/blob/master/lang/python3/patches/013-make-freeze-import-lib-into-an-override-able-var.patch

This patch is almost correct; it doesn't prevent _freeze_importlib from being built, which is needed in our case as it doesn't build cleanly on CloudABI.

- Second, if an out-of-tree build is performed, we need to make sure that Python/frozen.c imports the right copy of importlib.h. We must add -IPython to the compiler flags to realise.

Attached is a patch that contains the modifications that we've made to Makefile.pre.in to make cross compilation work for us.

Can you get your importlib changes so they are compatible with a native build? If so, maybe you can build the frozen module with a native build rather than a cross build.

“freeze_importlib is built using the compiler for the target”: I may be mistaken, but I thought we recently prevented this from being cross-compiled: Issue 27641, revision bc677cb34889, Jul 2016. Your patch seems to be based ontop of this change. Perhaps you have found some way to bypass that cross-compilation detection logic.

The addition of $(FREEZE_IMPORTLIB) is a bit like the outdated cross-override.patch I once proposed for Issue 22625.

The nice thing is that in our case, the importlib changes are already compatible with the native build. So yes, we can reuse the frozen module from the native build. :-)

Ah, yes. Issue 27641 already prevents that it's cross compiled. This patch was written prior to that and simply rebased.

Still, one problem remains: our strategy for creating importlib.h and importlib_external.h doesn't take out-of-tree builds into account. Even if we regenerate it, frozen.c won't pick it up because its directory is not part of the compiler search path.

Attached is a patch that makes us write the results of _freeze_importlib into $(srcdir)/Python/importlib*.h instead, so that even with an out-of-tree build, we actually update the header files used by frozen.c.

There are various tricky cases to be considered with the regenerated files like importlib.h. One of them is that you are supposed to be able to build Python from a read-only source tree. See Issue 15819. Writing files into $(srcdir) would break this.

Also, as I just wrote in that bug, the built Python/importlib.h is already supposed to be searchable. However the logic seems to be broken. Can you see if this patch helps?

It does. I can now cross build Python for CloudABI by copying importlib.h and importlib_external.h from the native build directory to the target build directory. Thanks!

New changeset c26dce72a4da by Martin Panter in branch '3.5':
Issue #28066: Fix include search directory logic for out-of-tree builds
https://hg.python.org/cpython/rev/c26dce72a4da

New changeset 3ef078f96494 by Martin Panter in branch 'default':
Issue #28066: Merge srcdir fix from 3.5
https://hg.python.org/cpython/rev/3ef078f96494