python39 installs with or without the universal variant, but when installed with the universal variant, it doesn't work. This is a regression from python38 and earlier versions which work fine universal or not.
For example when python39 is installed universal, py39-setuptools fails to build:
Traceback (most recent call last):
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/subprocess.py", line 73, in <module>
import msvcrt
ModuleNotFoundError: No module named 'msvcrt'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/local/var/macports/build/_Users_rschmidt_macports_macports-ports-ryandesign-fork_python_py-setuptools/py39-setuptools/work/setuptools-50.3.0/setup.py", line 10, in <module>
import setuptools
File "/opt/local/var/macports/build/_Users_rschmidt_macports_macports-ports-ryandesign-fork_python_py-setuptools/py39-setuptools/work/setuptools-50.3.0/setuptools/__init__.py", line 10, in <module>
import distutils.core
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/distutils/core.py", line 16, in <module>
from distutils.dist import Distribution
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/distutils/dist.py", line 19, in <module>
from distutils.util import check_environ, strtobool, rfc822_escape
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/distutils/util.py", line 14, in <module>
from distutils.spawn import spawn
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/distutils/spawn.py", line 11, in <module>
import subprocess
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/subprocess.py", line 78, in <module>
import _posixsubprocess
ModuleNotFoundError: No module named '_posixsubprocess'
This can be reproduced easily at the command line:
$ python3.9
Python 3.9.0 (default, Oct 6 2020, 19:05:10)
[Clang 9.1.0 (clang-902.0.39.2)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import subprocess
Traceback (most recent call last):
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/subprocess.py", line 73, in <module>
import msvcrt
ModuleNotFoundError: No module named 'msvcrt'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/subprocess.py", line 78, in <module>
import _posixsubprocess
ModuleNotFoundError: No module named '_posixsubprocess'
Examining the contents of python39, I see that when it is installed non-universal, it contains the file:
/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/lib-dynload/_posixsubprocess.cpython-39-darwin.so
But when it is installed universal, that file name contains "failed":
/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/lib-dynload/_posixsubprocess.cpython-39-darwin_failed.so
There are many other "failed" files in the universal install.
main.success.log.bz2 (166.7 KB) - added by ryandesign (Ryan Carsten Schmidt) 3 years ago.
successful build of python39 @3.9.2 with jmr's change
$ port -vq installed python39
python39 @3.9.1_0+universal (active) platform='darwin 17' archs='i386 x86_64' date='2020-12-18T07:33:31-0600'
$ python3.9
Python 3.9.1 (default, Dec 18 2020, 07:27:59)
[Clang 9.1.0 (clang-902.0.39.2)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import subprocess
Traceback (most recent call last):
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/subprocess.py", line 73, in <module>
import msvcrt
ModuleNotFoundError: No module named 'msvcrt'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/subprocess.py", line 78, in <module>
import _posixsubprocess
ModuleNotFoundError: No module named '_posixsubprocess'
A universal build of python39 with arm/x86_64 seems to work (on BigSur Intel at present):
% port -vq installed python39
python39 @3.9.1_0+universal (active) platform='darwin 20' archs='arm64 x86_64' date='2021-01-10T10:07:17-0800'
% python3.9
Python 3.9.1 (default, Jan 10 2021, 10:06:23)
[Clang 12.0.0 (clang-1200.0.32.28)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import subprocess
Version 0, edited 3 years ago
by kencu (Ken)
(next)
$ port -vq installed python39
python39 @3.9.1_0+universal (active) platform='darwin 10' archs='i386 x86_64' date='2021-01-10T18:29:47-0800'
$ file "/Applications/MacPorts/Python 3.9/IDLE.app/Contents/MacOS/Python"
/Applications/MacPorts/Python 3.9/IDLE.app/Contents/MacOS/Python: Mach-O universal binary with 2 architectures
/Applications/MacPorts/Python 3.9/IDLE.app/Contents/MacOS/Python (for architecture x86_64): Mach-O 64-bit executable x86_64
/Applications/MacPorts/Python 3.9/IDLE.app/Contents/MacOS/Python (for architecture i386): Mach-O executable i386
$ python3.9
Python 3.9.1 (default, Jan 10 2021, 18:21:38)
[Clang 9.0.1 ] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import subprocess
$ arch -i386 python3.9
Python 3.9.1 (default, Jan 10 2021, 18:21:38)
[Clang 9.0.1 ] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import subprocess
Traceback (most recent call last):
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/subprocess.py", line 73, in <module>
import msvcrt
ModuleNotFoundError: No module named 'msvcrt'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/subprocess.py", line 78, in <module>
import _posixsubprocess
ImportError: dynamic module does not define module export function (PyInit__posixsubprocess)
Not that much changed between 3.7.9 and 3.7.10: https://github.com/python/cpython/compare/v3.7.9..v3.7.10
or between 3.6.12 and 3.6.13: https://github.com/python/cpython/compare/v3.6.12..v3.6.13
Aha! Thank you for pointing that out. Then I think it is something I changed on my machine some months ago: I build with this change to base:
--- src/port1.0/portconfigure.tcl (revision 155969)
+++ src/port1.0/portconfigure.tcl (working copy)
@@ -235,8 +235,8 @@
configure.classpath
# compiler flags section
default configure.optflags -Os
-default configure.cflags {${configure.optflags}}
-default configure.objcflags {${configure.optflags}}
+default configure.cflags {${configure.optflags} -Werror=implicit-function-declaration}
+default configure.objcflags {${configure.optflags} -Werror=implicit-function-declaration}
default configure.cppflags {[portconfigure::configure_get_cppflags]}
proc portconfigure::configure_get_cppflags {} {
global prefix
If I rebuild python39 @3.9.2 +universal without that, then it works.
Let me get clean build logs both with and without that and let's see how they differ.
It looks like python's build system must do some kind of word wrapping when it creates the file _sysconfigdata__darwin_darwin.py (perhaps to comply with PEP-8). The Portfile uses reinplace
to remove -arch
flags from _sysconfigdata__darwin_darwin.py (see #39669 where that was first added) but because my CFLAGS
are longer, some of the -arch
flags happen to end up broken across multiple lines due to the word wrapping, so reinplace
doesn't remove all of them. (Even in the default build, it doesn't remove all of them—look at CONFIGURE_CFLAGS
, PY_CORE_CFLAGS
, and PY_STDMODULE_CFLAGS
.) Here's the diff between how that file got generated in a standard "vanilla" build and how it got generated with -Werror=implicit-function-declaration
added:
655 '-pipe -Os -Werror=implicit-function-declaration '
656 ' -std=c99 -Wextra '
657 '-Wno-unused-result -Wno-unused-parameter '
671 'x86_64 -pipe -Os '
672 '-std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter '
673 '-Wno-missing-field-initializers -Wstrict-prototypes '
674 '-Werror=implicit-function-declaration -fvisibility=hidden '
675 '-I./Include/internal -I. -I./Include -I/opt/local/include '
676 '-DPy_BUILD_CORE',
676 'x86_64 -pipe -Os '
677 '-Werror=implicit-function-declaration -arch '
678 'i386 -std=c99 -Wextra -Wno-unused-result '
679 '-Wno-unused-parameter -Wno-missing-field-initializers '
680 '-Wstrict-prototypes -Werror=implicit-function-declaration '
681 '-fvisibility=hidden -I./Include/internal -I. -I./Include '
682 '-I/opt/local/include -DPy_BUILD_CORE',
694 ' -pipe -Os '
695 '-Werror=implicit-function-declaration '
696 ' -std=c99 -Wextra -Wno-unused-result '
Note that in both main.logs some stuff gets compiled in the destroot phase, with that modified sysconfig. In main.2.log, you can see some things being compiled with only -arch i386
. The post-build
block tries to save off the sysconfig mtime before changing it and restoring it afterward so that things don't get rebuilt, and yet...
Summary:
python39 @3.9.0, python38 @3.8.7, python37 @3.7.10, python36 @3.6.13: +universal does not work →
python39 @3.9.0, python38 @3.8.7, python37 @3.7.10, python36 @3.6.13: +universal does not work with longer than normal CFLAGS
Ooh, thanks. I was just thinking about making this my project for the evening, but you beat me to it.
It looks good to me. Attaching my log with this change (plus a revbump to 1).
Changing the awk to a reinplace... is that just stylistic or is there a functional change?
Any idea why some stuff still gets built in the destroot phase?
I wonder if this will also fix #59466. (Edit: no, this issue only affects python34 and later but that issue was with python27.)
Prevent line wrapping in _sysconfigdata.py so that '-arch foo' can't be
split across multiple lines, preventing the reinplace from removing it
correctly. In practice this only seems to have happened with universal
builds.
All variables being on a single line also means that awk is no longer
needed to edit LINKFORSHARED.
Also fix implicit declaration errors in python35's configure script.
Closes: #61282