Static Python Build errors - Python Help - Discussions on Python.org

link管理
链接快照平台
输入网页链接，自动生成快照
标签化管理网页链接
相关文章推荐
爱听歌的茶叶 · 【恒有数入门手册】Python安装教程 - ...· 17 小时前 ·
安静的松树 · ImportError: DLL load ...· 15 小时前 ·
魁梧的排球 · Issue ...· 15 小时前 ·
机灵的手电筒 · Re:openvino.runtime - ...· 15 小时前 ·
粗眉毛的青蛙 · spark ...· 7 小时前 ·
豪情万千的吐司 · 云书网-开启美好生活· 4 月前 ·
瘦瘦的刺猬 · new expression - ...· 4 月前 ·
直爽的棒棒糖 · 2024年杭州市区中考招生政策发布· 5 月前 ·
火爆的电影票 · 孙南翔：法律域外适用体系建设中的管辖权：演化 ...· 7 月前 ·
一身肌肉的板栗 · SVG 颜色 – SVG Color – ...· 1 年前 ·
We need a static Python build. See the build flags below, but somehow the -shared flag keep being present in the gcc commands. Could somebody please please help us with our static python build getting working?
I’m aware I most likely also need to build openssl and libffi in a static manner. But let’s try to tackle the problem one by one.
We use a Ubuntu 20.04 LTS Docker image. And we try to build the latest version ( 3.12.2 ) from source at this moment in time.
This is the configure command:
./configure \
    --enable-optimizations \
    --with-ensurepip=install \
    --prefix="$rollout" \
    --disable-shared \
    --with-static-libpython \
    LDFLAGS="-static -static-libgcc" \
    CFLAGS="-static" \
    CPPFLAGS="-static" \
    CFLAGSFORSHARED="" CCSHARED="" LDSHARED="" LDCXXSHARED="" \
    MODULE_BUILDTYPE=static
Then I also added *static* for the Python modules (to this special Setup.local file):
echo "*static*" >> Modules/Setup.local
And finally the make command itself:
make LINKFORSHARED=" " CCSHARED="" LDSHARED="" LDCXXSHARED=""  -j 10
Now the gcc command will have -static flag but also still the -shared flag!? How to get rid of this -shared flag:
gcc -pthread -shared -static -static-libgcc  -fno-semantic-interposition -fprofile-generate  Modules/_testimportmultiple.o   -o Modules/_testimportmultiple.cpython-312-x86_64-linux-gnu.so
gcc -pthread -shared -static -static-libgcc  -fno-semantic-interposition -fprofile-generate  Modules/_testsinglephase.o   -o Modules/_testsinglephase.cpython-312-x86_64-linux-gnu.so
gcc -pthread -shared -static -static-libgcc  -fno-semantic-interposition -fprofile-generate  Modules/xxlimited.o   -o Modules/xxlimited.cpython-312-x86_64-linux-gnu.so
/usr/bin/ld: /usr/lib/gcc/x86_64-linux-gnu/9/crtbeginT.o: relocation R_X86_64_32 against hidden symbol `__TMC_END__' can not be used when making a shared object
/usr/bin/ld: Modules/_testimportmultiple.o: relocation R_X86_64_PC32 against symbol `PyInit__testimportmultiple_bar' can not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: final link failed: bad value
What do I wrong? Why has the gcc commands still -shared in the commands? Thanks in advance.
Ps. And no you don’t want to add -fPIC, since that is used for shared libs only.
Ps. Ps. Some commands seems fine like: gcc -pthread  -fno-strict-overflow -DNDEBUG -g -O3 -Wall -static   -fno-semantic-interposition -std=c11 -Werror=implicit-function-declaration -fvisibility=hidden -fprofile-generate -I./Include/internal  -I. -I./Include -static   -DPy_BUILD_CORE_BUILTIN -c ./Modules/itertoolsmodule.c -o Modules/itertoolsmodule.o. But then later it swiches to modules and it fails:
rm -f Modules/_hacl/libHacl_Hash_SHA2.a
rm -f Modules/expat/libexpat.a
ar rcs Modules/_hacl/libHacl_Hash_SHA2.a Modules/_hacl/Hacl_Hash_SHA2.o
ar rcs Modules/expat/libexpat.a Modules/expat/xmlparse.o Modules/expat/xmlrole.o Modules/expat/xmltok.o
gcc -pthread -shared -static -static-libgcc  -fno-semantic-interposition -fprofile-generate  Modules/_testimportmultiple.o   -o Modules/_testimportmultiple.cpython-312-x86_64-linux-gnu.so
gcc -pthread -shared -static -static-libgcc  -fno-semantic-interposition -fprofile-generate  Modules/_testsinglephase.o   -o Modules/_testsinglephase.cpython-312-x86_64-linux-gnu.so
gcc -pthread -shared -static -static-libgcc  -fno-semantic-interposition -fprofile-generate  Modules/xxlimited.o   -o Modules/xxlimited.cpython-312-x86_64-linux-gnu.so
gcc -pthread -shared -static -static-libgcc  -fno-semantic-interposition -fprofile-generate  Modules/xxlimited_35.o   -o Modules/xxlimited_35.cpython-312-x86_64-linux-gnu.so
/usr/bin/ld: /usr/lib/gcc/x86_64-linux-gnu/9/crtbeginT.o: relocation R_X86_64_32 against hidden symbol `__TMC_END__' can not be used when making a shared object
Regards,

Melroy van den Berg
              Which Python 3.11? I tried Python 3.11.0, however it might seems to continue futher indeed. If you look at the output, you might notice, still the wrong build flags sometimes (-shared flag again) as well as ld errors (but it’s non-blocking in Python 3.11 build):
2024-02-14T14:12:25.3323680Z gcc -pthread -fPIC -DNDEBUG -g -fwrapv -O3 -Wall -static -fno-semantic-interposition -std=c11 -Werror=implicit-function-declaration -fvisibility=hidden -fprofile-generate -I./Include/internal -I./Include -I. -I/usr/include/x86_64-linux-gnu -I/usr/local/include -I/__w/OpenRepo-Python/OpenRepo-Python/Python-3.11.0/Include -I/__w/OpenRepo-Python/OpenRepo-Python/Python-3.11.0 -c /__w/OpenRepo-Python/OpenRepo-Python/Python-3.11.0/Modules/readline.c -o build/temp.linux-x86_64-3.11/__w/OpenRepo-Python/OpenRepo-Python/Python-3.11.0/Modules/readline.o
2024-02-14T14:12:25.3324815Z building '_curses_panel' extension
2024-02-14T14:12:25.3326557Z gcc -pthread -fPIC -DNDEBUG -g -fwrapv -O3 -Wall -static -fno-semantic-interposition -std=c11 -Werror=implicit-function-declaration -fvisibility=hidden -fprofile-generate -I./Include/internal -DHAVE_NCURSESW=1 -I/usr/include/ncursesw -I./Include -I. -I/usr/include/x86_64-linux-gnu -I/usr/local/include -I/__w/OpenRepo-Python/OpenRepo-Python/Python-3.11.0/Include -I/__w/OpenRepo-Python/OpenRepo-Python/Python-3.11.0 -c /__w/OpenRepo-Python/OpenRepo-Python/Python-3.11.0/Modules/_curses_panel.c -o build/temp.linux-x86_64-3.11/__w/OpenRepo-Python/OpenRepo-Python/Python-3.11.0/Modules/_curses_panel.o
2024-02-14T14:12:25.3384170Z gcc -pthread -fPIC -DNDEBUG -g -fwrapv -O3 -Wall -static -fno-semantic-interposition -std=c11 -Werror=implicit-function-declaration -fvisibility=hidden -fprofile-generate -I./Include/internal -DHAVE_NCURSESW=1 -I/usr/include/ncursesw -I./Include -I. -I/usr/include/x86_64-linux-gnu -I/usr/local/include -I/__w/OpenRepo-Python/OpenRepo-Python/Python-3.11.0/Include -I/__w/OpenRepo-Python/OpenRepo-Python/Python-3.11.0 -c /__w/OpenRepo-Python/OpenRepo-Python/Python-3.11.0/Modules/_cursesmodule.c -o build/temp.linux-x86_64-3.11/__w/OpenRepo-Python/OpenRepo-Python/Python-3.11.0/Modules/_cursesmodule.o
2024-02-14T14:12:25.3433420Z building '_crypt' extension
2024-02-14T14:12:25.3434034Z building '_md5' extension
2024-02-14T14:12:25.3436365Z gcc -pthread -fPIC -DNDEBUG -g -fwrapv -O3 -Wall -static -fno-semantic-interposition -std=c11 -Werror=implicit-function-declaration -fvisibility=hidden -fprofile-generate -I./Include/internal -I./Include -I. -I/usr/include/x86_64-linux-gnu -I/usr/local/include -I/__w/OpenRepo-Python/OpenRepo-Python/Python-3.11.0/Include -I/__w/OpenRepo-Python/OpenRepo-Python/Python-3.11.0 -c /__w/OpenRepo-Python/OpenRepo-Python/Python-3.11.0/Modules/md5module.c -o build/temp.linux-x86_64-3.11/__w/OpenRepo-Python/OpenRepo-Python/Python-3.11.0/Modules/md5module.o
2024-02-14T14:12:25.3440023Z gcc -pthread -fPIC -DNDEBUG -g -fwrapv -O3 -Wall -static -fno-semantic-interposition -std=c11 -Werror=implicit-function-declaration -fvisibility=hidden -fprofile-generate -I./Include/internal -I./Include -I. -I/usr/include/x86_64-linux-gnu -I/usr/local/include -I/__w/OpenRepo-Python/OpenRepo-Python/Python-3.11.0/Include -I/__w/OpenRepo-Python/OpenRepo-Python/Python-3.11.0 -c /__w/OpenRepo-Python/OpenRepo-Python/Python-3.11.0/Modules/_cryptmodule.c -o build/temp.linux-x86_64-3.11/__w/OpenRepo-Python/OpenRepo-Python/Python-3.11.0/Modules/_cryptmodule.o
2024-02-14T14:12:25.3441602Z building '_sha1' extension
2024-02-14T14:12:25.3443897Z gcc -pthread -fPIC -DNDEBUG -g -fwrapv -O3 -Wall -static -fno-semantic-interposition -std=c11 -Werror=implicit-function-declaration -fvisibility=hidden -fprofile-generate -I./Include/internal -I./Include -I. -I/usr/include/x86_64-linux-gnu -I/usr/local/include -I/__w/OpenRepo-Python/OpenRepo-Python/Python-3.11.0/Include -I/__w/OpenRepo-Python/OpenRepo-Python/Python-3.11.0 -c /__w/OpenRepo-Python/OpenRepo-Python/Python-3.11.0/Modules/sha1module.c -o build/temp.linux-x86_64-3.11/__w/OpenRepo-Python/OpenRepo-Python/Python-3.11.0/Modules/sha1module.o
2024-02-14T14:12:25.3445961Z building '_sha256' extension
2024-02-14T14:12:25.3446696Z building '_sha512' extension
2024-02-14T14:12:25.4214020Z gcc -pthread -fPIC -DNDEBUG -g -fwrapv -O3 -Wall -static -fno-semantic-interposition -std=c11 -Werror=implicit-function-declaration -fvisibility=hidden -fprofile-generate -I./Include/internal -I./Include -I. -I/usr/include/x86_64-linux-gnu -I/usr/local/include -I/__w/OpenRepo-Python/OpenRepo-Python/Python-3.11.0/Include -I/__w/OpenRepo-Python/OpenRepo-Python/Python-3.11.0 -c /__w/OpenRepo-Python/OpenRepo-Python/Python-3.11.0/Modules/sha512module.c -o build/temp.linux-x86_64-3.11/__w/OpenRepo-Python/OpenRepo-Python/Python-3.11.0/Modules/sha512module.o
2024-02-14T14:12:25.4323142Z gcc -pthread -fPIC -DNDEBUG -g -fwrapv -O3 -Wall -static -fno-semantic-interposition -std=c11 -Werror=implicit-function-declaration -fvisibility=hidden -fprofile-generate -I./Include/internal -I./Include -I. -I/usr/include/x86_64-linux-gnu -I/usr/local/include -I/__w/OpenRepo-Python/OpenRepo-Python/Python-3.11.0/Include -I/__w/OpenRepo-Python/OpenRepo-Python/Python-3.11.0 -c /__w/OpenRepo-Python/OpenRepo-Python/Python-3.11.0/Modules/sha256module.c -o build/temp.linux-x86_64-3.11/__w/OpenRepo-Python/OpenRepo-Python/Python-3.11.0/Modules/sha256module.o
2024-02-14T14:12:25.5252775Z gcc -pthread -shared -static -static-libgcc -fno-semantic-interposition -fprofile-generate build/temp.linux-x86_64-3.11/__w/OpenRepo-Python/OpenRepo-Python/Python-3.11.0/Modules/_contextvarsmodule.o -L/usr/lib/x86_64-linux-gnu -L/usr/local/lib -o build/lib.linux-x86_64-3.11/_contextvars.cpython-311-x86_64-linux-gnu.so
2024-02-14T14:12:25.7213043Z gcc -pthread -shared -static -static-libgcc -fno-semantic-interposition -fprofile-generate build/temp.linux-x86_64-3.11/__w/OpenRepo-Python/OpenRepo-Python/Python-3.11.0/Modules/_statisticsmodule.o -L/usr/lib/x86_64-linux-gnu -L/usr/local/lib -lm -o build/lib.linux-x86_64-3.11/_statistics.cpython-311-x86_64-linux-gnu.so
2024-02-14T14:12:25.8214007Z gcc -pthread -shared -static -static-libgcc -fno-semantic-interposition -fprofile-generate build/temp.linux-x86_64-3.11/__w/OpenRepo-Python/OpenRepo-Python/Python-3.11.0/Modules/_typingmodule.o -L/usr/lib/x86_64-linux-gnu -L/usr/local/lib -o build/lib.linux-x86_64-3.11/_typing.cpython-311-x86_64-linux-gnu.so
2024-02-14T14:12:25.8273847Z gcc -pthread -shared -static -static-libgcc -fno-semantic-interposition -fprofile-generate build/temp.linux-x86_64-3.11/__w/OpenRepo-Python/OpenRepo-Python/Python-3.11.0/Modules/_opcode.o -L/usr/lib/x86_64-linux-gnu -L/usr/local/lib -o build/lib.linux-x86_64-3.11/_opcode.cpython-311-x86_64-linux-gnu.so
2024-02-14T14:12:25.8333112Z /usr/bin/ld: /usr/lib/gcc/x86_64-linux-gnu/9/crtbeginT.o: relocation R_X86_64_32 against hidden symbol `__TMC_END__' can not be used when making a shared object
2024-02-14T14:12:25.9212303Z collect2: error: ld returned 1 exit status
2024-02-14T14:12:25.9222126Z /usr/bin/ld: /usr/lib/gcc/x86_64-linux-gnu/9/crtbeginT.o: relocation R_X86_64_32 against hidden symbol `__TMC_END__' can not be used when making a shared object
2024-02-14T14:12:25.9222566Z collect2: error: ld returned 1 exit status
2024-02-14T14:12:25.9369774Z /usr/bin/ld: /usr/lib/gcc/x86_64-linux-gnu/9/crtbeginT.o: relocation R_X86_64_32 against hidden symbol `__TMC_END__' can not be used when making a shared object
2024-02-14T14:12:25.9376547Z collect2: error: ld returned 1 exit status
2024-02-14T14:12:25.9377158Z building '_sha3' extension
Same a bit later in the output:
gcc -pthread -shared -static -static-libgcc build/temp.linux-x86_64-3.11/__w/OpenRepo-Python/OpenRepo-Python/Python-3.11.0/Modules/_ctypes/_ctypes.o build/temp.linux-x86_64-3.11/__w/OpenRepo-Python/OpenRepo-Python/Python-3.11.0/Modules/_ctypes/callbacks.o build/temp.linux-x86_64-3.11/__w/OpenRepo-Python/OpenRepo-Python/Python-3.11.0/Modules/_ctypes/callproc.o build/temp.linux-x86_64-3.11/__w/OpenRepo-Python/OpenRepo-Python/Python-3.11.0/Modules/_ctypes/cfield.o build/temp.linux-x86_64-3.11/__w/OpenRepo-Python/OpenRepo-Python/Python-3.11.0/Modules/_ctypes/stgdict.o -L/sdev_shared/sdev_custom_scripts_tst/lib -L/usr/lib/x86_64-linux-gnu -L/usr/local/lib -lffi -ldl -o build/lib.linux-x86_64-3.11/_ctypes.cpython-311-x86_64-linux-gnu.so
/usr/bin/ld: /usr/lib/gcc/x86_64-linux-gnu/9/crtbeginT.o: relocation R_X86_64_32 against hidden symbol `__TMC_END__' can not be used when making a shared object
collect2: error: ld returned 1 exit status
Don’t you see this? Are you willing to share your configuration options? Is it my fault? Do I really do something wrong? What is your setup/ flags/commands?
              I used 3.11.8, but looking closer at the build output I do see the “relocation R_X86_64_32 ...” message when building several sub-components. Didn’t notice it before since the build continues, and eventually spits out a seemingly-functional python binary.
The exact configuration and build flags I used were
./configure LDFLAGS="-static -static-libgcc" CPPFLAGS="-static" --disable-shared
make -j$(nproc) LDFLAGS="-static" LINKFORSHARED=" "
              Thanks. Let me check again with those flags. As you noticed I kept adding more and more flags trying to get it working. If you get those link (ld) errors as well… Then you most likely also still got -shared flag in the gcc command!? Eg:
2024-02-14T14:12:25.8273847Z gcc -pthread -shared -static -static-libgcc -fno-semantic-interposition -fprofile-generate build/temp.linux-x86_64-3.11/__w/OpenRepo-Python/OpenRepo-Python/Python-3.11.0/Modules/_opcode.o -L/usr/lib/x86_64-linux-gnu -L/usr/local/lib -o build/lib.linux-x86_64-3.11/_opcode.cpython-311-x86_64-linux-gnu.so
I’m really the only person that sees this -shared flag still being present?
              No, I see the same thing during the 3.11.8 build for the failing modules (and for the 3.12.2 build).
gcc -pthread -shared -static -static-libgcc -static -static build/temp.linux-x86_64-3.11/home/abemtk/tmp/buildpython/Python-3.11.8/Modules/xxlimited_35.o -L/usr/lib/x86_64-linux-gnu -L/usr/local/lib -o build/lib.linux-x86_64-3.11/xxlimited_35.cpython-311-x86_64-linux-gnu.so
/usr/bin/ld: /usr/lib/gcc/x86_64-linux-gnu/9/crtbeginT.o: relocation R_X86_64_32 against hidden symbol `__TMC_END__' can not be used when making a shared object
collect2: error: ld returned 1 exit status
After reading up on this some more, I think the problem may be related to glibc. Specifically, I don’t think Ubuntu ships a static version of glibc, and there seem to be good reasons for that.
Here’s an old build script that uses a static build of musl libc in place of glibc. I haven’t tried it, but it might be a possible way forward.
Are you sure it’s glibc related. I still see -shared flag in the gcc command. Which can not be correct. This the whole problem, I suspect.
You can’t have static and shared flag together. But the python build is broken in that sense.
              Of course… But my first issue is with Python source build is that I can’t get rid of this -shared flag. Not in Python 3.11.x and not in 3.12.x. Not even in 3.10.x… Any idea how to get rid of this shared flag?
I have the feeling this bug is in the Python build system for several months/years. And nobody is validating the static build of Python, causing more and more regression over time. Which is a pity.
              In ye oldie times I used to use Autotools, so my “solution” might not work.
I think you can, after running ./configure script, open the Makefile and manually remove all -shared flags. Just use a find-and-delete tool in your favorite text editor 
Five minutes later…

Okay, I think I got something: when running ./configure add LDSHARED="gcc" (instead of gcc you can write your compiler of course)
Why I think it may work? Well, I have read the configure.ac from the repo (over 7000 lines of code o…m…g) and found this:
AC_MSG_CHECKING([LDSHARED])
if test -z "$LDSHARED"
Linux*|GNU*|QNX*|VxWorks*|Haiku*)
		LDSHARED='$(CC) -shared'
		LDCXXSHARED='$(CXX) -shared';;
In other words you MUST provide the LDSHARED= value that can’t be an empty value. If you don’t then the code that adds -shared flags are executed.
I must say this sounds counter-intuitive but it might work 
              Hi! Thanks for this question and for the hints. It’s very hard to find anything on the internet about static Python builds.
Python 3.12 removed distutils and setup.py from the repo and the build process is different.
Here in the configure script: https://github.com/python/cpython/blob/23ba96e2433d17e86f4770a64b94aaf9ad22a25b/configure#L32945 the modules are generated from those 4 files: Modules/Setup.local Modules/Setup.stdlib Modules/Setup.bootstrap $srcdir/Modules/Setup.

The file Modules/Setup.stdlib wasn’t used in Python 3.11 from my tests, but it is used now. I commented all of it, because it has the same libraries as Modules/Setup and you can enable/disable what you need in there.
I used the GitHub - jart/cosmopolitan: build-once run-anywhere c library compiler and I managed to generate a static build for 3.12, portable and multi-platform.

I will try to generate a static build with Clang and maybe I’ll make a Docker image with the recipe.