You signed in with another tab or window.
Reload
to refresh your session.
You signed out in another tab or window.
Reload
to refresh your session.
You switched accounts on another tab or window.
Reload
to refresh your session.
Already on GitHub?
Sign in
to your account
runc run failed: unable to start container process: can't copy bootstrap data to pipe: write init-p: broken pipe
#3937
runc run failed: unable to start container process: can't copy bootstrap data to pipe: write init-p: broken pipe
#3937
alexcb
opened this issue
Mar 25, 2024
· 8 comments
· Fixed by
#3944
There's three known cases of earthly not working, which has something to do with a failure starting runc:
https://github.com/earthly/earthly.git#main *failed* | Repeating the failure error...
https://github.com/earthly/earthly.git#main *failed* | runc run failed: unable to start container process: can't copy bootstrap data to pipe: write init-p: broken pipe
https://github.com/earthly/earthly.git#main *failed* | ERROR
https://github.com/earthly/earthly.git#main *failed* | The internal command
https://github.com/earthly/earthly.git#main *failed* | GET GIT META github.com/earthly/earthly/examples/tutorial/go:main
https://github.com/earthly/earthly.git#main *failed* | did not complete successfully. Exit code 1
After struggling with this for several days, I wasn't able to come up with a solution. On some machines it seems impossible to use Earthly when some unknown situation arises.
Unfortunately this is a very poor experience for a core build tool; anyone would say that it is unacceptable to have a dependency on a tool that simply cannot run on some developer's machines without any remediation.
Disabling "experimental features" in Docker Desktop's settings
Restarting the machine
Even a minimal Earthfile wouldn't build:
VERSION 0.8
FROM alpine
RUN echo hello
This was on an arm64 M1 MacBook running Earthly 0.8.5. There is no issue running Docker containers on the machine outside of Earthly.
These are the errors I saw:
# With Docker Desktop
runc run failed: unable to start container process: can't copy bootstrap data to pipe: write init-p: broken pipe
# With Orbstack
runc run failed: unable to start container process: waiting for init preliminary setup: read init-p: connection reset by peer
It contains two different versions: one for amd64 and another for arm64 -- I have tested this on both a linux amd64 host and M1, which work when running their own architecture.
However, when I run the arm64 version from my amd64 linux (ubuntu) host, I get the same error:
alex@surfperch:~/test$ git clone [email protected]:earthly/test-runc-in-earthly-buildkitd.git
Cloning into 'test-runc-in-earthly-buildkitd'...
remote: Enumerating objects: 169, done.
remote: Counting objects: 100% (169/169), done.
remote: Compressing objects: 100% (121/121), done.
remote: Total 169 (delta 14), reused 169 (delta 14), pack-reused 0
Receiving objects: 100% (169/169), 3.45 MiB | 3.86 MiB/s, done.
Resolving deltas: 100% (14/14), done.
alex@surfperch:~/test$ cd test-runc-in-earthly-buildkitd/
alex@surfperch:~/test/test-runc-in-earthly-buildkitd$ cd arm64/
alex@surfperch:~/test/test-runc-in-earthly-buildkitd/arm64$ git rev-parse HEAD
a3d0edd942c4fd052d6d0a464455084f3920f68b
alex@surfperch:~/test/test-runc-in-earthly-buildkitd/arm64$ docker build --platform linux/arm64 --tag foo . && docker run --rm --privileged --platform linux/arm64 foo
[+] Building 0.7s (10/10) FINISHED docker:default
=> [internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 196B 0.0s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [internal] load metadata for docker.io/earthly/buildkitd:v0.8.6 0.4s
=> [1/5] FROM docker.io/earthly/buildkitd:v0.8.6@sha256:d3bc052d1acb3bf00bffbfa747bd6dafd79242a573516a244db7fa28e7f57073 0.0s
=> [internal] load build context 0.1s
=> => transferring context: 8.41MB 0.0s
=> CACHED [2/5] COPY entrypoint.sh /root/entrypoint.sh 0.0s
=> CACHED [3/5] WORKDIR /test-runc 0.0s
=> [4/5] COPY rootfs rootfs 0.1s
=> [5/5] COPY config.json . 0.0s
=> exporting to image 0.1s
=> => exporting layers 0.1s
=> => writing image sha256:da6d557c823886b14f1fe9e462c3c7fb27afd99d4063cbf74e700dcc37578027 0.0s
=> => naming to docker.io/library/foo 0.0s
entrypoint.sh running under aarch64
detected cgroups v2; buildkit/entrypoint.sh running under pid=1 with controllers "cpuset cpu io memory hugetlb pids rdma misc" in group 0::/
moving 1 to my-test-group cgroup
running runc
NAME:
runc - Open Container Initiative runtime
runc is a command line client for running applications packaged according to
the Open Container Initiative (OCI) format and is a compliant implementation of the
Open Container Initiative specification.
runc integrates well with existing process supervisors to provide a production
container runtime environment for applications. It can be used with your
existing process monitoring tools and the container will be spawned as a
direct child of the process supervisor.
Containers are configured using bundles. A bundle for a container is a directory
that includes a specification file named "config.json" and a root filesystem.
The root filesystem contains the contents of the container.
To start a new instance of a container:
# runc run [ -b bundle ] <container-id>
Where "<container-id>" is your name for the instance of the container that you
are starting. The name you provide for the container instance must be unique on
your host. Providing the bundle directory using "-b" is optional. The default
value for "bundle" is the current directory.
USAGE:
buildkit-runc [global options] command [command options] [arguments...]
VERSION:
unknown
spec: 1.0.2-dev
go: go1.21.8
libseccomp: 2.5.5
COMMANDS:
checkpoint checkpoint a running container
... the --help command works, which shows the basic binary works
but then it fails:
time="2024-03-25T23:17:50Z" level=warning msg="invalid argument"
time="2024-03-25T23:17:50Z" level=fatal msg="nsexec[43]: could not ensure we are a cloned binary: No such file or directory"
time="2024-03-25T23:17:50Z" level=warning error="waiting for init preliminary setup: EOF"
time="2024-03-25T23:17:50Z" level=error msg="runc run failed: unable to start container process: can't copy bootstrap data to pipe: write init-p: broken pipe"
I can replicate this on a M1 by tricking earthly into running the amd64 version of earthly-buildkitd instead of the arm64 version:
It's also possible to reproduce this issue by not changing the config.yml, but instead by pre-pulling the incorrect platform image.
for example (running on a M1):
rm ~/.earthly/config.yml
docker rmi docker.io/earthly/buildkitd:v0.8.6
docker pull --platform linux/amd64 docker.io/earthly/buildkitd:v0.8.6
file /Users/administrator/bin/earthly-v0.8.6
# /Users/administrator/bin/earthly-v0.8.6: Mach-O 64-bit executable arm64
/Users/administrator/bin/earthly-v0.8.6 +base
Init 🚀
————————————————————————————————————————————————————————————————————————————————
buildkitd | Starting buildkit daemon as a docker container (earthly-buildkitd)...
buildkitd | ...Done
Streaming logs to https://cloud.earthly.dev/builds/1fba2d94-cfd9-401d-9daa-95f7837acc33
Build 🔧
————————————————————————————————————————————————————————————————————————————————
logbus | Setting organization "" and project ""
alpine | --> Load metadata alpine linux/amd64
+base | --> FROM alpine
+base | [----------] 100% FROM alpine
+base | --> RUN echo hello
+base | runc run failed: unable to start container process: waiting for init preliminary setup: read init-p: connection reset by peer
+base | ERROR Earthfile line 3:0
+base | The command
+base | RUN echo hello
+base | did not complete successfully. Exit code 1
================================== ❌ FAILURE ===================================
+base *failed* | Repeating the failure error...
+base *failed* | --> RUN echo hello
+base *failed* | runc run failed: unable to start container process: waiting for init preliminary setup: read init-p: connection reset by peer
+base *failed* | ERROR Earthfile line 3:0
+base *failed* | The command
+base *failed* | RUN echo hello
+base *failed* | did not complete successfully. Exit code 1
$ docker exec earthly-buildkitd uname -m
x86_64
It turns out that some users had DOCKER_DEFAULT_PLATFORM set to linux/amd64, but were using an arm64-based mac (aka M1/2/3)
For those users, they should delete all existing earthly/buildkitd images, and unset the default platform environment variable:
docker rm -f earthly-buildkitd
docker images | grep earthly/buildkitd | awk '{print $3}' | xargs docker rmi -f
unset DOCKER_DEFAULT_PLATFORM
When starting a new container:
buildkitd | Starting buildkit daemon as a docker container (earthly-dev-buildkitd)...
buildkitd | Warning: earthly-dev-buildkitd was started using architecture amd64, but host architecture is arm64; is DOCKER_DEFAULT_PLATFORM accidentally set?
When checking an existing container:
buildkitd | Found buildkit daemon as docker container (earthly-dev-buildkitd)
buildkitd | Warning: currently running earthly-dev-buildkitd under architecture amd64, but host architecture is arm64; is DOCKER_DEFAULT_PLATFORM accidentally set?
fixes#3937
Signed-off-by: Alex Couture-Beil <[email protected]>
