docker: Error response from daemon: failed to create shim: OCI runtime create failed:
container_linux.go:380: starting container process caused:
process_linux.go:545: container init caused:
rootfs_linux.go:76: mounting "proc" to rootfs at "/proc" caused:
mount through procfd: permission denied: unknown.
This happens inside the system container. I tried running the system container with --privileged, but the result remains the same.
Here's the complete output:
$ docker run --runtime=sysbox-runc -it --hostname=syscont nestybox/alpine-docker:latest
$ dockerd > /var/log/dockerd.log 2>&1 &
$ docker run -it busybox
Unable to find image 'busybox:latest' locally
latest: Pulling from library/busybox
5cc84ad355aa: Pull complete
Digest: sha256:5acba83a746c7608ed544dc1533b87c737a0b0fb730301639a0179f9344b1678
Status: Downloaded newer image for busybox:latest
docker: Error response from daemon: failed to create shim: OCI runtime create failed: container_linux.go:380: starting container process caused: process_linux.go:545: container init caused: rootfs_linux.go:76: mounting "proc" to rootfs at "/proc" caused: mount through procfd: permission denied: unknown.
ERRO[0004] error waiting for container: context canceled
          
@href, thanks for filing this one up.
Question. How do you enable nested virtualization in your VM? Either way, you shouldn't need to enable nested virtualization in order to do what you want. That's precisely one of the advantages of relying on container technology such as Sysbox.
I suspect that problem may be related to the docker version running within the sysbox container. Will try to reproduce and get back to you when I have a chance.
kernel: 5.11.0-43-generic
distro: Ubuntu 20.04.2 LTS
nested virtualization: disabled (see below)
/ # grep -cw vmx /proc/cpuinfo
Is this VM hosted on-premise or in a cloud vendor? If a vendor, which one?
          
This VM is hosted on-premise in a Proxmox cluster. I have admin rights on the cluster, so I was able to activate nested virtualization, with which it worked. The CPU is AMD Epyc Rome, if that matters.
Let me know if I can provide you with any other output. The VM is not in production so I can run what you like on it to get more debug output.
Launch the sysbox container
Start Docker inside of it (i.e., the inner Docker)
In a separate terminal, run the following strace command: sudo strace -f -p $(pidof sysbox-fs) -o sysboxfs-strace.txt
Reproduce the problem by launching an inner busybox container: docker run -it busybox
After this, the sysboxfs-strace.txt file should have all syscalls made by sysbox-fs. Please send us this to see if we spot the problem.
Thanks!
          
We're using Ubuntu 20.04.3 LTS, with apt install -y linux-generic-hwe-20.04 linux-tools-generic-hwe-20.04.
I tried creating the trace, but now I can't reproduce the problem, even if I disable nested virtualization again. I guess what solved it then was the reboot, not my activating of nested virtualization (which naturally required a reboot).
I'll have to roll sysbox out on a few more servers that are pretty much identical, so I might be able to reproduce the problem again. If I do I'll post a stacktrace and reopen the ticket. Thank you for your quick responses at any rate!
          
Hi @href, thanks for the update and glad to know the issue was resolved.
Please let us know if you see it again. Also, feel free to join the Sysbox slack channel if you have any other inquiries.
Thanks again for giving Sysbox a shot!
@ctalledo I am running into the same issue. I assume nested virtualization on AMD is supported too? (grep -cw vmx /proc/cpuinfo is for Intel, grep -cw svm /proc/cpuinfo returns "48" in my case)
Hi @maltegrosse, in theory nested virtualization should not make a difference at all when running Sysbox, because Sysbox does not use VMs in any way. It's all pure OS-virtualization (i.e., container technology).
Can you provide more info on your host environment, and the command you issued that led to the error? The more info we get, the better. Thanks!
          
@ctalledo thank you. Iam running Ubuntu 22.04 on the Host system, and Ubuntu 22.04 inside the VM. svm is available inside the VM. K8s 1.25 is deployed. Actually this issue happens by using the normal dind rootless docker container (privileged), but with the same issue as the author of this issue reported. So I was wondering where this proc issue comes from...
          
Thanks @maltegrosse.
Actually this issue happens by using the normal dind rootless docker container (privileged)
Ah interesting, it's not a sysbox issue then but rather something more generic then.
What happens if inside the VM, you do a simple procfs mount as follows:
sudo mkdir /root/proc
sudo mount -t proc proc /root/proc
Do you also see the permission denied? Of does procfs mount on /root/proc correctly?
You can cleanup with:
sudo umount /root/proc