[I have edited my original post to add information I forgot when first posting, I’ve marked my edits with cursif]
I had my Umbrel set up and running for a few months but recently moved it from my home, where it was connected to the router via Ethernet cable to a setup where I only have an LTE dingle without Ethernet ports.
Before moving it from the ethernet connected network I stopped being able to connect via Tor or Tailscale as I had before. Since I wasn’t in the same physical location as my Umbrel Pi I didn’t try logging in via umbrel.local
I Googled how to connect to WiFi and took the SD card out, added a wpa_supplicant.conf file in the root and tried to connect. This was unsuccessful, the Umbrel Raspberry Pi didn’t show up on my network and I was unable to connect to umbrel.local.
I then tried another method: connecting the Ethernet cable to another Raspberry Pi running Raspberian and I was able to get umbrel.local up but it was now showing the “
http://umbrel.local/
” screen. I tried taking the SD card out again and removing the wpa_supplicant.conf file but am still getting the same screen.
I also tried to SSH into my Umbrel Pi from the Raspberian device but the standard moneyprintergobrrr password does not work.
What else can I try to get my Umbrel Raspberry Pi back up?
I now have physical access to the Umbrel Pi and can connect it via ethernet to another Pi running Raspberian, but I do not know the password for the SSH connection. I might have changed the default password when setting up originally but I cannot remember.
Since I suspected a faulty SD card could be the issue, I bought a new card and first tried cloning the old card to the new one using ‘gdd’ command on my Mac. That resulted in the same error.
After that I tried installing the latest version of Umbrel on the new SD card, but then got the “Error: Failed to start containers” error.
Not sure what else I can do at this point.
I would hate to have to download the whole blockchain again as my limited data plan wouldn’t allow it, apart from the time it would take.
Any suggestions what else I could try?
Dec 1 2021 15:01:54
version 71bd3109023a0c8575585ba87cbb374d2eeb038f (clean) (release) (start)
Temperature
-----------
temp=36.5'C
Throttling
----------
throttled=0x0
Memory usage
------------
total used free shared buff/cache available
Mem: 3.8G 110M 3.1G 174M 545M 3.4G
Swap: 4.1G 0B 4.1G
And at the very end:
================
==== Result ====
================
The debug script did not automatically detect any issues with your Umbrel.
I realise there is a whole middle part missing from the logs, but between that first screen and the end bit I am getting a bunch of errors: https://pastebin.com/LBnWzunn (this is only as much as I could see on terminal screen)
One line I saw repeated in there a lot is this:
requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionRefusedError(111, 'Connection refused'))
The way I am connecting to my Umbrell box is via SSH from another Raspberry Pi (running Raspberian) via network cable (connecting network ports of both Raspberry Pies.
My Raspberian box is connected to the internet via WiFi but I assume my Umbrel Pi isn’t and I am unsure how to connect it to the wider internet. I assume that is where the above error is coming from.
Not sure how to further debug.
Or if I would need to reinstall from scratch, how to clear my hard drive. Although that would be the very last resort, not just because I already had quite a few Lightning channels open, but also because re-downloading the whole blockchain would be a lengthy and costly affair.
Hi Dhuyvetter,
Your SSH password matches the exact password of the Umbrel system password. So whatever you use to log into Umbrel on boot is what you’ll use for SSH.
Thanks,
I had a look at /umbrel/logs/status-monitor.log and the most recent errors are all write error: No space left on device
So I think I figured out what the issue is.
What would be safe to delete to free up some space on my SD card?
And how do I go about migrating to a bigger SD card? I have one, but how do I safely copy all the content to the new card?
I switched to a larger SD card and now running ~/umbrel/scripts/debug --upload gives less errors and more information:
=====================
= Umbrel debug info =
=====================
Umbrel version
--------------
0.5.1
Flashed OS version
-----------------
v0.4.17
Raspberry Pi Model
------------------
Revision : c03114
Serial : 10000000ee8d698e
Model : Raspberry Pi 4 Model B Rev 1.4
Firmware
--------
Dec 1 2021 15:01:54
version 71bd3109023a0c8575585ba87cbb374d2eeb038f (clean) (release) (start)
Temperature
-----------
temp=35.5'C
Throttling
----------
throttled=0x0
Memory usage
------------
total used free shared buff/cache available
Mem: 3.8G 103M 3.5G 8.0M 206M 3.6G
Swap: 0B 0B 0B
total: 2.7%
system: 2.7%
tor: 0%
lnd: 0%
electrs: 0%
bitcoin: 0%
Memory monitor logs
-------------------
tail: cannot open '/home/umbrel/umbrel/scripts/../logs/memory-monitor.log' for reading: No such file or directory
Filesystem information
----------------------
Filesystem Size Used Avail Use% Mounted on
/dev/root 59G 59G 0 100% /
/dev/root 59G 59G 0 100% /
Startup service logs
--------------------
-- Logs begin at Thu 2019-02-14 10:11:58 UTC, end at Thu 2019-02-14 10:23:50 UTC. --
Feb 14 10:12:03 umbrel systemd[1]: Dependency failed for Umbrel Startup Service.
Feb 14 10:12:03 umbrel systemd[1]: umbrel-startup.service: Job umbrel-startup.service/start failed with result 'dependency'.
External storage service logs
-----------------------------
-- Logs begin at Thu 2019-02-14 10:11:58 UTC, end at Thu 2019-02-14 10:23:50 UTC. --
Feb 14 10:12:03 umbrel systemd[1]: Starting External Storage Mounter...
Feb 14 10:12:03 umbrel external storage mounter[472]: /status-server/set-status: line 31: echo: write error: No space left on device
Feb 14 10:12:03 umbrel systemd[1]: umbrel-external-storage.service: Main process exited, code=exited, status=1/FAILURE
Feb 14 10:12:03 umbrel systemd[1]: umbrel-external-storage.service: Failed with result 'exit-code'.
Feb 14 10:12:03 umbrel systemd[1]: Failed to start External Storage Mounter.
External storage SD card update service logs
--------------------------------------------
-- Logs begin at Thu 2019-02-14 10:11:58 UTC, end at Thu 2019-02-14 10:23:50 UTC. --
Feb 14 10:12:03 umbrel systemd[1]: Dependency failed for External Storage SDcard Updater.
Feb 14 10:12:03 umbrel systemd[1]: umbrel-external-storage-sdcard-update.service: Job umbrel-external-storage-sdcard-update.service/start failed with result 'dependency'.
Karen logs
----------
tail: cannot open 'logs/karen.log' for reading: No such file or directory
Docker containers
-----------------
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Umbrel logs
-----------
ERROR: The Compose file './docker-compose.yml' is invalid because:
networks.default.ipam.config.subnet is invalid: should use the CIDR format
services.tor_proxy.ports contains an invalid type, it should be a number, or an object
services.auth.ports contains an invalid type, it should be a number, or an object
services.nginx.ports contains an invalid type, it should be a number, or an object
Tor logs
--------
ERROR: The Compose file './docker-compose.yml' is invalid because:
networks.default.ipam.config.subnet is invalid: should use the CIDR format
services.tor_proxy.ports contains an invalid type, it should be a number, or an object
services.auth.ports contains an invalid type, it should be a number, or an object
services.nginx.ports contains an invalid type, it should be a number, or an object
================
==== Result ====
================
The debug script did not automatically detect any issues with your Umbrel.
Hey @dhuyvetter,
Sorry to hear about your issue, good detective skills figuring out the filesystem out of space issue!
Would be great if we could get to the bottom of what caused the issue so we can prevent it happening to other users. Did you flash a fresh larger SD card or image your old SD card over to a new one?
Just wondering if we can run some commands in the original environment that ran out of space to figure out where the large files are.
You can run:
sudo du -h -a / 2>/dev/null | sort -h -r | head -n 100
To list 100 largest files on the filesystem (this might take a while to complete) if you still have access to the original environment.
I’m having this same problem with the Error: System service failed. Previously had the No Space left on device in debug logs. My Umbrel was been up and running on a RPi for some time and then something happened while I was out of town on vacation which led to my not being able to bootup. I’m able to login to the device via SSH. I’ve combed through the community Support and Troubleshooting posts, ran all kinds of commands that have led to no success. Only now I do not see the No Space Left on Device error in the debug logs from reboot Re-flashed the existing mSD card (not a new one). Not sure what is filling up the hard drive and causing the problem. Any help or suggestions on what to do moving forward would be greatly appreciated. I may have to reformat the SSD and start fresh and recover the node? I don’t know what else to try.
Below are the results from the debug report and when I ran the command @lukechilds recommended to @dhuyvetter above:
sudo du -h -a / 2>/dev/null | sort -h -r | head -n 100
1.8T /
916G /mnt/data
916G /mnt
517G /mnt/data/umbrel
517G /home/umbrel/umbrel
517G /home/umbrel
517G /home
482G /mnt/data/umbrel/bitcoin
482G /home/umbrel/umbrel/bitcoin
434G /mnt/data/umbrel/bitcoin/blocks
434G /home/umbrel/umbrel/bitcoin/blocks
396G /var/lib
396G /var
396G /mnt/data/docker
395G /var/lib/docker
376G /var/lib/docker/containers
376G /mnt/data/docker/containers
375G /var/lib/docker/containers/fb66b3fc74454f5f8c2db8e0cfa542bbf55e5141691be2e96844bc22e0b71fc6/fb66b3fc74454f5f8c2db8e0cfa542bbf55e5141691be2e96844bc22e0b71fc6-json.log
375G /var/lib/docker/containers/fb66b3fc74454f5f8c2db8e0cfa542bbf55e5141691be2e96844bc22e0b71fc6
375G /mnt/data/docker/containers/fb66b3fc74454f5f8c2db8e0cfa542bbf55e5141691be2e96844bc22e0b71fc6/fb66b3fc74454f5f8c2db8e0cfa542bbf55e5141691be2e96844bc22e0b71fc6-json.log
375G /mnt/data/docker/containers/fb66b3fc74454f5f8c2db8e0cfa542bbf55e5141691be2e96844bc22e0b71fc6
43G /mnt/data/umbrel/bitcoin/indexes
43G /home/umbrel/umbrel/bitcoin/indexes
36G /mnt/data/umbrel/bitcoin/indexes/txindex
36G /home/umbrel/umbrel/bitcoin/indexes/txindex
34G /mnt/data/umbrel/electrs/db/bitcoin
34G /mnt/data/umbrel/electrs/db
34G /mnt/data/umbrel/electrs
34G /home/umbrel/umbrel/electrs/db/bitcoin
34G /home/umbrel/umbrel/electrs/db
34G /home/umbrel/umbrel/electrs
20G /var/lib/docker/overlay2
20G /mnt/data/docker/overlay2
7.6G /mnt/data/umbrel/bitcoin/indexes/blockfilter/basic
7.6G /mnt/data/umbrel/bitcoin/indexes/blockfilter
7.6G /home/umbrel/umbrel/bitcoin/indexes/blockfilter/basic
7.6G /home/umbrel/umbrel/bitcoin/indexes/blockfilter
4.7G /mnt/data/umbrel/bitcoin/chainstate
4.7G /home/umbrel/umbrel/bitcoin/chainstate
4.1G /swap/swapfile
4.1G /swap
4.1G /mnt/data/swap/swapfile
4.1G /mnt/data/swap
2.5G /var/lib/docker/overlay2/bbaa23f2c98997e98fad25e482b5f836cc8462179339aaaeb3faa43a9af6516a/diff/relay/core
2.5G /var/lib/docker/overlay2/bbaa23f2c98997e98fad25e482b5f836cc8462179339aaaeb3faa43a9af6516a/diff/relay
2.5G /var/lib/docker/overlay2/bbaa23f2c98997e98fad25e482b5f836cc8462179339aaaeb3faa43a9af6516a/diff
2.5G /var/lib/docker/overlay2/bbaa23f2c98997e98fad25e482b5f836cc8462179339aaaeb3faa43a9af6516a
2.5G /mnt/data/docker/overlay2/bbaa23f2c98997e98fad25e482b5f836cc8462179339aaaeb3faa43a9af6516a/diff/relay/core
2.5G /mnt/data/docker/overlay2/bbaa23f2c98997e98fad25e482b5f836cc8462179339aaaeb3faa43a9af6516a/diff/relay
2.5G /mnt/data/docker/overlay2/bbaa23f2c98997e98fad25e482b5f836cc8462179339aaaeb3faa43a9af6516a/diff
2.5G /mnt/data/docker/overlay2/bbaa23f2c98997e98fad25e482b5f836cc8462179339aaaeb3faa43a9af6516a
1.6G /mnt/data/umbrel/app-data
1.6G /home/umbrel/umbrel/app-data
1.3G /usr
984M /var/lib/docker/containers/02b5e6c30e992627e03059501f11deeeb5bad4aebc17186375c702bd1bf2737c/02b5e6c30e992627e03059501f11deeeb5bad4aebc17186375c702bd1bf2737c-json.log
984M /var/lib/docker/containers/02b5e6c30e992627e03059501f11deeeb5bad4aebc17186375c702bd1bf2737c
984M /mnt/data/docker/containers/02b5e6c30e992627e03059501f11deeeb5bad4aebc17186375c702bd1bf2737c/02b5e6c30e992627e03059501f11deeeb5bad4aebc17186375c702bd1bf2737c-json.log
984M /mnt/data/docker/containers/02b5e6c30e992627e03059501f11deeeb5bad4aebc17186375c702bd1bf2737c
900M /var/lib/docker/overlay2/86f8fc246e58b81d11a5a4467abe65cf8bcc44d28121c744e7ad2864d8b47030/diff/usr/local
900M /var/lib/docker/overlay2/86f8fc246e58b81d11a5a4467abe65cf8bcc44d28121c744e7ad2864d8b47030/diff/usr
900M /var/lib/docker/overlay2/86f8fc246e58b81d11a5a4467abe65cf8bcc44d28121c744e7ad2864d8b47030/diff
900M /var/lib/docker/overlay2/86f8fc246e58b81d11a5a4467abe65cf8bcc44d28121c744e7ad2864d8b47030
900M /mnt/data/docker/overlay2/86f8fc246e58b81d11a5a4467abe65cf8bcc44d28121c744e7ad2864d8b47030/diff/usr/local
900M /mnt/data/docker/overlay2/86f8fc246e58b81d11a5a4467abe65cf8bcc44d28121c744e7ad2864d8b47030/diff/usr
900M /mnt/data/docker/overlay2/86f8fc246e58b81d11a5a4467abe65cf8bcc44d28121c744e7ad2864d8b47030/diff
900M /mnt/data/docker/overlay2/86f8fc246e58b81d11a5a4467abe65cf8bcc44d28121c744e7ad2864d8b47030
896M /var/lib/docker/overlay2/86f8fc246e58b81d11a5a4467abe65cf8bcc44d28121c744e7ad2864d8b47030/diff/usr/local/lib/python3.9/site-packages
896M /var/lib/docker/overlay2/86f8fc246e58b81d11a5a4467abe65cf8bcc44d28121c744e7ad2864d8b47030/diff/usr/local/lib/python3.9
896M /var/lib/docker/overlay2/86f8fc246e58b81d11a5a4467abe65cf8bcc44d28121c744e7ad2864d8b47030/diff/usr/local/lib
896M /mnt/data/docker/overlay2/86f8fc246e58b81d11a5a4467abe65cf8bcc44d28121c744e7ad2864d8b47030/diff/usr/local/lib/python3.9/site-packages
896M /mnt/data/docker/overlay2/86f8fc246e58b81d11a5a4467abe65cf8bcc44d28121c744e7ad2864d8b47030/diff/usr/local/lib/python3.9
896M /mnt/data/docker/overlay2/86f8fc246e58b81d11a5a4467abe65cf8bcc44d28121c744e7ad2864d8b47030/diff/usr/local/lib
866M /mnt/data/umbrel/app-data/nextcloud/data
866M /mnt/data/umbrel/app-data/nextcloud
866M /home/umbrel/umbrel/app-data/nextcloud/data
866M /home/umbrel/umbrel/app-data/nextcloud
718M /var/log
685M /var/lib/docker/overlay2/2ffaccce0562aa633a97cfa5f6878388900eb63c0e97e9db73afe46537f1d71e/diff/RTL/node_modules
685M /var/lib/docker/overlay2/2ffaccce0562aa633a97cfa5f6878388900eb63c0e97e9db73afe46537f1d71e/diff/RTL
685M /var/lib/docker/overlay2/2ffaccce0562aa633a97cfa5f6878388900eb63c0e97e9db73afe46537f1d71e/diff
685M /var/lib/docker/overlay2/2ffaccce0562aa633a97cfa5f6878388900eb63c0e97e9db73afe46537f1d71e
685M /mnt/data/docker/overlay2/2ffaccce0562aa633a97cfa5f6878388900eb63c0e97e9db73afe46537f1d71e/diff/RTL/node_modules
685M /mnt/data/docker/overlay2/2ffaccce0562aa633a97cfa5f6878388900eb63c0e97e9db73afe46537f1d71e/diff/RTL
685M /mnt/data/docker/overlay2/2ffaccce0562aa633a97cfa5f6878388900eb63c0e97e9db73afe46537f1d71e/diff
685M /mnt/data/docker/overlay2/2ffaccce0562aa633a97cfa5f6878388900eb63c0e97e9db73afe46537f1d71e
664M /mnt/data/umbrel/app-data/nextcloud/data/nextcloud
664M /home/umbrel/umbrel/app-data/nextcloud/data/nextcloud
625M /mnt/data/docker/overlay2/e15c0e6ef3a2301aeaac02ad0507a163efd53a9ef4f8f2f350ba8b84171c6fb4/diff
625M /mnt/data/docker/overlay2/e15c0e6ef3a2301aeaac02ad0507a163efd53a9ef4f8f2f350ba8b84171c6fb4
609M /mnt/data/docker/overlay2/e15c0e6ef3a2301aeaac02ad0507a163efd53a9ef4f8f2f350ba8b84171c6fb4/diff/usr
601M /var/lib/docker/overlay2/e15c0e6ef3a2301aeaac02ad0507a163efd53a9ef4f8f2f350ba8b84171c6fb4/diff
601M /var/lib/docker/overlay2/e15c0e6ef3a2301aeaac02ad0507a163efd53a9ef4f8f2f350ba8b84171c6fb4
584M /var/lib/docker/overlay2/e15c0e6ef3a2301aeaac02ad0507a163efd53a9ef4f8f2f350ba8b84171c6fb4/diff/usr
583M /mnt/data/docker/overlay2/136ef7f4a678574f85a1582d3fd8839f45e04426aca0c1b5f7047c122884023d/diff/build
583M /mnt/data/docker/overlay2/136ef7f4a678574f85a1582d3fd8839f45e04426aca0c1b5f7047c122884023d/diff
583M /mnt/data/docker/overlay2/136ef7f4a678574f85a1582d3fd8839f45e04426aca0c1b5f7047c122884023d
576M /var/lib/docker/overlay2/136ef7f4a678574f85a1582d3fd8839f45e04426aca0c1b5f7047c122884023d/diff/build
576M /var/lib/docker/overlay2/136ef7f4a678574f85a1582d3fd8839f45e04426aca0c1b5f7047c122884023d/diff
576M /var/lib/docker/overlay2/136ef7f4a678574f85a1582d3fd8839f45e04426aca0c1b5f7047c122884023d
529M /var/log/daemon.log.1
0.4.18
v0.5.0
Revision : d03114
Serial : 10000000e1ca6660
Model : Raspberry Pi 4 Model B Rev 1.4
Dec 1 2021 15:01:54
version 71bd3109023a0c8575585ba87cbb374d2eeb038f (clean) (release) (start)
temp=42.3’C
throttled=0x0
total used free shared buff/cache available
Mem: 7.8G 114M 7.1G 170M 611M 7.4G
Swap: 4.1G 0B 4.1G
total: 1.5%
system: 1.5%
vaultwarden: 0%
urbit: 0%
uptime-kuma: 0%
tor: 0%
thunderhub: 0%
syncthing: 0%
synapse: 0%
sphinx-relay: 0%
simple-torrent: 0%
ride-the-lightning: 0%
pi-hole: 0%
photoprism: 0%
node-red: 0%
nextcloud: 0%
mempool: 0%
lnmarkets: 0%
lndg: 0%
lnd: 0%
lnbits: 0%
lightning-terminal: 0%
krystal-bull: 0%
home-assistant: 0%
element: 0%
electrs: 0%
btc-rpc-explorer: 0%
btcpay-server: 0%
bluewallet: 0%
bitfeed: 0%
bitcoin: 0%
Filesystem Size Used Avail Use% Mounted on
/dev/root 59G 3.9G 53G 7% /
/dev/sda1 916G 916G 0 100% /home/umbrel/umbrel
Sep 25 20:08:39 umbrel umbrel startup[32365]: The NGINX_PORT variable is not set. Defaulting to a blank string.
Sep 25 20:08:39 umbrel umbrel startup[32365]: The NGINX_IP variable is not set. Defaulting to a blank string.
Sep 25 20:08:39 umbrel umbrel startup[32365]: The BITCOIN_P2P_PORT variable is not set. Defaulting to a blank string.
Sep 25 20:08:39 umbrel umbrel startup[32365]: The BITCOIN_IP variable is not set. Defaulting to a blank string.
Sep 25 20:08:39 umbrel umbrel startup[32365]: The LND_REST_PORT variable is not set. Defaulting to a blank string.
Sep 25 20:08:39 umbrel umbrel startup[32365]: The LND_GRPC_PORT variable is not set. Defaulting to a blank string.
Sep 25 20:08:39 umbrel umbrel startup[32365]: The LND_IP variable is not set. Defaulting to a blank string.
Sep 25 20:08:39 umbrel umbrel startup[32365]: The DASHBOARD_IP variable is not set. Defaulting to a blank string.
Sep 25 20:08:39 umbrel umbrel startup[32365]: The MIDDLEWARE_IP variable is not set. Defaulting to a blank string.
Sep 25 20:08:39 umbrel umbrel startup[32365]: The BITCOIN_RPC_PORT variable is not set. Defaulting to a blank string.
Sep 25 20:08:39 umbrel umbrel startup[32365]: The BITCOIN_RPC_USER variable is not set. Defaulting to a blank string.
Sep 25 20:08:39 umbrel umbrel startup[32365]: The BITCOIN_RPC_PASS variable is not set. Defaulting to a blank string.
Sep 25 20:08:39 umbrel umbrel startup[32365]: The BITCOIN_NETWORK variable is not set. Defaulting to a blank string.
Sep 25 20:08:39 umbrel umbrel startup[32365]: The MANAGER_IP variable is not set. Defaulting to a blank string.
Sep 25 20:08:39 umbrel umbrel startup[32365]: The NEUTRINO_SWITCHER_IP variable is not set. Defaulting to a blank string.
Sep 25 20:08:39 umbrel umbrel startup[32365]: The ELECTRUM_PORT variable is not set. Defaulting to a blank string.
Sep 25 20:08:39 umbrel umbrel startup[32365]: The ELECTRUM_IP variable is not set. Defaulting to a blank string.
Sep 25 20:08:39 umbrel umbrel startup[32365]: The NETWORK_IP variable is not set. Defaulting to a blank string.
Sep 25 20:08:39 umbrel umbrel startup[32365]: The Compose file ‘./docker-compose.yml’ is invalid because:
Sep 25 20:08:39 umbrel umbrel startup[32365]: networks.default.ipam.config.subnet is invalid: should use the CIDR format
Sep 25 20:08:39 umbrel umbrel startup[32365]: services.tor_proxy.ports contains an invalid type, it should be a number, or an object
Sep 25 20:08:39 umbrel umbrel startup[32365]: services.bitcoin.ports contains an invalid type, it should be a number, or an object
Sep 25 20:08:39 umbrel umbrel startup[32365]: services.electrs.ports contains an invalid type, it should be a number, or an object
Sep 25 20:08:39 umbrel umbrel startup[32365]: services.lnd.ports contains an invalid type, it should be a number, or an object
Sep 25 20:08:39 umbrel umbrel startup[32365]: services.nginx.ports contains an invalid type, it should be a number, or an object
Sep 25 20:08:39 umbrel umbrel startup[32365]: services.lnd.ports value [‘9735:9735’, ‘:’, ‘:’] has non-unique elements
Sep 25 20:08:39 umbrel umbrel startup[32365]: Failed to start containers
Sep 25 20:08:39 umbrel systemd[1]: umbrel-startup.service: Control process exited, code=exited, status=1/FAILURE
Sep 25 20:08:39 umbrel systemd[1]: umbrel-startup.service: Failed with result ‘exit-code’.
Sep 25 20:08:39 umbrel systemd[1]: Failed to start Umbrel Startup Service.
– Logs begin at Sun 2022-09-25 08:17:57 UTC, end at Sun 2022-09-25 20:08:39 UTC. –
Sep 25 08:18:02 umbrel systemd[1]: Starting External Storage Mounter…
Sep 25 08:18:02 umbrel external storage mounter[485]: Running external storage mount script…
Sep 25 08:18:02 umbrel external storage mounter[485]: Found device “External USB3.0”
Sep 25 08:18:02 umbrel external storage mounter[485]: Blacklisting USB device IDs against UAS driver…
Sep 25 08:18:02 umbrel external storage mounter[485]: Rebinding USB drivers…
Sep 25 08:18:02 umbrel external storage mounter[485]: Checking USB devices are back…
Sep 25 08:18:02 umbrel external storage mounter[485]: Waiting for USB devices…
Sep 25 08:18:03 umbrel external storage mounter[485]: Waiting for USB devices…
Sep 25 08:18:04 umbrel external storage mounter[485]: Waiting for USB devices…
Sep 25 08:18:05 umbrel external storage mounter[485]: Checking if the device is ext4…
Sep 25 08:18:05 umbrel external storage mounter[485]: Yes, it is ext4
Sep 25 08:18:05 umbrel external storage mounter[485]: Checking if device contains an Umbrel install…
Sep 25 08:18:05 umbrel external storage mounter[485]: Yes, it contains an Umbrel install
Sep 25 08:18:05 umbrel external storage mounter[485]: Bind mounting external storage over local Umbrel installation…
Sep 25 08:18:05 umbrel external storage mounter[485]: Bind mounting external storage over local Docker data dir…
Sep 25 08:18:05 umbrel external storage mounter[485]: Bind mounting external storage to /swap
Sep 25 08:18:05 umbrel external storage mounter[485]: Bind mounting SD card root at /sd-card…
Sep 25 08:18:05 umbrel external storage mounter[485]: Checking Umbrel root is now on external storage…
Sep 25 08:18:06 umbrel external storage mounter[485]: Checking /var/lib/docker is now on external storage…
Sep 25 08:18:06 umbrel external storage mounter[485]: Checking /swap is now on external storage…
Sep 25 08:18:06 umbrel external storage mounter[485]: Setting up swapfile
Sep 25 08:18:07 umbrel external storage mounter[485]: Setting up swapspace version 1, size = 4 GiB (4294963200 bytes)
Sep 25 08:18:07 umbrel external storage mounter[485]: no label, UUID=79029681-e317-4a7c-b90d-8fd0982ff0ad
Sep 25 08:18:07 umbrel external storage mounter[485]: Checking SD Card root is bind mounted at /sd-root…
Sep 25 08:18:07 umbrel external storage mounter[485]: Starting external drive mount monitor…
Sep 25 08:18:07 umbrel external storage mounter[485]: Mount script completed successfully!
Sep 25 08:18:07 umbrel systemd[1]: Started External Storage Mounter.
– Logs begin at Sun 2022-09-25 08:17:57 UTC, end at Sun 2022-09-25 20:08:39 UTC. –
Sep 25 08:18:14 umbrel systemd[1]: Starting External Storage SDcard Updater…
Sep 25 08:18:14 umbrel external storage updater[811]: Checking if SD card Umbrel is newer than external storage…
Sep 25 08:18:14 umbrel external storage updater[811]: Yes, SD version is newer.
Sep 25 08:18:14 umbrel external storage updater[811]: Checking if the external storage version “0.4.18” satisfies update requirement “>=0.4.18”…
Sep 25 08:18:14 umbrel external storage updater[811]: Yes, it does, attempting an automatic update…
Sep 25 08:18:14 umbrel external storage updater[811]: =======================================
Sep 25 08:18:14 umbrel external storage updater[811]: =============== UPDATE ================
Sep 25 08:18:14 umbrel external storage updater[811]: =======================================
Sep 25 08:18:14 umbrel external storage updater[811]: ========== Stage: Download ============
Sep 25 08:18:14 umbrel external storage updater[811]: =======================================
Sep 25 08:18:14 umbrel external storage updater[811]: Creating lock
Sep 25 08:18:14 umbrel external storage updater[811]: Cleaning up any previous mess
Sep 25 08:18:14 umbrel external storage updater[811]: Copying Umbrel 0.5.0 from /sd-root/home/umbrel/umbrel
Sep 25 08:18:14 umbrel external storage updater[811]: mkdir: cannot create directory ‘/home/umbrel/umbrel/.umbrel-0.5.0’: No space left on device
Sep 25 08:18:14 umbrel external storage updater[811]: cp: cannot create directory ‘/home/umbrel/umbrel/.umbrel-0.5.0’: No space left on device
Sep 25 08:18:14 umbrel external storage updater[811]: /home/umbrel/umbrel/scripts/update/update: line 113: cd: /home/umbrel/umbrel/.umbrel-0.5.0: No such file or directory
Sep 25 08:18:14 umbrel external storage updater[811]: Running update install scripts of the new release
Sep 25 08:18:14 umbrel external storage updater[811]: /home/umbrel/umbrel/scripts/update/update: line 118: cd: scripts/update: No such file or directory
Sep 25 08:18:14 umbrel external storage updater[811]: ls: cannot access ‘*-run.sh’: No such file or directory
Sep 25 08:18:14 umbrel external storage updater[811]: Deleting cloned repository
Sep 25 08:18:14 umbrel external storage updater[811]: Removing lock
Sep 25 08:18:22 umbrel systemd[1]: Started External Storage SDcard Updater.
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
ERROR: The Compose file ‘./docker-compose.yml’ is invalid because:
networks.default.ipam.config.subnet is invalid: should use the CIDR format
services.tor_proxy.ports contains an invalid type, it should be a number, or an object
services.bitcoin.ports contains an invalid type, it should be a number, or an object
services.electrs.ports contains an invalid type, it should be a number, or an object
services.lnd.ports contains an invalid type, it should be a number, or an object
services.lnd.ports contains an invalid type, it should be a number, or an object
services.nginx.ports contains an invalid type, it should be a number, or an object
services.lnd.ports value [‘9735:9735’, ‘:’, ‘:’] has non-unique elements
ERROR: The Compose file ‘./docker-compose.yml’ is invalid because:
networks.default.ipam.config.subnet is invalid: should use the CIDR format
services.tor_proxy.ports contains an invalid type, it should be a number, or an object
services.bitcoin.ports contains an invalid type, it should be a number, or an object
services.electrs.ports contains an invalid type, it should be a number, or an object
services.lnd.ports contains an invalid type, it should be a number, or an object
services.lnd.ports contains an invalid type, it should be a number, or an object
services.nginx.ports contains an invalid type, it should be a number, or an object
services.lnd.ports value [‘9735:9735’, ‘:’, ‘:’] has non-unique elements
ERROR: The Compose file ‘./docker-compose.yml’ is invalid because:
networks.default.ipam.config.subnet is invalid: should use the CIDR format
services.tor_proxy.ports contains an invalid type, it should be a number, or an object
services.bitcoin.ports contains an invalid type, it should be a number, or an object
services.electrs.ports contains an invalid type, it should be a number, or an object
services.lnd.ports contains an invalid type, it should be a number, or an object
services.lnd.ports contains an invalid type, it should be a number, or an object
services.nginx.ports contains an invalid type, it should be a number, or an object
services.lnd.ports value [‘9735:9735’, ‘:’, ‘:’] has non-unique elements
ERROR: The Compose file ‘./docker-compose.yml’ is invalid because:
networks.default.ipam.config.subnet is invalid: should use the CIDR format
services.tor_proxy.ports contains an invalid type, it should be a number, or an object
services.bitcoin.ports contains an invalid type, it should be a number, or an object
services.electrs.ports contains an invalid type, it should be a number, or an object
services.lnd.ports contains an invalid type, it should be a number, or an object
services.lnd.ports contains an invalid type, it should be a number, or an object
services.nginx.ports contains an invalid type, it should be a number, or an object
services.lnd.ports value [‘9735:9735’, ‘:’, ‘:’] has non-unique elements
ERROR: The Compose file ‘./docker-compose.yml’ is invalid because:
networks.default.ipam.config.subnet is invalid: should use the CIDR format
services.tor_proxy.ports contains an invalid type, it should be a number, or an object
services.bitcoin.ports contains an invalid type, it should be a number, or an object
services.electrs.ports contains an invalid type, it should be a number, or an object
services.lnd.ports contains an invalid type, it should be a number, or an object
services.lnd.ports contains an invalid type, it should be a number, or an object
services.nginx.ports contains an invalid type, it should be a number, or an object
services.lnd.ports value [‘9735:9735’, ‘:’, ‘:’] has non-unique elements
bitfeed
ERROR: Couldn’t find env file: /home/umbrel/umbrel/.env
bluewallet
ERROR: Couldn’t find env file: /home/umbrel/umbrel/.env
btc-rpc-explorer
ERROR: Couldn’t find env file: /home/umbrel/umbrel/.env
btcpay-server
ERROR: Couldn’t find env file: /home/umbrel/umbrel/.env
element
ERROR: Couldn’t find env file: /home/umbrel/umbrel/.env
home-assistant
ERROR: Couldn’t find env file: /home/umbrel/umbrel/.env
krystal-bull
ERROR: Couldn’t find env file: /home/umbrel/umbrel/.env
lightning-terminal
ERROR: Couldn’t find env file: /home/umbrel/umbrel/.env
lnbits
ERROR: Couldn’t find env file: /home/umbrel/umbrel/.env
ERROR: Couldn’t find env file: /home/umbrel/umbrel/.env
lnmarkets
ERROR: Couldn’t find env file: /home/umbrel/umbrel/.env
mempool
ERROR: Couldn’t find env file: /home/umbrel/umbrel/.env
nextcloud
ERROR: Couldn’t find env file: /home/umbrel/umbrel/.env
node-red
ERROR: Couldn’t find env file: /home/umbrel/umbrel/.env
photoprism
ERROR: Couldn’t find env file: /home/umbrel/umbrel/.env
pi-hole
ERROR: Couldn’t find env file: /home/umbrel/umbrel/.env
ride-the-lightning
ERROR: Couldn’t find env file: /home/umbrel/umbrel/.env
simple-torrent
ERROR: Couldn’t find env file: /home/umbrel/umbrel/.env
sphinx-relay
ERROR: Couldn’t find env file: /home/umbrel/umbrel/.env
synapse
ERROR: Couldn’t find env file: /home/umbrel/umbrel/.env
syncthing
ERROR: Couldn’t find env file: /home/umbrel/umbrel/.env
thunderhub
ERROR: Couldn’t find env file: /home/umbrel/umbrel/.env
uptime-kuma
ERROR: Couldn’t find env file: /home/umbrel/umbrel/.env
urbit
ERROR: Couldn’t find env file: /home/umbrel/umbrel/.env
vaultwarden
The debug script did not automatically detect any issues with your Umbrel.
Thanks for any help.
Some more info. When running:
~/umbrel $ sudo ./scripts/configure
======================================
=========== RECONFIGURING ============
========= UMBREL (mainnet) ===========
cp: error writing ‘./templates/lnd.conf’: No space left on device
sudo systemctl restart umbrel-startup
Job for umbrel-startup.service failed because the control process exited with error code.
See “systemctl status umbrel-startup.service” and “journalctl -xe” for details.
systemctl status umbrel-startup.service
Unit service.service could not be found.
● umbrel-startup.service - Umbrel Startup Service
Loaded: loaded (/etc/systemd/system/umbrel-startup.service; enabled; vendor preset: enabled)
Active: activating (auto-restart) (Result: exit-code) since Sun 2022-09-25 20:50:00 UTC; 5s ago
Process: 23249 ExecStart=/home/umbrel/umbrel/scripts/start (code=exited, status=1/FAILURE)
journalctl -xe
– A start job for unit umbrel-startup.service has begun execution.
– The job identifier is 388418.
Sep 25 20:50:59 umbrel umbrel startup[25379]: ======================================
Sep 25 20:50:59 umbrel umbrel startup[25379]: ============= STARTING ===============
Sep 25 20:50:59 umbrel umbrel startup[25379]: ============== UMBREL ================
Sep 25 20:50:59 umbrel umbrel startup[25379]: ======================================
Sep 25 20:50:59 umbrel umbrel startup[25379]: Setting environment variables…
Sep 25 20:50:59 umbrel umbrel startup[25379]: Starting karen…
Sep 25 20:50:59 umbrel umbrel startup[25379]: Starting status monitors…
Sep 25 20:50:59 umbrel umbrel startup[25379]: Starting memory monitor…
Sep 25 20:50:59 umbrel umbrel startup[25379]: Starting backup monitor…
Sep 25 20:50:59 umbrel umbrel startup[25379]: Starting decoy backup trigger…
Sep 25 20:50:59 umbrel umbrel startup[25379]: Starting Docker services…
Sep 25 20:51:00 umbrel umbrel startup[25379]: The TOR_PROXY_PORT variable is not set. Defaulting to a blank string.
Sep 25 20:51:00 umbrel umbrel startup[25379]: The TOR_PROXY_IP variable is not set. Defaulting to a blank string.
Sep 25 20:51:00 umbrel umbrel startup[25379]: The TOR_SERVER_IP variable is not set. Defaulting to a blank string.
Sep 25 20:51:00 umbrel umbrel startup[25379]: The NGINX_PORT variable is not set. Defaulting to a blank string.
Sep 25 20:51:00 umbrel umbrel startup[25379]: The NGINX_IP variable is not set. Defaulting to a blank string.
Sep 25 20:51:00 umbrel umbrel startup[25379]: The BITCOIN_P2P_PORT variable is not set. Defaulting to a blank string.
Sep 25 20:51:00 umbrel umbrel startup[25379]: The BITCOIN_IP variable is not set. Defaulting to a blank string.
Sep 25 20:51:00 umbrel umbrel startup[25379]: The LND_REST_PORT variable is not set. Defaulting to a blank string.
Sep 25 20:51:00 umbrel umbrel startup[25379]: The LND_GRPC_PORT variable is not set. Defaulting to a blank string.
Sep 25 20:51:00 umbrel umbrel startup[25379]: The LND_IP variable is not set. Defaulting to a blank string.
Sep 25 20:51:00 umbrel umbrel startup[25379]: The DASHBOARD_IP variable is not set. Defaulting to a blank string.
Sep 25 20:51:00 umbrel umbrel startup[25379]: The MIDDLEWARE_IP variable is not set. Defaulting to a blank string.
Sep 25 20:51:00 umbrel umbrel startup[25379]: The BITCOIN_RPC_PORT variable is not set. Defaulting to a blank string.
Sep 25 20:51:00 umbrel umbrel startup[25379]: The BITCOIN_RPC_USER variable is not set. Defaulting to a blank string.
Sep 25 20:51:00 umbrel umbrel startup[25379]: The BITCOIN_RPC_PASS variable is not set. Defaulting to a blank string.
Sep 25 20:51:00 umbrel umbrel startup[25379]: The BITCOIN_NETWORK variable is not set. Defaulting to a blank string.
Sep 25 20:51:00 umbrel umbrel startup[25379]: The MANAGER_IP variable is not set. Defaulting to a blank string.
Sep 25 20:51:00 umbrel umbrel startup[25379]: The NEUTRINO_SWITCHER_IP variable is not set. Defaulting to a blank string.
Sep 25 20:51:00 umbrel umbrel startup[25379]: The ELECTRUM_PORT variable is not set. Defaulting to a blank string.
Sep 25 20:51:00 umbrel umbrel startup[25379]: The ELECTRUM_IP variable is not set. Defaulting to a blank string.
Sep 25 20:51:00 umbrel umbrel startup[25379]: The NETWORK_IP variable is not set. Defaulting to a blank string.
Sep 25 20:51:00 umbrel umbrel startup[25379]: The Compose file ‘./docker-compose.yml’ is invalid because:
Sep 25 20:51:00 umbrel umbrel startup[25379]: networks.default.ipam.config.subnet is invalid: should use the CIDR format
Sep 25 20:51:00 umbrel umbrel startup[25379]: services.tor_proxy.ports contains an invalid type, it should be a number, or an object
Sep 25 20:51:00 umbrel umbrel startup[25379]: services.bitcoin.ports contains an invalid type, it should be a number, or an object
Sep 25 20:51:00 umbrel umbrel startup[25379]: services.electrs.ports contains an invalid type, it should be a number, or an object
Sep 25 20:51:00 umbrel umbrel startup[25379]: services.lnd.ports contains an invalid type, it should be a number, or an object
Sep 25 20:51:00 umbrel umbrel startup[25379]: services.nginx.ports contains an invalid type, it should be a number, or an object
Sep 25 20:51:00 umbrel umbrel startup[25379]: services.lnd.ports value [‘9735:9735’, ‘:’, ‘:’] has non-unique elements
Sep 25 20:51:01 umbrel umbrel startup[25379]: Failed to start containers
Sep 25 20:51:01 umbrel systemd[1]: umbrel-startup.service: Control process exited, code=exited, status=1/FAILURE
– Subject: Unit process exited
– Defined-By: systemd
– Support: https://www.debian.org/support
– An ExecStart= process belonging to unit umbrel-startup.service has exited.
– The process’ exit code is ‘exited’ and its exit status is 1.
Sep 25 20:51:01 umbrel systemd[1]: umbrel-startup.service: Failed with result ‘exit-code’.
– Subject: Unit failed
– Defined-By: systemd
– Support: https://www.debian.org/support
– The unit umbrel-startup.service has entered the ‘failed’ state with result ‘exit-code’.
Sep 25 20:51:01 umbrel systemd[1]: Failed to start Umbrel Startup Service.
– Subject: A start job for unit umbrel-startup.service has failed
– Defined-By: systemd
– Support: https://www.debian.org/support
– A start job for unit umbrel-startup.service has finished with a failure.
– The job identifier is 388418 and the job result is failed.
lines 2460-2530/2530 (END)
control z
[1]+ Stopped journalctl -xe
umbrel@umbrel:~/umbrel $ sudo ./scripts/update/update --repo getumbrel/umbrel#v0.4.17
Cloning into ‘/tmp/umbrel-update’…
remote: Enumerating objects: 4740, done.
remote: Total 4740 (delta 0), reused 0 (delta 0), pack-reused 4740
Receiving objects: 100% (4740/4740), 1.24 MiB | 5.53 MiB/s, done.
Resolving deltas: 100% (2943/2943), done.
Note: checking out ‘c630cd9f27ac955631ba1f4570e24ab319fadc63’.
You are in ‘detached HEAD’ state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.
If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:
git checkout -b
=======================================
=============== UPDATE ================
========== Stage: Download ============
Creating lock
Cleaning up any previous mess
Copying Umbrel 0.4.17 from /tmp/umbrel-update/
mkdir: cannot create directory ‘/home/umbrel/umbrel/.umbrel-0.4.17’: No space left on device
cp: cannot create directory ‘/home/umbrel/umbrel/.umbrel-0.4.17’: No space left on device
./scripts/update/update: line 113: cd: /home/umbrel/umbrel/.umbrel-0.4.17: No such file or directory
Running update install scripts of the new release
== Begin Update Script 00-run.sh ==
=======================================
=============== UPDATE ================
========= Stage: Pre-update ===========
Installing Umbrel 0.4.17 at /home/umbrel/umbrel
Fixing permissions
Backing up existing directory tree
rsync: failed to open include file /home/umbrel/umbrel/.umbrel-0.4.17/scripts/update/.updateinclude: No such file or directory (2)
rsync error: error in file IO (code 11) at exclude.c(1207) [client=3.1.3]
== End Update Script 00-run.sh ==
== Begin Update Script 01-run.sh ==
=======================================
=============== UPDATE ================
=========== Stage: Install ============
=============================================
Installing on Umbrel OS v0.5.0
Replacing /sd-root/home/umbrel/umbrel on SD card with the new release
rsync: failed to open include file /home/umbrel/umbrel/.umbrel-0.4.17/scripts/update/.updateinclude: No such file or directory (2)
rsync error: error in file IO (code 11) at exclude.c(1207) [client=3.1.3]
== End Update Script 01-run.sh ==
== Begin Update Script 02-run.sh ==
=======================================
=============== UPDATE ================
========= Stage: Post-update ==========
== End Update Script 02-run.sh ==
== Begin Update Script 03-run.sh ==
=======================================
=============== UPDATE ================
=========== Stage: Success ============
Removing backup
Successfully installed Umbrel 0.4.17
== End Update Script 03-run.sh ==
Deleting cloned repository
Removing lock
`sudo ./scripts/update/update --repo getumbrel/umbrel#v0.5.0
Cloning into ‘/tmp/umbrel-update’…
remote: Enumerating objects: 5168, done.
remote: Counting objects: 100% (36/36), done.
remote: Compressing objects: 100% (2/2), done.
remote: Total 5168 (delta 35), reused 34 (delta 34), pack-reused 5132
Receiving objects: 100% (5168/5168), 1.56 MiB | 5.76 MiB/s, done.
Resolving deltas: 100% (3149/3149), done.
Note: checking out ‘7c9084e12f06652b1a193136fe50d70e711a2dbc’.
You are in ‘detached HEAD’ state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.
If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:
git checkout -b
=======================================
=============== UPDATE ================
========== Stage: Download ============
Creating lock
Cleaning up any previous mess
Copying Umbrel 0.5.0 from /tmp/umbrel-update/
mkdir: cannot create directory ‘/home/umbrel/umbrel/.umbrel-0.5.0’: No space left on device
cp: cannot create directory ‘/home/umbrel/umbrel/.umbrel-0.5.0’: No space left on device
./scripts/update/update: line 113: cd: /home/umbrel/umbrel/.umbrel-0.5.0: No such file or directory
Running update install scripts of the new release
== Begin Update Script 00-run.sh ==
=======================================
=============== UPDATE ================
========= Stage: Pre-update ===========
Installing Umbrel 0.5.0 at /home/umbrel/umbrel
Fixing permissions
Backing up existing directory tree
rsync: failed to open include file /home/umbrel/umbrel/.umbrel-0.5.0/scripts/update/.updateinclude: No such file or directory (2)
rsync error: error in file IO (code 11) at exclude.c(1207) [client=3.1.3]
== End Update Script 00-run.sh ==
== Begin Update Script 01-run.sh ==
=======================================
=============== UPDATE ================
=========== Stage: Install ============
=============================================
Installing on Umbrel OS v0.5.0
Replacing /sd-root/home/umbrel/umbrel on SD card with the new release
rsync: failed to open include file /home/umbrel/umbrel/.umbrel-0.5.0/scripts/update/.updateinclude: No such file or directory (2)
rsync error: error in file IO (code 11) at exclude.c(1207) [client=3.1.3]
== End Update Script 01-run.sh ==
== Begin Update Script 02-run.sh ==
=======================================
=============== UPDATE ================
========= Stage: Post-update ==========
== End Update Script 02-run.sh ==
== Begin Update Script 03-run.sh ==
=======================================
=============== UPDATE ================
=========== Stage: Success ============
Removing backup
Successfully installed Umbrel 0.5.0
== End Update Script 03-run.sh ==
Deleting cloned repository
Removing lock
UPDATE: I flashed a new mSD card and it does not yet seemed to have resolved the issue with the No space left on device
Sorry for the late reply, I did not have access to my Umbrel box for the longest time. This command is not working, because the disk is full:
sudo du -h -a / 2>/dev/null | sort -h -r | head -n 100 sort: write failed: /tmp/sort8WesRh: No space left on device
I’m not sure how I can find out what is filling up the device space and what to do about it. I would prefer not to have to start over from scratch and reinstall everything, so if there are any tips on what my options are I would greatly appreciate it!
@dhuyvetter Hi, just curious if you were able to get your node recovered? I’m currently going through pretty much the same experience and haven’t found a solution yet.
Tedzo
Same here, anyone can assist on this please?
I also have Error: System service failed and I guess these logs are the issue:
Mar 06 14:15:14 umbrel systemd[1]: Dependency failed for Umbrel Startup Service.
Mar 06 14:15:14 umbrel systemd[1]: umbrel-startup.service: Job umbrel-startup.service/start failed with result ‘dependency’.
Thanks a lot
