I’ve been having recent issues with my Umbrel node and it just got worse. Quick background - my node stopped working a few weeks ago, but I was still able to log in. No details would load (LND details, Bitcoin core details, Bitcoin wallet details, etc.). It just said synchronizing on all three of those apps, but no new blocks were coming in. My node had been previously synchronized with the blockchain for about a few months before these issues started happening too.
I tried resetting a few times, but no luck. Umbrel v0.4.11 came out during this time, so I tried updating, but I’m pretty sure the updated failed. After that, I’m unable to login to the node via the web. When I go to the login page it says Error: System service failed and gives me the option to restart or shutdown, both of which I’ve done multiple times but still get the same error (tried to paste image below).
I am able to ssh into my Node. Can someone help give me troubleshooting tips? I think the original issue may have been with LND, but I’m not sure. Now I think it’s an issue with the upgrade failing. I’d like to avoid reflashing the SD card if possible, as I don’t want to lose my lightning channels. (I do have my seed phrase and LND backup file just in case though). Can someone give me some simple steps I can do via ssh to look up log files that maybe someone can help me understand to resolve this. I was able to get the debug log (via /umbrel/scripts/debug) but it’s 33 pages long so wasn’t sure if there was a particular section I should post here or not to help troubleshoot. Thank you so much in advance!!
Looking through the logs, here are some excerpts that may look like issues:
Startup service logs
Feb 14 10:15:21 umbrel umbrel startup[7917]: Starting Docker services…
Feb 14 10:15:24 umbrel umbrel startup[7917]: Starting electrs …
Feb 14 10:15:24 umbrel umbrel startup[7917]: Starting umbrel_app_2_tor_1 …
Feb 14 10:15:24 umbrel umbrel startup[7917]: Starting umbrel_app_tor_1 …
Feb 14 10:15:24 umbrel umbrel startup[7917]: Starting umbrel_app_3_tor_1 …
Feb 14 10:15:24 umbrel umbrel startup[7917]: Starting dashboard …
Feb 14 10:15:24 umbrel umbrel startup[7917]: Starting tor …
Feb 14 10:15:25 umbrel umbrel startup[7917]: Starting umbrel_app_tor_1 … error
Feb 14 10:15:25 umbrel umbrel startup[7917]: ERROR: for umbrel_app_tor_1 Cannot start service app_tor: mkdir /var/lib/containerd/io.containerd.runtime.v2.task/moby/ca0ad933d9b99a96426a3f807a4d0db6fe61fd432a64e423eaeb8a1e2b6941de: no space left on device: unknown
Feb 14 10:15:25 umbrel umbrel startup[7917]: Starting umbrel_app_3_tor_1 … error
Feb 14 10:15:25 umbrel umbrel startup[7917]: ERROR: for umbrel_app_3_tor_1 Cannot start service app_3_tor: mkdir /var/lib/containerd/io.containerd.runtime.v2.task/moby/23ed2c19681c87be2a8df9b7d479e01036972d539bfa3e8e309afafd119e1b49: no space left on device: unknown
Feb 14 10:15:25 umbrel umbrel startup[7917]: Starting dashboard … error
Feb 14 10:15:25 umbrel umbrel startup[7917]: ERROR: for dashboard Cannot start service dashboard: mkdir /var/lib/containerd/io.containerd.runtime.v2.task/moby/ce4c8d7462f23e8917da297ddafaead1c48b977c141bb7e442f85bed9fba3189: no space left on device: unknown
Feb 14 10:15:25 umbrel umbrel startup[7917]: Starting umbrel_app_2_tor_1 … error
Feb 14 10:15:25 umbrel umbrel startup[7917]: ERROR: for umbrel_app_2_tor_1 Cannot start service app_2_tor: mkdir /var/lib/containerd/io.containerd.runtime.v2.task/moby/8468dbe420adc859489b627668bdf028701b3655a7b8eb0c4b710eb2e764beaf: no space left on device: unknown
Feb 14 10:15:25 umbrel umbrel startup[7917]: Starting electrs … error
Feb 14 10:15:25 umbrel umbrel startup[7917]: ERROR: for electrs Cannot start service electrs: mkdir /var/lib/containerd/io.containerd.runtime.v2.task/moby/6ca9105c1ad0a64d11d9251a3b8c5f311095631e34634587fc39f24395fdb536: no space left on device: unknown
Feb 14 10:15:25 umbrel umbrel startup[7917]: Starting tor … error
Feb 14 10:15:25 umbrel umbrel startup[7917]: ERROR: for tor Cannot start service tor: mkdir /var/lib/containerd/io.containerd.runtime.v2.task/moby/11667858fe3100cdd32120c8b436d99053a0188bc4fb0a75b1743e0dbf73dabf: no space left on device: unknown
Feb 14 10:15:25 umbrel umbrel startup[7917]: ERROR: for app_tor Cannot start service app_tor: mkdir /var/lib/containerd/io.containerd.runtime.v2.task/moby/ca0ad933d9b99a96426a3f807a4d0db6fe61fd432a64e423eaeb8a1e2b6941de: no space left on device: unknown
Feb 14 10:15:25 umbrel umbrel startup[7917]: ERROR: for app_3_tor Cannot start service app_3_tor: mkdir /var/lib/containerd/io.containerd.runtime.v2.task/moby/23ed2c19681c87be2a8df9b7d479e01036972d539bfa3e8e309afafd119e1b49: no space left on device: unknown
Feb 14 10:15:25 umbrel umbrel startup[7917]: ERROR: for dashboard Cannot start service dashboard: mkdir /var/lib/containerd/io.containerd.runtime.v2.task/moby/ce4c8d7462f23e8917da297ddafaead1c48b977c141bb7e442f85bed9fba3189: no space left on device: unknown
Feb 14 10:15:25 umbrel umbrel startup[7917]: ERROR: for app_2_tor Cannot start service app_2_tor: mkdir /var/lib/containerd/io.containerd.runtime.v2.task/moby/8468dbe420adc859489b627668bdf028701b3655a7b8eb0c4b710eb2e764beaf: no space left on device: unknown
Feb 14 10:15:25 umbrel umbrel startup[7917]: ERROR: for electrs Cannot start service electrs: mkdir /var/lib/containerd/io.containerd.runtime.v2.task/moby/6ca9105c1ad0a64d11d9251a3b8c5f311095631e34634587fc39f24395fdb536: no space left on device: unknown
Feb 14 10:15:25 umbrel umbrel startup[7917]: ERROR: for tor Cannot start service tor: mkdir /var/lib/containerd/io.containerd.runtime.v2.task/moby/11667858fe3100cdd32120c8b436d99053a0188bc4fb0a75b1743e0dbf73dabf: no space left on device: unknown
Feb 14 10:15:25 umbrel umbrel startup[7917]: Encountered errors while bringing up the project.
Feb 14 10:15:26 umbrel umbrel startup[7917]: Failed to start containers
Feb 14 10:15:26 umbrel systemd[1]: umbrel-startup.service: Control process exited, code=exited, status=1/FAILURE
Feb 14 10:15:26 umbrel systemd[1]: umbrel-startup.service: Failed with result ‘exit-code’.
Feb 14 10:15:26 umbrel systemd[1]: Failed to start Umbrel Startup Service.
Umbrel logs
Attaching to middleware, manager
middleware | LndUnlocker: Wallet failed to unlock!
manager | ::ffff:10.21.21.2 - - [Fri, 14 Jan 2022 04:54:16 GMT] “GET /v1/system/update-status HTTP/1.0” 304 - “-” “Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.99 Safari/537.36”
manager |
manager | umbrel-manager
manager | ::ffff:10.21.21.2 - - [Fri, 14 Jan 2022 04:55:16 GMT] “GET /v1/system/update-status HTTP/1.0” 304 - “-” “Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.99 Safari/537.36”
(this error repeats multiple times)
Attaching to electrs
electrs | [2022-01-13T00:20:02.741Z INFO electrs::daemon] waiting for RPC warmup: Verifying blocks…
electrs | [2022-01-13T00:20:03.744Z INFO electrs::daemon] waiting for RPC warmup: Verifying blocks…
electrs | [2022-01-13T00:20:04.752Z INFO electrs::daemon] waiting for RPC warmup: Verifying blocks…
electrs | [2022-01-13T00:20:05.755Z INFO electrs::daemon] waiting for RPC warmup: Verifying blocks…
electrs | [2022-01-13T00:20:06.757Z INFO electrs::daemon] waiting for RPC warmup: Verifying blocks…
electrs | [2022-01-13T00:20:07.760Z INFO electrs::daemon] waiting for RPC warmup: Verifying blocks…
electrs | [2022-01-13T00:20:08.762Z INFO electrs::daemon] waiting for RPC warmup: Verifying blocks…
electrs | [2022-01-13T00:20:09.765Z INFO electrs::daemon] waiting for RPC warmup: Verifying blocks…
electrs | [2022-01-13T00:20:10.767Z INFO electrs::daemon] waiting for RPC warmup: Verifying blocks…
electrs | [2022-01-13T00:20:11.770Z INFO electrs::daemon] waiting for RPC warmup: Verifying blocks…
electrs | [2022-01-13T00:20:12.772Z INFO electrs::daemon] waiting for RPC warmup: Verifying blocks…
electrs | [2022-01-13T00:20:13.775Z INFO electrs::daemon] waiting for RPC warmup: Verifying blocks…
electrs | [2022-01-13T00:20:14.777Z INFO electrs::daemon] waiting for RPC warmup: Verifying blocks…
electrs | [2022-01-13T00:20:15.780Z INFO electrs::daemon] waiting for RPC warmup: Verifying blocks…
electrs | [2022-01-13T00:20:16.785Z INFO electrs::daemon] waiting for RPC warmup: Verifying blocks…
electrs | [2022-01-13T00:20:17.787Z INFO electrs::daemon] waiting for RPC warmup: Verifying blocks…
electrs | [2022-01-13T00:20:18.793Z INFO electrs::daemon] waiting for RPC warmup: Verifying blocks…
electrs | [2022-01-13T00:20:19.796Z INFO electrs::daemon] waiting for RPC warmup: Verifying blocks…
electrs | [2022-01-13T00:20:20.798Z INFO electrs::daemon] waiting for RPC warmup: Verifying blocks…
electrs | [2022-01-13T00:20:21.801Z INFO electrs::daemon] waiting for RPC warmup: Verifying blocks…
electrs | [2022-01-13T00:20:22.803Z INFO electrs::daemon] waiting for RPC warmup: Verifying blocks…
electrs | [2022-01-13T00:20:23.805Z INFO electrs::daemon] waiting for RPC warmup: Loading P2P addresses…
electrs | [2022-01-13T00:20:27.703Z INFO electrs::daemon] waiting for 44 blocks to download (IBD)
electrs | [2022-01-13T00:20:50.377Z INFO electrs::db] closing DB at /data/db/bitcoin
electrs | Error: electrs failed
electrs |
electrs | Caused by:
electrs | 0: bitcoind RPC polling failed
electrs | 1: daemon not available
electrs | 2: JSON-RPC error: transport error: Didn’t receive response data in time, timed out.
Tor logs
Attaching to umbrel_app_2_tor_1, umbrel_app_tor_1, tor, umbrel_app_3_tor_1
app_2_tor_1 | Jan 13 01:09:14.000 [warn] Received NETINFO cell with skewed time (OR:204.13.164.118:443): It seems that our clock is behind by 7 days, 20 hours, 16 minutes, or that theirs is ahead. Tor requires an accurate clock to work: please check your time, timezone, and date settings.
app_2_tor_1 | Jan 13 01:09:14.000 [warn] Problem bootstrapping. Stuck at 30% (loading_status): Loading networkstatus consensus. (Clock skew -677809 in NETINFO cell from OR; CLOCK_SKEW; count 82; recommendation warn; host 24E2F139121D4394C54B5BCC368B3B411857C413 at 204.13.164.118:443)
app_2_tor_1 | Jan 13 01:09:14.000 [warn] 42 connections have failed:
app_2_tor_1 | Jan 13 01:09:14.000 [warn] 38 connections died in state connect()ing with SSL state (No SSL object)
app_2_tor_1 | Jan 13 01:09:14.000 [warn] 4 connections died in state handshaking (TLS) with SSL state error in HANDSHAKE
app_2_tor_1 | Jan 13 01:09:14.000 [warn] Received directory with skewed time (DIRSERV:204.13.164.118:443): It seems that our clock is behind by 7 days, 20 hours, 16 minutes, or that theirs is ahead. Tor requires an accurate clock to work: please check your time, timezone, and date settings.
Look at the tor section of the debug log I noticed an issue about incorrect time. I just noticed the time when I login to my Umbrel node on ssh is incorrect. It says last login was “Thu Feb 14 10:13:06 2019” when it should have been a few minutes ago. Could this be the issue, and if so, how do I correct it?
Revision : d03114
Serial : 10000000b925c688
Model : Raspberry Pi 4 Model B Rev 1.4
Firmware
Aug 3 2021 18:14:56
version 40787ee5905644f639a2a0f6e00ae12e517a2211 (clean) (release) (start)
Temperature
temp=44.8’C
Throttling
throttled=0x0
Memory usage
total used free shared buff/cache available
Mem: 7.8G 163M 7.3G 16M 350M 7.5G
Swap: 4.1G 0B 4.1G
total: 2.1%
system: 2.1%
tor: 0%
thunderhub: 0%
sphinx-relay: 0%
ride-the-lightning: 0%
mempool: 0%
lnd: 0%
lnbits: 0%
lightning-terminal: 0%
electrs: 0%
btc-rpc-explorer: 0%
bluewallet: 0%
bitcoin: 0%
Memory monitor logs
2019-02-14 10:13:13 Memory monitor running!
2019-02-14 10:13:27 Memory monitor running!
2019-02-14 10:13:42 Memory monitor running!
2019-02-14 10:13:56 Memory monitor running!
2019-02-14 10:14:10 Memory monitor running!
2019-02-14 10:14:24 Memory monitor running!
2019-02-14 10:14:38 Memory monitor running!
2019-02-14 10:14:52 Memory monitor running!
2019-02-14 10:15:07 Memory monitor running!
2019-02-14 10:15:21 Memory monitor running!
Filesystem information
Filesystem Size Used Avail Use% Mounted on
/dev/root 30G 30G 0 100% /
/dev/sda1 916G 497G 374G 58% /home/umbrel/umbrel
Startup service logs
Feb 14 10:15:21 umbrel umbrel startup[7917]: Starting Docker services…
Feb 14 10:15:24 umbrel umbrel startup[7917]: Starting electrs …
Feb 14 10:15:24 umbrel umbrel startup[7917]: Starting umbrel_app_2_tor_1 …
Feb 14 10:15:24 umbrel umbrel startup[7917]: Starting umbrel_app_tor_1 …
Feb 14 10:15:24 umbrel umbrel startup[7917]: Starting umbrel_app_3_tor_1 …
Feb 14 10:15:24 umbrel umbrel startup[7917]: Starting dashboard …
Feb 14 10:15:24 umbrel umbrel startup[7917]: Starting tor …
Feb 14 10:15:25 umbrel umbrel startup[7917]: Starting umbrel_app_tor_1 … error
Feb 14 10:15:25 umbrel umbrel startup[7917]: ERROR: for umbrel_app_tor_1 Cannot start service app_tor: mkdir /var/lib/containerd/io.containerd.runtime.v2.task/moby/ca0ad933d9b99a96426a3f807a4d0db6fe61fd432a64e423eaeb8a1e2b6941de: no space left on device: unknown
Feb 14 10:15:25 umbrel umbrel startup[7917]: Starting umbrel_app_3_tor_1 … error
Feb 14 10:15:25 umbrel umbrel startup[7917]: ERROR: for umbrel_app_3_tor_1 Cannot start service app_3_tor: mkdir /var/lib/containerd/io.containerd.runtime.v2.task/moby/23ed2c19681c87be2a8df9b7d479e01036972d539bfa3e8e309afafd119e1b49: no space left on device: unknown
Feb 14 10:15:25 umbrel umbrel startup[7917]: Starting dashboard … error
Feb 14 10:15:25 umbrel umbrel startup[7917]: ERROR: for dashboard Cannot start service dashboard: mkdir /var/lib/containerd/io.containerd.runtime.v2.task/moby/ce4c8d7462f23e8917da297ddafaead1c48b977c141bb7e442f85bed9fba3189: no space left on device: unknown
Feb 14 10:15:25 umbrel umbrel startup[7917]: Starting umbrel_app_2_tor_1 … error
Feb 14 10:15:25 umbrel umbrel startup[7917]: ERROR: for umbrel_app_2_tor_1 Cannot start service app_2_tor: mkdir /var/lib/containerd/io.containerd.runtime.v2.task/moby/8468dbe420adc859489b627668bdf028701b3655a7b8eb0c4b710eb2e764beaf: no space left on device: unknown
Feb 14 10:15:25 umbrel umbrel startup[7917]: Starting electrs … error
Feb 14 10:15:25 umbrel umbrel startup[7917]: ERROR: for electrs Cannot start service electrs: mkdir /var/lib/containerd/io.containerd.runtime.v2.task/moby/6ca9105c1ad0a64d11d9251a3b8c5f311095631e34634587fc39f24395fdb536: no space left on device: unknown
Feb 14 10:15:25 umbrel umbrel startup[7917]: Starting tor … error
Feb 14 10:15:25 umbrel umbrel startup[7917]: ERROR: for tor Cannot start service tor: mkdir /var/lib/containerd/io.containerd.runtime.v2.task/moby/11667858fe3100cdd32120c8b436d99053a0188bc4fb0a75b1743e0dbf73dabf: no space left on device: unknown
Feb 14 10:15:25 umbrel umbrel startup[7917]: ERROR: for app_tor Cannot start service app_tor: mkdir /var/lib/containerd/io.containerd.runtime.v2.task/moby/ca0ad933d9b99a96426a3f807a4d0db6fe61fd432a64e423eaeb8a1e2b6941de: no space left on device: unknown
Feb 14 10:15:25 umbrel umbrel startup[7917]: ERROR: for app_3_tor Cannot start service app_3_tor: mkdir /var/lib/containerd/io.containerd.runtime.v2.task/moby/23ed2c19681c87be2a8df9b7d479e01036972d539bfa3e8e309afafd119e1b49: no space left on device: unknown
Feb 14 10:15:25 umbrel umbrel startup[7917]: ERROR: for dashboard Cannot start service dashboard: mkdir /var/lib/containerd/io.containerd.runtime.v2.task/moby/ce4c8d7462f23e8917da297ddafaead1c48b977c141bb7e442f85bed9fba3189: no space left on device: unknown
Feb 14 10:15:25 umbrel umbrel startup[7917]: ERROR: for app_2_tor Cannot start service app_2_tor: mkdir /var/lib/containerd/io.containerd.runtime.v2.task/moby/8468dbe420adc859489b627668bdf028701b3655a7b8eb0c4b710eb2e764beaf: no space left on device: unknown
Feb 14 10:15:25 umbrel umbrel startup[7917]: ERROR: for electrs Cannot start service electrs: mkdir /var/lib/containerd/io.containerd.runtime.v2.task/moby/6ca9105c1ad0a64d11d9251a3b8c5f311095631e34634587fc39f24395fdb536: no space left on device: unknown
Feb 14 10:15:25 umbrel umbrel startup[7917]: ERROR: for tor Cannot start service tor: mkdir /var/lib/containerd/io.containerd.runtime.v2.task/moby/11667858fe3100cdd32120c8b436d99053a0188bc4fb0a75b1743e0dbf73dabf: no space left on device: unknown
Feb 14 10:15:25 umbrel umbrel startup[7917]: Encountered errors while bringing up the project.
Feb 14 10:15:26 umbrel umbrel startup[7917]: Failed to start containers
Feb 14 10:15:26 umbrel systemd[1]: umbrel-startup.service: Control process exited, code=exited, status=1/FAILURE
Feb 14 10:15:26 umbrel systemd[1]: umbrel-startup.service: Failed with result ‘exit-code’.
Feb 14 10:15:26 umbrel systemd[1]: Failed to start Umbrel Startup Service.
External storage service logs
– Logs begin at Thu 2019-02-14 10:11:58 UTC, end at Thu 2019-02-14 10:15:26 UTC. –
Feb 14 10:12:03 umbrel systemd[1]: Starting External Storage Mounter…
Feb 14 10:12:03 umbrel external storage mounter[506]: Running external storage mount script…
Feb 14 10:12:03 umbrel external storage mounter[506]: Found device “ATA CT1000MX500SSD1”
Feb 14 10:12:03 umbrel external storage mounter[506]: Blacklisting USB device IDs against UAS driver…
Feb 14 10:12:03 umbrel external storage mounter[506]: Rebinding USB drivers…
Feb 14 10:12:03 umbrel external storage mounter[506]: Checking USB devices are back…
Feb 14 10:12:03 umbrel external storage mounter[506]: Waiting for USB devices…
Feb 14 10:12:04 umbrel external storage mounter[506]: Waiting for USB devices…
Feb 14 10:12:05 umbrel external storage mounter[506]: Checking if the device is ext4…
Feb 14 10:12:06 umbrel external storage mounter[506]: Yes, it is ext4
Feb 14 10:12:07 umbrel external storage mounter[506]: Checking if device contains an Umbrel install…
Feb 14 10:12:07 umbrel external storage mounter[506]: Yes, it contains an Umbrel install
Feb 14 10:12:07 umbrel external storage mounter[506]: Bind mounting external storage over local Umbrel installation…
Feb 14 10:12:07 umbrel external storage mounter[506]: Bind mounting external storage over local Docker data dir…
Feb 14 10:12:07 umbrel external storage mounter[506]: Bind mounting external storage to /swap
Feb 14 10:12:07 umbrel external storage mounter[506]: Bind mounting SD card root at /sd-card…
Feb 14 10:12:07 umbrel external storage mounter[506]: Checking Umbrel root is now on external storage…
Feb 14 10:12:08 umbrel external storage mounter[506]: Checking /var/lib/docker is now on external storage…
Feb 14 10:12:08 umbrel external storage mounter[506]: Checking /swap is now on external storage…
Feb 14 10:12:08 umbrel external storage mounter[506]: Setting up swapfile
Feb 14 10:12:23 umbrel external storage mounter[506]: Setting up swapspace version 1, size = 4 GiB (4294963200 bytes)
Feb 14 10:12:23 umbrel external storage mounter[506]: no label, UUID=50ecec99-37bf-474e-b2b1-299995ab7203
Feb 14 10:12:23 umbrel external storage mounter[506]: Checking SD Card root is bind mounted at /sd-root…
Feb 14 10:12:23 umbrel external storage mounter[506]: Starting external drive mount monitor…
Feb 14 10:12:23 umbrel external storage mounter[506]: Mount script completed successfully!
Feb 14 10:12:23 umbrel systemd[1]: Started External Storage Mounter.
External storage SD card update service logs
Feb 14 10:12:41 umbrel external storage updater[1376]: =======================================
Feb 14 10:12:41 umbrel external storage updater[1376]: =============================================
Feb 14 10:12:41 umbrel external storage updater[1376]: Installing on Umbrel OS v0.4.1
Feb 14 10:12:41 umbrel external storage updater[1376]: =============================================
Feb 14 10:12:41 umbrel external storage updater[1376]: Replacing /sd-root/home/umbrel/umbrel on SD card with the new release
Feb 14 10:12:41 umbrel external storage updater[1376]: sending incremental file list
Feb 14 10:12:41 umbrel external storage updater[1376]: sent 6,173 bytes received 123 bytes 12,592.00 bytes/sec
Feb 14 10:12:41 umbrel external storage updater[1376]: total size is 367,682 speedup is 58.40
Feb 14 10:12:41 umbrel external storage updater[1376]: Fixing permissions
Feb 14 10:12:44 umbrel external storage updater[1376]: ./01-run.sh: line 81: cannot create temp file for here-document: No space left on device
Feb 14 10:12:44 umbrel external storage updater[1376]: == End Update Script 01-run.sh ==
Feb 14 10:12:44 umbrel external storage updater[1376]: == Begin Update Script 02-run.sh ==
Feb 14 10:12:44 umbrel external storage updater[1376]: =======================================
Feb 14 10:12:44 umbrel external storage updater[1376]: =============== UPDATE ================
Feb 14 10:12:44 umbrel external storage updater[1376]: =======================================
Feb 14 10:12:44 umbrel external storage updater[1376]: ========= Stage: Post-update ==========
Feb 14 10:12:44 umbrel external storage updater[1376]: =======================================
Feb 14 10:12:44 umbrel external storage updater[1376]: == End Update Script 02-run.sh ==
Feb 14 10:12:44 umbrel external storage updater[1376]: == Begin Update Script 03-run.sh ==
Feb 14 10:12:44 umbrel external storage updater[1376]: =======================================
Feb 14 10:12:44 umbrel external storage updater[1376]: =============== UPDATE ================
Feb 14 10:12:44 umbrel external storage updater[1376]: =======================================
Feb 14 10:12:44 umbrel external storage updater[1376]: =========== Stage: Success ============
Feb 14 10:12:44 umbrel external storage updater[1376]: =======================================
Feb 14 10:12:44 umbrel external storage updater[1376]: Removing backup
Feb 14 10:12:44 umbrel external storage updater[1376]: ./03-run.sh: line 19: cannot create temp file for here-document: No space left on device
Feb 14 10:12:44 umbrel external storage updater[1376]: == End Update Script 03-run.sh ==
Feb 14 10:12:44 umbrel external storage updater[1376]: Deleting cloned repository
Feb 14 10:12:44 umbrel external storage updater[1376]: Removing lock
Feb 14 10:12:44 umbrel systemd[1]: Started External Storage SDcard Updater.
Karen logs
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
karen is running in /home/umbrel/umbrel/events
Docker containers
NAMES STATUS
Umbrel logs
Attaching to middleware, manager
middleware | LndUnlocker: Wallet failed to unlock!
middleware | LndUnlocker: Wallet failed to unlock!
middleware | LndUnlocker: Wallet failed to unlock!
middleware | LndUnlocker: Wallet failed to unlock!
middleware | LndUnlocker: Wallet failed to unlock!
middleware | LndUnlocker: Wallet failed to unlock!
middleware | LndUnlocker: Wallet failed to unlock!
middleware | LndUnlocker: Wallet failed to unlock!
middleware | LndUnlocker: Wallet failed to unlock!
middleware | LndUnlocker: Wallet failed to unlock!
middleware | LndUnlocker: Wallet failed to unlock!
middleware | LndUnlocker: Wallet failed to unlock!
middleware | LndUnlocker: Wallet failed to unlock!
middleware | LndUnlocker: Wallet failed to unlock!
middleware | LndUnlocker: Wallet failed to unlock!
middleware | LndUnlocker: Wallet failed to unlock!
middleware | LndUnlocker: Wallet failed to unlock!
middleware | LndUnlocker: Wallet failed to unlock!
middleware | LndUnlocker: Wallet failed to unlock!
middleware | LndUnlocker: Wallet failed to unlock!
middleware | LndUnlocker: Wallet failed to unlock!
middleware | LndUnlocker: Wallet failed to unlock!
middleware | LndUnlocker: Wallet failed to unlock!
middleware | LndUnlocker: Wallet failed to unlock!
middleware | LndUnlocker: Wallet failed to unlock!
middleware | LndUnlocker: Wallet failed to unlock!
middleware | LndUnlocker: Wallet failed to unlock!
manager | ::ffff:10.21.21.2 - - [Fri, 14 Jan 2022 04:54:16 GMT] “GET /v1/system/update-status HTTP/1.0” 304 - “-” “Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.99 Safari/537.36”
manager |
manager | umbrel-manager
manager | ::ffff:10.21.21.2 - - [Fri, 14 Jan 2022 04:55:16 GMT] “GET /v1/system/update-status HTTP/1.0” 304 - “-” “Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.99 Safari/537.36”
manager |
manager | umbrel-manager
manager | ::ffff:10.21.21.2 - - [Fri, 14 Jan 2022 04:56:16 GMT] “GET /v1/system/update-status HTTP/1.0” 304 - “-” “Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.99 Safari/537.36”
manager |
manager | umbrel-manager
manager | ::ffff:10.21.21.2 - - [Fri, 14 Jan 2022 04:57:16 GMT] “GET /v1/system/update-status HTTP/1.0” 304 - “-” “Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.99 Safari/537.36”
manager |
manager | umbrel-manager
manager | ::ffff:10.21.21.2 - - [Fri, 14 Jan 2022 04:58:16 GMT] “GET /v1/system/update-status HTTP/1.0” 304 - “-” “Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.99 Safari/537.36”
manager |
manager | umbrel-manager
manager | ::ffff:10.21.21.2 - - [Fri, 14 Jan 2022 04:59:16 GMT] “GET /v1/system/update-status HTTP/1.0” 304 - “-” “Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.99 Safari/537.36”
manager |
manager | umbrel-manager
manager | ::ffff:10.21.21.2 - - [Fri, 14 Jan 2022 05:00:16 GMT] “GET /v1/system/update-status HTTP/1.0” 304 - “-” “Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.99 Safari/537.36”
manager |
manager | umbrel-manager
manager | ::ffff:10.21.21.2 - - [Fri, 14 Jan 2022 05:01:16 GMT] “GET /v1/system/update-status HTTP/1.0” 304 - “-” “Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.99 Safari/537.36”
manager |
manager | umbrel-manager
manager | ::ffff:10.21.21.2 - - [Fri, 14 Jan 2022 05:02:16 GMT] “GET /v1/system/update-status HTTP/1.0” 304 - “-” “Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.99 Safari/537.36”
manager |
middleware | LndUnlocker: Wallet failed to unlock!
middleware | LndUnlocker: Wallet failed to unlock!
middleware | LndUnlocker: Wallet failed to unlock!
manager | umbrel-manager
manager | ::ffff:10.21.21.2 - - [Fri, 14 Jan 2022 05:03:16 GMT] “GET /v1/system/update-status HTTP/1.0” 304 - “-” “Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.99 Safari/537.36”
manager |
manager | umbrel-manager
Bitcoin Core logs
Attaching to bitcoin
bitcoin | 2022-01-13T00:21:59Z UpdateTip: new best=000000000000000000093200fa1b790bf5f3bf6a25c14cac0486f49c57debaa1 height=718357 version=0x3fffe000 log2_work=93.282615 tx=702279507 date=‘2022-01-12T21:15:57Z’ progress=0.999961 cache=41.6MiB(305833txo)
bitcoin | 2022-01-13T00:22:00Z UpdateTip: new best=000000000000000000086cc940582ee30b4a6f65c000890ac43dd579f30ef218 height=718358 version=0x3fff0000 log2_work=93.282628 tx=702279959 date=‘2022-01-12T21:17:51Z’ progress=0.999962 cache=41.8MiB(307030txo)
bitcoin | 2022-01-13T00:22:01Z UpdateTip: new best=000000000000000000075210e77e045168b6212eaf40f3402eccd472f71b18f7 height=718359 version=0x20600004 log2_work=93.282640 tx=702282571 date=‘2022-01-12T21:29:04Z’ progress=0.999964 cache=42.6MiB(313487txo)
bitcoin | 2022-01-13T00:22:02Z UpdateTip: new best=000000000000000000094bca7b169c051ddbe8b58733c640015ad23321042724 height=718360 version=0x2059c004 log2_work=93.282653 tx=702283835 date=‘2022-01-12T21:34:15Z’ progress=0.999965 cache=42.9MiB(316622txo)
bitcoin | 2022-01-13T00:22:03Z UpdateTip: new best=00000000000000000005a753cde50e3b9d61ac7db211119d67230beb34dac4a1 height=718361 version=0x20600000 log2_work=93.282665 tx=702284571 date=‘2022-01-12T21:37:21Z’ progress=0.999966 cache=43.2MiB(318739txo)
bitcoin | 2022-01-13T00:22:05Z UpdateTip: new best=00000000000000000005c421fb5a30a6d10ce62559019b42c074bd99eae9ec87 height=718362 version=0x20400000 log2_work=93.282678 tx=702287781 date=‘2022-01-12T21:53:26Z’ progress=0.999969 cache=44.3MiB(327379txo)
bitcoin | 2022-01-13T00:22:06Z UpdateTip: new best=00000000000000000000601484bd5ef681a868a29f7ea7a9c0533197b30ad83a height=718363 version=0x20000004 log2_work=93.282690 tx=702288251 date=‘2022-01-12T21:54:26Z’ progress=0.999969 cache=44.4MiB(328145txo)
bitcoin | 2022-01-13T00:22:07Z UpdateTip: new best=0000000000000000000092fa4e317c507f97cd5ed32743eee68b6b135ff4474a height=718364 version=0x20800000 log2_work=93.282703 tx=702289594 date=‘2022-01-12T22:00:22Z’ progress=0.999971 cache=44.8MiB(331885txo)
bitcoin | 2022-01-13T00:22:09Z UpdateTip: new best=00000000000000000000541ca26ce7a2aa18b7b1db9e2c2655b2976f8f055f80 height=718365 version=0x20006004 log2_work=93.282715 tx=702290621 date=‘2022-01-12T22:07:06Z’ progress=0.999972 cache=45.3MiB(335653txo)
bitcoin | 2022-01-13T00:22:10Z UpdateTip: new best=000000000000000000095a87f47c2f6b611b62dddfc9179830f6f9cb32837a45 height=718366 version=0x20400004 log2_work=93.282728 tx=702292383 date=‘2022-01-12T22:13:28Z’ progress=0.999973 cache=46.0MiB(341377txo)
bitcoin | 2022-01-13T00:22:11Z UpdateTip: new best=0000000000000000000562b43e66758a2e797b3d149b7a72ef33df0d6e4eaa94 height=718367 version=0x2c53e000 log2_work=93.282740 tx=702293490 date=‘2022-01-12T22:18:42Z’ progress=0.999974 cache=46.4MiB(344295txo)
bitcoin | 2022-01-13T00:22:11Z Leaving InitialBlockDownload (latching to false)
bitcoin | 2022-01-13T00:22:11Z Imported mempool transactions from disk: 0 succeeded, 0 failed, 7952 expired, 0 already there, 0 waiting for initial broadcast
bitcoin | 2022-01-13T00:22:11Z loadblk thread exit
bitcoin | 2022-01-13T00:25:24Z Loading addresses from DNS seed seed.bitcoin.jonasschnelli.ch
bitcoin | 2022-01-13T00:25:24Z Loading addresses from DNS seed dnsseed.bluematt.me
bitcoin | 2022-01-13T00:25:24Z Loading addresses from DNS seed seed.bitcoin.sprovoost.nl
bitcoin | 2022-01-13T00:25:24Z Waiting 300 seconds before querying DNS seeds.
bitcoin | 2022-01-20T20:26:43Z Potential stale tip detected, will try using extra outbound peer (last tip update: 677072 seconds ago)
bitcoin | 2022-01-20T20:30:05Z New outbound peer connected: version: 70015, blocks=719673, peer=0 (outbound-full-relay)
bitcoin | 2022-01-20T20:30:58Z UpdateTip: new best=000000000000000000084c2a4adf25d9c4fd29615ed6143acbdba616e888374b height=718368 version=0x20006000 log2_work=93.282753 tx=702294423 date=‘2022-01-12T22:23:12Z’ progress=0.997642 cache=46.7MiB(346635txo)
bitcoin | 2022-01-20T20:30:58Z FlushStateToDisk: write coins cache to disk (346635 coins, 48916kB) started
bitcoin | 2022-01-20T20:31:01Z FlushStateToDisk: write coins cache to disk (346635 coins, 48916kB) completed (2.72s)
bitcoin | 2022-01-20T20:31:26Z UpdateTip: new best=00000000000000000000a3bca8e279a8b246474930aeff61187507f21e48a452 height=718369 version=0x20000004 log2_work=93.282765 tx=702296095 date=‘2022-01-12T22:31:43Z’ progress=0.997644 cache=5.1MiB(9259txo)
bitcoin | 2022-01-20T20:31:28Z Loading addresses from DNS seed seed.bitcoin.sipa.be
bitcoin | 2022-01-20T20:31:28Z Loading addresses from DNS seed dnsseed.bitcoin.dashjr.org
bitcoin | 2022-01-20T20:31:28Z Loading addresses from DNS seed seed.btc.petertodd.org
bitcoin | 2022-01-20T20:31:28Z Waiting 300 seconds before querying DNS seeds.
bitcoin | 2022-01-20T20:31:45Z UpdateTip: new best=000000000000000000043059d232ce44b4fd147d32580dfc8f6d121f7955e7a1 height=718370 version=0x20000004 log2_work=93.282778 tx=702296801 date=‘2022-01-12T22:35:32Z’ progress=0.997645 cache=5.6MiB(12958txo)
bitcoin | 2022-01-20T20:32:06Z UpdateTip: new best=0000000000000000000657bf84cfee7dce02db81d29027f9721d7a17c1e312af height=718371 version=0x27ffe004 log2_work=93.282791 tx=702297654 date=‘2022-01-12T22:39:36Z’ progress=0.997645 cache=6.1MiB(17339txo)
LND logs
Attaching to lnd
lnd | 2022-01-20 20:04:27.702 [INF] WTCL: (anchor) Client stats: tasks(received=0 accepted=5347 ineligible=0) sessions(acquired=5 exhausted=11)
lnd | 2022-01-20 20:05:27.643 [INF] WTCL: (legacy) Client stats: tasks(received=0 accepted=258 ineligible=0) sessions(acquired=0 exhausted=1)
lnd | 2022-01-20 20:05:27.701 [INF] WTCL: (anchor) Client stats: tasks(received=0 accepted=5347 ineligible=0) sessions(acquired=5 exhausted=11)
lnd | 2022-01-20 20:06:27.642 [INF] WTCL: (legacy) Client stats: tasks(received=0 accepted=258 ineligible=0) sessions(acquired=0 exhausted=1)
lnd | 2022-01-20 20:06:27.702 [INF] WTCL: (anchor) Client stats: tasks(received=0 accepted=5347 ineligible=0) sessions(acquired=5 exhausted=11)
lnd | 2022-01-20 20:07:27.642 [INF] WTCL: (legacy) Client stats: tasks(received=0 accepted=258 ineligible=0) sessions(acquired=0 exhausted=1)
lnd | 2022-01-20 20:07:27.702 [INF] WTCL: (anchor) Client stats: tasks(received=0 accepted=5347 ineligible=0) sessions(acquired=5 exhausted=11)
lnd | 2022-01-20 20:08:27.642 [INF] WTCL: (legacy) Client stats: tasks(received=0 accepted=258 ineligible=0) sessions(acquired=0 exhausted=1)
lnd | 2022-01-20 20:08:27.702 [INF] WTCL: (anchor) Client stats: tasks(received=0 accepted=5347 ineligible=0) sessions(acquired=5 exhausted=11)
lnd | 2022-01-20 20:08:39.226 [ERR] WTCL: (anchor) SessionQueue(02d0bb60dfacf626ad7f220a234f56208f77185b47bd99ce8e9da7b9989faabcf1) unable to dial tower at 0301135932e89600b3582513c648d46213dc425c7666e3380faa7dbb51f7e6a3d6@tower4excc3jdaoxcqzbw7gzipoknzqn3dbnw2kfdfhpvvbxagrzmfad.onion:9911: dial proxy failed: socks connect tcp 10.21.21.11:9050->tower4excc3jdaoxcqzbw7gzipoknzqn3dbnw2kfdfhpvvbxagrzmfad.onion:9911: unknown error TTL expired
lnd | 2022-01-20 20:09:27.642 [INF] WTCL: (legacy) Client stats: tasks(received=0 accepted=258 ineligible=0) sessions(acquired=0 exhausted=1)
lnd | 2022-01-20 20:09:27.701 [INF] WTCL: (anchor) Client stats: tasks(received=0 accepted=5347 ineligible=0) sessions(acquired=5 exhausted=11)
lnd | 2022-01-20 20:10:27.642 [INF] WTCL: (legacy) Client stats: tasks(received=0 accepted=258 ineligible=0) sessions(acquired=0 exhausted=1)
lnd | 2022-01-20 20:10:27.702 [INF] WTCL: (anchor) Client stats: tasks(received=0 accepted=5347 ineligible=0) sessions(acquired=5 exhausted=11)
lnd | 2022-01-20 20:11:27.642 [INF] WTCL: (legacy) Client stats: tasks(received=0 accepted=258 ineligible=0) sessions(acquired=0 exhausted=1)
lnd | 2022-01-20 20:11:27.702 [INF] WTCL: (anchor) Client stats: tasks(received=0 accepted=5347 ineligible=0) sessions(acquired=5 exhausted=11)
lnd | 2022-01-20 20:12:27.643 [INF] WTCL: (legacy) Client stats: tasks(received=0 accepted=258 ineligible=0) sessions(acquired=0 exhausted=1)
lnd | 2022-01-20 20:12:27.702 [INF] WTCL: (anchor) Client stats: tasks(received=0 accepted=5347 ineligible=0) sessions(acquired=5 exhausted=11)
lnd | 2022-01-20 20:13:27.642 [INF] WTCL: (legacy) Client stats: tasks(received=0 accepted=258 ineligible=0) sessions(acquired=0 exhausted=1)
lnd | 2022-01-20 20:13:27.701 [INF] WTCL: (anchor) Client stats: tasks(received=0 accepted=5347 ineligible=0) sessions(acquired=5 exhausted=11)
lnd | 2022-01-20 20:14:27.642 [INF] WTCL: (legacy) Client stats: tasks(received=0 accepted=258 ineligible=0) sessions(acquired=0 exhausted=1)
lnd | 2022-01-20 20:14:27.701 [INF] WTCL: (anchor) Client stats: tasks(received=0 accepted=5347 ineligible=0) sessions(acquired=5 exhausted=11)
lnd | 2022-01-20 20:15:27.642 [INF] WTCL: (legacy) Client stats: tasks(received=0 accepted=258 ineligible=0) sessions(acquired=0 exhausted=1)
lnd | 2022-01-20 20:15:27.702 [INF] WTCL: (anchor) Client stats: tasks(received=0 accepted=5347 ineligible=0) sessions(acquired=5 exhausted=11)
lnd | 2022-01-20 20:15:39.974 [ERR] WTCL: (anchor) SessionQueue(02d0bb60dfacf626ad7f220a234f56208f77185b47bd99ce8e9da7b9989faabcf1) unable to dial tower at 0301135932e89600b3582513c648d46213dc425c7666e3380faa7dbb51f7e6a3d6@tower4excc3jdaoxcqzbw7gzipoknzqn3dbnw2kfdfhpvvbxagrzmfad.onion:9911: dial proxy failed: socks connect tcp 10.21.21.11:9050->tower4excc3jdaoxcqzbw7gzipoknzqn3dbnw2kfdfhpvvbxagrzmfad.onion:9911: unknown error TTL expired
lnd | 2022-01-20 20:16:27.642 [INF] WTCL: (legacy) Client stats: tasks(received=0 accepted=258 ineligible=0) sessions(acquired=0 exhausted=1)
lnd | 2022-01-20 20:16:27.701 [INF] WTCL: (anchor) Client stats: tasks(received=0 accepted=5347 ineligible=0) sessions(acquired=5 exhausted=11)
lnd | 2022-01-20 20:17:27.642 [INF] WTCL: (legacy) Client stats: tasks(received=0 accepted=258 ineligible=0) sessions(acquired=0 exhausted=1)
lnd | 2022-01-20 20:17:27.701 [INF] WTCL: (anchor) Client stats: tasks(received=0 accepted=5347 ineligible=0) sessions(acquired=5 exhausted=11)
lnd | failed to load config: ValidateConfig: error parsing tor dns: lookup nodes.lightning.directory on 127.0.0.11:53: read udp 127.0.0.1:39278->127.0.0.11:53: i/o timeout
electrs logs
Attaching to electrs
electrs | [2022-01-13T00:20:02.741Z INFO electrs::daemon] waiting for RPC warmup: Verifying blocks…
electrs | [2022-01-13T00:20:03.744Z INFO electrs::daemon] waiting for RPC warmup: Verifying blocks…
electrs | [2022-01-13T00:20:04.752Z INFO electrs::daemon] waiting for RPC warmup: Verifying blocks…
electrs | [2022-01-13T00:20:05.755Z INFO electrs::daemon] waiting for RPC warmup: Verifying blocks…
electrs | [2022-01-13T00:20:06.757Z INFO electrs::daemon] waiting for RPC warmup: Verifying blocks…
electrs | [2022-01-13T00:20:07.760Z INFO electrs::daemon] waiting for RPC warmup: Verifying blocks…
electrs | [2022-01-13T00:20:08.762Z INFO electrs::daemon] waiting for RPC warmup: Verifying blocks…
electrs | [2022-01-13T00:20:09.765Z INFO electrs::daemon] waiting for RPC warmup: Verifying blocks…
electrs | [2022-01-13T00:20:10.767Z INFO electrs::daemon] waiting for RPC warmup: Verifying blocks…
electrs | [2022-01-13T00:20:11.770Z INFO electrs::daemon] waiting for RPC warmup: Verifying blocks…
electrs | [2022-01-13T00:20:12.772Z INFO electrs::daemon] waiting for RPC warmup: Verifying blocks…
electrs | [2022-01-13T00:20:13.775Z INFO electrs::daemon] waiting for RPC warmup: Verifying blocks…
electrs | [2022-01-13T00:20:14.777Z INFO electrs::daemon] waiting for RPC warmup: Verifying blocks…
electrs | [2022-01-13T00:20:15.780Z INFO electrs::daemon] waiting for RPC warmup: Verifying blocks…
electrs | [2022-01-13T00:20:16.785Z INFO electrs::daemon] waiting for RPC warmup: Verifying blocks…
electrs | [2022-01-13T00:20:17.787Z INFO electrs::daemon] waiting for RPC warmup: Verifying blocks…
electrs | [2022-01-13T00:20:18.793Z INFO electrs::daemon] waiting for RPC warmup: Verifying blocks…
electrs | [2022-01-13T00:20:19.796Z INFO electrs::daemon] waiting for RPC warmup: Verifying blocks…
electrs | [2022-01-13T00:20:20.798Z INFO electrs::daemon] waiting for RPC warmup: Verifying blocks…
electrs | [2022-01-13T00:20:21.801Z INFO electrs::daemon] waiting for RPC warmup: Verifying blocks…
electrs | [2022-01-13T00:20:22.803Z INFO electrs::daemon] waiting for RPC warmup: Verifying blocks…
electrs | [2022-01-13T00:20:23.805Z INFO electrs::daemon] waiting for RPC warmup: Loading P2P addresses…
electrs | [2022-01-13T00:20:27.703Z INFO electrs::daemon] waiting for 44 blocks to download (IBD)
electrs | [2022-01-13T00:20:50.377Z INFO electrs::db] closing DB at /data/db/bitcoin
electrs | Error: electrs failed
electrs |
electrs | Caused by:
electrs | 0: bitcoind RPC polling failed
electrs | 1: daemon not available
electrs | 2: JSON-RPC error: transport error: Didn’t receive response data in time, timed out.
Tor logs
Attaching to umbrel_app_2_tor_1, umbrel_app_tor_1, tor, umbrel_app_3_tor_1
app_2_tor_1 | Jan 13 01:09:14.000 [warn] Received NETINFO cell with skewed time (OR:204.13.164.118:443): It seems that our clock is behind by 7 days, 20 hours, 16 minutes, or that theirs is ahead. Tor requires an accurate clock to work: please check your time, timezone, and date settings.
app_2_tor_1 | Jan 13 01:09:14.000 [warn] Problem bootstrapping. Stuck at 30% (loading_status): Loading networkstatus consensus. (Clock skew -677809 in NETINFO cell from OR; CLOCK_SKEW; count 82; recommendation warn; host 24E2F139121D4394C54B5BCC368B3B411857C413 at 204.13.164.118:443)
app_2_tor_1 | Jan 13 01:09:14.000 [warn] 42 connections have failed:
app_2_tor_1 | Jan 13 01:09:14.000 [warn] 38 connections died in state connect()ing with SSL state (No SSL object)
app_2_tor_1 | Jan 13 01:09:14.000 [warn] 4 connections died in state handshaking (TLS) with SSL state error in HANDSHAKE
app_2_tor_1 | Jan 13 01:09:14.000 [warn] Received directory with skewed time (DIRSERV:204.13.164.118:443): It seems that our clock is behind by 7 days, 20 hours, 16 minutes, or that theirs is ahead. Tor requires an accurate clock to work: please check your time, timezone, and date settings.
app_2_tor_1 | Jan 13 01:09:14.000 [warn] Problem bootstrapping. Stuck at 30% (loading_status): Loading networkstatus consensus. (Clock skew -677809 in directory from DIRSERV; CLOCK_SKEW; count 83; recommendation warn; host ? at 204.13.164.118:443)
app_2_tor_1 | Jan 13 01:09:14.000 [warn] 42 connections have failed:
app_2_tor_1 | Jan 13 01:09:14.000 [warn] 38 connections died in state connect()ing with SSL state (No SSL object)
app_2_tor_1 | Jan 13 01:09:14.000 [warn] 4 connections died in state handshaking (TLS) with SSL state error in HANDSHAKE
tor | Jan 14 04:40:21.000 [warn] 107 connections have failed:
tor | Jan 14 04:40:21.000 [warn] 103 connections died in state handshaking (TLS) with SSL state error in HANDSHAKE
tor | Jan 14 04:40:21.000 [warn] 4 connections died in state connect()ing with SSL state (No SSL object)
tor | Jan 14 04:40:21.000 [warn] Our clock is 7 days, 20 hours, 19 minutes behind the time published in the consensus network status document (2022-01-22 01:00:00 UTC). Tor needs an accurate clock to work correctly. Please check your time and date settings!
tor | Jan 14 04:40:21.000 [warn] Received microdesc flavor consensus with skewed time (CONSENSUS): It seems that our clock is behind by 7 days, 20 hours, 19 minutes, or that theirs is ahead. Tor requires an accurate clock to work: please check your time, timezone, and date settings.
tor | Jan 14 04:40:21.000 [warn] Problem bootstrapping. Stuck at 30% (loading_status): Loading networkstatus consensus. (Clock skew -677979 in microdesc flavor consensus from CONSENSUS; CLOCK_SKEW; count 1074; recommendation warn; host ? at ?)
tor | Jan 14 04:40:21.000 [warn] 107 connections have failed:
tor | Jan 14 04:40:21.000 [warn] 103 connections died in state handshaking (TLS) with SSL state error in HANDSHAKE
tor | Jan 14 04:40:21.000 [warn] 4 connections died in state connect()ing with SSL state (No SSL object)
tor | Jan 14 04:40:21.000 [notice] I learned some more directory information, but not enough to build a circuit: We have no recent usable consensus.
app_3_tor_1 | Jan 14 01:17:42.000 [notice] Heartbeat: Tor’s uptime is 1 day 0:00 hours, with 0 circuits open. I’ve sent 199 kB and received 1010 kB. I’ve received 0 connections on IPv4 and 0 on IPv6. I’ve made 63 connections with IPv4 and 0 with IPv6.
app_3_tor_1 | Jan 14 01:17:42.000 [notice] While bootstrapping, fetched this many bytes: 596835 (consensus network-status fetch)
app_3_tor_1 | Jan 14 01:17:42.000 [notice] Average packaged cell fullness: 67.403%. TLS write overhead: 18%
app_3_tor_1 | Jan 14 01:17:42.000 [notice] Our onion services received 0 v2 and 0 v3 INTRODUCE2 cells and attempted to launch 0 rendezvous circuits.
app_3_tor_1 | Jan 14 04:06:55.000 [warn] Our clock is 7 days, 19 hours, 53 minutes behind the time published in the consensus network status document (2022-01-22 00:00:00 UTC). Tor needs an accurate clock to work correctly. Please check your time and date settings!
app_3_tor_1 | Jan 14 04:06:55.000 [warn] Received microdesc flavor consensus with skewed time (CONSENSUS): It seems that our clock is behind by 7 days, 19 hours, 53 minutes, or that theirs is ahead. Tor requires an accurate clock to work: please check your time, timezone, and date settings.
app_3_tor_1 | Jan 14 04:06:55.000 [warn] Problem bootstrapping. Stuck at 30% (loading_status): Loading networkstatus consensus. (Clock skew -676385 in microdesc flavor consensus from CONSENSUS; CLOCK_SKEW; count 102; recommendation warn; host ? at ?)
app_3_tor_1 | Jan 14 04:06:55.000 [warn] 5 connections have failed:
app_3_tor_1 | Jan 14 04:06:55.000 [warn] 5 connections died in state handshaking (TLS) with SSL state error in HANDSHAKE
app_3_tor_1 | Jan 14 04:06:55.000 [notice] I learned some more directory information, but not enough to build a circuit: We have no recent usable consensus.
app_tor_1 | Jan 14 01:07:55.000 [warn] Our clock is 7 days, 19 hours, 52 minutes behind the time published in the consensus network status document (2022-01-21 21:00:00 UTC). Tor needs an accurate clock to work correctly. Please check your time and date settings!
app_tor_1 | Jan 14 01:07:55.000 [warn] Received microdesc flavor consensus with skewed time (CONSENSUS): It seems that our clock is behind by 7 days, 19 hours, 52 minutes, or that theirs is ahead. Tor requires an accurate clock to work: please check your time, timezone, and date settings.
app_tor_1 | Jan 14 01:07:55.000 [warn] Problem bootstrapping. Stuck at 30% (loading_status): Loading networkstatus consensus. (Clock skew -676325 in microdesc flavor consensus from CONSENSUS; CLOCK_SKEW; count 65; recommendation warn; host ? at ?)
app_tor_1 | Jan 14 01:07:55.000 [warn] 3 connections have failed:
app_tor_1 | Jan 14 01:07:55.000 [warn] 3 connections died in state handshaking (TLS) with SSL state error in HANDSHAKE
app_tor_1 | Jan 14 01:07:55.000 [notice] I learned some more directory information, but not enough to build a circuit: We have no recent usable consensus.
app_tor_1 | Jan 14 01:17:42.000 [notice] Heartbeat: Tor’s uptime is 1 day 0:00 hours, with 0 circuits open. I’ve sent 158 kB and received 1.48 MB. I’ve received 0 connections on IPv4 and 0 on IPv6. I’ve made 42 connections with IPv4 and 0 with IPv6.
app_tor_1 | Jan 14 01:17:42.000 [notice] While bootstrapping, fetched this many bytes: 1203184 (consensus network-status fetch)
app_tor_1 | Jan 14 01:17:42.000 [notice] Average packaged cell fullness: 67.394%. TLS write overhead: 16%
app_tor_1 | Jan 14 01:17:42.000 [notice] Our onion services received 0 v2 and 0 v3 INTRODUCE2 cells and attempted to launch 0 rendezvous circuits.
App logs
bluewallet
lndhub_1 | https://github.com/browserslist/browserslist#browsers-data-updating
lndhub_1 | using config from env
lndhub_1 | using config {“rateLimit”:10000,“postRateLimit”:10000,“redis”:{“port”:6379,“host”:“10.21.21.31”,“family”:4,“password”:“moneyprintergobrrr”,“db”:0},“lnd”:{“url”:“10.21.21.9:10009”,“password”:""}}
lndhub_1 | updateDescribeGraph()
lndhub_1 | updateLightning()
lndhub_1 | 2022-01-13T00:19:46.435Z : info: [BOOTING UP] : “Listening on port 3008”
lndhub_1 | (node:45) Warning: Setting the NODE_TLS_REJECT_UNAUTHORIZED environment variable to ‘0’ makes TLS connections and HTTPS requests insecure by disabling certificate verification.
lndhub_1 | (node:45) [DEP0123] DeprecationWarning: Setting the TLS ServerName to an IP address is not permitted by RFC 6066. This will be ignored in a future version.
lndhub_1 | lnd failure
lndhub_1 | Error: 14 UNAVAILABLE: No connection established
lndhub_1 | at Object.callErrorFromStatus (/lndhub/node_modules/@grpc/grpc-js/src/call.ts:81:24)
lndhub_1 | at Object.onReceiveStatus (/lndhub/node_modules/@grpc/grpc-js/src/client.ts:334:36)
lndhub_1 | at Object.onReceiveStatus (/lndhub/node_modules/@grpc/grpc-js/src/client-interceptors.ts:426:34)
lndhub_1 | at Object.onReceiveStatus (/lndhub/node_modules/@grpc/grpc-js/src/client-interceptors.ts:389:48)
lndhub_1 | at /lndhub/node_modules/@grpc/grpc-js/src/call-stream.ts:276:24
lndhub_1 | at processTicksAndRejections (internal/process/task_queues.js:79:11) {
lndhub_1 | code: 14,
lndhub_1 | details: ‘No connection established’,
lndhub_1 | metadata: Metadata { internalRepr: Map {}, options: {} }
lndhub_1 | }
lndhub_1 | npm ERR! code ELIFECYCLE
lndhub_1 | npm ERR! errno 3
lndhub_1 | npm ERR! [email protected] start: babel-node index.js
lndhub_1 | npm ERR! Exit status 3
lndhub_1 | npm ERR!
lndhub_1 | npm ERR! Failed at the [email protected] start script.
lndhub_1 | npm ERR! This is probably not a problem with npm. There is likely additional logging output above.
lndhub_1 |
lndhub_1 | npm ERR! A complete log of this run can be found in:
lndhub_1 | npm ERR! /lndhub/.npm/_logs/2022-01-13T00_19_50_048Z-debug.log
btc-rpc-explorer
web_1 | 2022-01-14T05:02:59.585Z btcexp:app Verifying RPC connection…
web_1 | 2022-01-14T05:03:02.658Z btcexp:error Error RpcError-001: Error: connect EHOSTUNREACH 10.21.21.8:8332, json: {“errno”:“EHOSTUNREACH”,“code”:“EHOSTUNREACH”,“syscall”:“connect”,“address”:“10.21.21.8”,“port”:8332,“userData”:{“request”:“getnetworkinfo”}}, userData: [object Object] (json: {“request”:“getnetworkinfo”})
web_1 | 2022-01-14T05:03:02.658Z btcexp:errorVerbose Stack: Error: connect EHOSTUNREACH 10.21.21.8:8332
web_1 | at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1144:16)
web_1 | 2022-01-14T05:03:02.659Z btcexp:error Error 32ugegdfsde: Error: connect EHOSTUNREACH 10.21.21.8:8332, json: {“errno”:“EHOSTUNREACH”,“code”:“EHOSTUNREACH”,“syscall”:“connect”,“address”:“10.21.21.8”,“port”:8332,“userData”:{“request”:“getnetworkinfo”}}, userData: [object Object] (json: {})
web_1 | 2022-01-14T05:03:02.659Z btcexp:errorVerbose Stack: Error: connect EHOSTUNREACH 10.21.21.8:8332
web_1 | at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1144:16)
web_1 | 2022-01-14T05:03:02.659Z btcexp:error Error RpcError-001: Error: connect EHOSTUNREACH 10.21.21.8:8332, json: {“errno”:“EHOSTUNREACH”,“code”:“EHOSTUNREACH”,“syscall”:“connect”,“address”:“10.21.21.8”,“port”:8332,“userData”:{“request”:“getblockchaininfo”}}, userData: [object Object] (json: {“request”:“getblockchaininfo”})
web_1 | 2022-01-14T05:03:02.659Z btcexp:errorVerbose Stack: Error: connect EHOSTUNREACH 10.21.21.8:8332
web_1 | at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1144:16)
web_1 | 2022-01-14T05:03:29.589Z btcexp:app Verifying RPC connection…
web_1 | 2022-01-14T05:03:32.642Z btcexp:error Error RpcError-001: Error: connect EHOSTUNREACH 10.21.21.8:8332, json: {“errno”:“EHOSTUNREACH”,“code”:“EHOSTUNREACH”,“syscall”:“connect”,“address”:“10.21.21.8”,“port”:8332,“userData”:{“request”:“getnetworkinfo”}}, userData: [object Object] (json: {“request”:“getnetworkinfo”})
web_1 | 2022-01-14T05:03:32.643Z btcexp:errorVerbose Stack: Error: connect EHOSTUNREACH 10.21.21.8:8332
web_1 | at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1144:16)
web_1 | 2022-01-14T05:03:32.643Z btcexp:error Error 32ugegdfsde: Error: connect EHOSTUNREACH 10.21.21.8:8332, json: {“errno”:“EHOSTUNREACH”,“code”:“EHOSTUNREACH”,“syscall”:“connect”,“address”:“10.21.21.8”,“port”:8332,“userData”:{“request”:“getnetworkinfo”}}, userData: [object Object] (json: {})
web_1 | 2022-01-14T05:03:32.643Z btcexp:errorVerbose Stack: Error: connect EHOSTUNREACH 10.21.21.8:8332
web_1 | at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1144:16)
web_1 | 2022-01-14T05:03:32.644Z btcexp:error Error RpcError-001: Error: connect EHOSTUNREACH 10.21.21.8:8332, json: {“errno”:“EHOSTUNREACH”,“code”:“EHOSTUNREACH”,“syscall”:“connect”,“address”:“10.21.21.8”,“port”:8332,“userData”:{“request”:“getblockchaininfo”}}, userData: [object Object] (json: {“request”:“getblockchaininfo”})
web_1 | 2022-01-14T05:03:32.644Z btcexp:errorVerbose Stack: Error: connect EHOSTUNREACH 10.21.21.8:8332
web_1 | at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1144:16)
web_1 | 2022-01-14T05:03:59.594Z btcexp:app Verifying RPC connection…
web_1 | 2022-01-14T05:04:02.659Z btcexp:error Error RpcError-001: Error: connect EHOSTUNREACH 10.21.21.8:8332, json: {“errno”:“EHOSTUNREACH”,“code”:“EHOSTUNREACH”,“syscall”:“connect”,“address”:“10.21.21.8”,“port”:8332,“userData”:{“request”:“getnetworkinfo”}}, userData: [object Object] (json: {“request”:“getnetworkinfo”})
web_1 | 2022-01-14T05:04:02.659Z btcexp:errorVerbose Stack: Error: connect EHOSTUNREACH 10.21.21.8:8332
web_1 | at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1144:16)
web_1 | 2022-01-14T05:04:02.660Z btcexp:error Error 32ugegdfsde: Error: connect EHOSTUNREACH 10.21.21.8:8332, json: {“errno”:“EHOSTUNREACH”,“code”:“EHOSTUNREACH”,“syscall”:“connect”,“address”:“10.21.21.8”,“port”:8332,“userData”:{“request”:“getnetworkinfo”}}, userData: [object Object] (json: {})
web_1 | 2022-01-14T05:04:02.660Z btcexp:errorVerbose Stack: Error: connect EHOSTUNREACH 10.21.21.8:8332
web_1 | at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1144:16)
web_1 | 2022-01-14T05:04:02.660Z btcexp:error Error RpcError-001: Error: connect EHOSTUNREACH 10.21.21.8:8332, json: {“errno”:“EHOSTUNREACH”,“code”:“EHOSTUNREACH”,“syscall”:“connect”,“address”:“10.21.21.8”,“port”:8332,“userData”:{“request”:“getblockchaininfo”}}, userData: [object Object] (json: {“request”:“getblockchaininfo”})
web_1 | 2022-01-14T05:04:02.661Z btcexp:errorVerbose Stack: Error: connect EHOSTUNREACH 10.21.21.8:8332
web_1 | at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1144:16)
lightning-terminal
web_1 | 2022-01-20 20:15:07.205 [WRN] GRPC: [core] grpc: addrConn.createTransport failed to connect to {mailbox.terminal.lightning.today:443 mailbox.terminal.lightning.today:443 0 }. Err: connection error: desc = “transport: Error while dialing dial tcp: lookup mailbox.terminal.lightning.today on 127.0.0.11:53: read udp 127.0.0.1:56860->127.0.0.11:53: i/o timeout”. Reconnecting…
web_1 | 2022-01-20 20:16:18.981 [WRN] GRPC: [core] grpc: addrConn.createTransport failed to connect to {pool.lightning.finance:12010 pool.lightning.finance:12010 0 }. Err: connection error: desc = “transport: Error while dialing dial tcp: lookup pool.lightning.finance on 127.0.0.11:53: read udp 127.0.0.1:59088->127.0.0.11:53: i/o timeout”. Reconnecting…
web_1 | 2022-01-20 20:16:49.071 [WRN] GRPC: [core] grpc: addrConn.createTransport failed to connect to {pool.lightning.finance:12010 pool.lightning.finance:12010 0 }. Err: connection error: desc = “transport: Error while dialing dial tcp: lookup pool.lightning.finance on 127.0.0.11:53: read udp 127.0.0.1:51396->127.0.0.11:53: i/o timeout”. Reconnecting…
web_1 | 2022-01-20 20:16:58.369 [WRN] GRPC: [core] grpc: addrConn.createTransport failed to connect to {mailbox.terminal.lightning.today:443 mailbox.terminal.lightning.today:443 0 }. Err: connection error: desc = “transport: Error while dialing dial tcp: lookup mailbox.terminal.lightning.today on 127.0.0.11:53: read udp 127.0.0.1:56107->127.0.0.11:53: i/o timeout”. Reconnecting…
web_1 | 2022-01-20 20:17:14.770 [WRN] GRPC: [core] grpc: addrConn.createTransport failed to connect to {swap.lightning.today:11010 swap.lightning.today:11010 0 }. Err: connection error: desc = “transport: Error while dialing dial tcp: lookup swap.lightning.today on 127.0.0.11:53: read udp 127.0.0.1:48034->127.0.0.11:53: i/o timeout”. Reconnecting…
web_1 | 2022-01-13 00:19:18.240 [INF] SESS: Checking for schema update: latest_version=0, db_version=0
web_1 | 2022-01-13 00:19:18.243 [INF] LITD: Dialing lnd gRPC server at 10.21.21.9:10009
web_1 | 2022-01-13 00:19:18.243 [INF] SESS: Mailbox RPC server listening on mailbox:c714e9944451f8040d5b2003bf05d55cdb88faf87ad107aa682ebdf0bac5d4c60815b78ced1b822585221d45767d5bc9fb1f5040df03910448e8ef46358053d0@mailbox.terminal.lightning.today:443
web_1 | 2022-01-13 00:19:18.253 [WRN] GRPC: [core] grpc: addrConn.createTransport failed to connect to {10.21.21.9:10009 10.21.21.9:10009 0 }. Err: connection error: desc = “transport: Error while dialing dial tcp 10.21.21.9:10009: connect: connection refused”. Reconnecting…
web_1 | 2022-01-13 00:19:18.484 [INF] LITD: Listening for http_tls on: 127.0.0.1:8443
web_1 | 2022-01-13 00:19:18.486 [INF] LITD: Listening for http on: [::]:3004
web_1 | 2022-01-13 00:19:18.490 [WRN] GRPC: [core] grpc: addrConn.createTransport failed to connect to {10.21.21.9:10009 10.21.21.9:10009 0 }. Err: connection error: desc = “transport: Error while dialing dial tcp 10.21.21.9:10009: connect: connection refused”. Reconnecting…
web_1 | ----------------------------------------------------------
web_1 | Lightning Terminal (LiT) by Lightning Labs
web_1 |
web_1 | Operating mode remote
web_1 | Node status locked
web_1 | Alias ??? (node is locked)
web_1 | Version ??? (node is locked)
web_1 | Web interface 127.0.0.1:8443, 0.0.0.0:3004 (open https://127.0.0.1:8443 or http://localhost:3004 in your browser)
web_1 | ----------------------------------------------------------
web_1 | 2022-01-13 00:19:18.697 [INF] LNDC: Creating lnd connection to 10.21.21.9:10009
web_1 | 2022-01-13 00:19:18.700 [WRN] GRPC: [core] grpc: addrConn.createTransport failed to connect to {10.21.21.9:10009 10.21.21.9:10009 0 }. Err: connection error: desc = “transport: Error while dialing dial tcp 10.21.21.9:10009: connect: connection refused”. Reconnecting…
web_1 | 2022-01-13 00:19:18.701 [INF] LNDC: Connected to lnd
web_1 | 2022-01-13 00:19:18.702 [WRN] GRPC: [core] grpc: addrConn.createTransport failed to connect to {10.21.21.9:10009 10.21.21.9:10009 0 }. Err: connection error: desc = “transport: Error while dialing dial tcp 10.21.21.9:10009: connect: connection refused”. Reconnecting…
web_1 | 2022-01-13 00:19:18.704 [ERR] LITD: Could not start subservers: error subscribing to lnd wallet state: lnd version incompatible, need at least v0.13.0-beta, got error on state subscription: rpc error: code = Unavailable desc = connection error: desc = “transport: Error while dialing dial tcp 10.21.21.9:10009: connect: connection refused”
web_1 | 2022-01-13 00:19:18.705 [INF] SGNL: Received shutdown request.
web_1 | 2022-01-13 00:19:18.705 [INF] SGNL: Shutting down…
web_1 | 2022-01-13 00:19:18.705 [INF] SGNL: Gracefully shutting down.
web_1 | error subscribing to lnd wallet state: lnd version incompatible, need at least v0.13.0-beta, got error on state subscription: rpc error: code = Unavailable desc = connection error: desc = “transport: Error while dialing dial tcp 10.21.21.9:10009: connect: connection refused”
lnbits
Attaching to lnbits_web_1
web_1 | running migration tipjar.1
web_1 | /app/lnbits/app.py:60: RuntimeWarning: × The backend for LndWallet isn’t working properly: ‘<_InactiveRpcError of RPC that terminated with:
web_1 | status = StatusCode.UNKNOWN
web_1 | details = “wallet locked, unlock it to enable full RPC access”
web_1 | debug_error_string = “{“created”:”@1639740837.447188909",“description”:“Error received from peer ipv4:10.21.21.9:10009”,“file”:“src/core/lib/surface/call.cc”,“file_line”:1063,“grpc_message”:“wallet locked, unlock it to enable full RPC access”,“grpc_status”:2}"
web_1 | >’
web_1 | RuntimeWarning,
web_1 | [2021-12-17 11:33:57 +0000] [42] [INFO] Running on http://0.0.0.0:3007 (CTRL + C to quit)
web_1 | [2021-12-17 11:34:33 +0000] [41] [INFO] Running on http://0.0.0.0:3007 (CTRL + C to quit)
mempool
mariadb_1 | 2022-01-13 1:17:44 0 [Note] Server socket created on IP: ‘::’.
mariadb_1 | 2022-01-13 1:17:44 0 [Warning] ‘proxies_priv’ entry ‘@% root@f19e8c3af4eb’ ignored in --skip-name-resolve mode.
mariadb_1 | 2022-01-13 1:17:44 0 [Note] Reading of all Master_info entries succeeded
mariadb_1 | 2022-01-13 1:17:44 0 [Note] Added new Master_info ‘’ to hash table
mariadb_1 | 2022-01-13 1:17:44 0 [Note] mysqld: ready for connections.
mariadb_1 | Version: ‘10.5.12-MariaDB-1:10.5.12+maria~focal’ socket: ‘/run/mysqld/mysqld.sock’ port: 3306 mariadb.org binary distribution
mariadb_1 | 2022-01-13 1:17:44 0 [Note] InnoDB: Buffer pool(s) load completed at 220113 1:17:44
mariadb_1 | 2022-01-13 1:17:44 3 [Warning] Aborted connection 3 to db: ‘unconnected’ user: ‘unauthenticated’ host: ‘10.21.21.27’ (This connection closed normally without authentication)
api_1 | Jan 14 04:56:43 [51] DEBUG: updateMempoolInfo: connect EHOSTUNREACH 10.21.21.8:8332
api_1 | Jan 14 04:56:46 [51] WARN: runMainLoop error: connect EHOSTUNREACH 10.21.21.8:8332. Retrying in 60 sec.
api_1 | Jan 14 04:56:46 [51] DEBUG: {“errno”:“EHOSTUNREACH”,“code”:“EHOSTUNREACH”,“syscall”:“connect”,“address”:“10.21.21.8”,“port”:8332}
api_1 | Jan 14 04:57:49 [51] DEBUG: updateMempoolInfo: connect EHOSTUNREACH 10.21.21.8:8332
api_1 | Jan 14 04:57:52 [51] DEBUG: {“errno”:“EHOSTUNREACH”,“code”:“EHOSTUNREACH”,“syscall”:“connect”,“address”:“10.21.21.8”,“port”:8332}
api_1 | Jan 14 04:57:52 [51] WARN: runMainLoop error: connect EHOSTUNREACH 10.21.21.8:8332. Retrying in 60 sec.
api_1 | Jan 14 04:58:36 [51] DEBUG: Statistics cache created
api_1 | Jan 14 04:58:55 [51] DEBUG: updateMempoolInfo: connect EHOSTUNREACH 10.21.21.8:8332
api_1 | Jan 14 04:58:58 [51] WARN: runMainLoop error: connect EHOSTUNREACH 10.21.21.8:8332. Retrying in 60 sec.
api_1 | Jan 14 04:58:58 [51] DEBUG: {“errno”:“EHOSTUNREACH”,“code”:“EHOSTUNREACH”,“syscall”:“connect”,“address”:“10.21.21.8”,“port”:8332}
api_1 | Jan 14 05:00:01 [51] DEBUG: updateMempoolInfo: connect EHOSTUNREACH 10.21.21.8:8332
api_1 | Jan 14 05:00:04 [51] WARN: runMainLoop error: connect EHOSTUNREACH 10.21.21.8:8332. Retrying in 60 sec.
api_1 | Jan 14 05:00:04 [51] DEBUG: {“errno”:“EHOSTUNREACH”,“code”:“EHOSTUNREACH”,“syscall”:“connect”,“address”:“10.21.21.8”,“port”:8332}
api_1 | Jan 14 05:01:08 [51] DEBUG: updateMempoolInfo: connect EHOSTUNREACH 10.21.21.8:8332
api_1 | Jan 14 05:01:11 [51] DEBUG: {“errno”:“EHOSTUNREACH”,“code”:“EHOSTUNREACH”,“syscall”:“connect”,“address”:“10.21.21.8”,“port”:8332}
api_1 | Jan 14 05:01:11 [51] WARN: runMainLoop error: connect EHOSTUNREACH 10.21.21.8:8332. Retrying in 60 sec.
api_1 | Jan 14 05:02:14 [51] DEBUG: updateMempoolInfo: connect EHOSTUNREACH 10.21.21.8:8332
api_1 | Jan 14 05:02:17 [51] WARN: runMainLoop error: connect EHOSTUNREACH 10.21.21.8:8332. Retrying in 60 sec.
api_1 | Jan 14 05:02:17 [51] DEBUG: {“errno”:“EHOSTUNREACH”,“code”:“EHOSTUNREACH”,“syscall”:“connect”,“address”:“10.21.21.8”,“port”:8332}
api_1 | Jan 14 05:03:20 [51] DEBUG: updateMempoolInfo: connect EHOSTUNREACH 10.21.21.8:8332
api_1 | Jan 14 05:03:23 [51] WARN: runMainLoop error: connect EHOSTUNREACH 10.21.21.8:8332. Retrying in 60 sec.
api_1 | Jan 14 05:03:23 [51] DEBUG: {“errno”:“EHOSTUNREACH”,“code”:“EHOSTUNREACH”,“syscall”:“connect”,“address”:“10.21.21.8”,“port”:8332}
ride-the-lightning
web_1 |
web_1 | [2022-01-13T01:38:58.364Z] INFO: RTLConf => Initial RTL Configuration Received.
web_1 |
web_1 | [2022-01-13T01:38:58.581Z] INFO: RTLConf => Updating Selected Node…
web_1 |
loop_1 | 2022-01-13 00:18:48.840 [INF] LNDC: Creating lnd connection to 10.21.21.9:10009
loop_1 | 2022-01-13 00:18:50.092 [INF] LNDC: Connected to lnd
loop_1 | 2022-01-13 00:18:50.858 [INF] LNDC: Waiting for lnd to unlock
loop_1 | loopd exited with an error: rpc error: code = DeadlineExceeded desc = latest connection error: connection error: desc = “transport: Error while dialing dial tcp 10.21.21.9:10009: connect: connection refused”
web_1 | [2022-01-13T01:38:58.584Z] INFO: RTLConf => Selected Node Updated.
web_1 |
web_1 | [2022-01-13T01:39:11.439Z] INFO: Authenticate => Authenticating User…
web_1 |
web_1 | [2022-01-13T01:39:11.441Z] ERROR: Authenticate => Invalid Password! Failed IP ::ffff:192.168.86.43: {“error”:“Invalid password.”}
web_1 |
web_1 | [2022-01-13T01:39:23.334Z] INFO: Authenticate => Authenticating User…
web_1 |
web_1 | [2022-01-13T01:39:23.338Z] INFO: Authenticate => User Authenticated.
web_1 |
web_1 | [2022-01-13T01:39:23.352Z] INFO: RTLConf => Getting RTL Configuration…
web_1 |
web_1 | [2022-01-13T01:39:23.354Z] INFO: RTLConf => RTL Configuration Received.
web_1 |
web_1 | [2022-01-13T01:39:23.381Z] INFO: RTLConf => Updating Selected Node…
web_1 |
web_1 | [2022-01-13T01:39:23.381Z] INFO: RTLConf => Selected Node Updated.
web_1 |
web_1 | [2022-01-13T01:39:23.491Z] INFO: GetInfo => Getting LND Node Information…
web_1 |
web_1 | [2022-01-13T01:39:26.610Z] ERROR: GetInfo => Get Info Error: {“name”:“RequestError”,“message”:“Error: connect EHOSTUNREACH 10.21.21.9:8080”,“cause”:{“errno”:-113,“code”:“EHOSTUNREACH”,“syscall”:“connect”,“address”:“10.21.21.9”,“port”:8080},“error”:{“errno”:-113,“code”:“EHOSTUNREACH”,“syscall”:“connect”,“address”:“10.21.21.9”,“port”:8080},“options”:{“url”:“https://10.21.21.9:8080/v1/getinfo",“rejectUnauthorized”:false,“json”:true,“headers”:{},“method”:“GET”,“qs”:{},“simple”:true,“resolveWithFullResponse”:false,"transform2xxOnly”:false}}
sphinx-relay
sphinx-relay | => 2022-01-14 05:03:41 [lightning] reconnecting… attempt #32530
sphinx-relay | [lightning] ERROR Error: 14 UNAVAILABLE: failed to connect to all addresses
sphinx-relay | => 2022-01-14 05:03:44 [lightning] reconnecting… attempt #32531
sphinx-relay | [lightning] ERROR Error: 14 UNAVAILABLE: failed to connect to all addresses
sphinx-relay | => 2022-01-14 05:03:47 [lightning] reconnecting… attempt #32532
sphinx-relay | [lightning] [ ‘listChannels’ ]
sphinx-relay | [lightning] ERROR Error: 14 UNAVAILABLE: failed to connect to all addresses
sphinx-relay | => 2022-01-14 05:03:50 [lightning] reconnecting… attempt #32533
sphinx-relay | [lightning] ERROR Error: 14 UNAVAILABLE: failed to connect to all addresses
sphinx-relay | => 2022-01-14 05:03:54 [lightning] reconnecting… attempt #32534
sphinx-relay | [lightning] ERROR Error: 14 UNAVAILABLE: failed to connect to all addresses
sphinx-relay | => 2022-01-14 05:03:57 [lightning] reconnecting… attempt #32535
sphinx-relay | [lightning] ERROR Error: 14 UNAVAILABLE: failed to connect to all addresses
sphinx-relay | => 2022-01-14 05:04:00 [lightning] reconnecting… attempt #32536
sphinx-relay | [lightning] ERROR Error: 14 UNAVAILABLE: failed to connect to all addresses
sphinx-relay | => 2022-01-14 05:04:03 [lightning] reconnecting… attempt #32537
sphinx-relay | [lightning] ERROR Error: 14 UNAVAILABLE: failed to connect to all addresses
sphinx-relay | => 2022-01-14 05:04:06 [lightning] reconnecting… attempt #32538
sphinx-relay | [lightning] ERROR Error: 14 UNAVAILABLE: failed to connect to all addresses
sphinx-relay | => 2022-01-14 05:04:09 [lightning] reconnecting… attempt #32539
sphinx-relay | [lightning] ERROR Error: 14 UNAVAILABLE: failed to connect to all addresses
sphinx-relay | => 2022-01-14 05:04:12 [lightning] reconnecting… attempt #32540
sphinx-relay | [lightning] ERROR Error: 14 UNAVAILABLE: failed to connect to all addresses
sphinx-relay | => 2022-01-14 05:04:15 [lightning] reconnecting… attempt #32541
sphinx-relay | [lightning] ERROR Error: 14 UNAVAILABLE: failed to connect to all addresses
sphinx-relay | [lightning] [ ‘listChannels’ ]
sphinx-relay | => 2022-01-14 05:04:18 [lightning] reconnecting… attempt #32542
sphinx-relay | [lightning] ERROR Error: 14 UNAVAILABLE: failed to connect to all addresses
sphinx-relay | => 2022-01-14 05:04:21 [lightning] reconnecting… attempt #32543
sphinx-relay | [lightning] ERROR Error: 14 UNAVAILABLE: failed to connect to all addresses
Full logs (final)
thunderhub
web_1 | 2022-01-13 01:39:32 verbose [THUB]: No macaroon path provided
web_1 | 2022-01-13 01:39:32 error [THUB]: No file found at path: /lnd/data/chain/bitcoin/testnet/admin.macaroon
web_1 | 2022-01-13 01:39:32 error [THUB]: Account Umbrel Testnet has neither lnd directory, macaroon nor macaroon path specified.
web_1 | 2022-01-13 01:39:32 error [THUB]: No file found at path: /lnd/data/chain/bitcoin/regtest/admin.macaroon
web_1 | 2022-01-13 01:39:32 error [THUB]: Account Umbrel Regtest has neither lnd directory, macaroon nor macaroon path specified.
web_1 | 2022-01-13 01:39:32 info [THUB]: Server accounts that will be available: Umbrel
web_1 | 2022-01-13 01:39:32 verbose [THUB]: No cookie path provided
web_1 | 2022-01-13 01:39:39 error [THUB]: Authentication failed from ip: 192.168.86.43 - Invalid Password!
web_1 | 2022-01-13 01:39:52 debug [THUB]: Correct password for account 9d186644-fa19-5e7a-99b8-e0462d5d1a3d
web_1 | (node:25) [DEP0123] DeprecationWarning: Setting the TLS ServerName to an IP address is not permitted by RFC 6066. This will be ignored in a future version.
web_1 | (Use node --trace-deprecation ... to show where the warning was created)
web_1 | 2022-01-13 01:39:57 error [THUB]: Unable to connect to this node: [ 503, ‘FailedToConnectToDaemon’, [length]: 2 ]
web_1 | 2022-01-13 01:40:02 error [THUB]: Error doing graphql fetch: FetchError: request to https://api.thunderbase.io/v1 failed, reason: Proxy connection timed out
web_1 | at ClientRequest. (/app/node_modules/node-fetch/lib/index.js:1461:11)
web_1 | at ClientRequest.emit (events.js:315:20)
web_1 | at onerror (/app/node_modules/agent-base/dist/src/index.js:117:21)
web_1 | at callbackError (/app/node_modules/agent-base/dist/src/index.js:136:17)
web_1 | at processTicksAndRejections (internal/process/task_queues.js:93:5) {
web_1 | message: ‘request to https://api.thunderbase.io/v1 failed, reason: Proxy connection timed out’,
web_1 | type: ‘system’,
web_1 | errno: undefined,
web_1 | code: undefined,
web_1 | [stack]: ‘FetchError: request to https://api.thunderbase.io/v1 failed, reason: Proxy connection timed out\n’ +
web_1 | ’ at ClientRequest. (/app/node_modules/node-fetch/lib/index.js:1461:11)\n’ +
web_1 | ’ at ClientRequest.emit (events.js:315:20)\n’ +
web_1 | ’ at onerror (/app/node_modules/agent-base/dist/src/index.js:117:21)\n’ +
web_1 | ’ at callbackError (/app/node_modules/agent-base/dist/src/index.js:136:17)\n’ +
web_1 | ’ at processTicksAndRejections (internal/process/task_queues.js:93:5)’,
web_1 | name: ‘FetchError’
web_1 | }
==== Result ====
The debug script did not automatically detect any issues with your Umbrel.
Quick update - this looks to be resolved! Thank you to Jon Syu from the Telegram channel. I was able to reflash the SD card with the latest Umbrel OS from the web and boot back up. No need for seed phrase or to use lightning channel.backup. It’s still synching to the chain, so another hour or so to fully confirm I’m back up and running, but so far so good!
