[bitnami/ghost] Access denied for user 'root'@'localhost' (using password: YES) after restart mariadb
mariadb 08:26:50.27
mariadb 08:26:50.28 Welcome to the Bitnami mariadb container
mariadb 08:26:50.28 Subscribe to project updates by watching https://github.com/bitnami/bitnami-docker-mariadb
mariadb 08:26:50.28 Submit issues and feature requests at https://github.com/bitnami/bitnami-docker-mariadb/issues
mariadb 08:26:50.28 Send us your feedback at [email protected]
mariadb 08:26:50.28
mariadb 08:26:50.28 INFO ==> ** Starting MariaDB setup **
mariadb 08:26:50.34 INFO ==> Validating settings in MYSQL_*/MARIADB_* env vars
mariadb 08:26:50.34 INFO ==> Initializing mariadb database
mariadb 08:26:50.34 DEBUG ==> Custom configuration my.cnf detected
mariadb 08:26:50.35 DEBUG ==> Ensuring expected directories/files exist
mariadb 08:26:50.36 INFO ==> Using persisted data
mariadb 08:26:50.37 INFO ==> Running mysql_upgrade
mariadb 08:26:50.37 INFO ==> Starting mariadb in background
2020-05-25 8:26:50 0 [Note] /opt/bitnami/mariadb/sbin/mysqld (mysqld 10.3.22-MariaDB) starting as process 137 ...
2020-05-25 8:26:50 0 [Note] InnoDB: Using Linux native AIO
2020-05-25 8:26:50 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2020-05-25 8:26:50 0 [Note] InnoDB: Uses event mutexes
2020-05-25 8:26:50 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
2020-05-25 8:26:50 0 [Note] InnoDB: Number of pools: 1
2020-05-25 8:26:50 0 [Note] InnoDB: Using SSE2 crc32 instructions
2020-05-25 8:26:50 0 [Note] InnoDB: Initializing buffer pool, total size = 128M, instances = 1, chunk size = 128M
2020-05-25 8:26:50 0 [Note] InnoDB: Completed initialization of buffer pool
2020-05-25 8:26:50 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
2020-05-25 8:26:50 0 [Note] InnoDB: 128 out of 128 rollback segments are active.
2020-05-25 8:26:50 0 [Note] InnoDB: Creating shared tablespace for temporary tables
2020-05-25 8:26:50 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
2020-05-25 8:26:50 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
2020-05-25 8:26:50 0 [Note] InnoDB: 10.3.22 started; log sequence number 4536575; transaction id 2192
2020-05-25 8:26:50 0 [Note] InnoDB: Loading buffer pool(s) from /bitnami/mariadb/data/ib_buffer_pool
2020-05-25 8:26:50 0 [Note] InnoDB: Buffer pool(s) load completed at 200525 8:26:50
2020-05-25 8:26:50 0 [Note] Plugin 'FEEDBACK' is disabled.
2020-05-25 8:26:50 0 [Note] Server socket created on IP: '0.0.0.0'.
2020-05-25 8:26:50 0 [Warning] 'proxies_priv' entry '@% root@ghost-mariadb-0' ignored in --skip-name-resolve mode.
2020-05-25 8:26:50 0 [Note] Reading of all Master_info entries succeeded
2020-05-25 8:26:50 0 [Note] Added new Master_info '' to hash table
2020-05-25 8:26:50 0 [Note] /opt/bitnami/mariadb/sbin/mysqld: ready for connections.
Version: '10.3.22-MariaDB' socket: '/opt/bitnami/mariadb/tmp/mysql.sock' port: 3306 Source distribution
2020-05-25 8:26:51 10 [Warning] Access denied for user 'root'@'localhost' (using password: YES)
Version check failed. Got the following error when calling the 'mysql' command line client
ERROR 1045 (28000): Access denied for user 'root'@'localhost' (using password: YES)
FATAL ERROR: Upgrade failed
mariadb 08:26:51.41 INFO ==> Stopping mariadb
2020-05-25 8:26:51 0 [Note] /opt/bitnami/mariadb/sbin/mysqld (initiated by: unknown): Normal shutdown
2020-05-25 8:26:51 0 [Note] Event Scheduler: Purging the queue. 0 events
2020-05-25 8:26:51 0 [Note] InnoDB: FTS optimize thread exiting.
2020-05-25 8:26:51 0 [Note] InnoDB: Starting shutdown...
2020-05-25 8:26:51 0 [Note] InnoDB: Dumping buffer pool(s) to /bitnami/mariadb/data/ib_buffer_pool
2020-05-25 8:26:51 0 [Note] InnoDB: Buffer pool(s) dump completed at 200525 8:26:51
2020-05-25 8:26:52 0 [Note] InnoDB: Shutdown completed; log sequence number 4536584; transaction id 2193
2020-05-25 8:26:52 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"
2020-05-25 8:26:52 0 [Note] /opt/bitnami/mariadb/sbin/mysqld: Shutdown complete
To Reproduce
Steps to reproduce the behavior:
Deploy clear ghost chart
Use pvc for mariadb
Restart mariadb
Get error
Version of Helm and Kubernetes:
Output of helm version:
Client: &version.Version{SemVer:"v2.14.3", GitCommit:"0e7f3b6637f7af8fcfddb3d2941fcc7cbebb0085", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.16.6", GitCommit:"dd2e5695da88625b190e6b22e9542550ab503a47", GitTreeState:"clean"}
Output of kubectl version:
Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.3", GitCommit:"2e7996e3e2712684bc73f0dec0200d64eec7fe40", GitTreeState:"clean", BuildDate:"2020-05-21T14:51:23Z", GoVersion:"go1.14.3", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"16+", GitVersion:"v1.16.8-gke.15", GitCommit:"9cabee15e0922c3b36724de4866a98f6c2da5e6a", GitTreeState:"clean", BuildDate:"2020-05-01T21:47:04Z", GoVersion:"go1.13.8b4", Compiler:"gc", Platform:"linux/amd64"}
Additional context
GKE version: v1.16.8-gke.15
MariaDB version: docker.io/bitnami/mariadb:10.3.22-debian-10-r92
[bitnami/mariadb] ERROR 1045 (28000): Access denied for user 'root'@'localhost' (using password: YES)
#2634
helm install myghost bitnami/ghost --version 9.2.7 --set service.type=ClusterIP
# To simulate a restart of MariaDB, the pod is deleted.
kubectl delete pod myghost-mariadb-0
kubectl logs myghost-mariadb-0
MariaDB logs after restart (Click to unfold)
mariadb 07:22:02.70
mariadb 07:22:02.70 Welcome to the Bitnami mariadb container
mariadb 07:22:02.70 Subscribe to project updates by watching https://github.com/bitnami/bitnami-docker-mariadb
mariadb 07:22:02.70 Submit issues and feature requests at https://github.com/bitnami/bitnami-docker-mariadb/issues
mariadb 07:22:02.71 Send us your feedback at [email protected]
mariadb 07:22:02.71
mariadb 07:22:02.71 INFO ==> ** Starting MariaDB setup **
mariadb 07:22:02.74 INFO ==> Validating settings in MYSQL_*/MARIADB_* env vars
mariadb 07:22:02.75 INFO ==> Initializing mariadb database
mariadb 07:22:02.75 INFO ==> Using persisted data
mariadb 07:22:02.76 INFO ==> Running mysql_upgrade
mariadb 07:22:02.76 INFO ==> Starting mariadb in background
mariadb 07:22:04.10 INFO ==> Stopping mariadb
mariadb 07:22:06.10 INFO ==> ** MariaDB setup finished! **
mariadb 07:22:06.16 INFO ==> ** Starting MariaDB **
2020-05-27 7:22:06 0 [Note] /opt/bitnami/mariadb/sbin/mysqld (mysqld 10.3.22-MariaDB) starting as process 1 ...
2020-05-27 7:22:06 0 [Note] InnoDB: Using Linux native AIO
2020-05-27 7:22:06 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2020-05-27 7:22:06 0 [Note] InnoDB: Uses event mutexes
2020-05-27 7:22:06 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
2020-05-27 7:22:06 0 [Note] InnoDB: Number of pools: 1
2020-05-27 7:22:06 0 [Note] InnoDB: Using SSE2 crc32 instructions
2020-05-27 7:22:06 0 [Note] InnoDB: Initializing buffer pool, total size = 128M, instances = 1, chunk size = 128M
2020-05-27 7:22:06 0 [Note] InnoDB: Completed initialization of buffer pool
2020-05-27 7:22:06 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
2020-05-27 7:22:06 0 [Note] InnoDB: 128 out of 128 rollback segments are active.
2020-05-27 7:22:06 0 [Note] InnoDB: Creating shared tablespace for temporary tables
2020-05-27 7:22:06 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
2020-05-27 7:22:06 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
2020-05-27 7:22:06 0 [Note] InnoDB: Waiting for purge to start
2020-05-27 7:22:06 0 [Note] InnoDB: 10.3.22 started; log sequence number 2926307; transaction id 1495
2020-05-27 7:22:06 0 [Note] InnoDB: Loading buffer pool(s) from /bitnami/mariadb/data/ib_buffer_pool
2020-05-27 7:22:06 0 [Note] InnoDB: Buffer pool(s) load completed at 200527 7:22:06
2020-05-27 7:22:06 0 [Note] Plugin 'FEEDBACK' is disabled.
2020-05-27 7:22:06 0 [Note] Server socket created on IP: '0.0.0.0'.
2020-05-27 7:22:06 0 [Warning] 'proxies_priv' entry '@% root@myghost-mariadb-0' ignored in --skip-name-resolve mode.
2020-05-27 7:22:06 0 [Note] Reading of all Master_info entries succeeded
2020-05-27 7:22:06 0 [Note] Added new Master_info '' to hash table
2020-05-27 7:22:06 0 [Note] /opt/bitnami/mariadb/sbin/mysqld: ready for connections.
Version: '10.3.22-MariaDB' socket: '/opt/bitnami/mariadb/tmp/mysql.sock' port: 3306 Source distribution
We couldn't reproduce the issue. Did you provide any extra parameters when deploying the chart?
I actually had to start over with db. I think it has to with upgrading the chart and editing params.
One thing that is a bit confusing is if I should be setting my sequel passwords from the secrets within the values yaml and turning off the flag allow empty passwords.
Can you clarify please?
I think it had to do with that. I'm also unable to reproduce.
rootPassword: "password"
Tried uninstal chart deleting the PVC and running without any auth in values.yaml then re-install helm chart.
Same problem.
I wonder if the problem is related to the message "The mariadb configuration file '/opt/bitnami/mariadb/conf/my.cnf'"
�[38;5;6mmariadb �[38;5;5m17:35:40.49 �[0m
�[38;5;6mmariadb �[38;5;5m17:35:40.50 �[0m�[1mWelcome to the Bitnami mariadb container�[0m
�[38;5;6mmariadb �[38;5;5m17:35:40.50 �[0mSubscribe to project updates by watching �[1mhttps://github.com/bitnami/bitnami-docker-mariadb�[0m
�[38;5;6mmariadb �[38;5;5m17:35:40.50 �[0mSubmit issues and feature requests at �[1mhttps://github.com/bitnami/bitnami-docker-mariadb/issues�[0m
�[38;5;6mmariadb �[38;5;5m17:35:40.50 �[0m
�[38;5;6mmariadb �[38;5;5m17:35:40.51 �[0m�[38;5;2mINFO �[0m ==> ** Starting MariaDB setup **
�[38;5;6mmariadb �[38;5;5m17:35:40.52 �[0m�[38;5;2mINFO �[0m ==> Validating settings in MYSQL_*/MARIADB_* env vars
�[38;5;6mmariadb �[38;5;5m17:35:40.53 �[0m�[38;5;2mINFO �[0m ==> Initializing mariadb database
�[38;5;6mmariadb �[38;5;5m17:35:40.53 �[0m�[38;5;5mDEBUG�[0m ==> Ensuring expected directories/files exist
�[38;5;6mmariadb �[38;5;5m17:35:40.54 �[0m�[38;5;3mWARN �[0m ==> The mariadb configuration file '/opt/bitnami/mariadb/conf/my.cnf' is not writable. Configurations based on environment variables will not be applied for this file.
�[38;5;6mmariadb �[38;5;5m17:35:40.54 �[0m�[38;5;2mINFO �[0m ==> Using persisted data
�[38;5;6mmariadb �[38;5;5m17:35:40.57 �[0m�[38;5;2mINFO �[0m ==> Running mysql_upgrade
�[38;5;6mmariadb �[38;5;5m17:35:40.57 �[0m�[38;5;2mINFO �[0m ==> Starting mariadb in background
2021-11-25 17:35:40 0 [Note] /opt/bitnami/mariadb/sbin/mysqld (mysqld 10.5.13-MariaDB) starting as process 50 ...
2021-11-25 17:35:40 0 [Note] InnoDB: Uses event mutexes
2021-11-25 17:35:40 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
2021-11-25 17:35:40 0 [Note] InnoDB: Number of pools: 1
2021-11-25 17:35:40 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions
2021-11-25 17:35:40 0 [Note] mysqld: O_TMPFILE is not supported on /opt/bitnami/mariadb/tmp (disabling future attempts)
2021-11-25 17:35:40 0 [Note] InnoDB: Using Linux native AIO
2021-11-25 17:35:40 0 [Note] InnoDB: Initializing buffer pool, total size = 134217728, chunk size = 134217728
2021-11-25 17:35:40 0 [Note] InnoDB: Completed initialization of buffer pool
2021-11-25 17:35:40 0 [Note] InnoDB: 128 rollback segments are active.
2021-11-25 17:35:40 0 [Note] InnoDB: Creating shared tablespace for temporary tables
2021-11-25 17:35:40 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
2021-11-25 17:35:40 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
2021-11-25 17:35:40 0 [Note] InnoDB: 10.5.13 started; log sequence number 46134; transaction id 20
2021-11-25 17:35:40 0 [Note] Plugin 'FEEDBACK' is disabled.
2021-11-25 17:35:40 0 [Note] Server socket created on IP: '127.0.0.1'.
2021-11-25 17:35:40 0 [Note] InnoDB: Loading buffer pool(s) from /bitnami/mariadb/data/ib_buffer_pool
2021-11-25 17:35:40 0 [Warning] 'proxies_priv' entry '@% root@ssapis-mariadb-0' ignored in --skip-name-resolve mode.
2021-11-25 17:35:40 0 [Note] InnoDB: Buffer pool(s) load completed at 211125 17:35:40
2021-11-25 17:35:40 0 [Note] Reading of all Master_info entries succeeded
2021-11-25 17:35:40 0 [Note] Added new Master_info '' to hash table
2021-11-25 17:35:40 0 [Note] /opt/bitnami/mariadb/sbin/mysqld: ready for connections.
Version: '10.5.13-MariaDB' socket: '/opt/bitnami/mariadb/tmp/mysql.sock' port: 3306 Source distribution
2021-11-25 17:35:42 3 [Warning] Access denied for user 'root'@'localhost' (using password: YES)
Version check failed. Got the following error when calling the 'mysql' command line client
ERROR 1045 (28000): Access denied for user 'root'@'localhost' (using password: YES)
FATAL ERROR: Upgrade failed
�[38;5;6mmariadb �[38;5;5m17:35:42.61 �[0m�[38;5;2mINFO �[0m ==> Stopping mariadb
2021-11-25 17:35:42 0 [Note] /opt/bitnami/mariadb/sbin/mysqld (initiated by: unknown): Normal shutdown
2021-11-25 17:35:42 0 [Note] Event Scheduler: Purging the queue. 0 events
2021-11-25 17:35:42 0 [Note] InnoDB: FTS optimize thread exiting.
2021-11-25 17:35:42 0 [Note] InnoDB: Starting shutdown...
2021-11-25 17:35:42 0 [Note] InnoDB: Dumping buffer pool(s) to /bitnami/mariadb/data/ib_buffer_pool
2021-11-25 17:35:42 0 [Note] InnoDB: Buffer pool(s) dump completed at 211125 17:35:42
2021-11-25 17:35:42 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"
2021-11-25 17:35:42 0 [Note] InnoDB: Shutdown completed; log sequence number 46146; transaction id 21
2021-11-25 17:35:42 0 [Note] /opt/bitnami/mariadb/sbin/mysqld: Shutdown complete
�[38;5;6mmariadb �[38;5;5m17:35:43.62 �[0m�[38;5;5mDEBUG�[0m ==> Waiting for mariadb to unlock db files
Edit:
If I install using helm install my-mariadb bitnami/mariadb --version 10.0.1 it works.
But it doesn't work as a dependency above.
Did you try to deploy it without being a dependency but using the values you are using when it is deployed as a dependency?
$ helm install my-mariadb bitnami/mariadb --version 10.0.1 -f values.yaml
where values.yaml is
image:
debug: true
auth:
username: root
password: "password"
rootPassword: "password"
In the same way, there is a new version (10.0.2) just in case the new version fix a related bug
I also hit this using the latest Ghost bitnami chart
2022-05-17 17:52:52 0 [Warning] 'proxies_priv' entry '@% root@ghost-mariadb-0' ignored in --skip-name-resolv │
│ 2022-05-17 17:52:52 0 [Note] /opt/bitnami/mariadb/sbin/mysqld: ready for connections. │
│ Version: '10.6.7-MariaDB' socket: '/opt/bitnami/mariadb/tmp/mysql.sock' port: 3306 Source distribution │
│ 2022-05-17 17:52:54 3 [Warning] Access denied for user 'root'@'localhost' (using password: YES) │
│ Reading datadir from the MariaDB server failed. Got the following error when executing the 'mysql' command l │
│ ERROR 1045 (28000): Access denied for user 'root'@'localhost' (using password: YES) │
│ FATAL ERROR: Upgrade failed │
│ mariadb 17:52:54.82 INFO ==> Stopping mariadb
I did the following steps to reproduce it:
have a superuser with different than root name on master (that was replicated to slave)
run salve with --skip-grant-tables and give it time to boot, then stop it
run salve without --skip-grant-tables and we get the error Access denied for user 'root'@'localhost' (using password: YES) though I don't have a root user at all.
Maybe that will help to investigate the issue.
What helped me to resolve the issue:
Remove the data folder from persistent storage on slave, so it will start a fresh replication from the very beginning. You need to have all bin logs present on the master for this without any cleanup.
Possibly related: I was running into a similar issue using Bitnami's Matomo chart, which also uses MariaDB as a subchart
While typing out this long post, I discovered the solution to the problem in my case was that my encoded secret values contained additional linefeed characters. I had apparently created them with echo 'mariadbRootPassword' | base64 instead of using echo -n, which would remove the trailing linefeed.
Context
I was attempting to leverage the existingSecret to set the root password for MariaDB so that we can provide these via sealed-secrets.
Reproducing the Issue
Prerequisite: Delete namespace and/or delete any stale state/PVCs from previous runs
Create 2 test secrets - we can safely assume that these will exist on the cluster before installing the chart
# matomo.secret.yaml
apiVersion: v1
data:
# echo 'matomoPassword' | base64
matomo-password: bWF0b21vUGFzc3dvcmQK
kind: Secret
metadata:
name: matomo
namespace: matomo
type: Opaque
# matomo-mariadb.secret.yaml
apiVersion: v1
data:
# echo 'mariadbPassword' | base64
mariadb-password: bWFyaWFkYlBhc3N3b3JkCg==
# echo 'mariadbRootPassword' | base64
mariadb-root-password: bWFyaWFkYlJvb3RQYXNzd29yZAo=
# echo 'mariadbReplicationPassword' | base64
mariadb-replication-password: bWFyaWFkYlJlcGxpY2F0aW9uUGFzc3dvcmQK
kind: Secret
metadata:
name: matomo-mariadb
namespace: matomo
type: Opaque
# values.local.yaml
matomoUsername: "admin"
matomoEmail: ""
existingSecret: "matomo"
mariadb:
auth:
existingSecret: "matomo-mariadb"bitnami/matomo Helm chart with the above values
Specific Commands
Prerequisite: kubectl delete pvc -n matomo --all && kubectl delete ns matomo
kubectl create ns matomo && kubectl apply -n matomo -f mariadb.secret.yaml -f matomo-mariadb.secret.yaml
helm upgrade --install matomo -n matomo bitnami/matomo -f values.local.yaml
Debugging
MariaDB should start up successfully, after reading its own credentials from the existingSecret
Instead, MariaDB fails to start and goes into CrashLoopBackoff with the following error in the logs after each restart:
% kubectl logs -f matomo-mariadb-0 -n matomo
mariadb 21:39:50.50
mariadb 21:39:50.50 Welcome to the Bitnami mariadb container
mariadb 21:39:50.50 Subscribe to project updates by watching https://github.com/bitnami/containers
mariadb 21:39:50.50 Submit issues and feature requests at https://github.com/bitnami/containers/issues
mariadb 21:39:50.51
mariadb 21:39:50.51 INFO ==> ** Starting MariaDB setup **
mariadb 21:39:50.52 INFO ==> Validating settings in MYSQL_*/MARIADB_* env vars
mariadb 21:39:50.52 INFO ==> Initializing mariadb database
mariadb 21:39:50.53 DEBUG ==> Ensuring expected directories/files exist
mariadb 21:39:50.54 WARN ==> The mariadb configuration file '/opt/bitnami/mariadb/conf/my.cnf' is not writable. Configurations based on environment variables will not be applied for this file.
mariadb 21:39:50.54 INFO ==> Using persisted data
mariadb 21:39:50.57 INFO ==> Running mysql_upgrade
mariadb 21:39:50.58 INFO ==> Starting mariadb in background
2023-02-03 21:39:50 0 [Note] /opt/bitnami/mariadb/sbin/mysqld (server 10.6.11-MariaDB) starting as process 48 ...
2023-02-03 21:39:50 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
2023-02-03 21:39:50 0 [Note] InnoDB: Number of pools: 1
2023-02-03 21:39:50 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions
2023-02-03 21:39:50 0 [Note] mysqld: O_TMPFILE is not supported on /opt/bitnami/mariadb/tmp (disabling future attempts)
2023-02-03 21:39:50 0 [Note] InnoDB: Using Linux native AIO
2023-02-03 21:39:50 0 [Note] InnoDB: Initializing buffer pool, total size = 134217728, chunk size = 134217728
2023-02-03 21:39:50 0 [Note] InnoDB: Completed initialization of buffer pool
2023-02-03 21:39:50 0 [Note] InnoDB: 128 rollback segments are active.
2023-02-03 21:39:50 0 [Note] InnoDB: Creating shared tablespace for temporary tables
2023-02-03 21:39:50 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
2023-02-03 21:39:50 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
2023-02-03 21:39:50 0 [Note] InnoDB: 10.6.11 started; log sequence number 42300; transaction id 14
2023-02-03 21:39:50 0 [Note] Plugin 'FEEDBACK' is disabled.
2023-02-03 21:39:50 0 [Note] InnoDB: Loading buffer pool(s) from /bitnami/mariadb/data/ib_buffer_pool
2023-02-03 21:39:50 0 [Note] Server socket created on IP: '127.0.0.1'.
2023-02-03 21:39:50 0 [Note] InnoDB: Buffer pool(s) load completed at 230203 21:39:50
2023-02-03 21:39:50 0 [Warning] 'proxies_priv' entry '@% root@matomo-mariadb-0' ignored in --skip-name-resolve mode.
2023-02-03 21:39:50 0 [Note] /opt/bitnami/mariadb/sbin/mysqld: ready for connections.
Version: '10.6.11-MariaDB' socket: '/opt/bitnami/mariadb/tmp/mysql.sock' port: 3306 Source distribution
2023-02-03 21:39:52 3 [Warning] Access denied for user 'root'@'localhost' (using password: YES)
Reading datadir from the MariaDB server failed. Got the following error when executing the 'mysql' command line client
ERROR 1045 (28000): Access denied for user 'root'@'localhost' (using password: YES)
FATAL ERROR: Upgrade failed
mariadb 21:39:52.61 INFO ==> Stopping mariadb
2023-02-03 21:39:52 0 [Note] /opt/bitnami/mariadb/sbin/mysqld (initiated by: unknown): Normal shutdown
2023-02-03 21:39:52 0 [Note] InnoDB: FTS optimize thread exiting.
2023-02-03 21:39:52 0 [Note] InnoDB: Starting shutdown...
2023-02-03 21:39:52 0 [Note] InnoDB: Dumping buffer pool(s) to /bitnami/mariadb/data/ib_buffer_pool
2023-02-03 21:39:52 0 [Note] InnoDB: Buffer pool(s) dump completed at 230203 21:39:52
2023-02-03 21:39:52 0 [Note] InnoDB: Removed temporary tablespace data file: "./ibtmp1"
2023-02-03 21:39:52 0 [Note] InnoDB: Shutdown completed; log sequence number 42312; transaction id 15
2023-02-03 21:39:52 0 [Note] /opt/bitnami/mariadb/sbin/mysqld: Shutdown complete
mariadb 21:39:53.63 DEBUG ==> Waiting for mariadb to unlock db files
View the container's EnvVars
Checking the env for mariadb, we see that the MARIADB_ROOT_PASSWORD is actually set to the correct value (e.g. mariadbRootPassword from the above example):
% kubectl delete pod matomo-mariadb-0 -n matomo && sleep 1s && kubectl exec -it matomo-mariadb-0 -n matomo -- env
pod "matomo-mariadb-0" deleted
PATH=/opt/bitnami/common/bin:/opt/bitnami/common/sbin:/opt/bitnami/mariadb/bin:/opt/bitnami/mariadb/sbin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
HOSTNAME=matomo-mariadb-0
TERM=xterm
MARIADB_DATABASE=bitnami_matomo
BITNAMI_DEBUG=false
MARIADB_ROOT_PASSWORD=mariadbRootPassword
MARIADB_USER=bn_matomo
MARIADB_PASSWORD=mariadbPassword
MATOMO_MARIADB_PORT_3306_TCP_PORT=3306
MATOMO_MARIADB_PORT_3306_TCP_ADDR=10.103.6.168
MATOMO_SERVICE_PORT_HTTPS=443
Note the line feeds in the values above - this is where the error was coming from, in my case
Resolution
I didn't realize that echo was inserting a linefeed here
For example, try this with any string:
% echo 'mariadbRootPassword' | base64
bWFyaWFkYlJvb3RQYXNzd29yZAo=
% echo -n 'mariadbRootPassword' | base64
bWFyaWFkYlJvb3RQYXNzd29yZA==
Note the 1-character difference toward the end - this is what was causing the failure during startup, as the env value had an additional linefeed character in there
Encoding ALL of our secrets with echo -n instead appears to resolve the issue, and after repeating our steps everything comes up just fine:
% kubectl get pods -n matomo
NAME READY STATUS RESTARTS AGE
matomo-646dd7646f-bg4ww 1/1 Running 0 10m
matomo-mariadb-0 1/1 Running 0 10m
TL;DR
PSA for others running into similar errors: ensure that your envvar / secret values do not contain extra line feeds
Use echo -n instead of echo if using this method to manually encode secrets
I am deploying the basic chart v.19.9.0 , no auth given as envvar and yet i am facing the same issue. The only variable i am setting is service: type: "ClusterIP" and ghostPath, ghostHost. Everything else is on default.
What i notice in argo is that the secret changes after the initial sync. Resyncing them and restarting the containers doesn't help.
@bodom0015
