First of all, congrats for this amazing project. Until now, it was the most simplest and cheapest implementation that I found to implement and work with HA monitoring. Very well documented as well.
However I'm having a problem spining the second storage instance. I implemented the cluster version of VictoriaMetrics in Kubernetes, with one instance of each segment (storage, insert and select). Now I'm trying to spin up the second instance of storage to check if all works properly, but this second one it's stuck in unready status.
Error from server (BadRequest): container "vitoria-storage" in pod "vitoria-storage-statefulset-1" is waiting to start: ContainerCreating
I'm using rancher as well to managing k8's clusters, and there, I can see this error:
Containers with unready status: [vitoria-storage]
This is my storage yaml file:
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: vitoria-storage-statefulset
labels:
app: vitoria-storage
spec:
replicas: 3
selector:
matchLabels:
app: vitoria-storage
serviceName: vitoria-storage
template:
metadata:
labels:
app: vitoria-storage
name: vitoria-storage
spec:
containers:
- name: vitoria-storage
image: victoriametrics/vmstorage:latest
imagePullPolicy: IfNotPresent
args:
- -retentionPeriod=1
- -storageDataPath=/data/metrics
volumeMounts:
- name: data
mountPath: /data/metrics
ports:
- name: insert-port
protocol: TCP
containerPort: 8400
- name: select-port
protocol: TCP
containerPort: 8401
- name: http
protocol: TCP
containerPort: 8482
readinessProbe:
httpGet:
path: /health
port: http
initialDelaySeconds: 5
periodSeconds: 15
timeoutSeconds: 5
livenessProbe:
tcpSocket:
port: http
initialDelaySeconds: 5
periodSeconds: 15
timeoutSeconds: 5
volumeClaimTemplates:
- metadata:
name: data
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10Gi
Any help with this problem? Thanks
Second storage unit stuck in Unready Status
Kubernetes: second storage unit stuck in Unready Status
Jul 8, 2019
Humm... I see. The problem it's with the volumes.
> kubectl --kubeconfig=/home/user/Downloads/do-moni-kubeconfig.yaml describe po vitoria-storage-statefulset-1
Volumes:
data:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: data-vitoria-storage-statefulset-1
ReadOnly: false
default-token-bnbkp:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-bnbkp
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedAttachVolume 4m31s (x489 over 16h) attachdetach-controller AttachVolume.Attach failed for volume "pvc-63d28cb0-a160-11e9-b5f5-5681ebed6a42" : rpc error: code = NotFound desc = volume "64c299f9-a160-11e9-8b70-0a58ac14c058" not found
Warning FailedMount 2m36s (x434 over 16h) kubelet, k8s-pool-moni-orie Unable to mount volumes for pod "vitoria-storage-statefulset-1_default(067d2982-a197-11e9-b5f5-5681ebed6a42)": timeout expired waiting for volumes to attach or mount for pod "default"/"vitoria-storage-statefulset-1". list of unmounted volumes=[data]. list of unattached volumes=[data default-token-bnbkp]
> kubectl --kubeconfig=/home/user/Downloads/do-moni-kubeconfig.yaml get pvc
data-vitoria-storage-statefulset-0 Bound pvc-0b721856-a160-11e9-b5f5-5681ebed6a42 10Gi RWO do-block-storage 22h
data-vitoria-storage-statefulset-1 Bound pvc-63d28cb0-a160-11e9-b5f5-5681ebed6a42 10Gi RWO do-block-storage 22h
data-vitoria-storage-statefulset-2 Bound pvc-70a48984-a160-11e9-b5f5-5681ebed6a42 10Gi RWO do-block-storage 22h
I deleted them in the cloud provider dashboard, I pod can't get access to it. Just had to delete them from pvc list and redeploy pod again.
Thanks for your help guys.
