E0427 01:08:34.346036       1 controller.go:116] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: failed to retrieve openAPI spec, http error: ResponseCode: 500, Body: Internal Server Error: "/openapi/v2": the server could not find the requested resource, Header: map[Content-Length:[86] Content-Type:[text/plain] Date:[Wed, 27 Apr 2022 01:08:34 GMT] X-Content-Type-Options:[nosniff]]
I0427 01:08:34.346050       1 controller.go:129] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
E0427 01:10:34.343692       1 controller.go:116] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: failed to retrieve openAPI spec, http error: ResponseCode: 500, Body: Internal Server Error: "/openapi/v2": the server could not find the requested resource, Header: map[Content-Length:[86] Content-Type:[text/plain] Date:[Wed, 27 Apr 2022 01:10:34 GMT] X-Content-Type-Options:[nosniff]]
kube-controller logs
W0426 09:25:35.005422       1 garbagecollector.go:707] failed to discover some groups: map[metrics.k8s.io/v1beta1:an error on the server ("Internal Server Error: \"/apis/metrics.k8s.io/v1beta1\": the server could not find the requested resource") has prevented the request from succeeding]
What you expected to happen:

HPA functionality
Anything else we need to know?:
Metrics-server is fetching metrics correctly for pods and nodes with the top command
kubectl top pods -A
NAMESPACE     NAME                                   CPU(cores)   MEMORY(bytes)
default       hpa-demo-deployment-75f866567d-rwkpx   1m           7Mi
kube-system   coredns-64897985d-5vkk5                2m           17Mi
kube-system   etcd-vlab033302                        9m           278Mi
kube-system   kube-apiserver-vlab033302              26m          264Mi
kube-system   kube-controller-manager-vlab033302     8m           55Mi
kube-system   kube-flannel-ds-26cdw                  1m           22Mi
kube-system   kube-flannel-ds-lblbx                  2m           35Mi
kube-system   kube-proxy-bgb7q                       1m           18Mi
kube-system   kube-proxy-trgrt                       1m           26Mi
kube-system   kube-scheduler-vlab033302              2m           20Mi
kube-system   metrics-server-77ffdc8fbc-l55wn        1m           7Mi
kubectl get --raw /apis/metrics.k8s.io/v1beta1/namespaces/default/pods
{"kind":"PodMetricsList","apiVersion":"metrics.k8s.io/v1beta1","metadata":{"selfLink":"/apis/metrics.k8s.io/v1beta1/namespaces/default/pods"},"items":[{"metadata":{"name":"hpa-demo-deployment-75f866567d-lvl2s","namespace":"default","selfLink":"/apis/metrics.k8s.io/v1beta1/namespaces/default/pods/hpa-demo-deployment-75f866567d-lvl2s","creationTimestamp":"2022-04-27T11:37:39Z"},"timestamp":"2022-04-27T11:37:05Z","window":"30s","containers":[{"name":"hpa-demo-deployment","usage":{"cpu":"24707n","memory":"5884Ki"}}]}]}
kubectl get --raw /apis/metrics.k8s.io/v1beta1/
{"kind":"APIResourceList","apiVersion":"v1","groupVersion":"metrics.k8s.io/v1beta1","resources":[{"name":"nodes","singularName":"","namespaced":false,"kind":"NodeMetrics","verbs":["get","list"]},{"name":"pods","singularName":"","namespaced":true,"kind":"PodMetrics","verbs":["get","list"]}]}
However, when I am running kubectl describe hpa the current cpu usage comes back as  / 20% with the warnings:
Metrics:                                               ( current / target )
  resource cpu on pods  (as a percentage of request):  <unknown> / 20%
Min replicas:                                          1
Max replicas:                                          10
Deployment pods:                                       1 current / 0 desired
Conditions:
  Type           Status  Reason                   Message
  ----           ------  ------                   -------
  AbleToScale    True    SucceededGetScale        the HPA controller was able to get the target's current scale
  ScalingActive  False   FailedGetResourceMetric  the HPA was unable to compute the replica count: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: an error on the server ("Internal Server Error: \"/apis/metrics.k8s.io/v1beta1/namespaces/default/pods?labelSelector=name%!D(MISSING)hpa-demo-deployment\": the server could not find the requested resource") has prevented the request from succeeding (get pods.metrics.k8s.io)
Events:
  Type     Reason                   Age                  From                       Message
  ----     ------                   ----                 ----                       -------
  Warning  FailedGetResourceMetric  58s (x261 over 66m)  horizontal-pod-autoscaler  failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: an error on the server ("Internal Server Error: \"/apis/metrics.k8s.io/v1beta1/namespaces/default/pods?labelSelector=name%3Dhpa-demo-deployment\": the server could not find the requested resource") has prevented the request from succeeding (get pods.metrics.k8s.io)
kube api-server
spec:
  containers:
  - command:
    - kube-apiserver
    - --advertise-address=10.204.108.18
    - --allow-privileged=true
    - --authorization-mode=Node,RBAC
    - --client-ca-file=/etc/kubernetes/pki/ca.crt
    - --enable-admission-plugins=NodeRestriction
    - --enable-bootstrap-token-auth=true
    - --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt
    - --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt
    - --etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key
    - --etcd-servers=https://127.0.0.1:2379/
    - --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt
    - --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key
    - --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
    - --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt
    - --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key
    - --requestheader-allowed-names=front-proxy-client
    - --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt
    - --requestheader-extra-headers-prefix=X-Remote-Extra-
    - --requestheader-group-headers=X-Remote-Group
    - --requestheader-username-headers=X-Remote-User
    - --secure-port=6443
    - --service-account-issuer=https://kubernetes.default.svc.cluster.local/
    - --service-account-key-file=/etc/kubernetes/pki/sa.pub
    - --service-account-signing-key-file=/etc/kubernetes/pki/sa.key
    - --service-cluster-ip-range=10.96.0.0/12
    - --tls-cert-file=/etc/kubernetes/pki/apiserver.crt
    - --tls-private-key-file=/etc/kubernetes/pki/apiserver.key
    - --runtime-config=api/all=true
Environment:
OS: CentOS 7.8
Kubernetes distribution (GKE, EKS, Kubeadm, the hard way, etc.):  kubeadm
Container Network Setup (flannel, calico, etc.): Tried on both types facing same issue on both of them
Kubernetes version (use kubectl version):
Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.5", GitCommit:"c285e781331a3785a7f436042c65c5641ce8a9e9", GitTreeState:"clean", BuildDate:"2022-03-16T15:58:47Z", GoVersion:"go1.17.8", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.5", GitCommit:"c285e781331a3785a7f436042c65c5641ce8a9e9", GitTreeState:"clean", BuildDate:"2022-03-16T15:52:18Z", GoVersion:"go1.17.8", Compiler:"gc", Platform:"linux/amd64"}

Metrics Server manifest

   containers:
      - name: metrics-server
        image: k8s.gcr.io/metrics-server-amd64:v0.3.6
        imagePullPolicy: IfNotPresent
        command:
        - /metrics-server
        - --cert-dir=/tmp
        - --secure-port=4443
        - --kubelet-insecure-tls
        - --kubelet-preferred-address-types=InternalIP, ExternalIP, Hostname
        name: metrics-server
        ports:
        - containerPort: 4443
          name: https
          protocol: TCP
        resources:
          requests:
            cpu: 100m
            memory: 200Mi
        securityContext:
          allowPrivilegeEscalation: false
          readOnlyRootFilesystem: true
          runAsNonRoot: true
          runAsUser: 1000
        volumeMounts:
        - mountPath: /tmp
          name: tmp-dir
       hostNetwork: true
Kubelet config:
apiVersion: kubelet.config.k8s.io/v1beta1
authentication:
  anonymous:
    enabled: false
  webhook:
    cacheTTL: 3600s
    enabled: true
  x509:
    clientCAFile: /etc/kubernetes/pki/ca.crt
authorization:
  mode: Webhook
  webhook:
    cacheAuthorizedTTL: 10m10s
    cacheUnauthorizedTTL: 10s  
cgroupDriver: systemd
clusterDNS:
- 10.96.0.10
clusterDomain: cluster.local
cpuManagerReconcilePeriod: 0s
evictionPressureTransitionPeriod: 0s
fileCheckFrequency: 0s
healthzBindAddress: 127.0.0.1
healthzPort: 10248
httpCheckFrequency: 0s
imageMinimumGCAge: 0s
kind: KubeletConfiguration
logging:
  flushFrequency: 0
  options:
    json:
      infoBufferSize: "0"
  verbosity: 0
memorySwap: {}
nodeStatusReportFrequency: 0s
nodeStatusUpdateFrequency: 0s
rotateCertificates: true
runtimeRequestTimeout: 0s
shutdownGracePeriod: 0s
shutdownGracePeriodCriticalPods: 0s
staticPodPath: /etc/kubernetes/manifests
streamingConnectionIdleTimeout: 0s
syncFrequency: 0s
volumeStatsAggPeriod: 0s
Metrics server logs:
 I0427 11:03:13.715206       1 serving.go:312] Generated self-signed cert (/tmp/apiserver.crt, /tmp/apiserver.key)
I0427 11:03:14.423638       1 secure_serving.go:116] Serving securely on [::]:4443
E0427 11:03:15.579508       1 webhook.go:196] Failed to make webhook authorizer request: the server could not find the requested resource
E0427 11:03:15.579593       1 errors.go:77] the server could not find the requested resource
E0427 11:03:15.621591       1 webhook.go:196] Failed to make webhook authorizer request: the server could not find the requested resource
E0427 11:03:15.621668       1 errors.go:77] the server could not find the requested resource
Status of Metrics API:
kubectl describe apiservice v1beta1.metrics.k8s.io
Name:         v1beta1.metrics.k8s.io
Namespace:
Labels:       k8s-app=metrics-server
Annotations:  <none>
API Version:  apiregistration.k8s.io/v1
Kind:         APIService
Metadata:
  Creation Timestamp:  2022-04-27T10:58:18Z
  Resource Version:  1431305
  UID:               46388641-1f14-47f6-8cac-d67f3e7e3b78
Spec:
  Group:                     metrics.k8s.io
  Group Priority Minimum:    100
  Insecure Skip TLS Verify:  true
  Service:
    Name:            metrics-server
    Namespace:       kube-system
    Port:            443
  Version:           v1beta1
  Version Priority:  100
Status:
  Conditions:
    Last Transition Time:  2022-04-27T11:03:13Z
    Message:               all checks passed
    Reason:                Passed
    Status:                True
    Type:                  Available
Events:                    <none>
Any help is highly appreciated.
/kind bug
          
In manifests you can see you run 8s.gcr.io/metrics-server-amd64:v0.3.6 which is very old version that doesn't work newer K8s versions.
Please read https://github.com/kubernetes-sigs/metrics-server#compatibility-matrix
          
Are you sure the spec.containers.resources.requests.cpu value is specified inside the deployment?
In order for server-metrics to calculate the % of the cpu used by the pod, it needs to know the amount of cpu that was allocated for that pod. Same goes with memory.
I would try deploying this HPA demo by DigitalOcean to see if the metrics show up:
kubectl apply -f https://docs.digitalocean.com/products/kubernetes/resources/hpa.yaml
      Kubernetes metrics doesn't work with K8s v1.30.x and cgroup-driver=cgroupfs (the default)
      cri-o/cri-o#8034