Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.5", GitCommit:"32ac1c9073b132b8ba18aa830f46b77dcceb0723", GitTreeState:"clean", BuildDate:"2018-06-21T11:46:00Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.4", GitCommit:"c27b913fddd1a6c480c229191a087698aa92f0b1", GitTreeState:"clean", BuildDate:"2019-02-28T13:30:26Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"}
Docker version (use docker version):
Client:
Version: 18.09.3
API version: 1.39
Go version: go1.10.8
Git commit: 774a1f4
Built: Thu Feb 28 06:34:04 2019
OS/Arch: linux/amd64
Experimental: false
Server: Docker Engine - Community
Engine:
Version: 18.09.3
API version: 1.39 (minimum version 1.12)
Go version: go1.10.8
Git commit: 774a1f4
Built: Thu Feb 28 05:59:55 2019
OS/Arch: linux/amd64
Experimental: false
I have try this
https://rancher.com/docs/rancher/v2.x/en/troubleshooting/dns/
# Run kubectl commands inside here
# e.g. kubectl get all
> kubectl -n kube-system get pods -l k8s-app=kube-dns
NAME READY STATUS RESTARTS AGE
kube-dns-6d656445d5-wlv9t 3/3 Running 1 9h
> kubectl -n kube-system get svc -l k8s-app=kube-dns
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kube-dns ClusterIP 10.43.0.10 <none> 53/UDP,53/TCP 9h
> kubectl run -it --rm --restart=Never busybox --image=busybox:1.28 -- nslookup kubernetes.default
If you don't see a command prompt, try pressing enter.
nslookup: can't resolve 'kubernetes.default'
pod default/busybox terminated (Error)
> kubectl run -it --rm --restart=Never busybox --image=busybox:1.28 -- nslookup kubernetes.default
If you don't see a command prompt, try pressing enter.
nslookup: can't resolve 'kubernetes.default'
pod default/busybox terminated (Error)
For more informations ...
I have install rancher server and agent on clean server.
Host can ping and reach, host resolv.conf is set for 8.8.8.8 - 8.8.4.4. No firewall.
The only thing i do, it's launch a simple debian container, and try ping with shell.
The used resolv.conf will be shown in the output of the command shown in https://rancher.com/docs/rancher/v2.x/en/troubleshooting/dns/#check-upstream-nameservers-in-kubedns-container.
On that same page, this is mentioned:
Before running the DNS checks, make sure that the overlay network is functioning correctly as this can also be the reason why DNS resolution (partly) fails.
Already do this test
> kubectl -n kube-system get pods -l k8s-app=kube-dns --no-headers -o custom-columns=NAME:.metadata.name,HOSTIP:.status.hostIP | while read pod host; do echo "Pod ${pod} on host ${host}"; kubectl -n kube-system exec $pod -c kubedns cat /etc/resolv.conf; done
Pod kube-dns-6d656445d5-wlv9t on host x.x.x.x
nameserver 8.8.8.8
nameserver 8.8.4.4
I seem to be experiencing the same issue. The DNS resolution failure happens intermittently and appears to be random. It does recover after a brief period of time.
Running CentOS 7.6, Rancher 2.2.2 and the cluster is hyercube:1.11.9.
If it happens intermittently, does the overlay network still function correctly or does it also occur on the same host where DNS pods are running? Is the upstream nameserver still responsive?
@Hantse Did you check the overlay network as well, does that function correctly at all times? Does it work using a supported OS like Ubuntu?