Failed to list *v1.Endpoints: Get https://10.96.0.1:443/api/v1/endpoints?limit=500\u0026resourceVersion=0: dial tcp 10.96.0.1:443: connect: no route to host
I've found out that flanneld is using the wrong network interface, and changed it in the kube-flannel.yml file before deployment. However the outcome is still the same.
Any help is greatly appreciated.
–
–
I've solved the problem. The cause is a mixture of inexperience, lack of documentation and some old, no-longer-correct information.
The guy who will be using the installation told me that Docker's bridge needs to be in the same subnet with the Flannel network, hence I edited Docker's bridge network.
However, when Kubernetes started to use CNI, this requirement not only became unnecessary, but plain wrong. Having both cni0 and docker0 on the same network with same IP address always felt wrong, but since I'm a complete beginner in Kubernetes, I ignored my hunch.
As a result, I reset Docker's network to its default, tore down the cluster and rebuilt it. Now everything is working as it should.
TL;DR: Never, ever touch Docker's network parameters if you are setting up a recent Kubernetes release. Just install Docker, init the Kubernetes and deploy Flannel. Kubernetes and CNI will take care of container to Flannel transport.
This is basically saying that your coredns pod cannot talk to the kube-apiserver. The kube-apiserver is exposed in the pod through these environment variables: KUBERNETES_SERVICE_HOST=10.96.0.1 and KUBERNETES_SERVICE_PORT_HTTPS=443
I believe that the routes that you posted are routes on the host since this is what you get when you run ip routes in pod container:
root@xxxx-xxxxxxxxxx-xxxxx:/# ip route
default via 169.254.1.1 dev eth0
169.254.1.1 dev eth0 scope link
root@xxxx-xxxxxxxxxx-xxxxx:/#
In any case, you wouldn't see 10.96.0.1 since that's exposed in the cluster using iptables. So what is that address? It happens that is a service in the default namespace called kubernetes. That service's ClusterIP is 10.96.0.1 and it's listening on port 443, it also maps to targetPort 6443 which is where your kube-apiserver is running.
Since you can deploy pods, etc. It seems like the kube-apiserver is not down and that's not your problem. So most likely you are missing that service (or there's some iptable rule not allowing you to connect to it). You can see it here, for example:
$ kubectl get svc kubernetes
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 92d
The full output is something like this:
$ kubectl get svc kubernetes -o=yaml
apiVersion: v1
kind: Service
metadata:
creationTimestamp: 2018-07-23T21:10:22Z
labels:
component: apiserver
provider: kubernetes
name: kubernetes
namespace: default
resourceVersion: "24"
selfLink: /api/v1/namespaces/default/services/kubernetes
uid: xxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxx
spec:
clusterIP: 10.96.0.1
ports:
- name: https
port: 443
protocol: TCP
targetPort: 6443
sessionAffinity: None
type: ClusterIP
status:
loadBalancer: {}
So if you are missing it, you can create it like this:
cat <<EOF
apiVersion: v1
kind: Service
metadata:
labels:
component: apiserver
provider: kubernetes
name: kubernetes
namespace: default
spec:
clusterIP: 10.96.0.1
ports:
- name: https
port: 443
protocol: TCP
targetPort: 6443
sessionAffinity: None
type: ClusterIP
EOF | kubectl apply -f -
I met this before. The Firewalld had opened the port 6443 to my real LAN IPs, but it still disables others, so I tried to shut down the Firewall via the CMD :
systemctl stop firewalld
It works and all exceptions that coming from kubectl logs were gone, so the root cause is the firewall rules of your linux servers.
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
site design / logo © 2019 Stack Exchange Inc; user contributions licensed under cc by-sa 3.0
with attribution required.
rev 2019.4.29.33475
