This page documents the list of known issues in Kubermatic KubeOne along with
possible workarounds and recommendations.
This list applies to KubeOne 1.10 release. For KubeOne 1.9, please consider
the v1.9 version of this document. For earlier releases,
please consult the appropriate changelog.
Internal Kubernetes endpoints unreachable on vSphere with Cilium/Canal
Description
Symptoms
- Unable to perform CRUD operations on resources governed by webhooks (e.g. ValidatingWebhookConfiguration, MutatingWebhookConfiguration, etc.). The following error is observed:
Internal error occurred: failed calling webhook "webhook-name": failed to call webhook: Post "https://webhook-service-name.namespace.svc:443/webhook-endpoint": context deadline exceeded
- Unable to reach internal Kubernetes endpoints from pods/nodes.
- ICMP is working but TCP/UDP is not.
Cause
On recent enough VMware hardware compatibility version (i.e >=15 or maybe >=14), CNI connectivity breaks because of hardware segmentation offload. cilium-health status
has ICMP connectivity working, but not TCP connectivity. cilium-health status may also fail completely.
Recommendation
sudo ethtool -K ens192 tx-udp_tnl-segmentation off
sudo ethtool -K ens192 tx-udp_tnl-csum-segmentation off
These flags are related to the hardware segmentation offload done by the vSphere driver VMXNET3. We have observed this issue for both Cilium and Canal CNI running on Ubuntu 22.04.
We have two options to configure these flags for KubeOne installations:
References