Nodes are deployed with a secondary Interface, and the file
/etc/default/nodeip-configuration
has been updated to point at the desired secondary interface to be selected by
br-ex
during boot with
NODEIP_HINT=<secondary-interface>
Secondary interface has been assigned a default route rule in the
/etc/NetworkManager/system-connections/*.nmconnection
network definition file, or
/etc/sysconfig/network-scripts/*.ifcfg
files.
Nodes are deployed with HostNetwork=True, and IPForwarding=Global
properties:
ipForwarding:
description: IPForwarding controls IP forwarding for all traffic on OVN-Kubernetes managed interfaces (such as br-ex). By default this is set to Restricted, and Kubernetes related traffic is still forwarded appropriately, but other IP traffic will not be routed by the OCP node. If there is a desire to allow the host to forward traffic across OVN-Kubernetes managed interfaces, then set this field to "Global". The supported values are "Restricted" and "Global".
type: string
Resolution
This issue is being tracked in https://issues.redhat.com/browse/OCPBUGS-27821 Which will seek to resolve the problem by removing the requirement for a default gateway on a secondary IP:
If someone deploys with no default gw on br-ex, we allow OVNK to come up, and we dont configure any default gw in OVN GR. This was intentionally done some time ago. But we didn't consider the case with local gateway mode (routingViaHost=true), and kube-api is on another interface. So we plan on making a change where if there is no default gateway detected via br-ex, then i will add a default route to the fake masquerade IP 169.254.169.4. This will allow OVN to get the traffic out to br-ex, where it will be forwarded into the host and work correctly.
As a workaround, you can address this problem by forcibly defining your desired interface's route to have a lower metric for the route than 48/49 - which is set byconfigure-ovs.sh` see [1] in root cause below.
[core@testing ~]$ ip route | grep default
default via 11.xxx.xxx.yyy dev bond1 proto static metric 25 ##set value lower on our secondary interface to force priority over automatically set br-ex routing.
default via 10.xxx.xxx.yyy dev br-ex proto static metric 48 ## value is automatically set/cannot be overridden + has (generally) a lower value metric to force priority, but not the LOWEST value.
[core@testing ~]$
[core@testing ~]$ ip -6 route | grep default
default via 2001::::3 dev bond1 proto static metric 25 pref medium
default via 2001::::1 dev br-ex proto static metric 48 pref medium
[core@testing ~]$
[root@testing ~]# cat /etc/sysconfig/network-scripts/ifcfg-bond1 | grep METRIC
IPV4_ROUTE_METRIC=25
IPV6_ROUTE_METRIC=25
Root Cause
[1] This is pushed by the machine-config using the yaml and is mentioned in the github:
https://github.com/openshift/machine-config-operator/blob/master/templates/common/_base/files/configure-ovs-network.yaml#L140
// Notes from Engineering
# when creating the bridge, we use a value lower than NM's ethernet device default route metric
# (we pick 48 and 49 to be lower than anything that NM chooses by default)
Diagnostic Steps
Observe that when you have applied your configuration to use RoutingViaHost=true, on a node that is using a secondary IP for br-ex handling, the routes are misconfigured. br-ex is still putting priority routing on the original/primary interface and not the overridden interface you've selected to handle networking.
Observe that pods deployed as hostNetwork (and nodes themselves in this configuration) cannot reach the primary default service IP for kube-apiserver
You have ensured that the configuration for network also includes ipForwarding=global
You have ensured that your secondary interface includes a default gateway configuration to allow routing appropriately.
This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.
Here are the common uses of Markdown.
Code blocks
~~~
Code surrounded in tildes is easier to read
Links/URLs
[Red Hat Customer Portal](https://access.redhat.com)
Learn more
Are you sure you want to request a translation?
We appreciate your interest in having Red Hat content localized to your language. Please note that excessive use of this feature could cause delays in getting specific content you are interested in translated.
