centos 7 firewall d routing IB to eth network with multihomed server

Add the following to /etc/sysctl.conf: net.ipv4.ip_forward = 1
Apply the sysctl settings: sysctl -p

Add direct rules to firewalld. Add the --permanent option to keep these rules across restarts.
firewall-cmd --zone=public --add-masquerade --permanent
firewall-cmd --direct --add-rule ipv4 nat POSTROUTING 0 -o ens1.801 -j MASQUERADE
firewall-cmd --direct --add-rule ipv4 filter FORWARD 0 -i ib0:3 -o ens1.801 -j ACCEPT
firewall-cmd --direct --add-rule ipv4 filter FORWARD 0 -i ens1.801 -o  ib0:3 -m state --state RELATED,ESTABLISHED -j ACCEPT



credits goes to:
https://www.centos.org/forums/viewtopic.php?t=53819