Standalone Egress NAT

[chdxD1]

What

We are currently using coil + calico as our CNI. We'd also like to investigate if we could use calico as the main CNI and chain coil after to just provide the Egress NAT functionality. Is this something you are interested in having in Coil? We could try to contribute it but I'd like to have your opinion on it.

How

Chaining Coil after the main CNI. Detect pod IP address and add egress interface to pod. On the coil egress side the functionality will probably not change at all.

Checklist

• Finish implentation of the issue
• Test all functions
• Have enough logs to trace activities
• Notify developers of necessary actions

[terassyi]

Hi, @chdxD1
Thank you for your suggestion!
Your idea sounds reasonable to us.

We are also considering separating coil (ipam CNI) and coil egress, but the way to achieve it is under consideration.
Could you send your design of implementation before the actual implementation? We are afraid of discarding the PR if it doesn't match our plan.

The design document is here.
Please add your design to this document and submit it as a pull request.
https://github.com/cybozu-go/coil/blob/main/docs/design.md#on-demand-nat-for-egress-traffics

[chdxD1]

Sounds good. With contributing I also meant making a design proposal first :) I'll come back with a PR to the design document 👍

[chdxD1]

@terassyi Sorry for taking such a long time, we (@p-strusiewiczsurmacki-mobica and I) finally got some to look into this in more details.

We have two ideas and would like your opinion on them first. On a high level:

1. Keep the tunnel inside the pod, adding options to the CNI part of coil to only do IPAM or/and egress NAT (e.g. 'ipam: true' and 'egress: true') and splitting coil-controller into coil-ipam-controller and coil-egress-controller. In addition to that it might be interesting to look into Allow modifications to Egress destinations #203 by having coild changing the routing table of the pod dynamically. It would probably still make sense to stick to CNI and controller to have the egress NAT exist when the pod starts up (as a requirement) but also allow dynamic reconciliation.
2. Move the egress NAT tunnel that is currently in the pod to the node (but keep the egress NAT pods). This would need thorough investigation regarding CNI compatibility but also has some benefits, e.g. network policies are applied by the CNI before pushing traffic into the tunnel. This way we would need coild to handle ip-rule and ip-route on the node to send traffic into tunnel interfaces when source interface/pod IP and destination matches, otherwise do a fallthrough to other ip rules. I would still opt to do some parts on CNI_ADD to have it setup when the app starts.

I am interested in what you think would be the best option. Option 1 might be the easiest one but option 2 might be interesting as well (but has some drawbacks).

[chdxD1]

@terassyi pinging you regarding my questions above

[terassyi]

@chdxD1
Thank you for the reminder & the suggestion!

Both ways are interesting to me, and I think opinion 1 is better.
As you mentioned, it may be easier than opinion 2, and we don't have to change the existing NATing method.

FYI, we are also planning to do a PoC of a stand-alone egress NAT in a completely separate repository.
But we don't have any plans to merge it into coil.

https://github.com/cybozu-go/pona

So, if you want to introduce stand-alone egress NAT mode in coil, we can support it!

[chdxD1]

@terassyi Sounds good! I think that particular section of design.md already covers this if we move with option 1 as we are not changing the existing NATing method. However the other sections / documents might need some adjustments.

Pona sounds interesting, will watch that project for sure! (and maybe we can even contribute there as well)

[terassyi]

However the other sections / documents might need some adjustments.

Yes, please create a pull request for the design!