How to setup static outbound IP in DigitalOcean Kubernetes (DOKS)?
DigitalOcean VPC Gateway configuration for Droplets and Kubernetes Clusters.
There are plenty of reasons to route all of your traffic through a single IP such as IP Whitelisting and security. This solution can be achieved in DigitalOcean droplet using something like VPC Gateway. But what about configuring static outbound IP in DigitalOcean managed Kubernetes AKA as DOKS ? Luckily it is very similar to DigitalOcean VPC Gateway configuration. Let’s take a look at how it works. Check out official documentation of configuring VPC Gateway here. I will be referring DigitalOcean VPC Gateway as VPC Gateway and DOKS as Kubernetes[ Cluster] from now onwards for simplicity.
Prerequisites
- DigitalOcean Droplet/Droplets
- DigitalOcean Kubernetes Cluster
- Access to both Droplets and Cluster via SSH and Kubectl
DigitalOcean VPC Gateway Configuration
In a simple VPC config we have 3 components. First we have the VPC (Virtual Private Cloud) it self which belongs to a location provided by DO. Next we have two droplets belongs to the same VPC (Droplets inside the same VPC share a private network with associated private IPs). First we have our gateway droplet and then we have our backend droplet or multiple backend droplets according to your requirement. There is no configuration needed for VPC after creation. But configuration is involved in rest of the components.
Gateway Droplet Configuration
There are two steps in configuring the gateway droplet. First we need to enable IP forwarding using the command
sysctl -w net.ipv4.ip_forward=1To persist this change after reboot we need to add net.ipv4.ip_forward=1 to /etc/systcl.conf
Next we need to configure Network address translation (NAT) to route traffic between our backend resources and public internet. Using the following iptables command we can configure the gateway droplet to translate traffic from our VPC network’s subnet to the public IP of the gateway droplet. Replace the <vpc_network_prefix> with our VPC Network private IP prefix. (Looks something like 10.118.0.0/20 right next to our VPC name)
iptables -t nat -A POSTROUTING -s <vpc_network_prefix> -o eth0 -j MASQUERADEThe default name of the public network interface for DO droplet is
eth0. If you have changed this, useip -br acommand to get the public interface and replaceeth0with the new interface name.
To persist these changes after rebooting we have install iptables-persistent
sudo apt-get install iptables-persistentSelect “Yes” when iptables-persistent asks if you want to save the current IPV4 rules.
Backend Droplet Configuration
Now for the backend droplets. Since we are going to redirect backend droplet internet traffic through the gateway droplet, we have to login to the backend droplet through gateway droplet to avoid disconnection when we update the IP routes.
ssh -o ProxyCommand="ssh -W %h:%p root@<public_IP_of_gateway_Droplet>" root@<private_IP_of_backend_Droplet>Here we have to add/update two routing table entries. First we need to find out the default gateway of our droplet and add an IP route to the network configuration to retain the droplet’s access to the metadata endpoint(169.254.169.254) provided by DigitalOcean.
curl -s <http://169.254.169.254/metadata/v1/interfaces/public/0/ipv4/gateway>This command returns the current default gateway IP address of the droplet. Use the IP from the above command to replace <default-gateway-ip> when running following command.
ip route add 169.254.169.254 via <default-gateway-ip> dev eth0 # returns nothing as outputNext we can change the default gateway for the outward traffic to our gateway droplet by running the following command. (Private IP of the droplet will be available in the droplet DO dashboard)
ip route change default via <private_IP_of_gateway_Droplet>Next we need to edit the /etc/netplan/50-cloud-init.yaml file to persist the changes after reboot. We have to remove the routes portion from the eth0 interface and add the new route to the eth1 interface.
network:
version: 2
ethernets:
eth0:
addresses:
- 137.184.174.21/20
- 10.20.0.7/16
match:
macaddress: 86:bc:46:82:4d:95
mtu: 1500
nameservers:
addresses:
- 67.207.67.3
- 67.207.67.2
search: []
# remove the following
routes:
- to: 0.0.0.0/0
via: <original gateway address>
# till here
set-name: eth0
eth1:
addresses:
- 10.118.0.4/20
match:
macaddress: 5a:59:57:e0:f3:9d
mtu: 1500
nameservers:
addresses:
- 67.207.67.3
- 67.207.67.2
search: []
routes:
- to: 0.0.0.0/0
via: <gateway droplet private ip address>
set-name: eth1After the changing the network file. Run netplan apply to apply the changes. Run curl -4 <https://icanhazip.com/> to verify if the traffic is being routed through the gateway. Please remember that you will have to use the proxy ssh to access the backend droplet from now onwards.
DigitalOcean Kubernetes Egress Configuration.
To configure egress inside DOKS we need to two things. First we need a VPC gateway droplet configured. Checkout the previous section to see how to do that and come back once it’s configured properly. Next we need to configure our K8s cluster to route network traffic through previously configured gateway. Under the hood the process is really similar to how we did things in the simple gateway config with backend droplets.As you can see in the below image you can configure both droplets and k8s cluster to route your traffic through a single VPC gateway. This is very helpful and DigitalOcean doesn’t provide any simple tool to achieve this like other cloud providers. But it’s achievable through some configuration.
Static Routes Operator
DigitalOcean has released Static Routes Operator CRD for updating the linux routing table of each worker nodes inside our cluster. Once installed, it deploys the static route operator daemonset in each worker node to manage the linux routing table. As you can see this is very similar to the backend droplet configuration in the previous section.
Deploy Static Routes Operator by running the following command. Also check github release page to find the lates version of Static Routes Operator.
kubectl apply -f <https://raw.githubusercontent.com/digitalocean/k8s-staticroute-operator/main/releases/v1/k8s-staticroute-operator-v1.0.0.yaml>After deploying check pods inside static-routes namespace to see if everything is up and running.
kubectl get pods -n static-routesOutput should look similar to:
NAME READY STATUS RESTARTS AGE
k8s-staticroute-operator-77h4g 1/1 Running 0 28d
k8s-staticroute-operator-cxdfn 1/1 Running 0 28d
k8s-staticroute-operator-f558l 1/1 Running 0 28d
k8s-staticroute-operator-hdfv4 1/1 Running 0 28dNow Static Routes Operator is deployed and ready to use. Next we can create a static route to our gateway. Use the following StaticRoute config file provided by DigitalOcean and replace the gateway with our gateway private IP address. We are using public CIDRs to redirect our egress traffic through the gateway, but there’s caveat with this approach. Like in the backend droplet metadata API there are internal services inside k8s cluster that need access to the DigitalOcean APIs. (I assume that those APIs are using the IP of the droplet to identify the source and provide response accordingly and routing traffic through the gateway could break this functionality. This is purely my assumption.) So the solution is to avoid the CIDRs that overlap with DO Public API endpoints. These are already commented out in the default configuration provided by DO.
apiVersion: networking.digitalocean.com/v1
kind: StaticRoute
metadata:
name: public-egress
spec:
destinations:
- "0.0.0.0/5"
- "8.0.0.0/7"
- "11.0.0.0/8"
- "12.0.0.0/6"
- "16.0.0.0/4"
- "32.0.0.0/3"
# - 64.0.0.0/2 NOT TO BE USED! Overlaps with DO API endpoints.
# - 128.0.0.0/3 NOT TO BE USED! Overlaps with DOKS API endpoints.
- "160.0.0.0/5"
- "168.0.0.0/6"
- "172.0.0.0/12"
- "172.32.0.0/11"
- "172.64.0.0/10"
- "172.128.0.0/9"
- "173.0.0.0/8"
- "174.0.0.0/7"
- "176.0.0.0/4"
- "192.0.0.0/9"
- "192.128.0.0/11"
- "192.160.0.0/13"
- "192.169.0.0/16"
- "192.170.0.0/15"
- "192.172.0.0/14"
- "192.176.0.0/12"
- "192.192.0.0/10"
- "193.0.0.0/8"
- "194.0.0.0/7"
- "196.0.0.0/6"
- "200.0.0.0/5"
- "208.0.0.0/4"
gateway: "<YOUR_EGRESS_GW_DROPLET_PRIVATE_IP_HERE>"Save the default config as public-egress-example.yaml and run kubectl apply -f public-egress-example.yaml to deploy the static route. verify the deployment by running kubectl get staticroutes. Now you can ssh into any of the pods and run curl -4 <https://icanhazip.com/> to verify if the traffic is being routed through the gateway. Please note that if your traffic is not being routed to the gateway for a specific IP. That IP could be missing from the list of destination. You could add the CIDR or the IP to the list and it will be routed as expected. You can see all the IP ranges used by DigitalOcean here and here. Restrain from using these IP ranges in your configuration.
Conclusion
Static IP is a requirement when it comes to normal server instances as kubernetes is widely used these days, the same is required in kubernetes. Although there is no simple built in solution for that in DigitalOcean, it is possible with some configuration. It can be little more hands-on than other cloud providers but I prefer to do it this way for more control. If you are only looking to do this setup with your K8s cluster there is a simpler method where infrastructure orchestration tool like crossplane is deployed inside your cluster and used to interact with DigitalOcean to create gateway droplet as a k8s resource, where you can manage the droplet through k8s and kubectl. I prefer the non-crossplane way to avoid deploying crossplane in my cluster. You can find the DO documentation for crossplane setup here. Also there’s another extra layer you could add to make sure that static IP will never change. You can configure the gateway droplet to send outbound traffic through the reserved IP (formerly known as floating IP). Check out this documentation from DigtialOcean to do that. Please note that static IP does not support certain type of traffic such as SMTP. Please raise a ticket if you are having trouble with outbound traffic after configuring static IP.
