New
EuroGPT Enterprise is open source, runs in Europe, and keeps your data private. Try it now
*excluding tests 🙂
At Ubicloud, simplicity is a core value. Our networking infrastructure is built using just four open source components:
With these, we provide a wide range of networking features. These include public IPv4 and IPv6, private networking, firewalls and load balancers.
Our recent integration with Kamal enabled new use cases, pushing us to refine how we manage private subnets and firewalls.
Private Networking: We first assign a private subnet (/26 for IPv4 and a /64 for IPv6) per private network. A private subnet is only known to the VMs in the same private network and specifically configured so that the VMs can communicate in an encrypted, private way. We provide that by creating IPsec Tunnels in tunnel mode between resources within the same subnet. We rely on IPv6 for internal communication, using a dedicated prefix that simplifies tunnel mode operations.
Firewalls: We implement firewalls using Linux nftables. For simplicity, firewalls operate at the private subnet level. In essence, a private subnet is a security perimeter, and all resources inside it share the same firewall rules.
While this model works for many scenarios, customers often requested more granular control over resource communication inside private subnets. A typical use case might involve:
However, since all resources in a private network share the same firewall rules, our existing model couldn’t fully address this need. Customers began placing resources into separate subnets to enforce different firewall rules. But this led to a new problem: they couldn’t privately connect these separate subnets.
To solve the problem of running VMs and databases in isolated private networks, we needed to implement private network peering. We decided to use the “connected subnets” terminology, because that explains the situation better. Essentially, connected subnets are 2 private subnets that their resources are able to communicate with each other using their private ip addresses in an encrypted way.
Our private connections already use the following approach:
Private network peering could follow a similar process. By creating an IPsec tunnel between resources in different subnets, we could enable private communication, reusing our existing mechanisms. So, essentially, our latest architecture can be summarized with the following diagram:
In short, we started creating Tunnel 2 and 3 and it just worked!
To implement private network peering, we needed to:
This was straightforward. We use PostgreSQL for our backing database and Ruby Sequel for interaction. Here’s the commit that introduces the connected_subnets entity in 16 lines.
Creating IPsec Tunnels
Establishing and managing tunnels between resources in connected subnets was handled in just 52 lines of code, plus 100 lines of tests, we have 100% unit test coverage, so 🙂. Here’s the commit.
Integrating with Provisioning/Deprovisioning
To make private network peering seamless, we integrated it into the resource provisioning and deprovisioning workflows. A key challenge was ensuring that rekeying occurs in an online, uninterrupted manner across the entire graph of connected subnets. By reusing our existing implementation, we were able to address this efficiently. The entire process —covering provisioning, deprovisioning, and rekeying— was completed in just 6 lines of net code addition, supported by net 17 lines of test addition. For a detailed look, you can check out the commit here.
UI Updates
The UI changes required 103 lines of net code addition and the tests took an additional 38 lines of code. The commit is here.
E2E tests
The E2E tests costed us 224 lines of net code addition. Since we have the policy of 100% unit test coverage, we wrote unit tests against our E2E tests as well; and that added 336 lines more. The commit is here.
In total, we added 177 lines of code, excluding tests. We had set ourselves a challenge of implementing private network peering in under 200 lines; and we succeeded 🙂. We could do this because we had spent a lot of effort in keeping our private networking implementation simple. So, extending it for peering became a trivial task.
