The Ubiquiti Diaries: A Site-to-Site VPN Story
by Ganesh T S on December 21, 2022 8:00 AM EST- Posted in
- Networking
- Ubiquiti Networks
- UniFi
- VPN
The CGNAT Spanner in the Works
Carrier-Grade NAT (CGNAT) is used by ISPs to circumvent the lack of IPv4 addresses available for allocation to their customers. As more and more devices come online and connect to the Internet directly, it is not surprising that there is a dearth of IPv4 addresses to hand out. CGNAT is mostly encountered with WAN connections enabled by mobile service providers (think of smartphones connecting to a 4G or 5G network). However, many wired ISPs (such as Airtel in India) have also started rolling out CGNAT. The easiest way to identify if one is behind a CGNAT is to search for one's IP address on Google. Google helpfully displays your public IP address as the first result. If this address is different from the WAN IP reported by your gateway (and that is usually between 100.64.0.0/16 and 100.127.0.0/16), you are likely behind a CGNAT. This is different from a local IP gathered by a gateway placed behind ISP-supplied equipment that acts as a DHCP server itself.
CGNAT is rarely a problem for regular web browsing and content consumption. Problems start cropping up when dealing with services that require opening up ports on the gateway for bidirectional communication - such as VoIP services, multi-player gaming, and VPN setups. Even simple things taken for granted like systems in the network being able to communicate with a remote NTP server (for time synchronization) are not possible. Since the NAT happens at the ISP end, it is not possible for port forwarding to be configured either. Some services manage to find their way around CGNAT using STUN / TURN, but power users end up finding one roadblock after another. On the whole, CGNAT is a pain in the neck for users accustomed to obtaining public-facing WAN IPs.
NTP Synchronization Attempt behind CGNAT, and the 'Solution' (Using a 'Local' NTP Server)
Ideally speaking, configuring the Site-to-Site Manual IPSec VPN on the USG Pro 4 (having a public WAN IP) with a remote server address of 0.0.0.0, and providing the USG Pro 4's WAN IP as the remote server address on the UDM side (behind the CGNAT) should have worked by allowing the CGNAT side to initiate the site-to-site connection with the server. I did try out this suggestion made by an Ubiquiti employee many years ago - however, the initial key exchange / handshake process appeared to be break down with the response from the USG Pro 4 apparently getting dropped by the UDM's ISP.
An online search for enabling site-to-site VPNs with one site behind a CGNAT brought up two sets of results - one suggesting a connection via an intermediate VPS (cloud server hosted by a 3rd party), and the other trying to make the public WAN IP-possessing side to act as an OpenVPN server and the CGNAT side to act as a client and initiate the connection. I was not interested in bringing a VPS into the picture, and my trials with OpenVPN did not yield any promising results.
At this point, I realized that my Android phone was able to open up a Teleport connection with the UDM despite the gateway going behind CGNAT. So, my first attempt was to get a system to connect to the Teleport VPN and route traffic from specific devices through the system's Teleport VPN connection.
35 Comments
View All Comments
Zoolook - Monday, December 26, 2022 - link
Dead? Yeah sure.It's only up to 40% of users now on a global scale and the increase is a steady 5% (of total users) y/y increase, so I'm sure it's about to roll over and go away any time now.
Just because your ISP is shielding you so far doesn't mean it's not there.
PeachNCream - Monday, December 26, 2022 - link
Don't buy into the hype kids. Ubiquiti's best selling point is that it looks pretty when its sitting on the rear seat of your car as you drive it to the nearest electronics recycling facility.egc - Tuesday, December 27, 2022 - link
Both DDWRT and OpenWRT support Wireguard (and of course OpenVPN) it is very easy to setup a site-to-site setup.Runs on relative cheap hardware e.g. NetGear R7800 with WireGuard speeds of about 300 Mb/s
Threska - Wednesday, December 28, 2022 - link
Asus-merlin does as well. Having a choice helps with different situations.https://www.top10vpn.com/guides/wireguard-vs-openv...
zanon - Saturday, December 31, 2022 - link
Ubiquiti sadly become a development dumpster fire and been going downhill internally for a while. I really REALLY wouldn't recommend them for a greenfield deployment these days. In particular their gateway/services has sucked badly for a long time and I dumped it for OPNsense a few years ago, now starting the move to Omada for switching/WiFi. Wireguard is excellent and reliable, though a lot of people may find Nebula a better match since while still using Noise it does true meshing. A fixed IP (a small light VPS will work fine) is only necessary to help clients find each other and coordinate, then the clients talk directly after. Still fully self hosted, no 3rd party service needed.