The Ubiquiti Diaries: A Site-to-Site VPN Storyby Ganesh T S on December 21, 2022 8:00 AM EST
- Posted in
- Ubiquiti Networks
Ubiquiti Networks is a popular vendor of networking-related equipment in the SMB / SME space. Their gear is immensely popular among prosumers too, thanks to the combination of ease of use and the ability to customize for specific requirements. I have been running an Ubiquiti UniFi installation at home for the last five years or so, and recently had the opportunity to create a new deployment in another country. There were two main reasons to go with Ubiquiti for the new location - a single management plane for both sites, and the ability to easily create a site-to-site VPN.
The new installation was fairly smooth and the site-to-site VPN was up and running in a stable manner until the ISP at the remote site moved the gateway from a public-facing WAN IP to one behind a carrier-grade NAT (CGNAT). That started a deeper investigation into various options available for site-to-site VPNs with Ubiquiti's gear for different scenarios. In this process, I ended up encountering a host of issues worthy of documentation to help folks who might encounter them in their own installations. This article provides a recount of my trip down the rabbit hole - including a step-by-step guide detailing my attempts to work around the various pitfalls.
Ubiquiti Networks offers a range of products targeting the networking market. While wireless ISPs are a key market segment for the company (serviced by the airFiber line), today's piece is focused on their UniFi product line - a range of managed software-defined networking equipment for SMBs, SMEs, and prosumers. There are a number of reasons for UniFi's popularity products among tech-savvy consumers. The company had a first-mover advantage in offering a cost-effective managed SDN solution. Isolating functionality into different devices (security gateways, routers, switches, and wireless access points) allowed users to pick and choose different equipment based on their custom needs. The unified management plane for all the UniFi products enables easy maintenance while retaining deployment flexibility. Network scaling in response to requirement changes is also straightforward. The company started out with a local management controller, which has now been augmented with a cloud-based offering.
My first brush with Ubiquiti was their mFi product line (which has since been unfortunately EOL-ed). Their lineup of network-connected power outlets with energy and power monitoring, as well as remote relay control was (and continues to be) more flexible than anything else in the market - and this was without even taking the low pricing into account. I had purchased a few of their units for my home / AnandTech testing lab use, and written a short review after a couple of months of use (those units are still in deployment).
After I published the mFi review, Ubiquiti's PR department approached me with an offer to review their UniFi product line. Around that time back in 2017, I had the opportunity to lay out a wired Cat 6 backbone for all the rooms in my house here in California. I took up the offer to spec out a UniFi system for testing out. The USG Pro 4 gateway took up the routing duties with a UniFi Cloud Key (first generation) performing controller duties. Access points with varying capabilities were mounted around the house to avoid wireless dead-spots. A number of switches were placed in the media center and different lab locations. I ended up augmenting the system with additional PoE switches and in-wall APs on my own.
The system was configured with the usual guest wireless network, and a bunch of different VLANs (serving the IoT devices in the house, the home lab equipment, and another for devices such as the common family desktop, phones, etc.). On the whole, it was an overkill for a residential installation. That said, the deployment has held its own over five years of stressful usage (and still going strong). The only hiccup I had was when the CloudKey controller became inaccessible on the network a couple of years back. It turned out that a power interruption had ended up corrupting the database - nothing that a few SSH commands (thanks to the helpful community) couldn't resolve. Since then, I ended up investing in a UPS for the rack holding the UniFi equipment to avoid the recurrence of such scenarios.
Such issues are also the reason why I recommend Ubiquiti equipment only to tech-savvy users. In almost all cases, calling up the company's support line and creating a ticket ends up being a waste of time. There are innumerable resources online (both the company's own users forum, as well as countless prosumer bloggers such as Scott Hanselman and Troy Hunt. In light of reviews from such sources, there is not much for readers to gain from posting yet another review of the Ubiquiti UniFi lineup. Instead, I am hoping to take up specific use-cases and figure out how Ubiquiti's product lineup can address those in these series of articles.
Earlier this year, my parents back in India decided to downsize their home. I took the opportunity to revamp their home network from the ground-up. I had been intending to add features to the home network of my parents, but had never had the opportunity because my visits were becoming infrequent. However, with my first visit post-pandemic, I wanted to get a few things set up as part of their move:
- Easier remote management and troubleshooting of network issues without the need for port forwarding.
- Ability to seamlessly use their Indian home network during travel / visits over here to California
- Ability to perform secure remote offsite backups for my data without relying on an external cloud storage provider
- Ability to seamlessly utilize Indian OTT service subscriptions irrespective of user location either in California or in India
When I initially set up the Cloud Key back in 2017, there was no requirement to use a cloud account. Unfortunately, the UniFi Network mobile application user experience became quite onerous without a ui.com ID a couple of years back. I caved in and ended up associating my installation with a cloud ID just for this purpose. Since I was already managing my network through this ID, it became a straightforward decision to go with Ubiquiti for the deployment back in India.
The key to fulfilling the above requirements was a secure VPN tunnel between my home network here in California and my parents' network in India. Prior to traveling, I arranged for a Ubiquiti Dream Machine to be delivered to the new home. The Ubiquiti UniFi Dream Machine is an all-in-one solution / UniFi starter kit. It integrates a 4-port switch, a 4x4 802.11ac access point, a security gateway, and an integrated controller. The Annapurna Labs AL314-based solution comes with a single WAN port, and is an acceptable solution for most home networks in the the 1000 sq. ft - 1200 sq. ft range.
From my use-case perspective, I wanted a solution that would support simple VPN tunnel configuration and easy app-based access for both the US and Indian networks via a single interface.
The Evolution of UniFi - A Short Recap
Ubiquiti's UniFi lineup was launched after their lineup of edge-focused products for WISPs started gaining traction in other markets. These EdgeRouters and EdgeSwitches were based on Vyatta OS, and the UniFi products initially started out with the same EdgeOS firmware base. The UniFi Security Gateway Pro 4 in my primary deployment runs EdgeOS to date.
The USG Pro 4 is based on Cavium's OCTEON II networking SoC, with a MIPS64 application processor. However, Ubiquiti's latest gateways / routers / switches in the UniFi lineup now run a custom Debian-based Linux distribution. The UniFi Dream Machine uses the Annapurna Labs AL314, and runs a distribution meant for the AArch64 platform. The UniFi OS itself runs as a container using podman.
The end result is that there are quite a number of disconnects between the features available on EdgeOS and UbiOS / UniFi OS. Migration from the EdgeOS line to UniFi OS is not straightforward enough for heavily customized installs. With focus shifting to UbiOS / UniFi OS, the updates for the older equipment have become few and far apart. While that might not be a concern for stable networks, it has unfortunately not kept up to date with evolving network security practices. For example, Android's recent releases have completely dropped support for L2TP VPNs, while EdgeOS has L2TP as the recommended VPN server type. This brings us to the topic of VPNs.
VPN Server Options in Ubiquiti's Stack
Ubiquiti offers a range of VPN options depending on the gateway being used. At home here in California with the USG Pro 4, I have been running a L2TP VPN server (allowing me to connect to it from public coffee shops and airports for secure browsing purposes) for several years now. I had minimal trouble setting it up for access from a Windows notebook. However, as mentioned in the previous sub-section, this VPN server is of no use for my mobile phone running Android 12. The USG Pro 4 also supports PPTP VPN, but it is not recommended even by Ubiquiti themselves.
The primary option for a VPN server in the UniFi Dream Machine running UbiOS / UniFi OS is quite different.
Here, Teleport (Ubiquiti's customized Wireguard implementation) takes precedence. This is a one-click VPN more in tune with today's mobile-first ecosystem. Clients are authorized via invites that can be generated either from the configuration page (on the unifi.ui.com cloud, or via the machine's local IP) or the UniFi Network mobile app. The invites can be opened on the client device using the Wifiman mobile application. The unfortunate aspect here is that Windows users are out of luck. While MacOS, Android, and iOS are covered, Windows users are left in the lurch. This is a hugely disappointing situation given that the L2TP option in EdgeOS works with Windows clients, but not Android and the Teleport option in UbiOS / UniFi OS works with Android clients, but not Windows. It must be noted that the UDM still supports L2TP for Windows clients.
Under the Teleport & VPN section, Ubiquiti also provides an option to create site-to-site VPNs, which is where our story starts.
Post Your CommentPlease log in or sign up to comment.
View All Comments
prophet001 - Wednesday, December 21, 2022 - linkNot really a ubiquiti fan.
Threska - Wednesday, December 21, 2022 - linkUbiquiti vacuum.
OddballSix - Wednesday, December 21, 2022 - linkThere's no point in even talking about Ubiquiti, you can't buy most of their products. Some of them have been out of stock in the entire channel for months.
Entire parts and lines of products gone. You can't buy them. One breaks? You're screwed. Need to upgrade the firewall? Tough.
HalcyonDays - Wednesday, December 21, 2022 - linkI actually went down a similar path as you did. Years ago, when I moved out, I needed away to troubleshoot my parents network remotely when I inevitably get the dreaded phone call "internet is not working".
My requirements for this setup are as follows:
1. Bidirectional encrypted tunnel(s) - preferably peer-to-peer
2. No third-party cloud services
3. Each site access internet through their its own ISP
4. Router at each site will handle the VPN connection - no additional hardware
After attempting and investigating multiple methods, I eventually settled on "tinc" based on the suggestion from the openwrt forums.
"tinc" is a peer-to-peer VPN supported by Tomato, Openwrt, and asuswrt-merlin. It doesn't need all sites to have public IP to work. It just need one site to have public IP (I think). To handle dynamic IPs, I use a free DDNS service and assign a domain name to each of the site.
Since then, I have expanded VPN network to include the in-laws and parents' home in Taiwan. It just required the router at each site to have the public key of at least one other site and it'll be able to see all sites. This means that I can be at any of these sites and still see every site.
Some caveats: I am uncertain of the performance. From what I can tell, "tinc" is pretty lightweight but not as performant as wireguard. Because I don't stream anything over tinc tunnels, I can't vouch for how well it works for for that.
Give it a try.
Samus - Thursday, December 22, 2022 - linkAmazing hardware and stability totally ruined by crap software. The controller is trash. Relying on Java is already a red flag but the way the controller database functions is bazaar and totally insecure. Inheriting\adopting hardware into a new instance results in a mandatory config wipe. No fortune 500 or enterprise network would use this so what really separates it from a $100 consumer product? A consumer product that often has more basic functionality; Ubiquiti has to this day failed to implement MAC cloning, axing it from consideration to anybody who has AT&T or Verizon fiber that need to emulate their gateway from the ONT\media converter. Such a basic feature dating back to the Linksys routers of the 90's missing from a $300 prosumer product is embarrassing and should alone put the company underwater. I mean why?
Hamm Burger - Thursday, December 22, 2022 - linkMy ISP provides CGNAT by default, but one can pay extra (€1.95/month afair) for a non-fixed but routable address. Which I do. Of course, you have to to know that you can ask, because they don't advertise this feature.
Samus - Saturday, December 24, 2022 - linkThat is cheap. Commercial block IP's are rarely offered in the US to residential subscribers, and even 'business' internet plans find a way to screw you out of $15 minimum for a "usable" static address.
It's worth noting over the years I've seen most IP addresses - even for residential internet - have become statically assigned to subscribers, but they are non-routable.
ballsystemlord - Thursday, December 22, 2022 - link@Ganesh , why not just contact the ISP and tell them that you were paying for an IP address that is *not* behind CGNAT? I mean, if you're spending the money for the IP you should get it.
Jorgp2 - Thursday, December 22, 2022 - linkYup, or just pay for a /29 or something.
coburn_c - Thursday, December 22, 2022 - linkipv6 is dead and rightly so