Dual Stack Lite (IPv6-only PPPoE) on an UniFi Security Gateway

2018-06-01

TL;DR

Enter your PPPoE credentials and set up IPv6 according to the UniFi IPv6 guide. Copy listing 2 to /config/scripts/ppp/ipv6-up.d/ipv6-only-pppoe-workaround, change the AFTR address accordingly and make it executable with chmod +x /config/scripts/ppp/ipv6-up.d/ipv6-only-pppoe-workaround. Reboot your USG and you should be done. Please mind the drawbacks.

How it works

As the UniFi system does not support Dual Stack Lite by default and I could not find any solutions in the community forum I started by looking at how Dual Stack Lite actually works. RFC 6333 is a very good starting point with a detailed description. Basically, DS Lite is built on IPv4-in-IPv6 tunnels. The customer element is called Basic Bridging BroadBand (B4) element whereas the provider element is referred to as Address Family Transition Router (AFTR).

The B4 element initiates the 4in6 tunnel to an AFTR IPv6 address given by the provider. Inside the tunnel, the AFTR element always has the address 192.0.0.1. By default, the B4 element uses the address 192.0.0.2 but can also select another address from the 192.0.0.0/29 subnet if required. After the tunnel is established an IPv4 default route via the AFTR is set. Great! That’s all you need to know from a customer side.

IPv6 setup

Before we can set up the 4in6 tunnel there must be a native IPv6 connection present. I started by entering the PPPoE credentials and then followed the UniFi IPv6 guide. If you are unsure about the Prefix Delegation size ask your ISP or plug in a DS Lite capable router (e.g. an AVM Fritz!Box) and scan the event log after the connection is established. The assigned prefix including its CIDR length should appear in the log. A very common value is /56, you can try that first if you do not want to fiddle with it.

Unfortunately, nothing worked. I started by examining the PPPoE connection. The USG uses the very common PPP daemon. The connection attempts are logged at /var/log/vyatta/ppp_pppoe0.log. I quickly realized that at least the PPPoE connection was successfully established. I just did not receive an IPv6 prefix. Fortunately, there is also a log file for the DHCPv6 client at /var/log/dhcp6c.log which contained the following information:

Jun/01/2018 15:38:20: ifreset: invalid interface(pppoe0): No such device
Jun/01/2018 15:38:20: main: failed to initialize pppoe0

As you can see, the DHCPv6 client expects the PPPoE interface at pppoe0. However, the interface still got its default name ppp0 and was not renamed. The PPP daemon executes scripts under /etc/ppp/ at certain events which can be found in the Scripts section of the manual. The directory /etc/ppp/ip-pre-up.d/ contains a script called 0002rename-pppoe which is apparently responsible for renaming the ppp0 interface. It is only executed if an IPv4 address has been assigned via the PPPoE connection which is not the case with DS Lite.

Since there is no corresponding IPv6 counterpart we have to come up with something else. The /etc/ppp/ipv6-up script is executed when the interface is already up and the link is available. This is not the best place to rename an interface because it is messing with the PPP daemon as we will see later but it is our best bet. As the sixth argument, the PPP daemon passes the ipparam value specified in the configuration which, in our case, corresponds to the desired interface name. First, we start the script that renames the interface, bring it back up and restart the DHCPv6 daemon. By placing our script at /config/scripts/ppp/ipv6-up.d/ we ensure that it remains intact even after a firmware upgrade. Also do not forget to make it executable with chmod +x /config/scripts/ppp/ipv6-up.d/ipv6-only-pppoe-workaround.

/config/scripts/ppp/ipv6-up.d/ipv6-only-pppoe-workaround

#!/bin/sh

INTERFACE="${6//[[:blank:]]/}" # Remove space at the end

/etc/ppp/ip-pre-up.d/0002rename-pppoe $*
ip link set dev $INTERFACE up

# start dhcpv6
if pidof dhcp6c; then
    dhcp6ctl start interface $INTERFACE
else
    /etc/ppp/ip-up.d/dhcpv6-pd-ppp $INTERFACE
fi

Following a reboot, the first part is done and you should now have an IPv6 connection. All we have to do now is create the 4in6 tunnel and change the default route. This could theoretically be done using a custom config.gateway.json on the controller but this is not practical since our IPv6 prefix changes every time we dial in and we would have to constantly adapt the local IPv6 address of our tunnel.

IPv4 setup

After we have instructed the DHCPv6 client to renew, we wait until an IPv6 address has been assigned to an interface. If that has not happened within two minutes we give up. If successful, we create the 4in6 tunnel using our local address and the AFTR address specified by our provider. Then we, the B4 element, assign ourselves the IPv4 address 192.0.0.2. Eventually we instruct Quagga to forget the IPv4 default route through the pppoe0 interface and use our v6tun0 interface instead. The complete script looks like this:

/config/scripts/ppp/ipv6-up.d/ipv6-only-pppoe-workaround

#!/bin/sh

AFTR_FQDN="<AFTR_FQDN>"
# AFTR might only resolve via provider resolver
RESOLVER="2620:fe::fe"

INTERFACE="${6//[[:blank:]]/}"
PATH=$PATH:/usr/bin

/etc/ppp/ip-pre-up.d/0002rename-pppoe $*
ip link set dev $INTERFACE up

# start dhcpv6
if pidof dhcp6c; then
    dhcp6ctl start interface $INTERFACE
else
    /etc/ppp/ip-up.d/dhcpv6-pd-ppp $INTERFACE
fi

LOCAL=""
COUNT=0
while [ -z "$LOCAL" ]; do
    sleep 5
    LOCAL=$(ip -6 addr | grep inet6 | awk -F '[ \t]+|/' '{print $3}' | grep -v '^\(::1\|fe80\)' | head -n1)

    (( COUNT++ ))
    if [ $COUNT -ge 120 ]; then
        touch /tmp/v6up_failed
        exit 1
    fi
done

# setup AFTR tunnel
AFTR_ADDRESS=$(host -t AAAA $AFTR_FQDN $RESOLVER | awk '/has IPv6 address/ { print $5; exit }')

ip -6 tunnel del v6tun0 >/dev/null 2>&1
ip -6 tunnel add v6tun0 mode ipip6 local $LOCAL remote $AFTR_ADDRESS encaplimit none
ip link set v6tun0 up
ip addr add 192.0.0.2/29 dev v6tun0

# firewall rules                                  
## masquerading (optional)
iptables-save -t nat | sed -nr '/POSTROUTING .*match-set (corporate_network|remote_user_vpn_network|guest_network) src .* MASQUERADE/s/-A/iptables -t nat -D/e'

RULE_NUM=$(iptables -t nat -S POSTROUTING | wc -l)                  
RULE_NUM=$(expr $RULE_NUM - 1)

iptables -t nat -I POSTROUTING $RULE_NUM -o v6tun0 -m set --match-set corporate_network src -j MASQUERADE      
iptables -t nat -I POSTROUTING $RULE_NUM -o v6tun0 -m set --match-set remote_user_vpn_network src -j MASQUERADE
iptables -t nat -I POSTROUTING $RULE_NUM -o v6tun0 -m set --match-set guest_network src -j MASQUERADE

## MSS clamping
iptables -t mangle -A UBNT_FW_MSS_CLAMP -o v6tun0 -p tcp -m tcp --tcp-flags SYN,RST SYN -j TCPMSS --set-mss 1412
iptables -t mangle -A UBNT_FW_MSS_CLAMP -i v6tun0 -p tcp -m tcp --tcp-flags SYN,RST SYN -j TCPMSS --set-mss 1412

## hooks
iptables-save -t filter | sed -nr '/VYATTA_FW_(IN|LOCAL|OUT)_HOOK .* WAN_(IN|LOCAL|OUT)/s/-A/iptables -t filter -D/e'

iptables -t filter -A VYATTA_FW_IN_HOOK -i v6tun0 -j WAN_IN
iptables -t filter -A VYATTA_FW_LOCAL_HOOK -i v6tun0 -j WAN_LOCAL
iptables -t filter -A VYATTA_FW_OUT_HOOK -o v6tun0 -j WAN_OUT

# IPv4 default route
vtysh -c 'configure terminal' -c "no ip route 0.0.0.0/0 $INTERFACE" -c 'ip route 0.0.0.0/0 v6tun0'

Drawbacks

Unfortunately, this solution also has some disadvantages, which I will briefly discuss below.

Hardware offloading

There is no hardware offloading for 4in6 tunnels, so all IPv4 traffic that passes the tunnel cannot be offloaded. A 100 Mbps link can be maxed out. For untagged IPv6 traffic, the offloading works fine.

Deep Packet Inspection (DPI)

Deep Packet Inspection only works with NAT and hardware offloading turned off. With hardware offloading enabled, almost no traffic is classified.

PPP daemon log spam

By renaming the ppp0 interface after it has already been brought up, the PPP daemon gets confused and fills the log file with Couldn't get PPP statistics: No such device messages. This can blow up the log file in continuous operation and should be filtered. I published a patched pppd on my GitHub profile which supports the ifname option (click). Additionally, a Vyatta config template needs patching after each firmware upgrade:

sudo sed -i '/\tsudo sh -c \"echo ipparam/a sudo sh -c \"echo ifname \\\\\\\"pppoe\$VAR(@)\\\\\\\" >> /etc/ppp/peers/pppoe\$VAR(@)\"' /opt/vyatta/share/vyatta-cfg/templates/interfaces/ethernet/node.tag/vif/node.tag/pppoe/node.def