UniFi Security Gateway: Overcoming IPv6 hardware offloading limitations

2019-02-16

In UniFi Security Gateway: Dual Stack Lite (IPv6-only PPPoE) we learned how to use the UniFi Security Gateway on a dual stack lite or IPv6-only PPPoE internet connection.
As mentioned in the drawbacks, IPv4 offloading on the USG is not possible as we have to push all IPv4 data through an ipip6 tunnel. This post is not about offloading IPv4 traffic but IPv6 traffic with special requirements.

If your service provider requires you to tag your traffic with a VLAN ID and at the same time authenticate using PPPoE, the USG will not be able to offload IPv6 traffic. A (most likely software) limitation will prevent it from offloading the PPPoE and VLAN encapsulation at the same time. With IPv6, only one of the two encapsulation layers can be offloaded, but not both together. This limitation does not exist for IPv4.

We will reuse the Optical Network Terminal (ONT) of our fibre connection for VLAN tagging.

Requirements

To offload all IPv6 traffic I will use a service provider supplied Huawei EchoLife HG8012H for VLAN tagging. Any model of this series should work such as HG8010H or EG8010H.
The EchoLife ONT supports adding and removing VLAN tags in hardware. On our USG we can then offload the IPv6 PPPoE encapsulation.

For this to work, the user name and password for the ONT web interface must be known. Most providers do not modify the default credentials.

The EchoLife web interface

By default the EchoLife ONT listens on 192.168.100.1 on the local WAN ethernet port. With the USG connected to the ONT we can add a masquerading rule on the USG to get access to the ONT without much effort:

iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE

On the web interface at 192.168.100.1 we can try the default user name telecomadmin and default password admintelecom. A successful login will display the Device Information page.
On the LAN tab the subnet of the ONT can be changed and on the Security tab a MAC filter can be enabled for the local WAN ethernet port. Unfortunately there is no option to set a default VLAN ID for the local WAN port.

From the publicly available operator documentation we know that it is possible to set a default VLAN through the OLT with the command ont port native-vlan 1 0 eth 1 vlan <VLAN_ID>. As an end user we do not have access to the service providers OLT so we have to find a different solution.

On the System Tools tab downloading and uploading the configuration and upgrading the firmware is possible. After decrypting the configuration file and playing around I came to the conclusion that there is no field to set a local VLAN ID. I had to go deeper and take a look at the firmware of the HG8012H.
Eventually the OLT must provision the native VLAN somewhere on the device.

The EchoLife telnet interface

By modifying the configuration file or by using a Huawei support tool, telnet access can be enabled. Telnet will drop to a very limited shell with only management commands available. In the past it was possible to access a real shell by entering su in the management shell. On my firmware version none of the suggestions to drop a real shell worked. After trying dozens of commands I quickly realized that on my firmware version accessing a Linux shell is not possible anymore.

Also none of the available management commands allowed me to set the local native VLAN.

Firmware upgrading to root

The firmware upgrade function of the web interface was still very interesting and sounded promising. After obtaining a firmware upgrade payload from the Huawei support tool I tried searching for strings found in the payload. Maybe someone else already wrote an unpacker.
The string “duit9rr.sh” sounded very unique and ultimately led me to a chinese blog post by Xiaolan Lee. He or she not only wrote an unpacker and repacker but also managed to gain root access on an HG8120C! The payload described in the blog post looked almost exactly like the payload I had and I immediately had root access on my ONT.

Setting the native VLAN

After trying different solutions to tag all incoming packets with my desired VLAN ID, nothing worked without heavy CPU utilization or breaking everything. I had to understand how the OLT would provision the native VLAN command.
I searched for the string “native-vlan” in all non-standard binaries. The binary /bin/Bbspcmd attracted my interest. The help flag was not very useful and the flags I tried were always rejected so I had to disassemble the binary and look deeper into where the string is used.

After some time I was able to find the control flow that constructed the command which allows me to set the native VLAN for the local WAN ethernet port:

/bin/Bbspcmd user_ctp set_ctp ctpid 1 nativevlan <VLAN_ID> # Setting the VLAN ID for port id 1
/bin/Bbspcmd user_ctp set_ctp ctpid 1 nativevlan_enable 1 # Enable tagging for port id 1

And this really worked! I was able to use all the available bandwidth without noticeably influencing the load of the ONT.

As this change is not persistent I had to find a solution to make it persistent or reprovision it during the boot process of the USG. Otherwise I would not have internet after a power outage. I do not want to make persistent changes to the ONT as I can not easliy buy another unit if I break this unit which is owned by my service provider.

Provision the ONT after a power outage

Using Xiaolan Lee’s blog post I was able to create a payload with my desired commands to set the native VLAN ID. This payload is uploaded by the USG during boot. First I had to automate the login process to be able to trigger the firmware upgrade with my payload. The login process was a bit more complex than expected. The ONT sends a seed with the first HTTP request, which is used to hash the username and password and those hashes are then presented in a cookie.

On the USG, all files below /config/scripts/ are kept during an upgrade. Scripts stored in /config/scripts/pre-config.d/ are executed at the beginning of the boot process. It is therefore a good idea to place the following script there.

With the help of curl, sed and awk, which are all available by default on the USG, everything resulted in the following script:

/config/scripts/pre-config.d/20-provision_ont.sh

#!/bin/sh

ONTIP=192.168.100.1
UPDFILE="/config/scripts/ont/hg8012h_nv_40.bin"

ip addr add 192.168.100.25/24 dev eth0

RAND=$(curl -s "http://$ONTIP" | fgrep 'function GetRandCnt' | sed "s/.*'\([^']\+\)'.*/\1/")

HRAND=$(echo -n "$RAND" | sha256sum | cut -f1 -d " ")
HUSER=$(echo -n "telecomadmin$RAND" | sha256sum | cut -f1 -d " ")
MD5PASS=$(echo -n "admintelecom" | md5sum | cut -f1 -d " ")
SHAPASS=$(echo -n "$MD5PASS" | sha256sum | cut -f1 -d " ")
HPASS=$(echo -n "$SHAPASS$RAND" | sha256sum | cut -f1 -d " ")

COOKIE="rid=$HRAND$HUSER$HPASS:Language:english:id=-1;path=/"
COOKIE=$(curl -s --cookie "$COOKIE" -o /dev/null -c - "http://$ONTIP/login.cgi" | tail -n1 | awk '{ print $7 }')

TOKEN=$(curl -s --cookie "$COOKIE" "http://$ONTIP/index.asp" | fgrep 'hwonttoken' | sed "s/.*value=\"\([^']\+\)\".*/\1/")

curl -s --cookie "$COOKIE" -F "onttoken=$TOKEN" -F "browse=@$UPDFILE" "http://$ONTIP/html/management/Firmwareupload.cgi?RequestFile=html/management/reset.asp&FileType=image" > /tmp/ontstatus.txt

A payload for VLAN id 40 can be downloaded here.

Have fun and do not forget to remove all VLAN tags from the USG’s WAN interface!