VDSL2 overheads and CAKE

Malcolm 25 January 2021 at 2.17 p.m.

Do you know what the overheads are for Sky superfast broadband? I'm trying to setup SQM on my router but I can't find the correct settings.

Alasdair Smith 25 January 2021 at 3.34 p.m.

I believe that Sky use IP directly over Ethernet (IPoE) and DHCP with option 61 / client ID to assign IP addresses. PPP/PPPoE is not involved so you can exclude those 8 bytes from the table above, which would leave a total of 26 bytes additional per-packet overhead.

That equates to the bridged-ptm plus ether-vlan options in tc-cake. So the command would be something like this instead:

tc qdisc replace root dev wan cake bandwidth 19691kbit bridged-ptm ether-vlan noatm

David 31 January 2023 at 10.13 p.m.

Hi there

I don't really understand all this stuff but I'm trying. I have BT as my ISP with an FTTC connection. In my Asus router there is an option to select VDSL2 PPPoE PTM for wan packet overhead which I am more than sure should be correct for BT and most ISP in UK anyway.

Does this look ok? https://i.imgur.com/HShXuVQ.png

Thank you

Alasdair Smith 1 February 2023 at 12.01 a.m.

Hi David, both our ISPs use the same underlying BT Wholesale FTTC service and so the overheads should be the same.

The PTM mode in your configuration should account for the 64/65 octet encapsulation. Though it might easier on your router’s CPU to not use this mode and simply reduce your 10 Mb/s upload bandwidth by that fraction to 9.84 Mb/s.

If I’m right about the VLAN tag then your packet overhead should be increased to 34 bytes. It’s probably safer to slightly overestimate than underestimate.

David 1 February 2023 at 6.16 a.m.

Hi again

Just to confirm you would set something like this? https://i.imgur.com/MmpWmPK.png

I can't find much for MRU for BT. I'm also already being very generous on upload bandwidth allocation at 10. I can usually max out around 11.20 if no shaping is applied.

Alasdair Smith 1 February 2023 at 8.30 p.m.

That looks right to me. I have never configured an MPU value, as I haven’t been able to understand the purpose of it. There seems to be very little information available.

David 2 February 2023 at 7.53 a.m.

I tried to read the cake documentation again and I do think that 34 is correct. I can't see any reason why it's not. However it got me thinking about FTTP providers. Apparently the FTTP networks (BT anyway) drop the need for vlan tagging so the correct number should go back to 30. Additionally, I found this info about MPU for UK ISP: https://openwrt.org/docs/guide-user/network/traffic-shaping/sqm-details#more_hints_tips_info

"Similarly there's a minimum packet size limit, which empirically I'll set to 72 bytes, based on the smallest packets we ever see on the ingress side."

It's unclear if a bridged-ptm connection via Sky will corelate to a 72 MPU for PPPoE. I don't know enough to check.

David 2 February 2023 at 2.45 p.m.

Sorry to bombard you again but I actually came across this thread: https://forum.openwrt.org/t/sqm-link-layer-adaptation-question-vdsl2/11055

The poster made a good point. I'm not 100% sure but there might actually be 2 vlan tags. The 0 and the 101. Thus the value should actually be 38. I don't know if the 0 actually is a vlan tag but cake documentation does actually state you can use multiple ether-vlan commands if needed so it's not unheard of.

Alasdair Smith 2 February 2023 at 4.56 p.m.

Interesting info about MPU, thanks. I’ll post an update if I work out what this should be. I believe BT FTTC connections use a single VLAN tag (101). You’ll see in the various places that provide BT FTTC configurations for third-party routers that the second VLAN tag (Service VLAN) is not used.

There is also quite a bit of information in SIN 498, the specification for compatibility with BT's VDSL2 services; though it's not exactly light reading.

David 2 February 2023 at 6.57 p.m.

No problem. I think you're right. Also with so much reading sometimes you miss the easy stuff: https://openwrt.org/docs/guide-user/network/traffic-shaping/sqm-details

"Choose Ethernet with overhead: select for e.g. VDSL2 and set the Per-packet Overhead to 34 if you know you have a VDSL2 connection (this is sometimes called Fiber to the Cabinet, for example in the UK)."

I think I've nearly cracked MPU. Further down that page:

"Similarly there's a minimum packet size limit, which empirically I'll set to 72 bytes, based on the smallest packets we ever see on the ingress side."

So if bridged-ptm is 22 (no vlan tag) and vdsl2 pppoe ptm is 30 (no vlan tag) you should just have to add 8 to the mpu.

Alasdair Smith 3 February 2023 at 1.57 p.m.

I think I’ve figured out MPU. The smallest packet my internet interface has seen in the last 24 hours or so is 28 bytes, as shown below. When CAKE adds the 34 bytes of overhead to that, it thinks it will result in 62 bytes transmitted over the wire and estimates the time that transmission will take accordingly.

# tc -s qdisc show dev aaisp
qdisc cake 8001: root refcnt 2 bandwidth 19691Kbit diffserv3 triple-isolate nonat nowash no-ack-filter split-gso rtt 30.0ms noatm overhead 34 
 Sent 986258293 bytes 5222192 pkt (dropped 2720, overlimits 2137730 requeues 0) 
 backlog 0b 0p requeues 0
 capacity estimate: 19691Kbit
 min/max network layer size:           28 /    1500
 min/max overhead-adjusted size:       62 /    1534

However, Ethernet has a minimum frame size of 64 bytes. A 28 byte packet + 2 bytes PPP + 6 bytes PPPoE + 22 bytes Ethernet results in only 58 bytes, and so it will actually be padded to create a 64 byte Ethernet frame. Once the remaining 4 bytes of PTM overhead are added for transmission, the minimum size (MPU) must actually be 68 bytes.

David 4 February 2023 at 3.21 p.m.

Are you sure it's not 72? The 64 doesn't include the vlan tag in the calculation (why it's in brackets): https://i.imgur.com/43A60Ss.png

So really you have to do ptm (4) and vlan tag (4) added to the minimum 64.

I could be wrong though trying to figure out what you account for is getting pretty complex for me now.

Alasdair Smith 5 February 2023 at 12.12 a.m.

This diagram shows what I think is happening.

The Ethernet frame is following the 802.1Q standard that includes the VLAN tag in its header. This means the 64‐byte minimum frame size is already inclusive of the VLAN tag.

The physical transmission of the frame is by PTM, so there is no preamble, start frame delimiter or interpacket gap.

Alasdair Smith 5 February 2023 at 12.34 a.m.

Wikipedia's Ethernet Frame page agrees with this interpretation:

Payload is a variable-length field. Its minimum size is governed by a requirement for a minimum frame transmission of 64 octets (bytes). With header and FCS taken into account, the minimum payload is 42 octets when an 802.1Q tag is present and 46 octets when absent. When the actual payload is less than the minimum, padding octets are added accordingly.

David 5 February 2023 at 2.03 p.m.

That's a good spot. I missed that. 68 seems correct.

You know I started to think about more stuff and going down the rabbit hole so to speak. I'm just thinking out loud here but:

1: I wonder what happens with FTTP/FTTH services in UK. I have read that they don't use vlan tags but I think it's people incorrectly assuming that since the ONT does the tagging automatically, you don't need to. Would you still account for this as normal?

The wiki states for FTTH you should set wan packet overhead to 44 and an mpu of 88: https://openwrt.org/docs/guide-user/network/traffic-shaping/sqm-details#more_hints_tips_info

It's ofcourse a general rule they say but unless I'm mistaken there should be no reason for the values to change. Ofcourse it is still ptm so there should be no reason to involve layer 1 values (maybe this is why the wiki states these as not many fttp providers will use ptm to stick within layer 2).... I think that's how it works.

2: I doubt AAISP will change for awhile but I can maybe see BT and slightly large ISP like Talk Talk maybe moving to CGNAT in some years. Now I am not sure but maybe it's worth looking into the numbers if a change to ipv6 is required to steer clear of that. ipv6 ethernet frame:

https://app.netrounds.com/static/3.0/support/defs-notes/l2-eth-frame-sizes.html

Minimum frame size of 84 (I assume also padded) so 88 mpu due to ptm... as a first guess ofcourse.

moeller0 24 February 2024 at 9.39 p.m.

´´´ I have also used the noatm option to disable compensation for the 64/65 octet encoding, as this has already been accounted for by calculating a reduction in the line rate (bandwidth). ´´´ cake defaults to noatm, but explicitly requesting it will not hurt.

moeller0 24 February 2024 at 9.40 p.m.

2: I doubt AAISP will change for awhile but I can maybe see BT and slightly large ISP like Talk Talk maybe moving to CGNAT in some years. Now I am not sure but maybe it's worth looking into the numbers if a change to ipv6 is required to steer clear of that. ipv6 ethernet frame:

https://app.netrounds.com/static/3.0/support/defs-notes/l2-eth-frame-sizes.html

Minimum frame size of 84 (I assume also padded) so 88 mpu due to ptm... as a first guess ofcourse.

MPU really only applies if there is some padding in the on-the-wire packets that the linux kernel does not know about... which is true for i.e. ethernet frame padding (which in ethernet frames is added as part of the payload, so lives between MAC headers and FCS, and so all protocols that preserve the FCS typically inherit that padding from ethernet and hence have an MPU). The actual value of the MPU depends on the link layer, in the PTM case there is the kernel-invisible PTM header that gets added on top of the ethernet L2 frame for a result of 64+4 = 68...

But if the payload already exceeds the 42-46 bytes (as in the IPv6 example) cake's normal overhead accounting already solves the issue: IPv6+empty UDP 40+8 = 48 -> so cake will take this length of 48 and add the overhead of 34 resulting in 48+34 = 82 then is will take max(82, mpu) and with mpu at 68 the size of this packet will not be determined by the mpu... but if you have IPv4 pure ACK packets (and lots of those) you might run into issues.