Jumbo frames for L2 and L3 environments on Juniper EX series switches

When there is a need to use jumbo frames for switching and optionally routing on a EX series switch from Juniper there are a few things to observe to make it work correctly. (Follwing example is tested with JUNOS 11.4 and 12.x)

First a few things to consider when thinking about jumbo frames in you layer 2 environment. Within a layer 2 zone there is no problem to have jumbo frames enabled on the switches, as long as all switches are set to forward frames of same size, because the switches acts like a conductor between its attached devices. In fact it is the actual end points (the attached devices to switches) that decides the MTU for its traffic. So each server and client decides the MTU size it will use on its NIC. There is no way for these hosts to peform Path MTU Discovery (PMD) on layer 2, this is a layer 3 function only, so a host will try to use its max MTU within the layer 2 broadcast domain (VLAN) whether the its destination can handle it or not. So if jumbo frames are to be used, all hosts in the same VLAN need to have the same MTU configured on its interfaces.

Also remember that each 802.1Q tag added on top of an ethernet frames when passed on trunk interfaces between switches add 4 bytes, so in an environment with Q and QinQ tags, make sure that the hosts have a MTU set that is at least 8 bytes smaller than the max MTU of the jumbo frames on the switches to allow for these extra bytes. A normal ethernet frame is 1518 bytes with a IP MTU of the IP packet of 1500 bytes. The actual IP packet payload is then 1472 bytes, because IP headers add 20 bytes and layer 4 headers add 8 bytes. So if you have switches that supports the full 9216 bytes of jumbo frames, use a conservative IP MTU of 9000 on your hosts so there are no nasty surprises between your switches with trunk ports.

So when using jumbo frames for layer 2 operations on a EX series switch from Juniper, the MTU needs to be set on the physical interfaces. The command is for example: “set interfaces ge-0/0/0 mtu 9216”. This can be a bit cumbersome since a switch have many interfaces, so here an apply group can be used. Something like:

set groups jumbo-frames interfaces <ge-*/*/*> mtu 9216
set groups jumbo-frames interfaces <xe-*/*/*> mtu 9216
set apply-groups jumbo-frames

This will enable jumbo frames with the size of 9216 on all gigabit and ten gigabit interfaces.

If jumbo frames also want to be used on the routing layer, the switch first need on its global RVI have the maximum MTU set. Then per individual RVI on family inet the actual MTU is set for that RVI. According to documentation so far RVIs can be configured to 9216 bytes. Also when setting the RVI MTU for family inet, remember the dicussion above for IP MTU and layer 2 header additions. So be conservative, dont use IP MTU on the global RVI and family inet  RVI larger than 9000 bytes.

So for example on VLAN 142 and VLAN 131 the vlan.142 resp. vlan.131 is used as RVI, the config would look like this:

set interfaces vlan mtu 9000
set interfaces vlan.131 family inet mtu 9000
set interfaces vlan.142 family inet mtu 9000

Again this is a bit cumbersome, so a addition to the apply group jumbo-frames can be done something like this:

set groups jumbo-frames interfaces vlan mtu 9000
set groups jumbo-frames interfaces vlan unit <*> family inet mtu 1500

Now all the RVIs configured on the switch will automatically adapt to your jumbo frame environment for layer 3 operations aswell. One final note, to adapt the control plane of the switch to perform PMD for control traffic with a TTL larger than 1, also do: “set system internet-options path-mtu-discovery”

So a short quick configuration summary:

set groups jumbo-frames interfaces <ge-*/*/*> mtu 9216
set groups jumbo-frames interfaces <xe-*/*/*> mtu 9216
set groups jumbo-frames interfaces vlan mtu 9000
set groups jumbo-frames interfaces vlan unit <*> family inet mtu 1500
set apply-groups jumbo-frames
set system internet-options path-mtu-discovery

Good luck and happy switching!

This entry was posted in Juniper Networking, Networking. Bookmark the permalink.

Leave a Reply