RX/TX Buffers, Flow Hash and Others on Boot

After installing Suricata, some fine tuning of the network interface(s) used in the traffic capture is required to ensure every ounce of performance is extracted from the new IDPS installation. Those configurations need to be persisted when the system is power cycled. To do that on a Enterprise Linux based OS (e.g. RedHat, CentOS, Fedora, etc.) one can leverage the /sbin/ifup-local script.

This script is called per interface by the network configuration utility when the network is up and running (at least when using a static configuration). The performance oriented configurations that are usually needed are:

Set the network card RX/TX buffers to the maximum that the hardware supports
Balance the receive flow hash indirection table equally in all CPU’s
Set the receive network card and transmits IRQ affinity to one CPU each
Turn off offloading features (only if using PF_RING, AF_PACKET or similar)

The kernel network stack can also be tuned using the /sbin/ifup-local script, however the recommended approach is to use a file under the /etc/sysctl.d/ directory instead. The ifup-local file doesn’t usually exist, so it needs to be created and made executable.

1#!/bin/bash
2# Create the file
3touch /sbin/ifup-local
4
5# Make it executable
6chmod +x /sbin/ifup-local

Follows the contents of the script that I use.

The name of the interrupts for the network card may vary, and it may support more or less offloading features, adjust accordingly.

 1#!/bin/bash
 2set_buffers() {
 3    # Get the hardware RX/TX maximum and current
 4    PRESET=$(ethtool -g $1 | tr '\n' ' ' | sed 's/.*RX:\s\+\([0-9]\+\).*TX:\s\+\([0-9]\+\).*RX:\s\+\([0-9]\+\).*TX:\s\+\([0-9]\+\).*/\1 \2 \3 \4/g')
 5
 6    # Set receive and trasmit buffers to the hardware maximum
 7    ethtool -G $1 rx $(echo $PRESET | cut -f 1 -d " ") tx $(echo $PRESET | cut -f 2 -d " ")
 8}
 9
10balance_flowhash() {
11    # Balance evenly per CPU
12    ethtool -X $1 equal $(cat /proc/cpuinfo | grep processor | wc -l)
13}
14
15set_affinity() {
16    MAX=$(cat /proc/cpuinfo | grep processor | wc -l)
17
18    # Since the receive/transmit interrupts name index starts at 0, subtract 1 from the maximum
19    let "MAX=$MAX-1"
20
21    # The mask that will define the affinity
22    MASK=1
23
24    for INDEX in $(seq 0 1 $MAX); do
25        IRQ=$(cat /proc/interrupts | grep $1-rxtx-$INDEX"$" | sed 's/\s\([0-9]\+\)\(.*\)/\1/g')
26
27        # Apply the mask to the current IRQ
28        printf "%X" $MASK > /proc/irq/$IRQ/smp_affinity
29
30        # Duplicate the next mask value
31        let "MASK=$MASK+$MASK"
32    done
33}
34
35turnoff_offloading() {
36    ethtool -K $1 rx off
37    ethtool -K $1 tx off
38    ethtool -K $1 sg off
39    ethtool -K $1 tso off
40    ethtool -K $1 gso off
41    ethtool -K $1 gro off
42    ethtool -K $1 lro off
43    ethtool -K $1 rxvlan off
44    ethtool -K $1 txvlan off
45    ethtool -K $1 rxhash off
46}
47
48case "$1" in
49eth1)
50    # Update the receive and transmit buffers
51    set_buffers $1
52
53    # Balance receive flow hash indirection table
54    balance_flowhash $1
55
56    # Set CPU affinity for the interrupts
57    set_affinity $1
58
59    # Offloading features
60    turnoff_offloading $1
61;;
62esac
63
64exit 0

After a reboot, to verify if the configurations have been applied correctly (the system used in this example as 8 CPU’s) issue the following commands.

The network interface used in the commands bellow is eth1, change accordingly.

 1#!/bin/bash
 2# Verify the send and receive buffers, note how the current hardware values are the same as the pre-set maximum values
 3ethtool -g eth1
 4# Ring parameters for eth1:
 5# Pre-set maximums:
 6# RX:             4096
 7# RX Mini:        0
 8# RX Jumbo:       0
 9# TX:             4096
10# Current hardware settings:
11# RX:             4096
12# RX Mini:        0
13# RX Jumbo:       0
14# TX:             4096
15
16# Verify the flow hash indirection table
17ethtool -x eth1
18# RX flow hash indirection table for eth1 with 8 RX ring(s):
19#     0:      0     1     2     3     4     5     6     7
20#     8:      0     1     2     3     4     5     6     7
21#    16:      0     1     2     3     4     5     6     7
22#    24:      0     1     2     3     4     5     6     7
23
24# Verify that the IRQ affinity is set correctly, the output bellow shows only the first 4 CPU's
25cat /proc/interrupts | grep 'CPU\|eth1'
26#            CPU0       CPU1       CPU2       CPU3   (...)
27#  65:  107325835          0          3          0         eth1-rxtx-0
28#  66:          0  150380495          0          2         eth1-rxtx-1
29#  67:          0          0  107109972          0         eth1-rxtx-2
30#  68:          0          0          0   91046195         eth1-rxtx-3
31# (...)
32
33# Verify that the offloading features are off
34ethtool -k eth1
35# Features for eth1:
36# rx-checksumming: off
37# tx-checksumming: off
38#         tx-checksum-ipv4: off [fixed]
39#         tx-checksum-ip-generic: off
40#         tx-checksum-ipv6: off [fixed]
41#         tx-checksum-fcoe-crc: off [fixed]
42#         tx-checksum-sctp: off [fixed]
43# scatter-gather: off
44#         tx-scatter-gather: off
45#         tx-scatter-gather-fraglist: off [fixed]
46# tcp-segmentation-offload: off
47#         tx-tcp-segmentation: off
48#         tx-tcp-ecn-segmentation: off [fixed]
49#         tx-tcp6-segmentation: off
50# udp-fragmentation-offload: off [fixed]
51# generic-segmentation-offload: off
52# generic-receive-offload: off
53# large-receive-offload: off
54# rx-vlan-offload: off
55# tx-vlan-offload: off
56# (...)

Happy tuning!

April 25, 2015
Configuration, Linux, Network
Suricata

RX/TX Buffers, Flow Hash and Others on Boot

Categories

Tags