We got an error message in our customer’s AlmaLinux 8 server hosting cPanel where e1000 network driver hangs with “Detected Tx Unit Hang“. This is as shown in the screenshot below.

The issue happens when the system is experiencing high network traffic load and it results in intermittent issues when connecting to server.

• e1000 is the Intel e1000 network driver – mostly used in the Intel Gigabit LAN controllers.

The said error can result in network connectivity timeouts, slow speeds, or complete network failure in some situations.

The customer Virtual machine sits in VMware hypervisor. The solution to this issue is to disable TCP Segmentation Offloading (TSO).

Disable TCP Segmentation Offloading (TSO)

To disable TCP Segmentation Offloading (TSO) means turning off a network optimization feature when handling TCP/IP connections. In some cases TSO is known to cause network instabilities and latencies with specific network hardware or network configurations or hardware.

We can use ethtool command-line utility on Linux to disable TCP Segmentation Offloading (TSO).

Here is a list of commonly used commands to disable various offloading network interface features in Linux:

• rx off: Use to disable receive checksum offloading. It means network interface card (NIC) will not calculate the checksum for incoming packets but instead CPU will do the job.
• lro off: Use to disable Large Receive Offload (LRO). It enables the the network interface card to combine larger packets for more efficiency.
• gro off: Used to disable Generic Receive Offloading (GRO). It means your NIC will combine multiple small packets into a single larger packet before it delivers it to CPU. There can be an improved performance from this.
• tso off: It disables TCP Segmentation Offload (TSO). It allows your NIC to segment large outgoing data packets into smaller TCP segments, reducing CPU overhead.

List all network interfaces in your server.

$ sudo ip link show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: ens32: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000
    link/ether 00:0c:24:48:8e:b6 brd ff:ff:ff:ff:ff:ff
    altname enp2s0
3: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN mode DEFAULT group default qlen 1000
    link/ether 52:54:00:04:30:73 brd ff:ff:ff:ff:ff:ff

Here we will disable TSO on ens32 interface.

DEVICE=ens32
sudo ethtool -K $DEVICE tso off

To persist the change we can use one of the following methods.

1) Edit Network Configurations manually

Let’s open the server network configuration file.

sudo vim /etc/sysconfig/network-scripts/ifcfg-$DEVICE

Then we add ETHTOOL_OPTS parameter at the end like below.

TYPE=Ethernet
BOOTPROTO=none
DEFROUTE=yes
NAME=ens32
DEVICE=ens32
ONBOOT=yes
IPADDR=192.168.20.22
PREFIX=24
GATEWAY=192.168.20.1
DNS1=8.8.8.8
DNS3=1.1.1.1
ETHTOOL_OPTS="-K ${DEVICE} tso off"

2) Using ifup-local script

The ifup-local script is used in network configuration on RHEL based systems. Its work is to run additional commands to perform certain actions immediately a network interface comes up.

$ sudo vim /sbin/ifup-local
#!/bin/bash
if [ "$1" == "ens32" ]; then
  /sbin/ethtool -K ens32 tso off
fi

Make the script executable.

sudo chmod +x /sbin/ifup-local

Please note that custom commands in the /sbin/ifup-local script is only executed if the network device is controlled by the /etc/init.d/network service.

3) Using NetworkManager dispatcher script

NetworkManager gives you a way of running additional custom scripts. Your script file should be placed under the /etc/NetworkManager/dispatcher.d/ directory. The scripts are run in alphabetical order.

$ sudo vim /etc/NetworkManager/dispatcher.d/20-ethtool
#!/bin/bash

if [ "$1" == "ens32" ] && [ "$2" == "up" ]; then
    ethtool -K "$1" tso off
fi

You can tune it to include other commands you want run by ethtool e.g

ethtool -K "$1" tso off rx off gro off lro off

Conclusion

In this post our Engineers have demonstrated how we can use ethtool in a Linux system to disable specific network card features like TCP Segmentation Offloading (TSO).

CloudSpinx exists to empower businesses of all sizes. We are always available to help you troubleshoot and fix issues in your IT Infrastructure so that you can focus on your Business. You can chat with us now!.