https://unix.stackexchange.com/questions/337456/is-tcp-pacing-enabled-by-default-on-linux

Asked 5 years ago
Active 2 years, 10 months ago
Viewed 3k times

Q

I want to ask a basic question: Is TCP PACING enabled by default on Linux? I'm using Ubuntu right now, kernel 4.4.0.

I saw that it can be enabled/disabled using TC-FQ, but is it enabled by default using TC only, or the default is disabled?

Answer

Short answer: for kernel 4.4.0 TCP PACING is enabled by default and is set to ~34.36Gb/s per socket. Since kernel 4.13.0-rc1, TCP PACING is disabled by default. On both cases, since TSO is enabled, TCP PACING would have no effect when pacing rate is set to the default value.

TCP PACING is set for a socket using the sock_setsockopt() function (net/core/sock.c). The flag for doing so is SO_MAX_PACING_RATE, and the handling code is as follow:

case SO_MAX_PACING_RATE:
        sk->sk_max_pacing_rate = val;
        sk->sk_pacing_rate = min(sk->sk_pacing_rate,
                     sk->sk_max_pacing_rate);
        break;
On socket creation the relevant fields are set to ~0U in sock_init_data() (net/core/sock.c):

sk->sk_max_pacing_rate = ~0U;
sk->sk_pacing_rate = ~0U;
So the default values for pacing rate and max pacing rate is max unsigned int value, which is (2^32 - 1), or 4,294,967,295 Bytes per second, which are 34.36Gb/s.

It is possible, however, to get to higher rates even with a single socket by using TCP segmentation offload (TSO). The TSO takes the pacing rate into consideration, but isn't really bounded by the default rate. In tcp_tso_autosize() function (net/ipv4/tcp_output.c) the pacing rate might reduce the size of the TSO session, but since the default value is ~4.3GB and even after shifting right by 10 you still get ~4MB, it is much bigger than the typical TSO session.

static u32 tcp_tso_autosize(const struct sock *sk, unsigned int mss_now)
{
    u32 bytes, segs;

    bytes = min(sk->sk_pacing_rate >> 10,
            sk->sk_gso_max_size - 1 - MAX_TCP_HEADER);

    /* Goal is to send at least one packet per ms,
     * not one big TSO packet every 100 ms.
     * This preserves ACK clocking and is consistent
     * with tcp_tso_should_defer() heuristic.
     */
    segs = max_t(u32, bytes / mss_now, sysctl_tcp_min_tso_segs);

    return min_t(u32, segs, sk->sk_gso_max_segs);
}
I tried actually hitting the 34Gb/s limitation, but unfortunately, without TSO CPU utilization on TX side limit the single socket BW to ~23Gb/s.

A patch by Eric Dumazet tcp: internal implementation for pacing was introduced on kernel 4.13.0-rc1. This patch enhances TCP PACING abilities. For once, it allow for pacing without using TC-FQ. It also disables pacing entirely if the sk_pacing_rate field is set to 0 or ~0U (default). See the patch cover and code for more details.