1

I need to proxy a lot of MQTT connections (over 200K) and distribute them over a number of ports at the backend server. Below is my Nginx configuration.

load_module /usr/lib/nginx/modules/ngx_stream_module.so;

user www-data;
pid /run/nginx.pid;


worker_rlimit_nofile 204800;
worker_cpu_affinity auto;
worker_processes auto;


events {
    accept_mutex on;             # Multiple workers should accept connection
    worker_connections 206200;   # Total connections with upstream and clients
    multi_accept off;            # Accept one connection in each worker
}


stream {
    upstream backend {
          server backendserver.com:1885;
          server backendserver.com:1886;
          server backendserver.com:1887;
          server backendserver.com:1888;
          server backendserver.com:1889;
          server backendserver.com:1890;
          server backendserver.com:1891;
          server backendserver.com:1892;
          server backendserver.com:1893;
          server backendserver.com:1894;
          server backendserver.com:1895;
          server backendserver.com:1896;
          server backendserver.com:1897;
          server backendserver.com:1898;
          server backendserver.com:1899;
          server backendserver.com:1900;
          server backendserver.com:1901;
          server backendserver.com:1902;
          server backendserver.com:1903;
          server backendserver.com:1904;
    }


server {
    listen 8883;
    proxy_pass backend;
}



}

I also have set the following system configurations:

sysctl -w fs.file-max=11000000
sysctl -w fs.nr_open=11000000
sysctl -w net.core.somaxconn=65535
sysctl -w net.ipv4.tcp_max_syn_backlog=65535
sysctl -w net.ipv4.ip_local_port_range="1025 65535"
sysctl -w net.ipv4.tcp_mem="100000000 100000000 100000000"
sysctl -w net.ipv4.tcp_rmem='2048 2048 2048'
sysctl -w net.ipv4.tcp_wmem='4096 4096 4096'
sysctl -w net.core.rmem_default=4096
sysctl -w net.core.wmem_default=4608
sysctl -w net.core.rmem_max=4096
sysctl -w net.core.wmem_max=4608

Also I have set ulimit -n 999999.

But I only can create around 128K-130K connections. After that, Nginx logs erros like below:

2023/08/16 06:46:32 [error] 4172305#4172305: *206569 recv() failed (104: Unknown error) while proxying and reading from upstream, client: 72.xxx.xxx.xxx, server: 0.0.0.0:8885, upstream: "139.xxx.xxx.xxx:8889", bytes from/to client:984/5224, bytes from/to upstream:5224/984

I checked directly connecting to all of the backend ports and they work fine.

Furthermore, when I only had one entry in upstream backend block, I could only create around 64K connections. When I add the second entry, I could go upto around 128K-130K. But then adding more entries does not let me go beyond that number of connections.

I have already tried doing below:

  1. Set worker_rlimit_nofile and worker_connections to 999999.
  2. Add more server blocks with different ports.
  3. 10x tcp_rmem, tcp_wmem, rmem_default, wmem_default, rmem_max, wmem_max.
  4. Setting multi_accept on;

How can I make my Nginx server to proxy more than 130K connections (at least 200K)?

Yehan Pemarathne
  • 11

1 Answers1

0

I got 100k WS proxy but not WSS proxy. Not testing 200k due to resources.

Technical explains:

  • TCP/IP outgoing connection need a Network Interface, a Source IP address, and a High Port
  • High Port number, even after sysctl to change, should be at most from 1024 to 65535, thus ~64k connections per source IP
  • Since nginx (and almost all proxy) cannot load balance to backend with binding a range of IP addresses, we can only do it by the kernel routing
  • Create multiple subnets, with netmask 255.255.255.252, for P-t-P communication. Find a Network Engineer to learn the details, if you want.
  • In the example below, I have 10 P-t-P subnet between nginx and MQTT. Nginx pick 1 from the 10 upstreams, the routing table will pick up the corresponding alias interface and source IP automatically by routing
  • Thus we have 10 x 64k = 640k slots for connecting from nginx to MQTT, in theory. Since we cannot control how nginx pick upstreams, just do excessive config first.
  • I can reach 100k from HTTP/80 to WS/8083. But I cannot go over 10k if the frontend is HTTPS/443.

1/ System Config (sysctl) is still required. Here's mine. I copied different configs from everywhere on the Internet talking about c100k. Not sure if they're all required or not.

# fs.file-max = 9223372036854775807
fs.nr_open=2097152
net.core.somaxconn = 262144
net.ipv4.ip_local_port_range = 1024 65535
net.ipv4.tcp_max_syn_backlog = 262144
net.ipv4.tcp_rmem = 1024 4096 16777216
net.ipv4.tcp_wmem = 1024 4096 16777216
net.netfilter.nf_conntrack_tcp_timeout_time_wait=30
net.ipv4.tcp_max_tw_buckets=8192
net.ipv4.tcp_fin_timeout=15
net.ipv4.tcp_max_orphans = 262144
net.core.netdev_max_backlog = 262144
net.core.rmem_default=262144
net.core.wmem_default=262144
net.core.rmem_max=16777216
net.core.wmem_max=16777216
net.core.optmem_max=16777216
net.ipv4.tcp_timestamps = 0
net.ipv4.tcp_synack_retries = 1
net.ipv4.tcp_syn_retries = 1
net.ipv4.tcp_fin_timeout = 5
net.ipv4.tcp_keepalive_time = 30

2/ Also limits

File: /etc/security/limits.d/tuning.conf

* hard nofile 1048576
* soft nofile 1048576
root hard nofile 1048576
root soft nofile 1048576
* hard nproc 1048576
* soft nproc 1048576
root hard nproc 1048576
root soft nproc 1048576

3/ If you're using systemd, like me using Ubuntu, need to override the systemd:

# systemctl edit nginx.service

[Service]
LimitNOFILE=1048576

4/ The backend MQTT server configure more alias IP, e.g.

ens18:0 192.168.0.1/30
ens18:1 192.168.1.1/30
ens18:2 192.168.2.1/30
ens18:3 192.168.3.1/30
...
ens18:9 192.168.9.100/30

5/ The nginx server configure more alias IP towards the MQTT server

ens19:0 192.168.0.2/30
ens19:1 192.168.1.2/30
ens19:2 192.168.2.2/30
ens19:3 192.168.3.2/30
...
ens19:9 192.168.9.2/30

6/ Nginx config: upstream backend (my MQTT use 8083 for WS)

upstream backend {
    192.168.0.1:8083;
    192.168.1.1:8083;
    192.168.2.1:8083;
    ...
    192.168.9.1:8083;
};

map $http_upgrade $connection_upgrade {
    default upgrade;
    '' close;
}

7/ Finally, do the proxy

location /mqtt {
    proxy_pass http://backend/mqtt;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection $connection_upgrade;
}

IF you run nginx and MQTT on one single server / VM, you can create dummy network interface. In this case the nginx upstream servers simply the same as the dummy interface IP address will work. Don't use localhost, 127.0.0.1, etc. That won't work.

I've try running nginx and MQTT inside two docker containers but fails.

Now I have nginx on the host system and MQTT inside docker (inside one VM) and works good.

Ken Cheung
  • 164