Server suddenly became too slow with timeout error

Question

My server suddenly became slow, all the websites in it at the http level i have checked all the logs but there is nothing that indicate why it's happening if anyone have any suggestion please

My setup :

Centos 6.8 Nginx 1.10 php-fpm 7.0 Mariadb 10.1 Memory 32gb with 13 gb free

1 year later still the same problem

Note: The only way i was able to reduce load to 10 seconds was by moving all cache to memory tmpfs - Enabling Nginx cache jump the load back to 100s either adding cache folder to memory or not (Which i couldn't understand as it was working fine before)

Caching system: Opcache Filecache (Codeigniter) Memcached Redis

netstat -an | wc -l
22445

netstat -an | grep :80 | wc -l
51196

netstat -ant | awk '{print $6}' | sort | uniq -c | sort -n
  1 established)
  1 FIN_WAIT1
  1 Foreign
  2 SYN_RECV
  3 LAST_ACK
  6 LISTEN
21144 ESTABLISHED
30908 TIME_WAIT

netstat -i 15

Kernel Interface table
Iface       MTU Met    RX-OK RX-ERR RX-DRP RX-OVR    TX-OK TX-ERR TX-DRP TX-OVR Flg
eth0       1500   0 218927712      0      0      0 257329662      0      0      0 BMRU
lo        65536   0 310876055      0      0      0 310876055      0      0      0 LRU
eth0       1500   0 218994531      0      0      0 257405202      0      0      0 BMRU
lo        65536   0 311006988      0      0      0 311006988      0      0      0 LRU
eth0       1500   0 219060371      0      0      0 257479672      0      0      0 BMRU
lo        65536   0 311139650      0      0      0 311139650      0      0      0 LRU
eth0       1500   0 219131683      0      0      0 257561956      0      0      0 BMRU
lo        65536   0 311270631      0      0      0 311270631      0      0      0 LRU
eth0       1500   0 219201250      0      0      0 257640969      0      0      0 BMRU
lo        65536   0 311407443      0      0      0 311407443      0      0      0 LRU
eth0       1500   0 219268062      0      0      0 257717388      0      0      0 BMRU
lo        65536   0 311534133      0      0      0 311534133      0      0      0 LRU
eth0       1500   0 219335153      0      0      0 257792649      0      0      0 BMRU
lo        65536   0 311666747      0      0      0 311666747      0      0      0 LRU

iostat 15

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          10.98    0.52    3.15    0.30    0.00   85.05

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda               20.79 833.21       490.35   76292548   44898504

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          28.67    0.00    9.33    0.83    0.00   61.17

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda              46.67        32.00       873.60        480      13104

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          27.82    0.00   10.18    0.15    0.00   61.85

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda              25.47        33.60      1692.80        504      25392

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          28.84    0.00   10.21    0.06    0.00   60.89

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda              11.27        32.53       712.00        488      10680

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          27.83    0.00   10.45    0.95    0.00   60.76

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda              58.67        18.67      1861.33        280      27920

vmstat 15

procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu-----
r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
12  0  0 11291076 1118568 12297544    0    0    17    10    8    4 11  3 
85  0  0
3  0   0 11248336 1118568 12299980    0    0     9   399 29547 52458 27  8 65  0  0
11  0  0 11224000 1118568 12303336    0    0    26   683 29830 52331 27  
9 63  1  0
21  0  0 11193280 1118568 12306744    0    0    39   347 31722 67277 31 10 59  0  0
16  0  0 11158436 1118568 12310428    0    0    17   809 30677 62257 27 
10 62  1  0
10  0  0 11129768 1118568 12314088    0    0    32   472 32385 61133 30 10 60  0  0
7  2   0 11120616 1118572 12317448    0    0    18   761 31739 66259 29 10 60  0  0

top -b -n 1 | head

top - 20:17:07 up 1 day,  1:12,  1 user,  load average: 3.95, 4.11, 4.29
Tasks: 641 total,  15 running, 626 sleeping,   0 stopped,   0 zombie
Cpu(s): 10.8%us,  2.9%sy,  0.5%ni, 85.3%id,  0.3%wa,  0.0%hi,  0.2%si,  0.0%st
Mem:  32865012k total, 21206392k used, 11658620k free,  1118552k buffers
Swap:  8388604k total,        0k used,  8388604k free, 12165180k cached

PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
27709 mysql     20   0 5189m 1.0g  13m S 559.9  3.1   1800:22 mysqld
2750 nginx     10 -10  437m  42m 1488 R 19.2  0.1  46:05.13 nginx
3135 nginx     10 -10  433m  41m 1496 S 15.4  0.1  46:45.90 nginx

iotp

Total DISK READ: 64.98 K/s | Total DISK WRITE: 667.90 K/s
TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN     IO>    COMMAND
884 be/3 root        0.00 B/s  491.00 K/s  0.00 %  4.80 % [jbd2/sda5-8]
6091 be/4 nginx      64.98 K/s   64.98 K/s  0.00 %  0.16 % php-fpm: pool www
16943 be/4 mysql       0.00 B/s    3.61 K/s  0.00 %  0.00 % mysqld
27829 be/4 mysql       0.00 B/s    7.22 K/s  0.00 %  0.00 % mysqld
27862 be/4 mysql       0.00 B/s    0.00 B/s  0.00 %  0.00 % mysqld 
27887 be/4 mysql       0.00 B/s  238.28 K/s  0.00 %  0.00 % mysqld 
2745 be/2 nginx       0.00 B/s    3.61 K/s  0.00 %  0.00 % nginx: worker process
2747 be/2 nginx       0.00 B/s    7.22 K/s  0.00 %  0.00 % nginx: worker process
2748 be/2 nginx       0.00 B/s   10.83 K/s  0.00 %  0.00 % nginx: worker process
2750 be/2 nginx       0.00 B/s    3.61 K/s  0.00 %  0.00 % nginx: worker process
2751 be/2 nginx       0.00 B/s    7.22 K/s  0.00 %  0.00 % nginx: worker process
2788 be/2 nginx       0.00 B/s   14.44 K/s  0.00 %  0.00 % nginx: worker process
2872 be/2 nginx       0.00 B/s   10.83 K/s  0.00 %  0.00 % nginx: worker process
2919 be/2 nginx       0.00 B/s    7.22 K/s  0.00 %  0.00 % nginx: worker process
2920 be/2 nginx       0.00 B/s   10.83 K/s  0.00 %  0.00 % nginx: worker process
2991 be/2 nginx       0.00 B/s    3.61 K/s  0.00 %  0.00 % nginx: worker process
3038 be/2 nginx       0.00 B/s    3.61 K/s  0.00 %  0.00 % nginx: worker process
3121 be/2 nginx       0.00 B/s    7.22 K/s  0.00 %  0.00 % nginx: worker process
3135 be/2 nginx       0.00 B/s   14.44 K/s  0.00 %  0.00 % nginx: worker process
3218 be/2 nginx       0.00 B/s    3.61 K/s  0.00 %  0.00 % nginx: worker process
3256 be/2 nginx       0.00 B/s    3.61 K/s  0.00 %  0.00 % nginx: worker process
3258 be/2 nginx       0.00 B/s    3.61 K/s  0.00 %  0.00 % nginx: worker process
3316 be/2 nginx       0.00 B/s   10.83 K/s  0.00 %  0.00 % nginx: worker process
3325 be/2 nginx       0.00 B/s   14.44 K/s  0.00 %  0.00 % nginx: worker process
3390 be/2 nginx       0.00 B/s    7.22 K/s  0.00 %  0.00 % nginx: worker process
28264 be/4 mysql       0.00 B/s   57.76 K/s  0.00 %  0.00 % mysqld 
27959 be/4 mysql       0.00 B/s    0.00 B/s  0.00 %  0.00 % mysqld 
28000 be/4 mysql       0.00 B/s    0.00 B/s  0.00 %  0.00 % mysqld
5892 be/4 nginx       0.00 B/s   25.27 K/s  0.00 %  0.00 % php-fpm: pool www
5918 be/4 nginx       0.00 B/s    3.61 K/s  0.00 %  0.00 % php-fpm: pool www
5938 be/4 nginx       0.00 B/s    3.61 K/s  0.00 %  0.00 % php-fpm: pool www
5958 be/4 nginx       0.00 B/s   28.88 K/s  0.00 %  0.00 % php-fpm: pool www
5981 be/4 nginx       0.00 B/s   43.32 K/s  0.00 %  0.00 % php-fpm: pool www
5990 be/4 nginx       0.00 B/s   57.76 K/s  0.00 %  0.00 % php-fpm: pool www
6037 be/4 nginx       0.00 B/s   28.88 K/s  0.00 %  0.00 % php-fpm: pool www
6062 be/4 nginx       0.00 B/s   21.66 K/s  0.00 %  0.00 % php-fpm: pool www
6114 be/4 nginx       0.00 B/s    3.61 K/s  0.00 %  0.00 % php-fpm: pool www
1 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % init
2 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [kthreadd]
3 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [migration/0]
4 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [ksoftirqd/0]
5 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [stopper/0]
6 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [watchdog/0]
7 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [migration/1]
8 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [stopper/1]
9 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [ksoftirqd/1]
10 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [watchdog/1]
11 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [migration/2]
12 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [stopper/2]
13 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [ksoftirqd/2]
14 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [watchdog/2]
15 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [migration/3]
16 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [stopper/3]
17 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [ksoftirqd/3]
18 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [watchdog/3]
19 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [migration/4]
20 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [stopper/4]
21 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [ksoftirqd/4]
22 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [watchdog/4]
23 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [migration/5]
24 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [stopper/5]

Nginx configuration

user  nginx nginx; 
worker_processes  24; 
worker_rlimit_nofile 300000;
pid  /var/run/nginx.pid;
events {
use                 epoll;
worker_connections  20192;
multi_accept        on;
   }
http {
server_tokens off;
include       /etc/nginx/mime.types;
default_type  application/octet-stream;
set_real_ip_from 127.0.0.1;
real_ip_header X-Forwarded-For;
sendfile        on;
tcp_nopush      on;
charset UTF-8;
fastcgi_buffers 8 16k;
fastcgi_buffer_size 32k;
proxy_http_version 1.1;
proxy_set_header Connection "";
proxy_connect_timeout  600s;
proxy_send_timeout  600s;
proxy_read_timeout  600s;
fastcgi_send_timeout 600s;
fastcgi_read_timeout 600s;
keepalive_timeout  60;
keepalive_disable   none;
max_ranges                1;
reset_timedout_connection on;
tcp_nodelay               on;
open_file_cache max=10000 inactive=5m;
open_file_cache_valid 10m;
open_file_cache_min_uses 5;
open_file_cache_errors off;
client_body_buffer_size  1m;

client_header_buffer_size 1m;

client_max_body_size 1m;

large_client_header_buffers 2 1m;
gzip  on;
gzip_comp_level 5;
gzip_min_length 1000;
gzip_proxied any;
gzip_vary on;
gzip_types
  application/atom+xml
  application/javascript
  application/json
  application/rss+xml
  application/vnd.ms-fontobject
  application/x-font-ttf
  application/x-web-app-manifest+json
  application/xhtml+xml
  application/xml
  font/opentype
  image/svg+xml
  image/x-icon
  text/css
  text/plain
  text/x-component;
#limit_req_zone  $binary_remote_addr  zone=gulag:1m   rate=50r/m;
Upstream to abstract backend connection(s) for PHP.
upstream php {
server unix:/tmp/php-fpm.sock;
server 127.0.0.1:9000;
}
include /etc/nginx/conf.d/*;
 }

php-fpm config http://pasted.co/d6a2fa9c

mysql Configuration

port = 3306
#bind-address    =  
default-storage-engine = ARIA 
aria-pagecache-buffer-size= 2048M 
key_buffer = 2024M 
query_cache_size = 64M 
query_cache_limit = 512M 
max_connections = 3000 
thread_cache_size = 512 
query_cache_min_res_unit = 0 
tmp_table_size = 32M 
max_heap_table_size = 32M 
table_cache=1024 
concurrent_insert=2
max_allowed_packet = 64M 
sort_buffer_size = 128K 
read_buffer_size = 512K 
read_rnd_buffer_size = 512K 
net_buffer_length = 2K 
thread_stack = 512K 
wait_timeout = 300 
table_definition_cache = 4000 
thread_handling = pool-of-threads 
host_cache_size = 2000 
skip_name_resolve 
thread-pool-max-threads= 1000

php.in http://pasted.co/523a81db

nginx error log nothing beside

[warn] 10063#10063: conflicting server name

mysql error log

also empty beside restarting log

php-fpm error log

[01-Jun-2016 21:54:00] WARNING: [pool www] child 9596, script '/var/www/index.php' (request: "POST /index.php") execution timed out (302.124548 sec), terminating 
[01-Jun-2016 21:54:00] WARNING: [pool www] child 9596 exited on signal 15 (SIGTERM) after 1203.573297 seconds from start 
[01-Jun-2016 21:54:00] NOTICE: [pool www] child 10116 started 
[01-Jun-2016 22:07:20] WARNING: [pool www] child 9450, script '/var/www/index.php' (request: "GET /index.php") execution timed out (331.919262 sec), terminating 
[01-Jun-2016 22:07:25] WARNING: [pool www] child 9450 exited on signal 15 (SIGTERM) after 2009.277917 seconds from start 
[01-Jun-2016 22:07:25] NOTICE: [pool www] >     >     child 10271 started

Top

top - 22:46:35 up  1:20,  1 user,  load average: 306.50, 283.23,
290.78 Tasks: 801 total,  89 running, 712 sleeping,   0 stopped,   0 zombie Cpu(s): 85.3%us, 14.2%sy,  0.0%ni,  0.0%id,  0.0%wa,  0.0%hi, 
0.5%si,  0.0%st Mem:  32865348k total, 13868888k used, 18996460k free,    28104k buffers Swap:  8388604k total,   727164k used,  7661440k free, 
394328k cached
PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND 
7962 mysql     20   0 3082m 321m 4072 S 114.1  1.0  82:07.36 mysqld 
9405 php       20   0 1387m 924m 8156 R  4.9  2.9   1:33.70 php-fpm 
9505 php       20   0  714m 252m 8116 R  4.9  0.8   1:29.31 php-fpm 
9448 php       20   0  475m  21m  14m S  4.6  0.1   1:34.66 php-fpm 
9533 php       20   0  472m  19m  12m D  4.6  0.1   1:34.74 php-fpm 
9628 php       20   0  516m  62m  14m R  4.6  0.2   1:40.33 php-fpm 
9685 php       20   0  670m 210m 8156 R  4.6  0.7   1:31.48 php-fpm 
9380 php       20   0  712m 253m 7896 R  4.3  0.8   1:28.64 php-fpm 
9387 php       20   0  507m  48m 7564 R  4.3  0.2   1:31.58 php-fpm 
9395 php       20   0  473m  20m  12m D  4.3  0.1   1:36.94 php-fpm 
9435 php       20   0 1597m 1.1g  12m R  4.3  3.6   1:34.38 php-fpm 
9461 php       20   0  500m  47m  13m R  4.3  0.1   1:36.33 php-fpm 
9493 php       20   0  491m  32m 7560 R  4.3  0.1   1:41.10 php-fpm 
9514 php       20   0  477m  25m  14m R  4.3  0.1   1:32.29 php-fpm 
9554 php       20   0  513m  59m  13m R  4.3  0.2   1:33.10 php-fpm 
9556 php       20   0  512m  57m  12m R  4.3  0.2   1:24.28 php-fpm 
9566 php       20   0  473m  15m 7524 D  4.3  0.0   1:33.42 php-fpm 
9598 php       20   0  531m  71m 7764 R  4.3  0.2   1:32.87 php-fpm 
9620 php       20   0  473m  21m  14m S  4.3  0.1   1:20.62 php-fpm 
9653 php       20   0  509m  54m  11m R  4.3  0.2   1:36.90 php-fpm 
9677 php       20   0  513m  23m 7392 R  4.3  0.1   1:34.66 php-fpm 
9689 php       20   0  714m 258m  14m R  4.3  0.8   1:30.03 php-fpm 
9695 php       20   0 1431m 973m  12m R  4.3  3.0   1:31.20 php-fpm 
9408 php       20   0  472m  13m 7456 S  4.1  0.0   1:35.02 php-fpm 
9463 php       20   0  514m  60m  12m R  4.1  0.2   1:33.01 php-fpm 
9504 php       20   0  505m  48m  21m R  4.1  0.1   1:36.00 php-fpm 
9507 php       20   0  491m  38m  12m R  4.1  0.1   1:37.42 php-fpm 
9516 php       20   0  493m  32m 8108 R  4.1  0.1   1:35.84 php-fpm 
9522 php       20   0  627m 166m 7816 R  4.1  0.5   1:30.21 php-fpm 
9524 php       20   0  936m 480m  13m R  4.1  1.5   1:33.89 php-fpm 
9534 php       20   0  475m  15m 8060 D  4.1  0.0   1:38.95 php-fpm 
9638 php       20   0  504m  45m 7652 S  4.1  0.1   1:35.04 php-fpm 
9641 php       20   0  777m 314m 8508 R  4.1  1.0   1:37.97 php-fpm 
9655 php       20   0  507m  56m  14m R  4.1  0.2   1:28.89 php-fpm 
9671 php       20   0  513m  54m 7544 R  4.1  0.2   1:34.73 php-fpm 
9684 php       20   0  514m  61m  14m R  4.1  0.2   1:36.00 php-fpm 
9793 php       20   0  472m  19m  13m R  4.1  0.1   1:28.19 php-fpm 
9806 php       20   0  515m  55m 8172 R  4.1  0.2   1:35.00 php-fpm 
9818 php       20   0  771m 308m 8068 R  4.1  1.0   1:31.61 php-fpm 
9825 php       20   0  473m  13m 8740 S  4.1  0.0   1:29.41 php-fpm 
9850 php       20   0  475m  18m  11m S  4.1  0.1   1:35.34 php-fpm 
9861 php       20   0  473m  14m 7556 S  4.1  0.0   1:32.32 php-fpm 
9425 php       20   0  475m  21m  13m S  3.8  0.1   1:34.69 php-fpm 
9446 php       20   0  471m  12m 7640 S  3.8  0.0   1:36.17 php-fpm 
9579 php       20   0  471m  12m 7592 D  3.8  0.0   1:37.13 php-fpm 
9588 php       20   0  491m  32m 7468 R  3.8  0.1   1:36.44 php-fpm

[root@davidsm01 ~]# vmstat

procs -----------memory---------- ---swap-- -----io---- --system-------cpu-----  
r  b   swpd   free   buff  cache   si   so    bi    bo   in  cs us sy id wa st 
273  0 669112 13713228  76480 541424   21   30  69    50  567  165 83 15  1  1  0

tcpdump -i eth0

22:55:11.703006 IP 141.101.70.49.36344 >
expressvpn16.clients.netelligent.ca.http: Flags [.], ack 2921, win 77,
length 0 22:55:11.703017 IP 141.101.70.49.36344 >
expressvpn16.clients.netelligent.ca.http: Flags [.], ack 3718, win 80,
length 0 22:55:11.717115 IP 108.162.214.215.23885 >
expressvpn16.clients.netelligent.ca.http: Flags [.], ack 3878216429,
win 29, length 0 22:55:11.717173 IP 108.162.214.215.23885 >
expressvpn16.clients.netelligent.ca.http: Flags [P.], seq 0:485, ack
1, win 29, length 485 22:55:11.717192 IP
expressvpn16.clients.netelligent.ca.http > 108.162.214.215.23885:
Flags [.], ack 485, win 123, length 0
15493 packets captured 
43347 packets received by filter 
27820 packets dropped by kernel

mpstat -P ALL

Linux 2.6.32-642.1.1.el6.x86_64 (davidsm01.localdomain)         06/01/2016      _x86_64_        (12 CPU)
11:14:56 PM  CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest   %idle
11:14:56 PM  all   83.76    0.00   14.16    1.08    0.00    0.56    0.00    0.00    0.43
11:14:56 PM    0   84.53    0.00   13.89    1.18    0.00    0.12    0.00    0.00    0.28
11:14:56 PM    1   83.44    0.00   14.19    1.14    0.00    0.85    0.00    0.00    0.38
11:14:56 PM    2   83.38    0.00   14.23    1.16    0.00    0.85    0.00    0.00    0.38
11:14:56 PM    3   84.59    0.00   13.76    1.15    0.00    0.10    0.00    0.00    0.39
11:14:56 PM    4   84.30    0.00   14.04    1.14    0.00    0.13    0.00    0.00    0.39
11:14:56 PM    5   84.66    0.00   13.73    1.10    0.00    0.11    0.00    0.00    0.40
11:14:56 PM    6   83.24    0.00   14.39    1.16    0.00    0.84    0.00    0.00    0.37
11:14:56 PM    7   84.44    0.00   13.94    0.55    0.00    0.15    0.00    0.00    0.92
11:14:56 PM    8   83.07    0.00   14.54    1.17    0.00    0.87    0.00    0.00    0.35
11:14:56 PM    9   83.07    0.00   14.43    1.09    0.00    0.97    0.00    0.00    0.45
11:14:56 PM   10   83.11    0.00   14.50    1.05    0.00    0.84    0.00    0.00    0.50
11:14:56 PM   11   83.31    0.00   14.33    1.13    0.00    0.84    0.00    0.00    0.39

ps -eo pcpu,pid,user,args | sort -k 1 -r | head -10

%CPU   PID USER     COMMAND
 2.3 10772 php      php-fpm: pool www
 2.3 10490 php      php-fpm: pool www
 2.2  9869 php      php-fpm: pool www
 2.2  9868 php      php-fpm: pool www
 2.2  9864 php      php-fpm: pool www
 2.2  9854 php      php-fpm: pool www
 2.2  9850 php      php-fpm: pool www
 2.2  9827 php      php-fpm: pool www
 2.2  9824 php      php-fpm: pool www

ss -s

 Total: 14878 (kernel 15172)
 TCP:   30203 (estab 11194, closed 16290, orphaned 89, synrecv 0, timewait >      16281/0), ports 4092
Transport Total     IP        IPv6

    15172     -         -



RAW       0         0         0
 UDP       0         0         0
 TCP       13913     13908     5
 INET      13913     13908     5
 FRAG      0         0         0

Device eth0 [] (1/5):

 Incoming:
Curr: 1.61 MBit/s
 Avg: 1.78 MBit/s
 Min: 886.95 kBit/s
 Max: 5.11 MBit/s
 Ttl: 2.06 GByte
Outgoing
Curr: 4.82 MBit/s
 Avg: 5.08 MBit/s
 Min: 2.10 MBit/s
 Max: 14.50 MBit/s
 Ttl: 5.59 GByte

score 2 · Answer 1 · edited Apr 13 '17 at 12:14

Your server is suffering because database cannot provide data quickly enough to your web application.

There can be several reasons why this happens:

1) There aren't proper indices in the database, which means MySQL has to do full table scans to retrieve data which takes a long time.

2) Caching isn't implemented properly in the web application.

3) Your site is under DDoS attack, which could be succesful because of the facts above.

From your web server log comment, option 3 might be the reason. You should analyse the logs with some analysis software to see how it is different from a "normal" day, and therefore deduce if there is some unwanted traffic and where it originates from.

You can read I am under DDoS. What can I do? to see what options you have for DDoS mitigation.

score 0 · Answer 2 · answered Jul 25 '17 at 12:34

There is an awful lot going wrong here. And there is a lot missing from your analysis/question.

The most relevant part of what you have posted here is the output of top. With only a single snapshot its hard to tell exactly what's going on, but the system is not coping with demand. i.e. this is an issue with capacity.

That you have made no mention of the obvious candidates for improving performance suggests that there is a lot of scope for improving the capacity of the system by improving its performance.

Based on the single top snapshot (which means a lot of guesswork) mysql is only using 10% (112/12) of the available CPU, implying that php-fpm is using up the rest. This implies that its your PHP code and pre-emptive context switching which is killing your performance.

But your assertion that "Enabling Nginx cache jump the load back to 100s" (which I take to mean that moving the NGinx cache from ram to filesystem causes the page load time to increase to 100s) suggests that the issue lies in the Nginx cache or storage I/O. Is your nginx disk cache on the same block device as the database?

Unless you are giving away bitcoins, it is unlikely that your users are actually putting up with load times like this and still hosing your site. Suggesting that you are getting DOSsed - it should be easy to see this from some simple log analysis. However DOS attacks rarely last for more than a few hours. If you've been under attack for over a year, then you must have done something to really upset the NSA/CIA/FSB.

Your software stack, Linux, nginx, PHP & MySQL, does not make good use of big iron - and such an architecture is only cost effective when running multiple servers in a colocation facility.

Server suddenly became too slow with timeout error

Upstream to abstract backend connection(s) for PHP.

mysql Configuration

2 Answers2