First thank you, I am not very knowledgeable in MYSQL and am purely self taught.
I moved from metal to AWS(EC2, with maxed out IOPS:3000) and in the process we also upgrade from MYSQL5.6 to MYSQL8.0.28.
We have had major memory issues with MYSQL8 and cannot seem to track it down. It takes several days (~5-10) to exhaust memory on our production server (~1k requests per second) MYSQL is takes up over 100% of our 16GB of memory.
It slowly creeps up with a few random spikes.
we have 16GB of memory, the old metal server also had 16GB.
all tables are innodb.
I reduced innodb_buffer_pool_size from 8GB to 4GB to try and stabilize the server with no joy.
any advise would be a lot of help for me to understand what is going on.
UPDATE 4-25-2022
I wanted to let this run for a few days to let it get back to an add state. I changed a bunch of settings based on another post recommendation, so i removed the previous information for this thread.
mysqlc.cnf https://pastebin.com/FRuh55Lx
The paste bins request by Wilson below:
A)439
B) https://pastebin.com/86Tj6YBe
C) https://pastebin.com/Jn19dyu7
D)
mysql> SHOW FULL PROCESSLIST;
+---------+-----------------+----------------------+-------+---------+--------+------------------------+-----------------------+
| Id | User | Host | db | Command | Time | State | Info |
+---------+-----------------+----------------------+-------+---------+--------+------------------------+-----------------------+
| 5 | event_scheduler | localhost | NULL | Daemon | 357369 | Waiting on empty queue | NULL |
| 2208882 | root | localhost | NULL | Sleep | 330 | | NULL |
| 2210947 | root | localhost | NULL | Sleep | 144 | | NULL |
| 2211462 | calypso | <removed>:33362 | <removed>| Sleep | 52 | | NULL |
| 2211998 | freeradius | localhost | <removed>| Sleep | 0 | | NULL |
| 2212151 | calypso | <removed>:33992 | <removed>| Sleep | 52 | | NULL |
| 2212502 | freeradius | localhost | ikeja | Sleep | 0 | | NULL |
| 2212641 | root | localhost | NULL | Query | 0 | init | SHOW FULL PROCESSLIST |
+---------+-----------------+----------------------+-------+---------+--------+------------------------+-----------------------+
8 rows in set (0.00 sec)
E)
mysql> STATUS;
--------------
mysql Ver 8.0.28-0ubuntu0.20.04.3 for Linux on x86_64 ((Ubuntu))
Connection id: 2212641
Current database:
Current user: root@localhost
SSL: Not in use
Current pager: stdout
Using outfile: ''
Using delimiter: ;
Server version: 8.0.28-0ubuntu0.20.04.3 (Ubuntu)
Protocol version: 10
Connection: Localhost via UNIX socket
Server characterset: utf8mb4
Db characterset: utf8mb4
Client characterset: utf8mb4
Conn. characterset: utf8mb4
UNIX socket: /var/run/mysqld/mysqld.sock
Binary data as: Hexadecimal
Uptime: 4 days 3 hours 18 min 15 sec
Threads: 12 Questions: 37878605 Slow queries: 35 Opens: 1647 Flush tables: 3 Open tables: 1501 Queries per second avg: 105.955
