Yesterday I updated my Wordpress website to the latest version of WordPress (6.0), and I updated several other plugins to their latest versions. After the updates everything appeared to be working fine, so I disabled my maintenance page. Several hours later my remote MySQL server (used for the WP database) crashed and this morning my WebServer (Nginx) couldn’t connect to the database.
I restored my production servers from backup, and I’ve attached the potentially corrupted volume from the MySQL server to a new instance. MySQL won’t start, and I’ve been reviewing the logs, but they are well above my level of expertise.
Below are two sections of the logs. The first describes potential corruption? I added innodb_force_recovery = 6 to my CNF file and that allows the MySQL to start, but I don’t really know what this means. Does that mean there is a 100% guarantee of corruption and if so, how would I find out the cause / where?
The second section has a buffer pool error, which is a little easier for me to understand. I commented out my innodb_buffer_pool_size setting in my CNF file, but that had no affect on MySQL starting without innodb_force_recovery set to 6.
I’m hoping someone can explain the errors and potentially how to find out the cause/s?
MySQL8.0.29 R6G.Large (2 cores, 16gb ram) Buffer pool: 12000M
Section 1:
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/8.0/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
18:49:04 UTC - mysqld got signal 6 ;
Most likely, you have hit a bug, but this error can also be caused by malfunctioning hardware.
Thread pointer: 0xfffc28000b20
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = fffc488565f0 thread_stack 0x100000
/usr/sbin/mysqld(my_print_stacktrace(unsigned char const*, unsigned long)+0x44) [0x1d0e2a4]
/usr/sbin/mysqld(print_fatal_signal(int)+0x28c) [0xe3278c]
/usr/sbin/mysqld(my_server_abort()+0xa0) [0xe32920]
/usr/sbin/mysqld(my_abort()+0x14) [0x1d08244]
/usr/sbin/mysqld(ut_dbg_assertion_failed(char const*, char const*, unsigned long)+0x290) [0x1f79ba0]
/usr/sbin/mysqld() [0x1f7c42c]
/usr/sbin/mysqld(page_copy_rec_list_end_no_locks(buf_block_t*, buf_block_t*, unsigned char*, dict_index_t*, mtr_t*)+0x2dc) [0x1ec1dec]
/usr/sbin/mysqld(page_copy_rec_list_end(buf_block_t*, buf_block_t*, unsigned char*, dict_index_t*, mtr_t*)+0x300) [0x1ec2270]
/usr/sbin/mysqld(btr_compress(btr_cur_t*, bool, mtr_t*)+0x624) [0x1fa53f4]
/usr/sbin/mysqld(btr_cur_pessimistic_delete(dberr_t*, bool, btr_cur_t*, unsigned int, bool, unsigned long, unsigned long, unsigned long, mtr_t*, btr_pcur_t*, purge_node_t*)+0x1cc) [0x1fb130c]
/usr/sbin/mysqld() [0x1f02bf8]
/usr/sbin/mysqld(row_purge_step(que_thr_t*)+0x4f4) [0x1f05364]
/usr/sbin/mysqld(que_run_threads(que_thr_t*)+0x578) [0x1ece5e8]
/usr/sbin/mysqld(srv_worker_thread()+0x21c) [0x1f3184c]
/usr/sbin/mysqld(std::thread::_State_impl<std::thread::_Invoker<std::tuple<Detached_thread, void (*)()> > >::_M_run()+0xd4) [0x1e85774]
/usr/sbin/mysqld() [0x244623c]
/lib64/libpthread.so.0(+0x722c) [0xffffab1ec22c]
/lib64/libc.so.6(+0xd2e5c) [0xffffaaa99e5c]
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0): Connection ID (thread ID): 0
Status: NOT_KILLED
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
2022-05-25T18:49:04.565524Z 0 [Warning] [MY-010918] [Server] 'default_authentication_plugin' is deprecated and will be removed in a future release. Please use authentication_policy instead.
2022-05-25T18:49:04.565544Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.29) starting as process 6679
2022-05-25T18:49:04.571539Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
2022-05-25T18:49:05.807854Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended.
2022-05-25T18:49:06.622221Z 0 [System] [MY-010229] [Server] Starting XA crash recovery...
2022-05-25T18:49:06.625449Z 0 [System] [MY-010232] [Server] XA crash recovery finished.
2022-05-25T18:49:07.251794Z 0 [Warning] [MY-010068] [Server] CA certificate ca.pem is self signed.
2022-05-25T18:49:07.251829Z 0 [System] [MY-013602] [Server] Channel mysql_main configured to support TLS. Encrypted connections are now supported for this channel.
2022-05-25T18:49:07.272581Z 0 [System] [MY-011323] [Server] X Plugin ready for connections. Bind-address: '::' port: 33060, socket: /var/run/mysqld/mysqlx.sock
2022-05-25T18:49:07.272631Z 0 [System] [MY-010931] [Server] /usr/sbin/mysqld: ready for connections. Version: '8.0.29' socket: '/var/lib/mysql/mysql.sock' port: 3306 MySQL Community Server - GPL.
2022-05-25T18:49:07.347411Z 0 [ERROR] [MY-012687] [InnoDB] [FATAL] Rec offset 99, cur1 offset 4038, cur2 offset 16004
2022-05-25T18:49:07.347449Z 0 [ERROR] [MY-013183] [InnoDB] Assertion failure: page0page.cc:502:ib::fatal triggered thread 281457662619536
Section 2:
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
2022-05-26T12:27:29.518146Z 0 [Warning] [MY-010918] [Server] 'default_authentication_plugin' is deprecated and will be removed in a future release. Please use authentication_policy instead.
2022-05-26T12:27:29.518165Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.29) starting as process 17135
2022-05-26T12:27:29.524074Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
2022-05-26T13:20:40.613066Z 0 [Warning] [MY-010918] [Server] 'default_authentication_plugin' is deprecated and will be removed in a future release. Please use authentication_policy instead.
2022-05-26T13:20:40.613087Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.29) starting as process 1086
2022-05-26T13:20:41.866918Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
2022-05-26T13:20:49.824464Z 0 [Warning] [MY-012681] [InnoDB] page_aligned_alloc mmap(137232384 bytes) failed; errno 12
2022-05-26T13:20:49.844869Z 0 [Warning] [MY-012681] [InnoDB] page_aligned_alloc mmap(137232384 bytes) failed; errno 12
2022-05-26T13:20:49.961232Z 1 [ERROR] [MY-012956] [InnoDB] Cannot allocate memory for the buffer pool
2022-05-26T13:20:49.961279Z 1 [ERROR] [MY-012930] [InnoDB] Plugin initialization aborted with error Generic error.
2022-05-26T13:20:49.962080Z 1 [ERROR] [MY-010334] [Server] Failed to initialize DD Storage Engine
2022-05-26T13:20:49.962240Z 0 [ERROR] [MY-010020] [Server] Data Dictionary initialization failed.
2022-05-26T13:20:49.962355Z 0 [ERROR] [MY-010119] [Server] Aborting
2022-05-26T13:20:49.966630Z 0 [System] [MY-010910] [Server] /usr/sbin/mysqld: Shutdown complete (mysqld 8.0.29) MySQL Community Server - GPL.
Any help would be great.
Update: I ran mysqlcheck on my faulty database and it found corruption. Looks like the cause is WooCommerce?
Here is the error mysqlcheck returns:
production.wp_wc_admin_note_actions
Warning : InnoDB: The B-tree of index PRIMARY is corrupted.
Warning : InnoDB: Index 'note_id' contains 1004 entries, should be 18446744073709551615.
error : Corrupt
Jack
