I am running a migration script which copies a large amount of data from one table into another and the process is now running for more than 24 hours and failing, as well as generating alarms from the AWS Cloudwatch monitor regarding high IOPS and through-put.
Server version: 10.2.12-MariaDB-log MariaDB Server
I also see these errors:
2019-04-23 16:09:44 47291578124032 [Warning] Aborted connection 17189 to db: 'mydb' user: 'APPDBA' host: '1.2.3.4' (The table 'modification' is full)
2019-04-23 16:09:44 47291578124032 [ERROR] WSREP: failed to initialize io-cache
2019-04-23 16:09:44 47291578124032 [Warning] WSREP: binlog trx cache not empty (0 bytes) @ connection close 17189
2019-04-23 16:09:44 47291578124032 [Warning] WSREP: binlog stmt cache not empty (0 bytes) @ connection close 17189
and (although I've no idea if it's connected - it is much later):
2019-04-24 13:33:24 47292002281216 [Warning]
Aborted connection 18175 to db: 'mydb' user: 'service_user_a' host: '1.2.3.4' (Got an error reading communication packets)
These are the tables and the data copy SQL:
CREATE TABLE old_mod (
id INT UNSIGNED NOT NULL AUTO_INCREMENT,
userId INT UNSIGNED NOT NULL,
epochTime BIGINT NOT NULL,
forecastId INT UNSIGNED NOT NULL,
description CHAR(12) NOT NULL,
auxText TEXT NULL,
auxDate DATE NULL,
CONSTRAINT pk_modification PRIMARY KEY (id),
CONSTRAINT fk_modification_forecastId FOREIGN KEY(forecastId)
REFERENCES forecast(id),
CONSTRAINT fk_modification_userId FOREIGN KEY (userId)
REFERENCES user(id)
) ENGINE=InnoDB
MAX_ROWS 12345678;
CREATE INDEX ix_modification_query_by_user_and_forecast
ON old_mod (userId, forecastId, description, auxDate, epochTime);
CREATE INDEX ix_modification_last_save_query
ON old_mod (forecastId, description, auxDate);
I drop the indices on the original table before creating the new one, so that I can re-use the index names - I didn't think that would slow it down - or does it?
In my operation, originally the table above is modification but I rename it to old_mod. I would have just altered it in place, but I kept getting other errors - see here if interested: MariaDB "ERROR 1034 (HY000) at line 41: Index for table 'xyz' is corrupt" on importing dump
The operation then creates this table and inserts into it:
CREATE TABLE modification (
id INT UNSIGNED NOT NULL AUTO_INCREMENT,
username CHAR(30) NOT NULL,
display_name VARCHAR(255) NULL,
epoch_time BIGINT NOT NULL,
forecast_id INT UNSIGNED NOT NULL,
forecast_symbol VARCHAR(255) NOT NULL,
group_id INT UNSIGNED NOT NULL,
group_name CHAR(100) NOT NULL,
description CHAR(12) NOT NULL,
aux_text TEXT NULL,
aux_date DATE NULL,
CONSTRAINT pk_modification PRIMARY KEY (id),
KEY ix_modification_query_by_user_and_forecast
(username, forecast_id, description, aux_date, epoch_time),
KEY ix_modification_last_save_query
(forecast_id, description, aux_date),
KEY ix_modification_forecast_symbol (forecast_symbol)
) ENGINE=InnoDB
DEFAULT CHARSET=utf8mb4
COLLATE=utf8mb4_unicode_ci
MAX_ROWS 500123123;
-- add username to modification via userid
INSERT INTO modification (username, epoch_time, forecast_id, forecast_symbol,
group_id, group_name,
description, aux_text, aux_date)
SELECT u.name, m.epochTime, m.forecastId, f.name, 1, 'Not recorded',
m.description, m.auxText, m.auxDate
FROM old_mod m
LEFT JOIN `user` u ON u.id = m.userId
LEFT JOIN forecast f ON f.id = m.forecastId;
I'm not sure what config I need to adjust, or whether I can improve my table schema.
My config:
aria_block_size 8192
aria_checkpoint_interval 30
aria_checkpoint_log_activity 1048576
aria_encrypt_tables OFF
aria_force_start_after_recovery_failures 0
aria_group_commit none
aria_group_commit_interval 0
aria_log_file_size 1073741824
aria_log_purge_type immediate
aria_max_sort_file_size 9223372036853727232
aria_page_checksum ON
aria_pagecache_age_threshold 300
aria_pagecache_buffer_size 134217728
aria_pagecache_division_limit 100
aria_pagecache_file_hash_size 512
aria_recover_options BACKUP,QUICK
aria_repair_threads 1
aria_sort_buffer_size 268434432
aria_stats_method nulls_unequal
aria_sync_log_dir NEWFILE
aria_used_for_temp_tables ON
auto_increment_increment 1
auto_increment_offset 1
autocommit ON
automatic_sp_privileges ON
back_log 558
basedir /rdsdbbin/mysql/
big_tables OFF
binlog_annotate_row_events ON
binlog_cache_size 32768
binlog_checksum CRC32
binlog_commit_wait_count 0
binlog_commit_wait_usec 100000
binlog_direct_non_transactional_updates OFF
binlog_format MIXED
binlog_optimize_thread_scheduling ON
binlog_row_image FULL
binlog_stmt_cache_size 32768
bulk_insert_buffer_size 8388608
check_constraint_checks ON
collation_connection latin1_swedish_ci
collation_database latin1_swedish_ci
collation_server latin1_swedish_ci
completion_type NO_CHAIN
concurrent_insert AUTO
connect_timeout 10
datadir /rdsdbdata/db/
default_storage_engine InnoDB
default_tmp_storage_engine
default_week_format 0
delay_key_write ON
delayed_insert_limit 100
delayed_insert_timeout 300
delayed_queue_size 1000
div_precision_increment 4
encrypt_binlog OFF
encrypt_tmp_disk_tables OFF
encrypt_tmp_files OFF
enforce_storage_engine
event_scheduler OFF
expensive_subquery_limit 100
expire_logs_days 0
explicit_defaults_for_timestamp OFF
extra_max_connections 1
extra_port 0
flush OFF
flush_time 0
foreign_key_checks ON
general_log OFF
general_log_file /rdsdbdata/log/general/mysql-general.log
group_concat_max_len 1048576
have_compress YES
have_crypt YES
have_dynamic_loading YES
have_geometry YES
have_openssl YES
have_profiling YES
have_query_cache YES
have_rtree_keys YES
have_ssl YES
have_symlink YES
histogram_size 0
histogram_type SINGLE_PREC_HB
host_cache_size 730
hostname ip-172-20-2-15
ignore_builtin_innodb OFF
ignore_db_dirs
init_connect
init_file
init_slave
innodb_adaptive_flushing ON
innodb_adaptive_flushing_lwm 10.000000
innodb_adaptive_hash_index ON
innodb_adaptive_hash_index_partitions 8
innodb_adaptive_hash_index_parts 8
innodb_adaptive_max_sleep_delay 150000
innodb_autoextend_increment 64
innodb_autoinc_lock_mode 1
innodb_background_scrub_data_check_interval 3600
innodb_background_scrub_data_compressed OFF
innodb_background_scrub_data_interval 604800
innodb_background_scrub_data_uncompressed OFF
innodb_buf_dump_status_frequency 0
innodb_buffer_pool_chunk_size 134217728
innodb_buffer_pool_dump_at_shutdown ON
innodb_buffer_pool_dump_now OFF
innodb_buffer_pool_dump_pct 25
innodb_buffer_pool_filename ib_buffer_pool
innodb_buffer_pool_instances 8
innodb_buffer_pool_load_abort OFF
innodb_buffer_pool_load_at_startup ON
innodb_buffer_pool_load_now OFF
innodb_buffer_pool_populate OFF
innodb_buffer_pool_size 24696061952
innodb_change_buffer_max_size 25
innodb_change_buffering all
innodb_checksum_algorithm crc32
innodb_checksums ON
innodb_cleaner_lsn_age_factor DEPRECATED
innodb_cmp_per_index_enabled OFF
innodb_commit_concurrency 0
innodb_compression_algorithm zlib
innodb_compression_default OFF
innodb_compression_failure_threshold_pct 5
innodb_compression_level 6
innodb_compression_pad_pct_max 50
innodb_concurrency_tickets 5000
innodb_corrupt_table_action deprecated
innodb_data_file_path ibdata1:12M:autoextend
innodb_data_home_dir /rdsdbdata/db/innodb
innodb_deadlock_detect ON
innodb_default_encryption_key_id 1
innodb_default_row_format dynamic
innodb_defragment OFF
innodb_disable_sort_file_cache OFF
innodb_disallow_writes OFF
innodb_doublewrite ON
innodb_empty_free_list_algorithm DEPRECATED
innodb_encrypt_log OFF
innodb_encrypt_tables OFF
innodb_encryption_rotate_key_age 1
innodb_encryption_rotation_iops 100
innodb_encryption_threads 0
innodb_fake_changes OFF
innodb_fast_shutdown 1
innodb_fatal_semaphore_wait_threshold 600
innodb_file_format Barracuda
innodb_file_format_check ON
innodb_file_format_max Barracuda
innodb_file_per_table ON
innodb_fill_factor 100
innodb_flush_log_at_timeout 1
innodb_flush_log_at_trx_commit 0
innodb_flush_method O_DIRECT
innodb_flush_neighbors 1
innodb_flush_sync ON
innodb_flushing_avg_loops 30
innodb_force_load_corrupted OFF
innodb_force_primary_key OFF
innodb_force_recovery 0
innodb_foreground_preflush DEPRECATED
innodb_ft_aux_table
innodb_ft_cache_size 8000000
innodb_ft_enable_diag_print OFF
innodb_ft_enable_stopword ON
innodb_ft_max_token_size 84
innodb_ft_min_token_size 3
innodb_ft_num_word_optimize 2000
innodb_ft_result_cache_limit 2000000000
innodb_ft_server_stopword_table
innodb_ft_sort_pll_degree 2
innodb_ft_total_cache_size 640000000
innodb_ft_user_stopword_table
innodb_idle_flush_pct 100
innodb_immediate_scrub_data_uncompressed OFF
innodb_instrument_semaphores OFF
innodb_io_capacity 200
innodb_io_capacity_max 2000
innodb_kill_idle_transaction 0
innodb_large_prefix ON
innodb_lock_schedule_algorithm vats
innodb_lock_wait_timeout 50
innodb_locking_fake_changes OFF
innodb_locks_unsafe_for_binlog OFF
innodb_log_arch_dir
innodb_log_arch_expire_sec 0
innodb_log_archive OFF
innodb_log_block_size 0
innodb_log_buffer_size 8388608
innodb_log_checksum_algorithm DEPRECATED
innodb_log_checksums ON
innodb_log_compressed_pages ON
innodb_log_file_size 134217728
innodb_log_files_in_group 2
innodb_log_group_home_dir /rdsdbdata/log/innodb
innodb_log_write_ahead_size 8192
innodb_lru_scan_depth 1024
innodb_max_bitmap_file_size 0
innodb_max_changed_pages 0
innodb_max_dirty_pages_pct 75.000000
innodb_max_dirty_pages_pct_lwm 0.000000
innodb_max_purge_lag 0
innodb_max_purge_lag_delay 0
innodb_max_undo_log_size 10485760
innodb_mirrored_log_groups 0
innodb_monitor_disable
innodb_monitor_enable
innodb_monitor_reset
innodb_monitor_reset_all
innodb_mtflush_threads 8
innodb_numa_interleave OFF
innodb_old_blocks_pct 37
innodb_old_blocks_time 1000
innodb_online_alter_log_max_size 134217728
innodb_open_files 2000
innodb_optimize_fulltext_only OFF
innodb_page_cleaners 4
innodb_page_size 16384
innodb_prefix_index_cluster_optimization OFF
innodb_purge_batch_size 300
innodb_purge_rseg_truncate_frequency 128
innodb_purge_threads 4
innodb_random_read_ahead OFF
innodb_read_ahead_threshold 56
innodb_read_io_threads 4
innodb_read_only OFF
innodb_replication_delay 0
innodb_rollback_on_timeout OFF
innodb_rollback_segments 128
innodb_sched_priority_cleaner 0
innodb_scrub_log OFF
innodb_scrub_log_speed 256
innodb_show_locks_held 0
innodb_show_verbose_locks 0
innodb_sort_buffer_size 1048576
innodb_spin_wait_delay 6
innodb_status_output OFF
innodb_status_output_locks OFF
innodb_strict_mode ON
innodb_support_xa ON
innodb_sync_array_size 1
innodb_sync_spin_loops 30
innodb_table_locks ON
innodb_temp_data_file_path ibtmp1:12M:autoextend
innodb_thread_concurrency 0
innodb_thread_sleep_delay 10000
innodb_tmpdir
innodb_track_changed_pages OFF
innodb_track_redo_log_now OFF
innodb_undo_directory ./
innodb_undo_log_truncate OFF
innodb_undo_logs 128
innodb_undo_tablespaces 0
innodb_use_atomic_writes ON
innodb_use_fallocate OFF
innodb_use_global_flush_log_at_trx_commit OFF
innodb_use_mtflush OFF
innodb_use_native_aio ON
innodb_use_stacktrace OFF
innodb_use_trim ON
innodb_version 5.7.20
innodb_write_io_threads 4
interactive_timeout 28800
join_buffer_size 262144
join_buffer_space_limit 2097152
join_cache_level 2
keep_files_on_create OFF
key_buffer_size 16777216
key_cache_age_threshold 300
key_cache_block_size 1024
key_cache_division_limit 100
key_cache_file_hash_size 512
key_cache_segments 0
large_files_support ON
large_page_size 0
large_pages OFF
local_infile ON
lock_wait_timeout 86400
locked_in_memory OFF
log_bin ON
log_bin_basename /rdsdbdata/log/binlog/mysql-bin-changelog
log_bin_compress OFF
log_bin_compress_min_len 256
log_bin_index /rdsdbdata/log/binlog/mysql-bin-changelog.index
log_bin_trust_function_creators ON
log_error /rdsdbdata/log/error/mysql-error.log
long_query_time 10.000000
low_priority_updates OFF
lower_case_file_system OFF
lower_case_table_names 0
master_verify_checksum OFF
max_allowed_packet 562036736
max_binlog_cache_size 18446744073709547520
max_binlog_size 134217728
max_binlog_stmt_cache_size 18446744073709547520
max_connect_errors 100
max_connections 2540
max_delayed_threads 20
max_digest_length 1024
max_error_count 64
max_heap_table_size 16777216
max_insert_delayed_threads 20
max_join_size 18446744073709551615
max_length_for_sort_data 1024
max_long_data_size 16777216
max_prepared_stmt_count 16382
max_recursive_iterations 4294967295
max_relay_log_size 134217728
max_seeks_for_key 4294967295
max_session_mem_used 9223372036854775807
max_sort_length 1024
max_sp_recursion_depth 0
max_statement_time 0.000000
max_tmp_tables 32
max_user_connections 0
max_write_lock_count 4294967295
metadata_locks_cache_size 1024
metadata_locks_hash_instances 8
min_examined_row_limit 0
mrr_buffer_size 262144
multi_range_count 256
mysql56_temporal_format ON
net_buffer_length 16384
net_read_timeout 30
net_retry_count 10
net_write_timeout 60
open_files_limit 65535
optimizer_prune_level 1
optimizer_search_depth 62
optimizer_selectivity_sampling_limit 100
optimizer_switch index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,de
optimizer_use_condition_selectivity 1
performance_schema OFF
pid_file /rdsdbdata/log/mysql-3306.pid
plugin_dir /rdsdbbin/mariadb-10.2.12.R1/lib/plugin/
plugin_maturity unknown
port 3306
preload_buffer_size 32768
profiling OFF
profiling_history_size 15
progress_report_time 5
protocol_version 10
query_alloc_block_size 16384
query_cache_limit 1048576
query_cache_min_res_unit 4096
query_cache_size 1048576
query_cache_strip_comments OFF
query_cache_type OFF
query_cache_wlock_invalidate OFF
query_prealloc_size 24576
range_alloc_block_size 4096
read_binlog_speed_limit 0
read_buffer_size 262144
read_only OFF
read_rnd_buffer_size 524288
rowid_merge_buff_size 8388608
secure_auth ON
secure_file_priv /tmp/
server_id 1853500884
session_track_schema ON
session_track_state_change OFF
session_track_system_variables
session_track_transaction_info OFF
skip_external_locking ON
skip_name_resolve OFF
skip_networking OFF
skip_show_database OFF
slow_launch_time 2
slow_query_log OFF
slow_query_log_file /rdsdbdata/log/slowquery/mysql-slowquery.log
socket /tmp/mysql.sock
sort_buffer_size 2097152
sql_auto_is_null OFF
sql_big_selects ON
sql_buffer_result OFF
sql_log_bin ON
sql_log_off OFF
sql_safe_updates OFF
sql_select_limit 18446744073709551615
sql_slave_skip_counter 0
sql_warnings OFF
standard_compliant_cte ON
storage_engine InnoDB
stored_program_cache 256
strict_password_validation ON
table_definition_cache 400
table_open_cache 2000
table_open_cache_instances 8
thread_cache_size 256
thread_concurrency 10
thread_handling one-thread-per-connection
thread_pool_idle_timeout 60
thread_pool_max_threads 65536
thread_pool_oversubscribe 3
thread_pool_prio_kickup_timer 1000
thread_pool_priority auto
thread_pool_size 4
thread_pool_stall_limit 500
thread_stack 262144
tmp_disk_table_size 18446744073709551615
tmp_memory_table_size 16777216
tmp_table_size 16777216
tmpdir /rdsdbdata/tmp
transaction_alloc_block_size 8192
transaction_prealloc_size 4096
tx_isolation REPEATABLE-READ
tx_read_only OFF
unique_checks ON
updatable_views_with_limit YES
use_stat_tables NEVER
userstat OFF
version 10.2.12-MariaDB-log
wait_timeout 28800
Update: here's the query plan.
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+------+-------------+-------+--------+---------------+---------+---------+---------------------+--------+----------+-------+
| 1 | SIMPLE | m | ALL | NULL | NULL | NULL | NULL | 441833 | 100.00 | |
| 1 | SIMPLE | u | eq_ref | PRIMARY | PRIMARY | 4 | myDb.m.userId | 1 | 100.00 | |
| 1 | SIMPLE | f | eq_ref | PRIMARY | PRIMARY | 4 | myDb.m.forecastId | 1 | 100.00 | |
