Exception : Can not read response from server. Expected to read 4 bytes, read 0 bytes before connection was unexpectedly lost

Question

There is a java based cron, which does some processing and at the end it inserts some rows to a InnoDB.

This cron runs 24 times a day at each hour, but fails at some same particular time everyday since quite few days. Exception :

The last packet successfully received from the server was 465,902 milliseconds ago.  The last packet sent successfully to the server was 8 milliseconds ago.
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
    at com.mysql.jdbc.Util.handleNewInstance(Util.java:400)
    at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1038)
    at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3434)
    at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3334)
    at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3774)
    at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2447)
    at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2594)
    at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2541)
    at com.mysql.jdbc.StatementImpl.executeUpdate(StatementImpl.java:1604)
    at com.mysql.jdbc.StatementImpl.executeUpdate(StatementImpl.java:1535)
    at DBConnection.QueryUpdate(DBConnection.java:41)
    at Mailer.insertDetailstoDB(Mailer.java:203)
    at Mailer.run(Mailer.java:64)
    at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.EOFException: Can not read response from server. Expected to read 4 bytes, read 0 bytes before connection was unexpectedly lost.
    at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:2926)
    at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3344)
    ... 11 more
com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure

Experiments done:

Issue happens only while insertion. Update/select query on other table works fine. Even select query on same ( issue ) table works fine.
JDBC URL : DB_URL=URL+DB+"?zeroDateTimeBehavior=convertToNull&autoReconnect=true&characterEncoding=UTF-8&characterSetResults=UTF-8&failOverReadOnly=true";
Took sql varibale dump, but both are same even when there is no issue.

Variable DUMP : 
auto_increment_increment.........2
auto_increment_offset.........1
autocommit.........ON
automatic_sp_privileges.........ON
back_log.........150
basedir........./usr/local/mysql6
big_tables.........OFF
bind_address.........*
binlog_cache_size.........32768
binlog_checksum.........CRC32
binlog_direct_non_transactional_updates.........OFF
binlog_error_action.........IGNORE_ERROR
binlog_format.........MIXED
binlog_gtid_simple_recovery.........OFF
binlog_max_flush_queue_time.........0
binlog_order_commits.........ON
binlog_row_image.........FULL
binlog_rows_query_log_events.........OFF
binlog_stmt_cache_size.........32768
binlogging_impossible_mode.........IGNORE_ERROR
block_encryption_mode.........aes-128-ecb
bulk_insert_buffer_size.........8388608
character_set_client.........latin1
character_set_connection.........latin1
character_set_database.........latin1
character_set_filesystem.........binary
character_set_results.........latin1
character_set_server.........latin1
character_set_system.........utf8
character_sets_dir........./usr/local/mysql-5.6.23-linux-glibc2.5-x86_64/share/charsets/
collation_connection.........latin1_swedish_ci
collation_database.........latin1_swedish_ci
collation_server.........latin1_swedish_ci
completion_type.........NO_CHAIN
concurrent_insert.........AUTO
connect_timeout.........10
core_file.........OFF
datadir........./usr/local/mysql6/data_99acres06/
date_format.........%Y-%m-%d
datetime_format.........%Y-%m-%d %H:%i:%s
default_storage_engine.........MyISAM
default_tmp_storage_engine.........InnoDB
default_week_format.........0
delay_key_write.........ON
delayed_insert_limit.........100
delayed_insert_timeout.........300
delayed_queue_size.........1000
disconnect_on_expired_password.........ON
div_precision_increment.........4
end_markers_in_json.........OFF
enforce_gtid_consistency.........OFF
eq_range_index_dive_limit.........10
event_scheduler.........OFF
expire_logs_days.........5
explicit_defaults_for_timestamp.........OFF
flush.........OFF
flush_time.........0
foreign_key_checks.........ON
ft_boolean_syntax.........+ -><()~*:""&|
ft_max_word_len.........84
ft_min_word_len.........4
ft_query_expansion_limit.........20
ft_stopword_file.........(built-in)
general_log.........OFF
general_log_file........./usr/local/mysql6/data/npl9dba08.log
group_concat_max_len.........1073741824
gtid_executed.........
gtid_mode.........OFF
gtid_owned.........
gtid_purged.........
have_compress.........YES
have_crypt.........YES
have_dynamic_loading.........YES
have_geometry.........YES
have_openssl.........DISABLED
have_profiling.........YES
have_query_cache.........YES
have_rtree_keys.........YES
have_ssl.........DISABLED
have_symlink.........YES
host_cache_size.........628
hostname.........npl9dba08.ieil.net
ignore_builtin_innodb.........OFF
ignore_db_dirs.........
init_connect.........
init_file.........
init_slave.........
innodb_adaptive_flushing.........ON
innodb_adaptive_flushing_lwm.........10
innodb_adaptive_hash_index.........ON
innodb_adaptive_max_sleep_delay.........150000
innodb_additional_mem_pool_size.........8388608
innodb_api_bk_commit_interval.........5
innodb_api_disable_rowlock.........OFF
innodb_api_enable_binlog.........OFF
innodb_api_enable_mdl.........OFF
innodb_api_trx_level.........0
innodb_autoextend_increment.........64
innodb_autoinc_lock_mode.........1
innodb_buffer_pool_dump_at_shutdown.........OFF
innodb_buffer_pool_dump_now.........OFF
innodb_buffer_pool_filename.........ib_buffer_pool
innodb_buffer_pool_instances.........40
innodb_buffer_pool_load_abort.........OFF
innodb_buffer_pool_load_at_startup.........OFF
innodb_buffer_pool_load_now.........OFF
innodb_buffer_pool_size.........42949672960
innodb_change_buffer_max_size.........25
innodb_change_buffering.........all
innodb_checksum_algorithm.........innodb
innodb_checksums.........ON
innodb_cmp_per_index_enabled.........OFF
innodb_commit_concurrency.........0
innodb_compression_failure_threshold_pct.........5
innodb_compression_level.........6
innodb_compression_pad_pct_max.........50
innodb_concurrency_tickets.........5000
innodb_data_file_path.........ibdata1:100M:autoextend
innodb_data_home_dir........./usr/local/mysql6/data
innodb_disable_sort_file_cache.........OFF
innodb_doublewrite.........ON
innodb_fast_shutdown.........1
innodb_file_format.........Antelope
innodb_file_format_check.........ON
innodb_file_format_max.........Antelope
innodb_file_per_table.........ON
innodb_flush_log_at_timeout.........1
innodb_flush_log_at_trx_commit.........1
innodb_flush_method.........O_DIRECT
innodb_flush_neighbors.........1
innodb_flushing_avg_loops.........30
innodb_force_load_corrupted.........OFF
innodb_force_recovery.........0
innodb_ft_aux_table.........
innodb_ft_cache_size.........8000000
innodb_ft_enable_diag_print.........OFF
innodb_ft_enable_stopword.........ON
innodb_ft_max_token_size.........84
innodb_ft_min_token_size.........3
innodb_ft_num_word_optimize.........2000
innodb_ft_result_cache_limit.........2000000000
innodb_ft_server_stopword_table.........
innodb_ft_sort_pll_degree.........2
innodb_ft_total_cache_size.........640000000
innodb_ft_user_stopword_table.........
innodb_io_capacity.........200
innodb_io_capacity_max.........2000
innodb_large_prefix.........OFF
innodb_lock_wait_timeout.........100
innodb_locks_unsafe_for_binlog.........OFF
innodb_log_buffer_size.........16777216
innodb_log_compressed_pages.........ON
innodb_log_file_size.........536870912
innodb_log_files_in_group.........2
innodb_log_group_home_dir........../
innodb_lru_scan_depth.........1024
innodb_max_dirty_pages_pct.........75
innodb_max_dirty_pages_pct_lwm.........0
innodb_max_purge_lag.........0
innodb_max_purge_lag_delay.........0
innodb_mirrored_log_groups.........1
innodb_monitor_disable.........
innodb_monitor_enable.........
innodb_monitor_reset.........
innodb_monitor_reset_all.........
innodb_old_blocks_pct.........37
innodb_old_blocks_time.........1000
innodb_online_alter_log_max_size.........134217728
innodb_open_files.........400
innodb_optimize_fulltext_only.........OFF
innodb_page_size.........16384
innodb_print_all_deadlocks.........OFF
innodb_purge_batch_size.........300
innodb_purge_threads.........1
innodb_random_read_ahead.........OFF
innodb_read_ahead_threshold.........56
innodb_read_io_threads.........4
innodb_read_only.........OFF
innodb_replication_delay.........0
innodb_rollback_on_timeout.........OFF
innodb_rollback_segments.........128
innodb_sort_buffer_size.........1048576
innodb_spin_wait_delay.........6
innodb_stats_auto_recalc.........ON
innodb_stats_method.........nulls_equal
innodb_stats_on_metadata.........OFF
innodb_stats_persistent.........ON
innodb_stats_persistent_sample_pages.........20
innodb_stats_sample_pages.........8
innodb_stats_transient_sample_pages.........8
innodb_status_output.........OFF
innodb_status_output_locks.........OFF
innodb_strict_mode.........OFF
innodb_support_xa.........ON
innodb_sync_array_size.........1
innodb_sync_spin_loops.........30
innodb_table_locks.........ON
innodb_thread_concurrency.........0
innodb_thread_sleep_delay.........10000
innodb_undo_directory..........
innodb_undo_logs.........128
innodb_undo_tablespaces.........0
innodb_use_native_aio.........ON
innodb_use_sys_malloc.........ON
innodb_version.........5.6.23
innodb_write_io_threads.........4
interactive_timeout.........28800
join_buffer_size.........8388608
keep_files_on_create.........OFF
key_buffer_size.........12884901888
key_cache_age_threshold.........300
key_cache_block_size.........1024
key_cache_division_limit.........100
large_files_support.........ON
large_page_size.........0
large_pages.........OFF
lc_messages.........en_US
lc_messages_dir........./usr/local/mysql-5.6.23-linux-glibc2.5-x86_64/share/
lc_time_names.........en_US
license.........GPL
local_infile.........ON
lock_wait_timeout.........31536000
locked_in_memory.........OFF
log_bin.........ON
log_bin_basename........./usr/local/mysql6/data/npl9dba08_99acres
log_bin_index........./usr/local/mysql6/data/npl9dba08_99acres.index
log_bin_trust_function_creators.........OFF
log_bin_use_v1_row_events.........OFF
log_error........./usr/local/mysql6/data/npl9dba08.ieil.net.err
log_output.........FILE
log_queries_not_using_indexes.........OFF
log_slave_updates.........ON
log_slow_admin_statements.........OFF
log_slow_slave_statements.........OFF
log_throttle_queries_not_using_indexes.........0
log_warnings.........1
long_query_time.........1.000000
low_priority_updates.........OFF
lower_case_file_system.........OFF
lower_case_table_names.........0
master_info_repository.........FILE
master_verify_checksum.........OFF
max_allowed_packet.........33554432
max_binlog_cache_size.........18446744073709547520
max_binlog_size.........1073741824
max_binlog_stmt_cache_size.........18446744073709547520
max_connect_errors.........100
max_connections.........500
max_delayed_threads.........20
max_error_count.........64
max_heap_table_size.........16777216
max_insert_delayed_threads.........20
max_join_size.........18446744073709551615
max_length_for_sort_data.........1024
max_prepared_stmt_count.........16382
max_relay_log_size.........0
max_seeks_for_key.........18446744073709551615
max_sort_length.........1024
max_sp_recursion_depth.........0
max_tmp_tables.........32
max_user_connections.........0
max_write_lock_count.........18446744073709551615
metadata_locks_cache_size.........1024
metadata_locks_hash_instances.........8
min_examined_row_limit.........0
multi_range_count.........256
myisam_data_pointer_size.........6
myisam_max_sort_file_size.........9223372036853727232
myisam_mmap_size.........18446744073709551615
myisam_recover_options.........OFF
myisam_repair_threads.........1
myisam_sort_buffer_size.........12582912
myisam_stats_method.........nulls_unequal
myisam_use_mmap.........OFF
net_buffer_length.........16384
net_read_timeout.........30
net_retry_count.........10
net_write_timeout.........60
new.........OFF
old.........OFF
old_alter_table.........OFF
old_passwords.........0
open_files_limit.........20000
optimizer_prune_level.........1
optimizer_search_depth.........62
optimizer_switch.........index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,engine_condition_pushdown=on,index_condition_pushdown=on,mrr=on,mrr_cost_based=on,block_nested_loop=on,batched_key_access=off,materialization=on,semijoin=on,loosescan=on,firstmatch=on,subquery_materialization_cost_based=on,use_index_extensions=on
optimizer_trace.........enabled=off,one_line=off
optimizer_trace_features.........greedy_search=on,range_optimizer=on,dynamic_range=on,repeated_subselect=on
optimizer_trace_limit.........1
optimizer_trace_max_mem_size.........16384
optimizer_trace_offset.........-1
performance_schema.........ON
performance_schema_accounts_size.........100
performance_schema_digests_size.........10000
performance_schema_events_stages_history_long_size.........10000
performance_schema_events_stages_history_size.........10
performance_schema_events_statements_history_long_size.........10000
performance_schema_events_statements_history_size.........10
performance_schema_events_waits_history_long_size.........10000
performance_schema_events_waits_history_size.........10
performance_schema_hosts_size.........100
performance_schema_max_cond_classes.........80
performance_schema_max_cond_instances.........3300
performance_schema_max_file_classes.........50
performance_schema_max_file_handles.........32768
performance_schema_max_file_instances.........30770
performance_schema_max_mutex_classes.........200
performance_schema_max_mutex_instances.........10000
performance_schema_max_rwlock_classes.........40
performance_schema_max_rwlock_instances.........5000
performance_schema_max_socket_classes.........10
performance_schema_max_socket_instances.........1020
performance_schema_max_stage_classes.........150
performance_schema_max_statement_classes.........168
performance_schema_max_table_handles.........800
performance_schema_max_table_instances.........12500
performance_schema_max_thread_classes.........50
performance_schema_max_thread_instances.........1100
performance_schema_session_connect_attrs_size.........512
performance_schema_setup_actors_size.........100
performance_schema_setup_objects_size.........100
performance_schema_users_size.........100
pid_file........./usr/local/mysql6/data/npl9dba08.ieil.net.pid
plugin_dir........./usr/local/mysql6/lib/plugin/
port.........3306
preload_buffer_size.........32768
profiling.........OFF
profiling_history_size.........15
protocol_version.........10
query_alloc_block_size.........8192
query_cache_limit.........1048576
query_cache_min_res_unit.........4096
query_cache_size.........268435456
query_cache_type.........ON
query_cache_wlock_invalidate.........OFF
query_prealloc_size.........8192
range_alloc_block_size.........4096
read_buffer_size.........16777216
read_only.........OFF
read_rnd_buffer_size.........262144
relay_log.........
relay_log_basename.........
relay_log_index.........
relay_log_info_file.........relay-log.info
relay_log_info_repository.........FILE
relay_log_purge.........ON
relay_log_recovery.........OFF
relay_log_space_limit.........0
report_host.........npl9dba0806
report_password.........
report_port.........3306
report_user.........
rpl_stop_slave_timeout.........31536000
secure_auth.........ON
secure_file_priv.........
server_id.........178806
server_id_bits.........32
server_uuid.........561d32a3-5544-11e5-b989-44a8423fc648
simplified_binlog_gtid_recovery.........OFF
skip_external_locking.........ON
skip_name_resolve.........ON
skip_networking.........OFF
skip_show_database.........OFF
slave_allow_batching.........OFF
slave_checkpoint_group.........512
slave_checkpoint_period.........300
slave_compressed_protocol.........ON
slave_exec_mode.........STRICT
slave_load_tmpdir........./usr/local/mysql6/data/tmp
slave_max_allowed_packet.........1073741824
slave_net_timeout.........3600
slave_parallel_workers.........0
slave_pending_jobs_size_max.........16777216
slave_rows_search_algorithms.........TABLE_SCAN,INDEX_SCAN
slave_skip_errors.........OFF
slave_sql_verify_checksum.........ON
slave_transaction_retries.........10
slave_type_conversions.........
slow_launch_time.........2
slow_query_log.........ON
slow_query_log_file.........npl9dba08.slow
socket........./tmp/mysql_06.sock
sort_buffer_size.........8388608
sql_auto_is_null.........OFF
sql_big_selects.........ON
sql_buffer_result.........OFF
sql_log_bin.........ON
sql_log_off.........OFF
sql_mode.........NO_ENGINE_SUBSTITUTION
sql_notes.........ON
sql_quote_show_create.........ON
sql_safe_updates.........OFF
sql_select_limit.........18446744073709551615
sql_slave_skip_counter.........0
sql_warnings.........OFF
ssl_ca.........
ssl_capath.........
ssl_cert.........
ssl_cipher.........
ssl_crl.........
ssl_crlpath.........
ssl_key.........
storage_engine.........MyISAM
stored_program_cache.........256
sync_binlog.........0
sync_frm.........ON
sync_master_info.........10000
sync_relay_log.........10000
sync_relay_log_info.........10000
system_time_zone.........IST
table_definition_cache.........600
table_open_cache.........400
table_open_cache_instances.........1
thread_cache_size.........400
thread_concurrency.........10
thread_handling.........one-thread-per-connection
thread_stack.........262144
time_format.........%H:%i:%s
time_zone.........Asia/Calcutta
timed_mutexes.........OFF
tmp_table_size.........3221225472
tmpdir........./usr/local/mysql6/data/tmp
transaction_alloc_block_size.........8192
transaction_prealloc_size.........4096
tx_isolation.........REPEATABLE-READ
tx_read_only.........OFF
unique_checks.........ON
updatable_views_with_limit.........YES
version.........5.6.23-log
version_comment.........MySQL Community Server (GPL)
version_compile_machine.........x86_64
version_compile_os.........linux-glibc2.5
wait_timeout.........200

Smells like connection leak. You might be exhausting connections in the pool after certain time as they might nor have been closed properly. Please share the code. — Roshith, Apr 15 '16 at 05:59

score 0 · Answer 1 · answered Apr 15 '16 at 05:59

0

query_cache_size.........268435456 -- Not more than about 50M, else the prunes will cause performance problems in INSERTs and other writes.

INSERT look like? One row? Many rows? INSERT..SELECT? Show us the INSERT.

Do you have at least 64GB of RAM?

You are using only MyISAM? Ouch!

40GB is wasted in the buffer_pool that is used only by InnoDB.

group_concat_max_len.........1073741824 -- get a few queries going at once, and you could run out of RAM!
Ditto for tmp_table_size.........3221225472.

Is a FULLTEXT index involved with the naughty INSERT?

answered Apr 15 '16 at 05:59

Rick James

135,179
13
127
222

Here is my code for inserting: private void insertDetailstoDB() { for ( int i =0; i < listpropTosend.size(); i++) { String sql = "insert into ALERT_TRACK(ADDED_ON,RID,SPID) VALUES ('"+today+"','"+rid+"','"+listpropTosend.get(i)+"')"; DBConnection.QueryUpdate(connectionObjProperty, sql); } } QueryUpdate() { Statement statement; try { statement = connection.createStatement(); if(sql!=null) { statement.executeUpdate(sql); statement.close(); } } } – Rajat Goyal Apr 18 '16 at 06:51
Is there a "catch" clause? Do the rows get inserted? – Rick James Apr 18 '16 at 14:28
Yes, it is with catch statement. I removed in comment, as it was exceeding limit. Yes, we want to insert the rows. – Rajat Goyal Apr 19 '16 at 06:03

Exception : Can not read response from server. Expected to read 4 bytes, read 0 bytes before connection was unexpectedly lost

Experiments done:

1 Answers1