1

I have a MySQL database with around 250GB of data running on Linux platform (12GB Ram). I am upgrading MySQL to 8.0.32 from 5.7.42-1 and to do so, altering one of the largest table with partitioning and MyISAM storage engine in my DB. Here is the table structure:

| devstats| CREATE TABLE `devstats` (
  `time` datetime(6) NOT NULL DEFAULT '0000-00-00 00:00:00.000000',
  `s_num` int(11) NOT NULL DEFAULT '0',
  `devid` mediumint(9) NOT NULL DEFAULT '0',
  `grpid` mediumint(9) NOT NULL DEFAULT '0',
  `subid` mediumint(9) NOT NULL DEFAULT '0',
  `value` double NOT NULL DEFAULT '0',
  KEY `devid` (`devid`),
  KEY `grpid` (`grpid`),
  KEY `time` (`time`)
) ENGINE=MyISAM DEFAULT CHARSET=latin1
/*!50100 PARTITION BY RANGE (to_days(time))
(PARTITION p0000 VALUES LESS THAN (0) ENGINE = MyISAM,
 PARTITION p20230522 VALUES LESS THAN (739028) ENGINE = MyISAM,
-----------------------
-----------------------

Following logical approach hence performing below operations:

  1. Alter table (change MyISAM to InnoDB)
  2. Backup the DB using mysqldump
  3. Upgrade mysql binaries
  4. Import back the DB data

configured innodb_buffer_pool_size = 8GB

With these all setup, the whole process takes almost couple of days. So, can someone please help me to speed up these all operations without loosing any data.

Thanks

Dams
  • 47
  • 7
  • Which step took the longest? Is the goal to use InnoDB? Why add Partitioning? Why only 2 partitions? Did the data start out in `time` order? Is there an explicit `PRIMARY KEY`? Is any combination of columns unique? – Rick James Jul 21 '23 at 20:27
  • Is your list missing a `DROP TABLE` some time between steps 2 and 4? – Rick James Jul 21 '23 at 20:49
  • Hi Rick James, Thanks for responding. Answers to your few queries: Which step took the longest? : Alter, export, and import dump operations take a lot of time. Is the goal to use InnoDB? : Yes, to comply with MySQL8, need to replace MyISAM with InnoDB. Why add Partitioning? : It’s the largest table in my DB and needs partitioning to improve the performance. Why only 2 partitions? : There are more than 2, sorry for confusion. Did the data start out in time order? : Did not get it. Is there an explicit PRIMARY KEY? : No. Is any combination of columns unique? : No – Dams Jul 23 '23 at 06:21
  • DROP table is not used, why do we need to drop any table? – Dams Jul 23 '23 at 06:22
  • I added responses in my Answer. – Rick James Jul 23 '23 at 06:51

1 Answers1

1

Skip steps 2 (except for having a backup) and 4.

Step 4 is redundant since the table is already InnoDB ready for use.

I see no PRIMARY KEY; that is possible but not advised. These will help in designing a PK that works well:

  • What are typical queries that are applied to the table?
  • Is there any 'churn', or is this a write-once, but never update or delete, type of dataset?
  • Is some combination of columns already Unique?
  • Something like WHERE devid=123 AND time BETWEEN... begs for a composite INDEX(devid, time)

More

  • Partitioning is unlikely to help performance. Let's see a query that you think would benefit from Partitioning. More on: Partitioning
  • My question about DROP was because the table(s) seem to still be intact as you start step 4.
Rick James
  • 135,179
  • 13
  • 127
  • 222
  • Hi Rick, I need to take dump just to make sure I can restore the DB in case of any failure in the Upgrade process (5.7 to 8.x) Mostly Select and insert queries are being used with that table. Yes, we never update the table, its just Select and Insert kind of Dataset. There is no combination of columns with Unique. – Dams Jul 23 '23 at 06:31
  • Thanks again for your time. This DB is associated with a legacy product so except changing storage engine, i am not allowed to do any other modifications and as i have to upgrade it to 8.x version, need to backup customer data as well, so i have to follow those 4 steps. But thanks for sharing the info regarding partitioning, that is really nice. – Dams Jul 23 '23 at 16:19
  • Dams - Changing the Engine can lead to problems in a few cases -- See my list: http://mysql.rjweb.org/doc.php/myisam2innodb@ – Rick James Jul 23 '23 at 18:00
  • Yes, Agree, but i have to do it just to comply with MySQL8 changes. Getting "Document '' is empty?" response from the link you shared, pls chk and reshare. – Dams Jul 24 '23 at 06:14
  • @Dams - Oops, typo: https://mysql.rjweb.org/doc.php/myisam2innodb – Rick James Jul 24 '23 at 06:30