0

I've about 30 tables in MySQL Data Warehouse database for analytical data. For now its around 2 Millions of data rows, but I'm sure in future it will become billions very soon. Challenge is, the queries should return the data in a fast way. I've following simple query which is taking over 60 seconds to process those 2 Million rows:

    SELECT count(distinct fact.dim_pageview_id) 
    FROM `datawarehouse_schema_alpha`.`fact_master` fact
    left join dim_visit visit on visit.dim_visit_id = fact.dim_visit_id
    left join dim_datetime datetim on datetim.dim_datetime_id = fact.dim_datetime_id
    where fact.dim_site_id = 552

Explain query result:

    1   SIMPLE  fact    ref fk_fact_bb_pageview_dim_bb_site,fk_fact_master_dim_site fk_fact_bb_pageview_dim_bb_site 5   const   17490   Using where
    1   SIMPLE  visit   eq_ref  PRIMARY PRIMARY 4   datawarehouse_schema_alpha.fact.dim_visit_id    1   Using index
    1   SIMPLE  datetim eq_ref  PRIMARY PRIMARY 4   datawarehouse_schema_alpha.fact.dim_datetime_id 1   Using index

I've following sample database structure:

    --
    -- Table structure for table `dim_datetime`
    --

    CREATE TABLE IF NOT EXISTS `dim_datetime` (
      `dim_datetime_id` int(11) NOT NULL AUTO_INCREMENT,
      `datetime_date` varchar(45) CHARACTER SET latin1 DEFAULT NULL,
      `datetime_year` varchar(45) CHARACTER SET latin1 DEFAULT NULL,
      `datetime_full` varchar(45) CHARACTER SET latin1 DEFAULT NULL,
      PRIMARY KEY (`dim_datetime_id`)
    ) ENGINE=InnoDB  DEFAULT CHARSET=big5 AUTO_INCREMENT=4568326 ;

    -- --------------------------------------------------------

    --
    -- Table structure for table `dim_visit`
    --

    CREATE TABLE IF NOT EXISTS `dim_visit` (
      `dim_visit_id` int(11) NOT NULL AUTO_INCREMENT,
      `visit_start_time` datetime DEFAULT NULL,
      `visit_end_time` datetime DEFAULT NULL,
      `visit_duration` varchar(45) DEFAULT NULL,
      PRIMARY KEY (`dim_visit_id`)
    ) ENGINE=InnoDB  DEFAULT CHARSET=big5 AUTO_INCREMENT=1295102 ;

    -- --------------------------------------------------------

    --
    -- Table structure for table `dim_site`
    --

    CREATE TABLE IF NOT EXISTS `dim_site` (
      `dim_site_id` int(11) NOT NULL AUTO_INCREMENT,
      `site_name` varchar(255) CHARACTER SET latin1 DEFAULT NULL,
      `site_url` text CHARACTER SET latin1,
      `site_key` text CHARACTER SET latin1,
      PRIMARY KEY (`dim_site_id`)
    ) ENGINE=InnoDB  DEFAULT CHARSET=big5 AUTO_INCREMENT=870 ;

    --
    -- Table structure for table `fact_master`
    --

    CREATE TABLE IF NOT EXISTS `fact_master` (
      `fact_master_id` int(11) NOT NULL AUTO_INCREMENT,
      `dim_pageview_id` int(11) DEFAULT NULL,
      `dim_visit_id` int(11) DEFAULT NULL,
      `dim_site_id` int(11) DEFAULT NULL,
      `dim_datetime_id` int(11) DEFAULT NULL,
      `master_ip` varchar(255) DEFAULT NULL,
      `master_spent_time` varchar(255) DEFAULT NULL,
      `master_datetime` datetime DEFAULT NULL,
      PRIMARY KEY (`fact_master_id`),
      KEY `fk_fact_bb_pageview_dim_bb_visit` (`dim_visit_id`),
      KEY `fk_fact_bb_pageview_dim_bb_datetime` (`dim_datetime_id`),
      KEY `fk_fact_bb_pageview_dim_bb_pageview` (`dim_pageview_id`),
      KEY `fk_fact_master_dim_pageview` (`dim_pageview_id`),
      KEY `fk_fact_master_dim_visit` (`dim_visit_id`),
      KEY `fk_fact_master_dim_datetime` (`dim_datetime_id`),
      KEY `fk_fact_master_dim_site` (`dim_site_id`),
    ) ENGINE=InnoDB DEFAULT CHARSET=big5 AUTO_INCREMENT=1 ;

    --
    -- Constraints for dumped tables
    --

    --
    -- Constraints for table `fact_master`
    --
    ALTER TABLE `fact_master`
      ADD CONSTRAINT `fk_fact_master_dim_datetime` FOREIGN KEY (`dim_datetime_id`) REFERENCES `dim_datetime` (`dim_datetime_id`) ON DELETE NO ACTION ON UPDATE NO ACTION,
      ADD CONSTRAINT `fk_fact_master_dim_pageview` FOREIGN KEY (`dim_pageview_id`) REFERENCES `dim_pageview` (`dim_pageview_id`) ON DELETE NO ACTION ON UPDATE NO ACTION,
      ADD CONSTRAINT `fk_fact_master_dim_site` FOREIGN KEY (`dim_site_id`) REFERENCES `dim_site` (`dim_site_id`) ON DELETE NO ACTION ON UPDATE NO ACTION,
      ADD CONSTRAINT `fk_fact_master_dim_visit` FOREIGN KEY (`dim_visit_id`) REFERENCES `dim_visit` (`dim_visit_id`) ON DELETE NO ACTION ON UPDATE NO ACTION;

Please let me know what problem could there be? How can I use Indexes and Views to get the data in a really really quick way? Any other suggestions than using Indexes or Views, to optimize the speed is also welcome. Thanks!

Edge
  • 901
  • 1
  • 12
  • 19

0 Answers0