I've about 30 tables in MySQL Data Warehouse database for analytical data. For now its around 2 Millions of data rows, but I'm sure in future it will become billions very soon. Challenge is, the queries should return the data in a fast way. I've following simple query which is taking over 60 seconds to process those 2 Million rows:
SELECT count(distinct fact.dim_pageview_id)
FROM `datawarehouse_schema_alpha`.`fact_master` fact
left join dim_visit visit on visit.dim_visit_id = fact.dim_visit_id
left join dim_datetime datetim on datetim.dim_datetime_id = fact.dim_datetime_id
where fact.dim_site_id = 552
Explain query result:
1 SIMPLE fact ref fk_fact_bb_pageview_dim_bb_site,fk_fact_master_dim_site fk_fact_bb_pageview_dim_bb_site 5 const 17490 Using where
1 SIMPLE visit eq_ref PRIMARY PRIMARY 4 datawarehouse_schema_alpha.fact.dim_visit_id 1 Using index
1 SIMPLE datetim eq_ref PRIMARY PRIMARY 4 datawarehouse_schema_alpha.fact.dim_datetime_id 1 Using index
I've following sample database structure:
--
-- Table structure for table `dim_datetime`
--
CREATE TABLE IF NOT EXISTS `dim_datetime` (
`dim_datetime_id` int(11) NOT NULL AUTO_INCREMENT,
`datetime_date` varchar(45) CHARACTER SET latin1 DEFAULT NULL,
`datetime_year` varchar(45) CHARACTER SET latin1 DEFAULT NULL,
`datetime_full` varchar(45) CHARACTER SET latin1 DEFAULT NULL,
PRIMARY KEY (`dim_datetime_id`)
) ENGINE=InnoDB DEFAULT CHARSET=big5 AUTO_INCREMENT=4568326 ;
-- --------------------------------------------------------
--
-- Table structure for table `dim_visit`
--
CREATE TABLE IF NOT EXISTS `dim_visit` (
`dim_visit_id` int(11) NOT NULL AUTO_INCREMENT,
`visit_start_time` datetime DEFAULT NULL,
`visit_end_time` datetime DEFAULT NULL,
`visit_duration` varchar(45) DEFAULT NULL,
PRIMARY KEY (`dim_visit_id`)
) ENGINE=InnoDB DEFAULT CHARSET=big5 AUTO_INCREMENT=1295102 ;
-- --------------------------------------------------------
--
-- Table structure for table `dim_site`
--
CREATE TABLE IF NOT EXISTS `dim_site` (
`dim_site_id` int(11) NOT NULL AUTO_INCREMENT,
`site_name` varchar(255) CHARACTER SET latin1 DEFAULT NULL,
`site_url` text CHARACTER SET latin1,
`site_key` text CHARACTER SET latin1,
PRIMARY KEY (`dim_site_id`)
) ENGINE=InnoDB DEFAULT CHARSET=big5 AUTO_INCREMENT=870 ;
--
-- Table structure for table `fact_master`
--
CREATE TABLE IF NOT EXISTS `fact_master` (
`fact_master_id` int(11) NOT NULL AUTO_INCREMENT,
`dim_pageview_id` int(11) DEFAULT NULL,
`dim_visit_id` int(11) DEFAULT NULL,
`dim_site_id` int(11) DEFAULT NULL,
`dim_datetime_id` int(11) DEFAULT NULL,
`master_ip` varchar(255) DEFAULT NULL,
`master_spent_time` varchar(255) DEFAULT NULL,
`master_datetime` datetime DEFAULT NULL,
PRIMARY KEY (`fact_master_id`),
KEY `fk_fact_bb_pageview_dim_bb_visit` (`dim_visit_id`),
KEY `fk_fact_bb_pageview_dim_bb_datetime` (`dim_datetime_id`),
KEY `fk_fact_bb_pageview_dim_bb_pageview` (`dim_pageview_id`),
KEY `fk_fact_master_dim_pageview` (`dim_pageview_id`),
KEY `fk_fact_master_dim_visit` (`dim_visit_id`),
KEY `fk_fact_master_dim_datetime` (`dim_datetime_id`),
KEY `fk_fact_master_dim_site` (`dim_site_id`),
) ENGINE=InnoDB DEFAULT CHARSET=big5 AUTO_INCREMENT=1 ;
--
-- Constraints for dumped tables
--
--
-- Constraints for table `fact_master`
--
ALTER TABLE `fact_master`
ADD CONSTRAINT `fk_fact_master_dim_datetime` FOREIGN KEY (`dim_datetime_id`) REFERENCES `dim_datetime` (`dim_datetime_id`) ON DELETE NO ACTION ON UPDATE NO ACTION,
ADD CONSTRAINT `fk_fact_master_dim_pageview` FOREIGN KEY (`dim_pageview_id`) REFERENCES `dim_pageview` (`dim_pageview_id`) ON DELETE NO ACTION ON UPDATE NO ACTION,
ADD CONSTRAINT `fk_fact_master_dim_site` FOREIGN KEY (`dim_site_id`) REFERENCES `dim_site` (`dim_site_id`) ON DELETE NO ACTION ON UPDATE NO ACTION,
ADD CONSTRAINT `fk_fact_master_dim_visit` FOREIGN KEY (`dim_visit_id`) REFERENCES `dim_visit` (`dim_visit_id`) ON DELETE NO ACTION ON UPDATE NO ACTION;
Please let me know what problem could there be? How can I use Indexes and Views to get the data in a really really quick way? Any other suggestions than using Indexes or Views, to optimize the speed is also welcome. Thanks!