1

Scenario: Following up from my previous question (Using a cursor in a stored procedure to loop rows MySQL) I am trying to do a nested prepare statement in which I input a date to the outer one, and that calls the inner, which gets data from a table.

Code:

-- Create temporary table for the Output:
drop temporary table if exists `stats`;
create temporary table `stats`
(
    col_name varchar(32) null,
    num_nulls int null,
    num_values int null
);


-- Procedure for the check:
drop procedure if exists `set_column_null_stats`;
delimiter $$
create procedure `set_column_null_stats`
(`p_col_name` varchar(128), `wanted_date` date)
begin

-- Set variables:
set @col_nme = `p_col_name`;
set @date1 = `wanted_date`;

prepare stmt from 'insert into `stats` (`col_name`) values (?);';
execute stmt using @col_nme;
deallocate prepare stmt;

-- count number of NULLS based on conditions:
set @sql_txt = concat(
            'update `stats` s join(
            select 
            count(1) as `nb`
            from `btc`
            where`btc`.`date` = ', @date1, ' and `btc`.`', @col_nme, '` is null)
            t set `num_nulls` = t.`nb` where `col_name` = \'', @col_nme, '\';');
prepare stmt from @sql_txt;
execute stmt;
deallocate prepare stmt;

-- count number of not NULLS based on conditions:
set @sql_txt = concat(
           'update `stats` s join(
            select
            count(1) as `nb`
            from `btc`
            where `btc`.`date` = ', @date1, ' and `btc`.`', @col_nme, '` is not null)
            -- t set `num_values` = t.`nb` where `col_name` = \'', @col_nme, '\';');        
set @sql_txt = concat('update `stats` s join (select count(1) as `nb` from `btc` where `', @col_nme, '` is not null) t set `num_values` = t.`nb` where `col_name` = \'', @col_nme, '\';');
prepare stmt from @sql_txt;
execute stmt;
deallocate prepare stmt;

end$$
delimiter ;


-- Procedure for looping through rows of `wanted_columns` table:
delimiter $$
drop procedure if exists `data_check_loop` $$
create procedure `data_check_loop`(`wanted_date` date)
begin

declare dateval date default null;
declare colval text default null;

-- boolean variable to indicate cursor is out of data
declare done tinyint default false;

-- declare a cursor to select the desired columns from the desired source table
declare cursor1
    cursor for
        select *
        from `wanted_columns`; 

-- catch exceptions
        declare continue handler for not found set done = true;

set dateval = `wanted_date`;

-- open the cursor
        open cursor1;
            my_loop: 
            loop
                fetch next from cursor1 into colval;
                if done then 
                    leave my_loop; 
                else  
                    call `set_column_null_stats`(colval, dateval);
                end if;
            end loop;
        close cursor1;

end $$
delimiter ;


-- Start the process with the wanted date:
call `data_check_loop`('2018-08-13');

select * from `stats`;

Issue: This code runs with no errors, but does not give me any result. If I run just the first prepared statement, feeding the variables directly one by one, it works fine. So I am guessing the problem is with my second statement.

Question: Any ideas on what I am doing wrong here?

Obs: The second code should loop rows from a table (wanted columns) and feed them to the first statement, one by one (alongside with the date, which should always be the same)

Obs2: My objective with this query is: from a table with a list of names as rows ("id1", "date1"...) I intend to read each row and use that value into another table where the names ("id1", "date1"...) are the columns, and get a sum for each of my wanted columns, of the NULL and not NULL values (also, given another constraint of the date input). Finally, for each of my original rows (table1) I will output a new row with #NULL and #notNULL.

Ex. Table 1:

Col_names
  Id1
  Name1
  Date1
  Process
  Time
  Class

Ex. Table 2:

Id1    Name1    Date1    Process    Time    Class
aa     test1    01/01       3       NULL      A
NULL   test2    01/02       4       NULL      b
bb     test3    NULL        3       NULL     NULL

Ex. Output:

Col_name    #Null    #notNull
  Id1         1          2
  Name1       0          3
  Date1       1          2 
  Process     0          3
  Time        3          0
  Class       1          2
DGMS89
  • 1,507
  • 6
  • 29
  • 60
  • 2
    You know you opening yourself to possible (blind) SQL injections vectors attacks in the prepared UPDATES queries right? – Raymond Nijland Nov 27 '18 at 13:19
  • @RaymondNijland No idea, I just recently started with SQL and am trying to put together the pieces I find online. I don't even know what injections mean. – DGMS89 Nov 27 '18 at 13:21
  • The first PREPARE (insert) is protected because you are using `execute stmt using @col_nme` you should do the same approach for the Updates. – Raymond Nijland Nov 27 '18 at 13:23
  • 1
    It is possible that you don't even need to use a cursor or dynamic SQL here. What would be wrong with doing a normal update? – Tim Biegeleisen Nov 27 '18 at 13:23
  • @RaymondNijland Ok I will try to read on that and try to fix. – DGMS89 Nov 27 '18 at 13:24
  • @TimBiegeleisen Sincerely, IDK. I just found other questions where cursors were use to loop the rows of a table, and that is what I tried to do. No idea if there is a better way. – DGMS89 Nov 27 '18 at 13:25
  • 1
    Can you provide example data and expected results? I and @TimBiegeleisen are pretty sure there is a more easy approach to this.. [help for providing data](https://meta.stackoverflow.com/questions/333952/why-should-i-provide-an-mcve-for-what-seems-to-me-to-be-a-very-simple-sql-query) – Raymond Nijland Nov 27 '18 at 13:27
  • @RaymondNijland Updated. – DGMS89 Nov 27 '18 at 13:38

1 Answers1

1

It's seams you want to convert columns into records end count where the values are null or not null for that column .
The process which converts columns into records is called unpivot in most database systems this is supported with UNPIVOT() but MySQL does not support this.
So normally this is done with UNION in combination with some aggregate functions like MAX(), MIX(), SUM() and COUNT() and a CASE END clause to simulate UNPIVOT() on MySQL.

Query

SELECT 
   'Id1' AS Col_name
 , SUM(CASE WHEN Table1.Id1 IS NULL THEN 1 ELSE 0 END) AS `#Null`
 , SUM(CASE WHEN Table1.Id1 IS NOT NULL THEN 1 ELSE 0 END) AS `#notNull` 
FROM 
 Table1

UNION ALL

SELECT 
  'Name1' AS Col_name
 , SUM(CASE WHEN Table1.Name1 IS NULL THEN 1 ELSE 0 END) AS `#Null`
 , SUM(CASE WHEN Table1.Name1 IS NOT NULL THEN 1 ELSE 0 END) AS `#notNull` 
FROM 
 Table1

UNION ALL

SELECT 
  'Date1' AS Col_name
 , SUM(CASE WHEN Table1.Date1 IS NULL THEN 1 ELSE 0 END) AS `#Null`
 , SUM(CASE WHEN Table1.Date1 IS NOT NULL THEN 1 ELSE 0 END) AS `#notNull` 
FROM 
 Table1

UNION ALL

SELECT 
  'Process' AS Col_name
 , SUM(CASE WHEN Table1.Process IS NULL THEN 1 ELSE 0 END) AS `#Null`
 , SUM(CASE WHEN Table1.Process IS NOT NULL THEN 1 ELSE 0 END) AS `#notNull` 
FROM 
 Table1

UNION ALL

SELECT 
  'Time' AS Col_name
 , SUM(CASE WHEN Table1.Time IS NULL THEN 1 ELSE 0 END) AS `#Null`
 , SUM(CASE WHEN Table1.Time IS NOT NULL THEN 1 ELSE 0 END) AS `#notNull` 
FROM 
 Table1

UNION ALL

SELECT 
  'Class' AS Col_name
 , SUM(CASE WHEN Table1.Class IS NULL THEN 1 ELSE 0 END) AS `#Null`
 , SUM(CASE WHEN Table1.Class IS NOT NULL THEN 1 ELSE 0 END) AS `#notNull` 
FROM 
 Table1

Results

| Col_name | #Null | #notNull |
| -------- | ----- | -------- |
| Id1      | 1     | 2        |
| Name1    | 0     | 3        |
| Date1    | 1     | 2        |
| Process  | 0     | 3        |
| Time     | 3     | 0        |
| Class    | 1     | 2        |

demo

Raymond Nijland
  • 11,488
  • 2
  • 22
  • 34
  • Thanks for the answer. Actually, I ran by this UNION ALL procedure, but since I have a very long list of columns, and they change continuously, I started to try an automatic procedure to read them, instead of me inputting one by one manually multiple times a day. – DGMS89 Nov 27 '18 at 14:04
  • 1
    No problem @DGMS89 this https://www.db-fiddle.com/f/fDzZWbxq9NX2e7h7YUnA7U/6 is a dynamic unpivot method which generates a SQL query based on the number of columns.. I made the steps visible so it more easy to follow whats going on. – Raymond Nijland Nov 27 '18 at 14:23
  • Many Thanks. I will try to add my date constraint and run it. – DGMS89 Nov 27 '18 at 14:25
  • One question: in the query, I see you left space for passing my "table1" which has the data (as rows) for the column names. I just do not understand how to connect to the table where I will retrieve the #nulls and #notnulls. – DGMS89 Nov 27 '18 at 14:50