1

I have almost 1000 tables and most of them have a common column ItemNumber. How do I search across all the tables in the database for a value or list of values that exist in this common column, such as 350 or (350, 465)? The tables have different schemas.

Table A100

ItemNumber Detail
230 Car
245 Plane

Table A1000

ItemNumber ProductDescription
350 Pie
465 Cherry
marc_s
  • 732,580
  • 175
  • 1,330
  • 1,459
Montreal
  • 15
  • 8
  • 1
    Fix the schema and put those values in the same table. – dfundako Aug 18 '22 at 17:27
  • The tables are all actually different but share a common column. @dfundako – Montreal Aug 18 '22 at 17:35
  • You'll have to script it out. Get a list of tables, loop through it, build you SQL to search for whatever you are searching for, run the sql that is produced, collect the results and write them somewhere and then look at the results. [This Q&A may help a bit](https://stackoverflow.com/questions/1325044/dynamic-sql-passing-table-name-as-parameter) Could be done in a stored procedure or outside scripting language like Python. There is no functionality like `SELECT * FROM * WHERE ItemNumber = 350` unfortunately. – JNevill Aug 18 '22 at 17:38
  • If this is a one time query then use `select t.TABLE_SCHEMA, TABLE_NAME from INFORMATION_SCHEMA.TABLES t` to get all the tables. You can use this to generate a SELECT statement (either with string concatenation in the query or in Excel or notepad) and then just run that big query. If this is something that is long(er) lived then use the above query to generate dynamic sql and execute it in a loop using `exec sp_executesql`. – Igor Aug 18 '22 at 17:45
  • I have a script that searches across all columns in all tables. However, I do not want to use it here and want to narrow down my work to a specific column. – Montreal Aug 18 '22 at 17:57
  • Then take that script and add a `WHERE` at the right location? Presumably it has to work column by column in any case. – Jeroen Mostert Aug 18 '22 at 18:10
  • @JeroenMostert I have tried that, it takes forever to run the query. The database is denormalized and there are hundreds of millions of records. It doesn't work. – Montreal Aug 18 '22 at 19:46

2 Answers2

0

This does not perform type checking, so you can get conversion errors if the target column is not the correct type. Also, this script uses LIKE, you would probably need to change that to a direct comparison.

SET NOCOUNT ON

DECLARE @ID NVARCHAR(100) = '2'
DECLARE @ColumnName NVARCHAR(100) ='UserID'

DECLARE @Sql NVARCHAR(MAX)=N'CREATE TABLE #TempResults(TableName NVARCHAR(50), ColumnName NVARCHAR(50), ItemCount INT)'

SELECT      
     @Sql = @Sql + N'INSERT INTO #TempResults SELECT * FROM (SELECT '''+ST.Name+''' AS TableName,  '''+C.Name+''' AS ColumnName, COUNT(*) AS ItemCount FROM '+ST.Name+' WHERE '+C.Name+'='+@ID+') AS X WHERE ItemCount > 0 '  
FROM        
    sys.columns C
    INNER JOIN sys.tables ST ON C.object_id = ST.object_id
WHERE      
    C.Name  LIKE '%'+@ColumnName+'%'

SET @Sql = @Sql + N'SELECT * FROM #TempResults'

exec sp_executesql @sql
Ross Bush
  • 14,648
  • 2
  • 32
  • 55
  • Thank you so much, let me try it and I will let you know. BTW the target column is of type "char" @Ross Bush – Montreal Aug 18 '22 at 19:23
  • It is counting the number of records in the tables. It is not using variable ID to search the columns. Can you please modify your script to include that? Thank you again! – Montreal Aug 18 '22 at 19:44
  • @Montreal - Doh! Sorry, small mental lapse. I updated it to return proper counts, not all counts. – Ross Bush Aug 18 '22 at 19:53
  • I am getting the conversion error message. "Conversion failed when converting the varchar value 'some value with number ' to data type int. – Montreal Aug 18 '22 at 20:06
  • I thought you might get that too. That is a large amount of columns and case may be some fields are not all INTs – Ross Bush Aug 18 '22 at 20:07
  • You can see if this will work -> WHERE '+C.Name+'='''+@ID+''' – Ross Bush Aug 18 '22 at 20:08
  • I am still getting the same conversion error message. :( The column is of type: (PK, char(31), not null) I really appreciate your help! Thanks! – Montreal Aug 18 '22 at 20:20
  • @Montreal - That is a viable conversion. How did you find out which column errored out? char() should convert with given cast. – Ross Bush Aug 19 '22 at 00:04
  • Let us [continue this discussion in chat](https://chat.stackoverflow.com/rooms/247378/discussion-between-ross-bush-and-montreal). – Ross Bush Aug 19 '22 at 01:05
0

You need to do this with dynamic SQL. You will need to query all 1000 tables, and make sure you are converting the values correctly if the columsn are different types.

You don't need a temp table for this, you can just script one giant UNION ALL query. You must make sure to quote all dynamic names correctly using QUOTENAME.

To be able to return data for multiple items, you should create a Table Valued Parameter, which you can pass in using sp_executesql.

First create a table type

CREATE TYPE dbo.IntList (Id int PRIMARY KEY);

Then you create a table variable containing them, and pass it in. You can also do this in a client application and pass in a TVP.

SET NOCOUNT ON;

DECLARE @Items dbo.IntList;
INSERT @Items (Id) VALUES(350),(465);


DECLARE @Sql nvarchar(max);

SELECT      
     @Sql = STRING_AGG(CONVERT(nvarchar(max), N'
SELECT
  ' + QUOTENAME(t.name, '''') + ' AS TableName,
  t.ItemNumber,
  COUNT(*) AS ItemCount
FROM ' + QUOTENAME(t.Name) + ' t
JOIN @items i ON i.Id = t.ItemNumber
GROUP BY
  t.ItemNumber
HAVING COUNT(*) > 0
'   ),
    N'
UNION ALL
'   )

FROM        
    sys.tables t
WHERE t.object_id IN (
    SELECT c.object_id
    FROM sys.columns c
    WHERE      
      c.Name = 'ItemNumber'
);

PRINT @sql;  -- your friend

EXEC sp_executesql
  @sql,
  N'@items dbo.IntList',
  @items = @items READONLY;

If you don't need to know the count, and only want to know if a value exists, you can change the dynamic SQL to an EXISTS

....
SELECT      
     @Sql = STRING_AGG(CONVERT(nvarchar(max), N'
SELECT
  ' + QUOTENAME(t.name, '''') + ' AS TableName,
  t.ItemNumber
FROM @items i
WHERE i.Id IN (
    SELECT t.ItemNumber
    FROM ' + QUOTENAME(t.Name) + ' t
)
'   ),
    N'
UNION ALL
'   )

....
Charlieface
  • 52,284
  • 6
  • 19
  • 43