Import 'xml' into Sql Server

Question

I have a file that is structured like so:

<?xml version="1.0" encoding="UTF-8"?>
<EventSchedule>
    <Event Uid="2" Type="Main Event">
        <IsFixed>True</IsFixed>
        <EventKind>MainEvent</EventKind>
        <Fields>
            <Parameter Name="Type" Value="TV_Show"/>
            <Parameter Name="Name" Value="The Muppets"/>
            <Parameter Name="Duration" Value="00:30:00"/>
        </Fields>
    </Event>
    <Event>
    ...and so on
    </Event>
</EventSchedule>

I'm not entirely sure if it is valid XML, however I need to import it into SQL Server but everything I try doesn't seem to work.

Please could anyone point me in the right direction either with some example code or a recommendation on which method to use?

I'd ideally like to get the raw data into a flat table, along the lines of:

Name        | Type    | Duration | EventKind

The Muppets | TV_Show | 00:30:00 | MainEvent

Finally this is coming from fairly large files and I will need to import the regularly.

Thanks, pugu

How do you want the data to be composed once you have imported it? — paul, May 30 '13 at 14:19
Would you import it entirely as XML or parse it, build a schema based on XML structure and so on...? — Francesco De Lisi, May 30 '13 at 14:22
I was hoping to get the data into a single table, accepting that in some cases there would be 'null' entries, for example if one of the Parameters wasn't present. Ideally I would like to do it with some T-SQL that I can run as a scheduled task. I tried using SSIS but I couldn't get it to recognise the XML. — pugu, May 30 '13 at 14:31

marc_s · Accepted Answer · 2013-05-30T16:01:28.203

63

Try this:

DECLARE @XML XML = '<EventSchedule>
    <Event Uid="2" Type="Main Event">
        <IsFixed>True</IsFixed>
        <EventKind>MainEvent</EventKind>
        <Fields>
            <Parameter Name="Type" Value="TV_Show"/>
            <Parameter Name="Name" Value="The Muppets"/>
            <Parameter Name="Duration" Value="00:30:00"/>
        </Fields>
    </Event>
    <Event Uid="3" Type="Secondary Event">
        <IsFixed>True</IsFixed>
        <EventKind>SecondaryEvent</EventKind>
        <Fields>
            <Parameter Name="Type" Value="TV_Show"/>
            <Parameter Name="Name" Value="The Muppets II"/>
            <Parameter Name="Duration" Value="00:30:00"/>
        </Fields>
    </Event>
</EventSchedule>'

SELECT
    EventUID = Events.value('@Uid', 'int'),
    EventType = Events.value('@Type', 'varchar(20)'),
    EventIsFixed =Events.value('(IsFixed)[1]', 'varchar(20)'),
    EventKind =Events.value('(EventKind)[1]', 'varchar(20)')
FROM
 @XML.nodes('/EventSchedule/Event') AS XTbl(Events)

Gives me an output of:

enter image description here

And of course, you can easily do an

INSERT INTO dbo.YourTable(EventUID, EventType, EventIsFixed, EventKind)
   SELECT 
         ......

to insert that data into a relational table.

Update: assuming you have your XML in files - you can use this code to load the XML file into an XML variable in SQL Server:

DECLARE @XmlFile XML

SELECT @XmlFile = BulkColumn
FROM  OPENROWSET(BULK 'path-to-your-XML-file', SINGLE_BLOB) x;

and then use the above code snippet to parse the XML.

Update #2: if you need the parameters, too - use this XQuery statement:

SELECT
    EventUID = Events.value('@Uid', 'int'),
    EventType = Events.value('@Type', 'varchar(20)'),
    EventIsFixed = Events.value('(IsFixed)[1]', 'varchar(20)'),
    EventKind = Events.value('(EventKind)[1]', 'varchar(20)'),
    ParameterType = Events.value('(Fields/Parameter[@Name="Type"]/@Value)[1]', 'varchar(20)'),
    ParameterName = Events.value('(Fields/Parameter[@Name="Name"]/@Value)[1]', 'varchar(20)'),
    ParameterDuration = Events.value('(Fields/Parameter[@Name="Duration"]/@Value)[1]', 'varchar(20)')
FROM
    @XML.nodes('/EventSchedule/Event') AS XTbl(Events)

Results in:

enter image description here

edited May 30 '13 at 16:01

answered May 30 '13 at 14:45

marc_s

732,580
175
1,330
1,459

That works great thanks. With this method would I use OPENROWSET, OPENXML or bulk load to get the data in from the file? – pugu May 30 '13 at 14:55
@pugu: updated my response - use `OPENROWSET(BULK .....)` to load the file from disk – marc_s May 30 '13 at 15:04
The problem with this method is that it doesn't include any of the parameters such as "The Muppets"... – pugu May 30 '13 at 15:30
Thanks @marc_s, one more thing and then I will leave you alone I promise. I need the "ParameterName = Params.value('@Name', 'varchar(20)')" to actually be the column name and the "ParameterValue = Params.value('@Value', 'varchar(20)')" to be the data in that column. Thanks for all your help on this. – pugu May 30 '13 at 15:56
@Pugu: updated - yet again! You better accept this answer now! :-) – marc_s May 30 '13 at 16:01
1

That's brilliant thanks @marc_s. I'm completely new to xpath/xquery but this has helped me start getting to grips with it as well as solving my immediate problem. Off to read Books Online! – pugu May 30 '13 at 16:05
@marc_s What happens in case if ` ` get repeated, In this solution it won't show in the table – Shabar Aug 22 '15 at 12:04
1

@MPH: that is true - if the `` node shows up multiple times, you'd have to have another `CROSS APPLY Events.nodes('Fields') AS XT(XFields)` or something like that – marc_s Aug 22 '15 at 13:37
@marc_s Thanks, Really appreciated – Shabar Aug 22 '15 at 14:37
Can you please explain where exactly to put the INSERT statement. I made to to run with my files but it says `0 rows affected` so nothing is inserting in my table. But no other errors. – zar Jan 30 '21 at 04:46

score 2 · Answer 2 · answered May 30 '13 at 14:50

You do it by creating a destination table, then a schema mapping file that maps the xml elements to table columns.

Yours might look a bit like this:

create table event (
    Type nvarchar(50),
    Name nvarchar(50),
    Duration nvarchar(50))

and this:

<?xml version="1.0" ?>
<Schema xmlns="urn:schemas-microsoft-com:xml-data" 
        xmlns:dt="urn:schemas-microsoft-com:xml:datatypes"  
        xmlns:sql="urn:schemas-microsoft-com:xml-sql" > 

   <ElementType name="Type" dt:type="string" />
   <ElementType name="Name" dt:type="string" />
   <ElementType name="Duration" dt:type="string" />

   <ElementType name="EventSchedule" sql:is-constant="1">
      <element type="Event" />
   </ElementType>

   <ElementType name="Event" sql:relation="Event">
      <element type="Type" sql:field="Type" />
      <element type="Name" sql:field="Name" />
      <element type="Duration" sql:field="Duration" />
   </ElementType>
</Schema>

Then you can load your XML into your table using the XML bulk loader.

http://support.microsoft.com/kb/316005

I get the error `Cannot create Active X Object` but related [link](https://stackoverflow.com/q/12804636/841330) about it doesn't seem to help. Any ideas how to fix it? I tried adding .4.0 too but still same error. But your answer is pretty old too. Is this method not used much now? — zar, Jan 30 '21 at 04:37

Stefan Steiger · Answer 3 · 2015-10-21T08:03:50.033

If you need to do it without XML variable (from string in table-valued function)

SELECT 
    --myTempTable.XmlCol.value('.', 'varchar(36)') AS val 
     myTempTable.XmlCol.query('./ID').value('.', 'varchar(36)') AS ID 
    ,myTempTable.XmlCol.query('./Name').value('.', 'nvarchar(MAX)') AS Name 
    ,myTempTable.XmlCol.query('./RFC').value('.', 'nvarchar(MAX)') AS RFC 
    ,myTempTable.XmlCol.query('./Text').value('.', 'nvarchar(MAX)') AS Text 
    ,myTempTable.XmlCol.query('./Desc').value('.', 'nvarchar(MAX)') AS Description 

    --,myTempTable.XmlCol.value('(Desc)[1]', 'nvarchar(MAX)') AS DescMeth2
FROM 
(
    SELECT  
        CAST('<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
        <data-set>
            <record>
                <ID>1</ID>
                <Name>A</Name>
                <RFC>RFC 1035[1]</RFC>
                <Text>Address record</Text>
                <Desc>Returns a 32-bit IPv4 address, most commonly used to map hostnames to an IP address of the host, but it is also used for DNSBLs, storing subnet masks in RFC 1101, etc.</Desc>
            </record>
            <record>
                <ID>2</ID>
                <Name>NS</Name>
                <RFC>RFC 1035[1]</RFC>
                <Text>Name server record</Text>
                <Desc>Delegates a DNS zone to use the given authoritative name servers</Desc>
            </record>
        </data-set>
        ' AS xml) AS RawXml
) AS b 
--CROSS APPLY b.RawXml.nodes('//record/ID') myTempTable(XmlCol);
CROSS APPLY b.RawXml.nodes('//record') myTempTable(XmlCol);

Or from file:

IF  EXISTS (SELECT * FROM sys.objects WHERE object_id = OBJECT_ID(N'[dbo].[tfu_RPT_SEL_XmlData]') AND type in (N'FN', N'IF', N'TF', N'FS', N'FT'))
DROP FUNCTION [dbo].[tfu_RPT_SEL_XmlData]
GO



CREATE FUNCTION [dbo].[tfu_RPT_SEL_XmlData]
(
     @in_language varchar(10) 
    ,@in_reportingDate datetime 
)
RETURNS TABLE
AS
RETURN 
(   

    SELECT 
        --myTempTable.XmlCol.value('.', 'varchar(36)') AS val 
         myTempTable.XmlCol.query('./ID').value('.', 'varchar(36)') AS ID 
        ,myTempTable.XmlCol.query('./Name').value('.', 'nvarchar(MAX)') AS Name 
        ,myTempTable.XmlCol.query('./RFC').value('.', 'nvarchar(MAX)') AS RFC 
        ,myTempTable.XmlCol.query('./Text').value('.', 'nvarchar(MAX)') AS Text 
        ,myTempTable.XmlCol.query('./Desc').value('.', 'nvarchar(MAX)') AS Description 
    FROM 
    (
        SELECT CONVERT(XML, BulkColumn) AS RawXml 
        FROM OPENROWSET(BULK 'D:\username\Desktop\MyData.xml', SINGLE_BLOB) AS MandatoryRowSetName 
    ) AS b 
    CROSS APPLY b.RawXml.nodes('//record') myTempTable(XmlCol)

)


GO


SELECT * FROM tfu_RPT_SEL_XmlData('DE', CURRENT_TIMESTAMP);

e.g.

DECLARE @bla varchar(MAX)
SET @bla = 'BED40DFC-F468-46DD-8017-00EF2FA3E4A4,64B59FC5-3F4D-4B0E-9A48-01F3D4F220B0,A611A108-97CA-42F3-A2E1-057165339719,E72D95EA-578F-45FC-88E5-075F66FD726C'

-- http://stackoverflow.com/questions/14712864/how-to-query-values-from-xml-nodes
SELECT 
    x.XmlCol.value('.', 'varchar(36)') AS val 
FROM 
(
    SELECT 
    CAST('<e>' + REPLACE(@bla, ',', '</e><e>') + '</e>' AS xml) AS RawXml
) AS b 
CROSS APPLY b.RawXml.nodes('e') x(XmlCol);

So you can have a function like

SELECT * FROM MyTable 
WHERE UID IN 
(
    SELECT 
        x.XmlCol.value('.', 'varchar(36)') AS val 
    FROM 
    (
        SELECT 
        CAST('<e>' + REPLACE(@bla, ',', '</e><e>') + '</e>' AS xml) AS RawXml
    ) AS b 
    CROSS APPLY b.RawXml.nodes('e') x(XmlCol)
)

Francesco De Lisi · Answer 4 · 2013-05-30T14:44:41.773

If you're trying to import your XML as a "pure" XML field you should create a table like this (obviously with many other fields as you want):

CREATE TABLE [dbo].[TableXML](
    [ID] [int] IDENTITY(1,1) NOT NULL,
    [XmlContent] [xml] NOT NULL -- specify [xml] type
)

Then you can easily insert your XML as a string:

INSERT INTO [dbo].[TableXML]
           ([XmlContent])
     VALUES
           ('<?xml version="1.0" encoding="UTF-8"?>
               <EventSchedule>
                 <Event Uid="2" Type="Main Event">
                   <IsFixed>True</IsFixed>
                   <EventKind>MainEvent</EventKind>
                   <Fields>
                     <Parameter Name="Type" Value="TV_Show"/>
                     <Parameter Name="Name" Value="The Muppets"/>
                     <Parameter Name="Duration" Value="00:30:00"/>
                   </Fields>
                </Event>
              </EventSchedule>')

Then to query start from MSDN t-SQL XML

If you prefer store it as string use a varchar(max) in place of [XML] column type and the same insert. But if you like to query easily I suggest [XML] type. With the flat string approach you need a lot of work unless you will implement some application code to parse it and store in a flat table. A good approach could be an XML storage in a "compress" TABLE and a VIEW for data retrieve with the flat field disposition.

If possible I'd like to get it into a table as pure data, not XML. Or is that going to be impractical? — pugu, May 30 '13 at 14:33

score 0 · Answer 5 · edited Oct 17 '19 at 11:52

How to load the below XML data into the SQL

<?xml version="1.0" encoding="utf-8"?> 
<DataTable xmlns="SmarttraceWS"> 
  <xs:schema id="NewDataSet" xmlns="" xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:msdata="urn:schemas-microsoft-com:xml-msdata"> 
    <xs:element name="NewDataSet" msdata:IsDataSet="true" msdata:MainDataTable="ActivityRecords" msdata:UseCurrentLocale="true"> 
      <xs:complexType> 
        <xs:choice minOccurs="0" maxOccurs="unbounded"> 
          <xs:element name="ActivityRecords"> 
            <xs:complexType> 
              <xs:sequence> 
                <xs:element name="ReferenceID" type="xs:long" minOccurs="0" /> 
                <xs:element name="IMEI" type="xs:string" minOccurs="0" /> 
                <xs:element name="Asset" type="xs:string" minOccurs="0" /> 
                <xs:element name="Driver" type="xs:string" minOccurs="0" /> 
                <xs:element name="DateTime" type="xs:string" minOccurs="0" /> 

              </xs:sequence> 
            </xs:complexType> 
          </xs:element> 
        </xs:choice> 
      </xs:complexType> 
    </xs:element> 
  </xs:schema> 
  <diffgr:diffgram xmlns:msdata="urn:schemas-microsoft-com:xml-msdata" xmlns:diffgr="urn:schemas-microsoft-com:xml-diffgram-v1"> 
    <DocumentElement xmlns=""> 
      <ActivityRecords diffgr:id="ActivityRecords1" msdata:rowOrder="0"> 
        <ReferenceID>2620443016</ReferenceID> 
        <IMEI>013795001360346</IMEI> 
        <Asset>L-93745</Asset> 
        <Driver>N/A</Driver> 
        <DateTime>2019-10-14 12:00:35</DateTime> 

              </ActivityRecords> 
    </DocumentElement> 
  </diffgr:diffgram> 
</DataTable>

Import 'xml' into Sql Server

5 Answers5

Linked

Related