Synchronizing XML file to MySQL database

Question

My company uses an internal management software for storing products. They want to transpose all the products in a MySql database so they can do available their products on the company website.

Notice: they will continue to use their own internal software. This software can exports all the products in various file format (including XML).

The syncronization not have to be in real time, they are satisfied to syncronize the MySql database once a day (late night).

Also, each product in their software has one or more images, then I have to do available also the images on the website.

Here is an example of an XML export:

<?xml version="1.0" encoding="UTF-8"?>
<export_management userid="78643">
    <product id="1234">
        <version>100</version>
        <insert_date>2013-12-12 00:00:00</insert_date>
        <warrenty>true</warrenty>
        <price>139,00</price>
        <model>
            <code>324234345</code>
            <model>Notredame</model>
            <color>red</color>
            <size>XL</size>
        </model>
        <internal>
            <color>green</color>
            <size>S</size>
        </internal>
        <options>
            <s_option>aaa</s_option>
            <s_option>bbb</s_option>
            <s_option>ccc</s_option>
            <s_option>ddd</s_option>
            <s_option>eee</s_option>
            <s_option>fff</s_option>
      ...
            <extra_option>ggg</extra_option>
            <extra_option>hhh</extra_option>
            <extra_option>jjj</extra_option>
            <extra_option>kkk</extra_option>
      ...
        </options>
        <images>
            <image>
                <small>1234_0.jpg</small>
            </image>
            <image>
                <small>1234_1.jpg</small>
            </image>
        </images>
    </product>
    <product id="5321">
    ...
    </product>
    <product id="2621">
    ...
    </product>
  ...
</export_management>

Some ideas for how can I do it?

Please let me know if my question is not clear. Thanks

---

EDIT: I used a SQL like this for each table to fill them with the XML datas:

LOAD XML LOCAL INFILE '/products.xml' INTO TABLE table_name ROWS IDENTIFIED BY '<tag_name>';

Then, checking the tables content I can see that the field "id" (primary key) automatically has mantained itself the same for each respective product row in each tables. That's correct and suprisingly awesome!

The problem now is for the parameter <options> because it contains sub-parameters with same name (<s_option> and <extra_option>). The values of these tags are always different (that is, there is no a specific list of values, they are inserted manually by an employee) and also I don't know how many are for each product. I read that storing them as an array is not so good but if it's the only simple solution I can get it.

Answer 1

The way that I would approach the problem in your case is:

1. Create a respective set of corresponding tables in the database which in turn will represent the company's Product model by extracting the modelling from your given XML.

2. Create and use a scheduled daily synchronization job, that probably will executes few SQL commands in order to refresh the data or introduce a new one by parsing the products XMLs into the created tables.

To be more practical about it all:

• As for the database's tables, I can easily identify three tables to be created based on your XML, look at the yellow marked elements:

  1. Products
  2. ProductsOptions
  3. ProductsImages

(This diagram created based on an XSD that was generated from your XML)

---

All rest can be considered as regular columns in the Products table since they're constitutes a 1-1 relationship only.

Next, create the required tables in your database (you can use an XSD2DB Schema converter tool to create the DDL script, I did it manually):

companydb.products

CREATE TABLE companydb.products (
  Id INT(11) NOT NULL,
  Version INT(11) DEFAULT NULL,
  InsertDate DATETIME DEFAULT NULL,
  Warrenty TINYINT(1) DEFAULT NULL,
  Price DECIMAL(19, 2) DEFAULT NULL,
  ModelCode INT(11) DEFAULT NULL,
  ModelColor VARCHAR(10) DEFAULT NULL,
  Model VARCHAR(255) DEFAULT NULL,
  ModelSize VARCHAR(10) DEFAULT NULL,
  InternalColor VARCHAR(10) DEFAULT NULL,
  InternalSize VARCHAR(10) DEFAULT NULL,
  PRIMARY KEY (Id)
)
ENGINE = INNODB
CHARACTER SET utf8
COLLATE utf8_general_ci
COMMENT = 'Company''s Products';

companydb.productsimages

CREATE TABLE companydb.productimages (
  Id INT(11) NOT NULL AUTO_INCREMENT,
  ProductId INT(11) DEFAULT NULL,
  Size VARCHAR(10) DEFAULT NULL,
  FileName VARCHAR(255) DEFAULT NULL,
  PRIMARY KEY (Id),
  CONSTRAINT FK_productsimages_products_Id FOREIGN KEY (ProductId)
    REFERENCES companydb.products(Id) ON DELETE RESTRICT ON UPDATE RESTRICT
)
ENGINE = INNODB
AUTO_INCREMENT = 1
CHARACTER SET utf8
COLLATE utf8_general_ci
COMMENT = 'Products'' Images';

companydb.productsoptions

CREATE TABLE companydb.productoptions (
  Id INT(11) NOT NULL AUTO_INCREMENT,
  ProductId INT(11) DEFAULT NULL,
  Type VARCHAR(255) DEFAULT NULL,
  `Option` VARCHAR(255) DEFAULT NULL,
  PRIMARY KEY (Id),
  CONSTRAINT FK_producstsoptions_products_Id FOREIGN KEY (ProductId)
    REFERENCES companydb.products(Id) ON DELETE RESTRICT ON UPDATE RESTRICT
)
ENGINE = INNODB
AUTO_INCREMENT = 1
CHARACTER SET utf8
COLLATE utf8_general_ci;

---

• As for the synchronisation job process to take place, you can easily create an MySql event and use the Event Scheduler to control it, I created the required event which is calling a stored-procedure that you'll find below (SyncProductsDataFromXML), look:

CREATE DEFINER = 'root'@'localhost' EVENT companydb.ProductsDataSyncEvent ON SCHEDULE EVERY '1' DAY STARTS '2014-06-13 01:27:38' COMMENT 'Synchronize Products table with Products XMLs' DO BEGIN SET @productsXml = LOAD_FILE('C:/MySqlXmlSync/products.xml'); CALL SyncProductsDataFromXML(@productsXml); END; ALTER EVENT companydb.ProductsDataSyncEvent ENABLE

Now the interesting part is taking place, here is the synchronization stored-procedure (note how the event above is calling it):

CREATE DEFINER = 'root'@'localhost'
PROCEDURE companydb.SyncProductsDataFromXML(IN productsXml MEDIUMTEXT)
BEGIN

  DECLARE totalProducts INT;
  DECLARE productIndex INT;

  SET totalProducts = ExtractValue(productsXml, 'count(//export_management/product)');

  SET productIndex = 1;
  WHILE productIndex <= totalProducts DO
        SET @productId = CAST(ExtractValue(productsXml, 'export_management/product[$productIndex]/@id') AS UNSIGNED);

        INSERT INTO products(`Id`, `Version`, InsertDate, Warrenty, Price, ModelCode, Model, ModelColor, ModelSize, InternalColor, InternalSize)
        VALUES(
                @productId,
                ExtractValue(productsXml, 'export_management/product[$productIndex]/version'),
                ExtractValue(productsXml, 'export_management/product[$productIndex]/insert_date'),
                CASE WHEN (ExtractValue(productsXml, 'export_management/product[$productIndex]/warrenty')) <> 'false' THEN 1 ELSE 0 END,
                CAST(ExtractValue(productsXml, 'export_management/product[$productIndex]/price') as DECIMAL),
                ExtractValue(productsXml, 'export_management/product[$productIndex]/model/code'),
                ExtractValue(productsXml, 'export_management/product[$productIndex]/model/model'),
                ExtractValue(productsXml, 'export_management/product[$productIndex]/model/color'),
                ExtractValue(productsXml, 'export_management/product[$productIndex]/model/size'),
                ExtractValue(productsXml, 'export_management/product[$productIndex]/internal/color'),
                ExtractValue(productsXml, 'export_management/product[$productIndex]/internal/size')
              );

        SET @totalImages = ExtractValue(productsXml, 'count(//export_management/product[$productIndex]/images/image)');
        SET @imageIndex = 1;
        WHILE (@imageIndex <= @totalImages) DO
          INSERT INTO productimages(ProductId, Size, FileName) VALUES(@productId, 'small', EXTRACTVALUE(productsXml, 'export_management/product[$productIndex]/images/image[$@imageIndex]/small'));
          SET @imageIndex = @imageIndex + 1;        
        END WHILE;

        SET @totalStandardOptions = ExtractValue(productsXml, 'count(//export_management/product[$productIndex]/options/s_option)');
        SET @standardOptionIndex = 1;
        WHILE (@standardOptionIndex <= @totalStandardOptions) DO
          INSERT INTO productoptions(ProductId, `Type`, `Option`) VALUES(@productId, 'Standard Option', EXTRACTVALUE(productsXml, 'export_management/product[$productIndex]/options/s_option[$@standardOptionIndex]'));
          SET @standardOptionIndex = @standardOptionIndex + 1;
        END WHILE;

        SET @totalExtraOptions = ExtractValue(productsXml, 'count(//export_management/product[$productIndex]/options/extra_option)');
        SET @extraOptionIndex = 1;
        WHILE (@extraOptionIndex <= @totalExtraOptions) DO
          INSERT INTO productoptions(ProductId, `Type`, `Option`) VALUES(@productId, 'Extra Option', EXTRACTVALUE(productsXml, 'export_management/product[$productIndex]/options/extra_option[$@extraOptionIndex]'));
          SET @extraOptionIndex = @extraOptionIndex + 1;        
        END WHILE;

        SET productIndex = productIndex + 1;

  END WHILE;
END

And you're done, this is the final expected results from this process:

---

---

NOTE: I've commit the entire code to one of my GitHub's repositories: XmlSyncToMySql

UPDATE:

Because your XML data might be larger then the maximum allowed for a TEXT field, I've changed the productsXml parameter to a MEDIUMTEXT. Look at this answer which outlines the various text datatypes max allowed size: Maximum length for MYSQL type text

Answer 2

As this smells like integration work, I would suggest a multi-pass, multi-step procedure with an interim format that is not only easy to import into mysql but which also helps you to wrap your mind around the problems this integration ships with and test a solution in small steps.

This procedure works well if you can flatten the tree structure that can or could be expressed within the XML export into a list of products with fixed named attributes.

• query all product elements with an xpath query from the XML, iterate the result of products
• query all product attributes relative to the context node of the product from the previous query. Use one xpath per each attribute again.
• store the result of all attributes per each product as one row into a CSV file.
• store the filenames in the CSV as well (the basenames), but the files into a folder of it's own
• create the DDL of the mysql table in form of an .sql file
• run that .sql file against mysql commandline.
• import the CSV file into that table via mysql commandline.

You should get quick results within hours. If it turns out that products can not be mapped on a single row because of attributes having multiple values (what you call an array in your question), consider to turn these into JSON strings if you can not prevent to drop them at all (just hope you don't need to display complex data in the beginning). Doing so would be violating to target a normal form, however as you describe the Mysql table is only intermediate here as well, I would aim for simpleness of the data-structure in the database as otherwise queries for a simple and fast display on the website will create the next burden.

So my suggestion here basically is: Turn the tree structure into a (more) flat list for both simplification of transition and easier templating for display.

Having an intermediate format here also allows you to replay in case things are going wrong.

It also allows you to mock the whole templating more easily.

Alterantively is is also possible to store the XML of each project inside the database (keep the chunks in a second table so you can keep varchar (variable length) fileds out of the first table) and keep some other columns as (flat) reference columns to query against. If it's for templating needs, turning the XML into a SimpleXMLElement is often very nice to have it being a structured, non-primitive data-type as view object you can traverse and loop over options. Would work similar with JSON however keeping the XML would not break a format boundary and XML can also express more structure than JSON.

Answer 3

You're taking a very technology-centered approach to this. I think it's wise to start by looking at the functional specifications.

It helps to have a simple UML class diagram of the business class Product. Show its attributes as the business sees them. So:

• How is Model related to Product? Can there be multiple Models for one Product or the other way around?
• What kind of data is stored in the Internal element? In particular: How can Internal's color and size be different from Model's color and size?

And specifically about the web application:

• Is the Web application the only application that will be interested in this export?
• Should the web application care about versioning or simply display the last available version?
• Which specific options are interesting to the web application. Like a discount property or vendor name property or others?
• What should the Product details page look like, what data needs to be displayed where?
• Will there be other pages on the website displaying product information, and what product information will they list?

Then you'll know which attributes need to be readily available to the web application (as columns in the Product table) and (perhaps) which ones may be simply stored in one big XML blob in the database.