8

Preamble:

  1. All data connection strings, connections, etc are created using DbProviderFactories.
  2. Code is mixed C# and VB.Net from mulitple libraries.

I am mapping a DbDataReader to entities and have some benchmarks:

[0] retrieved 159180 records in 45135 ms
[1] retrieved 159180 records in 45008 ms
[2] retrieved 159180 records in 44814 ms
[3] retrieved 159180 records in 44987 ms
[4] retrieved 159180 records in 44914 ms
[5] retrieved 159180 records in 45224 ms
[6] retrieved 159180 records in 45829 ms
[7] retrieved 159180 records in 60762 ms
[8] retrieved 159180 records in 52128 ms
[9] retrieved 159180 records in 47982 ms  

This is a significant amount of time and extremely poor considering it only takes 17 seconds to query from Sql Server Management Studio. My select statement:

"SELECT * FROM tbl_MyTable"

Table contains 43 fields and probably isn't indexed as best as it should; however, performing a select all, I wouldn't expect indexing to be problematic. So ... here is what I am doing:

Define an entity:

public class Concept
{
    #region Columns
    [DataParameter("ConceptID", DbType.Int32)]
    public Int32 ConceptID
    { get; set; }
    [DataParameter("ConceptName", DbType.String)]
    public string ConceptName
    { get; set; }
    [DataParameter("ConceptTypeID", DbType.Int32)]
    public Int32 ConceptTypeID
    { get; set; }
    [DataParameter("ActiveYN", DbType.Boolean)]
    public bool ActiveYN
    { get; set; }
    #endregion
}

Query DataReader:

for (int i = 0; i <= 99; i++)
{
    sw.Start();
    var results = session.QueryReader<Concept>(
        new SqlCommand(command), dr => new Concept());

    sw.Stop();

    Console.WriteLine("[{0}] retrieved {1} records in {2} ms", i, results.Count(), sw.ElapsedMilliseconds);
    sw.Reset();
}

... calling:

Public Function QueryReader(Of TEntity As {Class, New})(ByVal Command As DbCommand, _
                                                        ByVal Projection As Func(Of DbDataReader, TEntity)) _
                                                        As IEnumerable(Of TEntity)

    Dim list As IEnumerable(Of TEntity)

    Command.Connection = dataReader.NewConnection
    Command.Connection.Open()

    Using _reader As DbDataReader = Command.ExecuteReader()
        list = _reader.Query(Of TEntity)(Projection).ToList()
    End Using

    Command.Connection.Close()

    Return list
End Function

... and extension method QueryReader<T>: edit placement of new TEntity() - thanks @Henk

public static IEnumerable<TEntity> Query<TEntity>(this DbDataReader Reader,
    Func<DbDataReader, TEntity> Projection)
    where TEntity : class, new()
{
    //   moving this reflection to another class
    Dictionary<string, PropertyInfo> props;

    while (Reader.Read())
    {
        TEntity entity = new TEntity();

        if (!entities.TryGetValue(typeof(TEntity).ToString(), out props))
        {
            //  reflection over TEntity
            props = (from p in entity.GetType().GetProperties()
                     from a in p.GetCustomAttributes(typeof(DataParameterAttribute), false)
                     select p)
                     .ToDictionary(p => p.Name);

            entities.Add(typeof(TEntity).ToString(), props);
        }

        foreach (KeyValuePair<string, PropertyInfo> field in props)
        {
            if (null != Reader[field.Key] && Reader[field.Key] != DBNull.Value)
            { field.Value.SetValue(entity, Reader[field.Key], null); }
        }

        yield return entity;
    }
}

Any suggestions on increasing performance would be greatly appreciated ...


Update

I implemented dapper-dot-net as @EtienneT suggested - here are the retrieval times:

[0] retrieved 159180 records in 6874 ms
[1] retrieved 159180 records in 6866 ms
[2] retrieved 159180 records in 6570 ms
[3] retrieved 159180 records in 6785 ms
[4] retrieved 159180 records in 6693 ms
[5] retrieved 159180 records in 6735 ms
[6] retrieved 159180 records in 6627 ms
[7] retrieved 159180 records in 6739 ms
[8] retrieved 159180 records in 6569 ms
[9] retrieved 159180 records in 6666 ms
IAbstract
  • 19,551
  • 15
  • 98
  • 146
  • 160K records with 43 fields populating entities via reflection? I'd say 45-50 seconds is pretty decent, but maybe I'm too forgiving. – Anthony Pegram May 26 '11 at 17:35
  • @Anthony: I think it's *decent* ... but I want **wow** - especially since I use a dictionary to map the reader[field] to the propertyInfo ... but maybe my expectations are too high. ;) – IAbstract May 26 '11 at 17:40
  • 1
    Just for fun, try writing an old-fashioned `while(Reader.Read())` loop, Fx 2.0 style. It should give you an idea how expensive all this reflection and indirection is. – H H May 26 '11 at 17:40
  • And I'm perplexed the `new TEntity()` is outside the while. After this is done, are all your records the same? – H H May 26 '11 at 17:41
  • @Henk: Fx 2.0 style ??? I was thinking about trying to push into an anonymous type instead of a class; but having an IEnumerable is soooo much better than working with anonTypes or DataTables. Plus my TConcrete objects are quite mutable and I plan on using them with a similar implementation with a DataReader to write back to the data store ... if all that makes sense. ;) – IAbstract May 26 '11 at 17:44
  • @Henk: yup - I had been messing with it ... thanks – IAbstract May 26 '11 at 17:50
  • @Anthony: now I have my **wow** I was looking for. It's faster than MS Sql Mgr Studio. – IAbstract May 26 '11 at 19:42
  • 1
    Mixing your drinks? (c# and vb) – Aran Mulholland Aug 12 '14 at 06:22
  • 1
    @AranMulholland: unfortunately, yes :) at the time I had a DAL written in VB by another developer and I was building the backend in C#. The beauty of .Net! – IAbstract Aug 12 '14 at 19:39
  • Few things. 1) You're passing parameter "Projection" to your Query method. Then why not just return the projected form in place of all this reflection? Or was it a mistake? 2) Your strategy to not assign anything to property when the db value is DBNull can backfire. For e.g. assume your default constructor of entity looks like this: "public T() { P = someValue; }" .Now if property "P" is DbNull in db and you are not assigning anything to it in your mapping code, then the mapper returns a T where P = someValue (assigned in constructor) while in db it is DBNull. These are rarities but still.. – nawfal Jul 30 '15 at 06:37
  • Also you're reading multiple times from reader for a single value. I think this can be greatly improved. See: http://codereview.stackexchange.com/questions/58251/transform-datareader-to-listt-using-reflections. Ultimately I believe one should use expression trees here. – nawfal Aug 01 '15 at 10:18
  • Related: http://stackoverflow.com/questions/812034/fastest-way-to-use-reflection-for-converting-datareader-to-list, http://stackoverflow.com/questions/19841120/generic-dbdatareader-to-listt-mapping, http://stackoverflow.com/questions/1105549/c-sharp-idatareader-to-object-mapping-using-generics – nawfal Aug 01 '15 at 11:57

1 Answers1

6

Have you considered a micro ORM like dapper.net?

https://github.com/StackExchange/dapper-dot-net

It is made by the developers of StackOverflow and map an SQL query directly to your objects. It generates and caches IL code to map the SQL results to your objects. So the IL code is generated only one time per type. Never used this, but if you need performance to map your SQL results to .net objects, it is the library you need.

IAbstract
  • 19,551
  • 15
  • 98
  • 146
EtienneT
  • 5,045
  • 6
  • 36
  • 39
  • Wow, didn't know that existed. Looks nifty – Earlz May 26 '11 at 18:20
  • I'm looking at it now - I can give it a pretty quick test I think - I'll post some benchmarks as soon as I have it completed. – IAbstract May 26 '11 at 18:53
  • 1
    Something similar made by Rob Conery: http://blog.wekeroad.com/helpy-stuff/and-i-shall-call-it-massive Also pretty fast, but it deals more with dynamic objects. Dapper is still faster from what I read. – EtienneT May 26 '11 at 19:18
  • Only negative thing I saw about Dapper is that it generates IL to map your objects and then cache it. So if there's a bug in the generated IL, good luck. Look at the dapper code: https://github.com/SamSaffron/dapper-dot-net/blob/master/Dapper/SqlMapper.cs#L677 – EtienneT May 26 '11 at 19:20
  • 1
    So far, it seems to be quite fast and without issue. I don't expect much in the way of IL bugs being that Marc Gravell is involved. – IAbstract May 26 '11 at 20:07
  • @IAbstract `Dapper` GitHub page says: Hand coded using `SqlDataReader` has 2 ms more performance than `Dapper` , so its better to use [this](https://stackoverflow.com/a/1464929/2218697) extension method or [this](https://stackoverflow.com/a/8525024/2218697) to map to class. – Shaiju T Sep 06 '17 at 12:33