Generic Import method for EF

Question

I would like to make a generic method to import data into my application.

For example, say I have:

private static async Task<int> ImportAccount(string filename)
{
    var totalRecords = await GetLineCount(filename);
    var ctx = new AccountContext();
    var count = 0;
    var records = 0;
    using (var stream = File.Open(filename, FileMode.Open, FileAccess.Read, FileShare.ReadWrite))
    {
        using (var reader = new StreamReader(stream, Encoding.UTF8))
        {
            string line;
            while ((line = await reader.ReadLineAsync()) != null)
            {
                var data = line.Split('\t');
                var acc = new Account(data);
                await ctx.Accounts.AddAsync(acc);
                // need this to avoid using all the memory
                // maybe there is a smarter or beter way to do it
                // with 10k it uses about 500mb memory, 
                // files have million rows+
                if (count % 10000 == 1)
                {
                    records += result = await ctx.SaveChangesAsync();
                    if (result > 0)
                    {
                        ctx.Dispose();
                        ctx = new AccountContext();
                    }
                }
                count++;
            }
        }
    }
    await ctx.SaveChangesAsync();
    ctx.Dispose();
    return records;
}

In the above example I am importing data from a tab delimited file into the Accounts db.

Then I have properties, lands, and a whole lot of other db's I need to import.

Instead of having to make a method for each db like the above, I would like to make something like:

internal static readonly Dictionary<string, ??> FilesToImport = new Dictionary<string, ??>
{
    { "fullpath to file", ?? would be what I need to pass to T }
    ... more files ...
};
private static async Task<int> Import<T>(string filename)

Where T would be the DB in question.

All my classes have 1 thing in common, they all have a constructor that takes a string[] data.

But I have no idea how I could make a method that I would be able to accept:

private static async Task<int> Import<T>(string filename)

And then be able to do a:

var item = new T(data);
await ctx.Set<T>().AddAsync(item);

And if I recall correctly, I would not be able to instantiate T with a parameter.

How could I make this generic Import method and is it possible to achieve?

Avin Kavish · Accepted Answer · 2019-06-07T16:21:11.393

2

The easiest way to accomplish this is to pass a generic function that accepts the string line or a string array of split values and returns an object with values set. Use the ctx.AddAsync() method which supports generics and add the entity to the correct set.

private static async Task<int> Import<T>(string filename, Func<string, T> transform) where T : class
{
    var totalRecords = await GetLineCount(filename);
    var ctx = new AccountContext();
    var count = 0;
    var records = 0;
    using (var stream = File.Open(filename, FileMode.Open, FileAccess.Read, FileShare.ReadWrite))
    {
        using (var reader = new StreamReader(stream, Encoding.UTF8))
        {
            string line;
            while ((line = await reader.ReadLineAsync()) != null)
            {
                var data = line.Split("\t");
                var entity = transform(data);
                await ctx.AddAsync(entity);
                if (count % 10000 == 1)
                {
                    records += result = await ctx.SaveChangesAsync();
                    if (result > 0)
                    {
                        ctx.Dispose();
                        ctx = new AccountContext();
                    }
                }
                count++;
            }
        }
    }
    await ctx.SaveChangesAsync();
    ctx.Dispose();
    return records;
}

// Usage

Import(filename, splits => {
   / * do whatever you need to transform the data */
   return new Whatever(splits);
})

Since generic types cannot be constructed by passing a parameter, you will have to use a function as the second type in the dictionary.

Dictionary<string, Func<string, object>> FilesToImport = new Dictionary<string, Func<string, object>>{
  { "fullpath to file", data => new Account(data) },
  { "fullpath to file", data => new Whatever(data) },
  { "fullpath to file", data => new Whatever2(data) },
}

edited Jun 07 '19 at 16:21

answered Jun 07 '19 at 12:55

Avin Kavish

8,317
1
21
36

That looks very interesting but `var entity = transform(line)`says it can't convert string to T, – Guapo Jun 07 '19 at 15:22
oh sorry, edited. Changed signature to `Func` – Avin Kavish Jun 07 '19 at 15:27
now it says `The type 'T' must be a reference type in order to use it as parameter 'TEntity' in the generic type or method 'DbContext.AddAsync(TEntity, CancellationToken)'` but the transform works. – Guapo Jun 07 '19 at 15:34
If I add `where T : class` it goes away but I am not entirely sure that is right. – Guapo Jun 07 '19 at 15:40
But how would I create a new T from the type in my dictionary? otherwise I would still need to define the type myself for each import `return new Account(data);` should be `return new T(data);` or something along those lines. – Guapo Jun 07 '19 at 15:48
That's not possible unfortunately, https://stackoverflow.com/questions/1852837/is-there-a-generic-constructor-with-parameter-constraint-in-c. You have to pass a function that creates the object. – Avin Kavish Jun 07 '19 at 15:55
I just realised there is no way to build a generic dictionary of functions with different types. What is stopping you from writing a bunch of import statements? as far as I can see it takes the same effort as creating the dictionary in the first place. Is this a one time load or re-usable code? – Avin Kavish Jun 07 '19 at 16:16
Well my goal is to connect a filename to a specific class, so that I could just say make a dictionary that will import that file using that said class to insert into the database without, what would you suggest then? I originally had a switch that checks the file name and uses the class but that looks awful and I need to update for every new table with a new class. – Guapo Jun 07 '19 at 16:21
Oh I see what you did there, that looks interesting let me try – Guapo Jun 07 '19 at 16:22
Let us [continue this discussion in chat](https://chat.stackoverflow.com/rooms/194609/discussion-between-avin-kavish-and-guapo). – Avin Kavish Jun 07 '19 at 16:22

Roman Koliada · Answer 2 · 2019-06-10T07:25:49.893

C# have only new() restriction for generic type arguments. But unfortunately it's not possible to force a type to have a constructor with parameters.

One workaround for this is to define an interface like this:

interface IImportedEntity<T>
// where T: YourBaseClass
{
    T Init(string[] data);
}

In this case all implementation classes will have to implement such method:

class Account : /*YourBaseClass*/ IImportedEntity<Account>
{
    public Account()
    {
        // for EF
    }

    // can be made private or protected
    public Account(string[] data)
    {
        // your code
    }


    // public Account Init(string[] data) => { /*populate current instance*/  return this;};
    // can be implemented in base class
    public Account Init(string[] data) => new Account(data);
}

Finally you can restrict your generic Import method to deal only with imported entities:

private static async Task<int> Import<T>(string filename) 
      where T: class, IImportedEntity<T>, new()
{
    ....
    var item = new T();
    item = item.Init(data);
    await ctx.Set<T>().AddAsync(item);
    ...
}

Note, if you still want to use this with a dictionary it will be needed to use reflection(example).

This looks promising but I get a `Cannot implicitly convert type 'V' to 'T'` and `The type 'T' must be a reference type in order to use it as parameter 'TEntity' in the generic type or method 'DbContext.Set()'` on the respective lines `item = item.Init(data);` and `await ctx.Set().AddAsync(item);` — Guapo, Jun 07 '19 at 16:13

Generic Import method for EF

2 Answers2