48

I want to store an object that contains a List of primitives using EF.

public class MyObject {
    public int Id {get;set;}
    public virtual IList<int> Numbers {get;set;}
}

I know that EF cannot store this, but I'd like to know possible solutions to solve this problem.

The 2 Solutions I can think of are:

1.Create a Dummy object that has an Id and the Integervalue, e.g.

public class MyObject {
    public int Id {get;set;}
    public virtual IList<MyInt> Numbers {get;set;}
}

public class MyInt {
    public int Id {get;set;}
    public int Number {get;set;}
}

2.Store the list values as a blob, e.g.

public class MyObject {
    public int Id {get;set;}

    /// use NumbersValue to persist/load the list values
    public string NumbersValue {get;set;}

    [NotMapped]
    public virtual IList<int> Numbers {
         get {
              return NumbersValue.split(',');
         }
         set {
             NumbersValue = value.ToArray().Join(",");
         }
    }
}

The Problem with the 2. approach is, that I have to create a Custom IList implementation to keep track if someone modifies the returned collection.

Is there a better solution for this?

Bernhard Kircher
  • 4,132
  • 3
  • 32
  • 38
  • 1
    "1.Create a Dummy object" If EF did support this natively, this is what it would be doing behind the scenes. I'd just go with this. –  Aug 16 '12 at 10:31
  • 1
    I didn't like that because I just want to store a List of primitives and I don't want EF to perform a Join on another table (which would require at least an Id and the value) - which may not be a problem but it just feels not good. I found a workable solution for my problem (see my answer). - I don't pretend it's perfect. – Bernhard Kircher Aug 16 '12 at 15:48
  • But you want to store a separate list of ints for each `MyObject`. That maps to a separate list of ints for each `MyObject.Id` -- exactly the extra table you want to avoid. There are multiple reasons why you shouldn't just shove them all in a string, but the two most clear ones (IMO) are that it makes it impossible to query for those `MyObject`s for which `Number` contains (for example) 10, and it breaks badly if joining all numbers results in a string that is too long to fit in your database column. –  Aug 16 '12 at 17:08
  • I am not sure if I understand you correctly. By using ComplexType the List data (in this case the string) is **stored in the same table as MyObject**. I think the solution is an acceptable one for my scenario, but you are right: I cannot query for objects that contain some values (at least not without using string operations) - but in my case this is not required (I did not mention this). The number of entries that can be stored is another good argument, but I think the max. number of characters on nvarchar(max) allows [2^30-1 characters](http://msdn.microsoft.com/en-us/library/ms143432.aspx) – Bernhard Kircher Aug 16 '12 at 18:37
  • Yes, you're understanding me correctly. If you're using `nvarchar(max)` then that objection doesn't apply. I was thinking of SQL Server 2000, which has a `varchar(8000)`/`nvarchar(4000)` maximum, unless you use the `text`/`ntext` type (which behaves differently). (It's still stored inefficiently, but that's less of a issue.) FWIW, this is called a repeating group and making your database [1NF](https://en.wikipedia.org/wiki/1NF) (First normal form) means getting rid of repeating groups. (Don't take this as saying database normalisation is always good per definition, there are exceptions.) –  Aug 16 '12 at 18:47

4 Answers4

45

Although I do not like to answer my own question, but here is what solved my problem:

After I found this link about Complex Types I tried several implementations, and after some headache I ended up with this.

The List values get stored as a string on the table directly, so it's not required to perform several joins in order to get the list entries. Implementors only have to implement the conversation for each list entry to a persistable string (see the Code example).

Most of the code is handled in the Baseclass (PersistableScalarCollection). You only have to derive from it per datatype (int, string, etc) and implement the method to serialize/deserialize the value.

It's important to note, that you cannot use the the generic baseclass directly (when you remove the abstract). It seems that EF cannot work with that. You also have to make sure to annotate the derived class with the [ComplexType] attribute.

Also note that it seems not to be possible to implement a ComplexType for IList<T> because EF complains about the Indexer (therefore I went on with ICollection).

It's also important to note, that since everything is stored within one column, you cannot search for values in the Collection (at least on the database). In this case you may skip this implementation or denormalize the data for searching.

Example for a Collection of integers:

    /// <summary>
    /// ALlows persisting of a simple integer collection.
    /// </summary>
    [ComplexType]
    public class PersistableIntCollection : PersistableScalarCollection<int> {
        protected override int ConvertSingleValueToRuntime(string rawValue) {
            return int.Parse(rawValue);
        }

        protected override string ConvertSingleValueToPersistable(int value) {
            return value.ToString();
        }
    }

Usage example:

public class MyObject {
    public int Id {get;set;}
    public virtual PersistableIntCollection Numbers {get;set;}
}

This is the baseclass that handles the persistence aspect by storing the list entries within a string:

    /// <summary>
    /// Baseclass that allows persisting of scalar values as a collection (which is not supported by EF 4.3)
    /// </summary>
    /// <typeparam name="T">Type of the single collection entry that should be persisted.</typeparam>
    [ComplexType]
    public abstract class PersistableScalarCollection<T> : ICollection<T> {

        // use a character that will not occur in the collection.
        // this can be overriden using the given abstract methods (e.g. for list of strings).
        const string DefaultValueSeperator = "|"; 

        readonly string[] DefaultValueSeperators = new string[] { DefaultValueSeperator };

        /// <summary>
        /// The internal data container for the list data.
        /// </summary>
        private List<T> Data { get; set; }

        public PersistableScalarCollection() {
            Data = new List<T>();
        }

        /// <summary>
        /// Implementors have to convert the given value raw value to the correct runtime-type.
        /// </summary>
        /// <param name="rawValue">the already seperated raw value from the database</param>
        /// <returns></returns>
        protected abstract T ConvertSingleValueToRuntime(string rawValue);

        /// <summary>
        /// Implementors should convert the given runtime value to a persistable form.
        /// </summary>
        /// <param name="value"></param>
        /// <returns></returns>
        protected abstract string ConvertSingleValueToPersistable(T value);

        /// <summary>
        /// Deriving classes can override the string that is used to seperate single values
        /// </summary>        
        protected virtual string ValueSeperator {
            get {
                return DefaultValueSeperator;
            }
        }

        /// <summary>
        /// Deriving classes can override the string that is used to seperate single values
        /// </summary>        
        protected virtual string[] ValueSeperators {
            get {
                return DefaultValueSeperators;
            }
        }

        /// <summary>
        /// DO NOT Modeify manually! This is only used to store/load the data.
        /// </summary>        
        public string SerializedValue {
            get {
                var serializedValue = string.Join(ValueSeperator.ToString(),
                    Data.Select(x => ConvertSingleValueToPersistable(x))
                    .ToArray());
                return serializedValue;
            }
            set {
                Data.Clear();

                if (string.IsNullOrEmpty(value)) {
                    return;
                }

                Data = new List<T>(value.Split(ValueSeperators, StringSplitOptions.None)
                    .Select(x => ConvertSingleValueToRuntime(x)));
            }
        }

        #region ICollection<T> Members

        public void Add(T item) {
            Data.Add(item);
        }

        public void Clear() {
            Data.Clear();
        }

        public bool Contains(T item) {
            return Data.Contains(item);
        }

        public void CopyTo(T[] array, int arrayIndex) {
            Data.CopyTo(array, arrayIndex);
        }

        public int Count {
            get { return Data.Count; }
        }

        public bool IsReadOnly {
            get { return false; }
        }

        public bool Remove(T item) {
            return Data.Remove(item);
        }

        #endregion

        #region IEnumerable<T> Members

        public IEnumerator<T> GetEnumerator() {
            return Data.GetEnumerator();
        }

        #endregion

        #region IEnumerable Members

        IEnumerator IEnumerable.GetEnumerator() {
            return Data.GetEnumerator();
        }

        #endregion
    }
Bernhard Kircher
  • 4,132
  • 3
  • 32
  • 38
  • 1
    Nice! I think this is better than creating a table that holds a column of integers – Serj Sagan Mar 25 '13 at 03:57
  • Very nice implementation, one question though. Are you using the **ComplexType** attribute or **EdmComplexType** attribute? Can't seem to find the **ComplexType** attribute anywhere... – rumblefx0 Mar 25 '13 at 08:45
  • 1
    @Rumble the ComplexType attribute is in the `System.ComponentModel.DataAnnotations.Schema` namespace, in the `EntityFramework` assembly. I think some of this changed after upgrading from Entity Framework 4.3 to **Entity Framework 5.0** – Bernhard Kircher Mar 25 '13 at 14:34
  • 1
    I'm sorry, but why is it wrong to have a table that holds the Id and N Values? After all, it IS a one-to-many relationship. There's nothing wrong using one-to-many, since you will always access the data using na integer indexed column. – cidico Mar 29 '13 at 06:04
  • @cidico There is nothing wrong with such an relationship. I think it depends on the usecase and what you need the list's data for. If you need to search for single list entries on the database you need such a reationship, since you cannot search in the complex type. In my case I just needed to store integer values (and datetimes) which are only used in the Application itself, and it felt a little strange to require a seperate object that is just an integer. Also for my scenario, this would mean that EF would need to perform an additional join to get the data (which should not be a problem..) – Bernhard Kircher Mar 29 '13 at 06:35
  • Have you thought of using an Xml column + index and serialize the collection with some Xml serializer? Would that enable search with E? – Tasio Aug 14 '13 at 09:17
  • I tried this and it worked. But I don't understand how it works. I put a breakpoint in the getter of SerializedValue property but it seems to be called by EXTERNAL CODE, what framework is called SerializedValue? How does this work? Cheers +1 – Lzh Feb 26 '14 at 12:06
  • 1
    @Tasio Sorry for beeing late... XML Column with a schema/index would be an interesting idea. But EF does not support querying XML columns directly (no linq, e.g. filtering always has to happen in memory and not on the database), but you could use XPAth queries etc. within manual SQL, SP's ... and let EF map the result to your object (although you can search xml data, in my experience this was always slow and only an option for small datasets). So this would be an advantage to my example. As I mentioned, my example is only suitable if you do not want to search the data, just store information. – Bernhard Kircher Mar 09 '14 at 09:54
  • @Mzn The SerializedValue property is called by Entity Framework, when deserializing a database result and materializing the objects (= converting a Resultset from the DB to the objects). The SerializedValue property is just a mapped column that is used to store the information (like all other properties that you map with EF and get persisted). We only add a manual conversion from the serialized string (which gets saved/loaded by EF) to a collection. – Bernhard Kircher Mar 09 '14 at 09:58
  • @BernhardKircher Brilliant! Saved me a lot of hacking! – Shimmy Weitzhandler Mar 01 '15 at 04:06
  • Explanation how to configure EntityTypeConfiguration if you use code first aproach: Property(x => x.AttachmentsPaths.SerializedValue).HasColumnName("AttachmentsPaths"); https://visualstudiomagazine.com/articles/2014/04/01/making-complex-types-useful.aspx – Denis Oct 02 '16 at 16:28
  • Is it possible to define custom functions to be able to search the "list" in the database? – Samir Aguiar Apr 19 '17 at 14:08
  • @samir arguiar unfortunately this is not possible via an integrated way. Also as stated in the answer, search is very limited, since you only search in a blob field. In that case a separate entity would make more Sense in my opinion. – Bernhard Kircher Apr 20 '17 at 17:11
  • 1
    @BernhardKircher - Hi, I'm using EF Core and I'm getting this error `The property 'Cameras.YourList' is of type 'PersistableStringCollection' which is not supported by current database provider. Either change the property CLR type or manually configure the database type for it.` – Liran Friedman Jun 21 '17 at 13:37
  • 1
    @liran-friedman This question/answer is quite old know and I did not try it on EF core. Right now I do not if there is a better way to do it in EF Core, but I just add my thoughts: As far as I know EF Core does not support complex types (yet), which is used to store the serialized List content "inline" of the table instead of adding a relation. So if we would follow the path described in the answer, there would be 2 options: – Bernhard Kircher Jun 21 '17 at 13:58
  • @liran-friedman 1.) Add a persisted Object that is basically the same like the abstract abseclass but has an Id (= it will get persisted in its own table, depending on the configuration) Te advantage would be, that this should be quite easy to do, but one of the downsides would be, that we have an additional select (loading the "list" property from the new table), but it would still better than loading all list entries. There would be still all donwsides described in the anser (searchng, etc).... – Bernhard Kircher Jun 21 '17 at 14:01
  • @BernhardKircher - I tried to overcome it using NewtonSoft's JsonConvert: `public partial class Cameras { public string PossibleResolutions { get; set; } [NotMapped] public List PossibleResolutionsList; } devices.Cameras.ToList().ForEach(camera => { camera.PossibleResolutionsList = JsonConvert.DeserializeObject>(camera.PossibleResolutions); });` – Liran Friedman Jun 21 '17 at 14:06
  • @LiranFriedman 2.) add the serialzed string property to the class directly and use the behaviour like described in the PersitableCollection as non persisted class: – Bernhard Kircher Jun 21 '17 at 14:18
  • @LiranFriedman: `public class MyObject { public MyObject() { _numbers = new PersistableIntCollection(); } private PersistableIntCollection _numbers; public string SerializedNumbers { get { return _numbers.SerializedValue;} set {_numbes.SerializedValue = value;} } [NotMapped] public virtual IList Numbers { get {return _numbers;} set { _numbers.Clear(); _numbers.AddRange(value); } }` – Bernhard Kircher Jun 21 '17 at 14:18
  • @LiranFriedman not sure if i understand correctly: i do not think changing the serialization technology (json vs manually) solves the problem. If you have infrastructurecode (like a repository) that always loads and saves your class, you coul add the logic there too – Bernhard Kircher Jun 21 '17 at 14:26
  • @LiranFriedman One thing I just noticed: since the errormessage states that it tries t opersist the "list" class: you need to add the [NotMapped] attribute to the list property, since the persisted value is the string and we just use list during the runtime. – Bernhard Kircher Jun 21 '17 at 14:33
10

I'm using EF Core and had a similar problem but solved it in a simpler way.

The idea is to store the list of integers as a comma separated string in the database. I do that by specifying a ValueConverter in my entity type builder.

public class MyObjectBuilder : IEntityTypeConfiguration<MyObject>
{
    public void Configure(EntityTypeBuilder<MyObject> builder)
    {
        var intArrayValueConverter = new ValueConverter<int[], string>(
            i => string.Join(",", i),
            s => string.IsNullOrWhiteSpace(s) ? new int[0] : s.Split(new[] { ',' }).Select(v => int.Parse(v)).ToArray());

        builder.Property(x => x.Numbers).HasConversion(intArrayValueConverter);
    }
}

More information can be found here: https://entityframeworkcore.com/knowledge-base/37370476/how-to-persist-a-list-of-strings-with-entity-framework-core-

Johan Gov
  • 1,262
  • 1
  • 13
  • 26
1

Bernhard's answer is brilliant. I just couldn't help but refine it a little. Here's my two cents:

[ComplexType]
public abstract class EFPrimitiveCollection<T> : IList<T>
{
    [Key, DatabaseGenerated(DatabaseGeneratedOption.Identity)]
    public virtual int Id { get; set; }

    const string DefaultValueSeperator = "|";
    readonly string[] DefaultValueSeperators = new string[] { DefaultValueSeperator };

    [NotMapped]
    private List<T> _data;

    [NotMapped]
    private string _value;

    [NotMapped]
    private bool _loaded;

    protected virtual string ValueSeparator => DefaultValueSeperator;
    protected virtual string[] ValueSeperators => DefaultValueSeperators;

    [ShadowColumn, MaxLength]
    protected virtual string Value // Change this to public if you prefer not to use the ShadowColumnAttribute
    {
        get => _value;
        set
        {
            _data.Clear();
            _value = value;

            if (string.IsNullOrWhiteSpace(_value))
                return;

            _data = _value.Split(ValueSeperators, StringSplitOptions.None)
                        .Select(x => ConvertFromString(x)).ToList();

            if (!_loaded) _loaded = true;
        }
    }

    public EFPrimitiveCollection()
    {
        _data = new List<T>();
    }

    void UpdateValue()
    {
        _value = string.Join(ValueSeparator.ToString(),
                _data.Select(x => ConvertToString(x))
                .ToArray());
    }

    public abstract T ConvertFromString(string value);
    public abstract string ConvertToString(T value);

    #region IList Implementation
    public int Count
    {
        get
        {
            EnsureData();
            return _data.Count;
        }
    }

    public T this[int index]
    {
        get
        {
            EnsureData();
            return _data[index];
        }
        set
        {
            EnsureData();
            _data[index] = value;
        }
    }

    public bool IsReadOnly => false;

    void EnsureData()
    {
        if (_loaded)
            return;

        if (string.IsNullOrWhiteSpace(_value))
            return;

        if (_data.Count > 0) return;


        if (!_loaded) _loaded = true;
        _data = _value.Split(ValueSeperators, StringSplitOptions.None)
                        .Select(x => ConvertFromString(x)).ToList();
    }

    public void Add(T item)
    {
        EnsureData();

        _data.Add(item);
        UpdateValue();
    }

    public bool Remove(T item)
    {
        EnsureData();

        bool res = _data.Remove(item);
        UpdateValue();

        return res;
    }

    public void Clear()
    {
        _data.Clear();
        UpdateValue();
    }

    public bool Contains(T item)
    {
        EnsureData();
        return _data.Contains(item);
    }

    public void CopyTo(T[] array, int arrayIndex)
    {
        EnsureData();
        _data.CopyTo(array, arrayIndex);
    }

    public int IndexOf(T item)
    {
        EnsureData();
        return _data.IndexOf(item);
    }

    public void Insert(int index, T item)
    {
        EnsureData();
        _data.Insert(index, item);
        UpdateValue();
    }

    public void RemoveAt(int index)
    {
        EnsureData();
        _data.RemoveAt(index);
        UpdateValue();
    }

    public void AddRange(IEnumerable<T> collection)
    {
        EnsureData();
        _data.AddRange(collection);
        UpdateValue();
    }

    public IEnumerator<T> GetEnumerator()
    {
        EnsureData();
        return _data.GetEnumerator();
    }
    IEnumerator IEnumerable.GetEnumerator()
    {
        EnsureData();
        return _data.GetEnumerator();
    }
    #endregion
}

With that base class you can have as many derivations as you like:

[ComplexType]
public class EFIntCollection : EFPrimitiveCollection<int>
{
    public override int ConvertFromString(string value) => int.Parse(value);
    public override string ConvertToString(int value) => value.ToString();
}

[ComplexType]
public class EFInt64Collection : EFPrimitiveCollection<long>
{
    public override long ConvertFromString(string value) => long.Parse(value);
    public override string ConvertToString(long value) => value.ToString();
}

[ComplexType]
public class EFStringCollection : EFPrimitiveCollection<string>
{
    string _separator;
    protected override string ValueSeparator => _separator ?? base.ValueSeparator;

    public override string ConvertFromString(string value) => value;
    public override string ConvertToString(string value) => value;

    public EFStringCollection()
    {
    }
    public EFStringCollection(string separator)
    {
        _separator = separator;
    }
}

EFPrimitiveCollection works just like a list, so you shouldn't have any issues using it like a normal List. Also the data is loaded on demand. Here's an example:

if (store.AcceptedZipCodes == null)
    store.AcceptedZipCodes = new EFStringCollection();

store.AcceptedZipCodes.Clear();
store.AcceptedZipCodes.AddRange(codes.Select(x => x.Code));

Shadow Column

This attribute is being used to abstract away the Value property. If you do not see the need to do this, simply remove it and make the Value property public.

More information can be found on the ShadowColumnAttribute in my answer here

Prince Owen
  • 1,225
  • 12
  • 20
1

The simple solution would be in your DataContext (a class that implements DbContext) add this override:

protected override void OnModelCreating(ModelBuilder builder)
{
    builder.Entity<MyObject>()
        .Property(p => p.Numbers)
        .HasConversion(
            toDb => string.Join(",", toDb), 
            fromDb => fromDb.Split(',').Select(Int32.Parse).ToList() ?? new List<int>());
}
Yoro
  • 805
  • 8
  • 17