Split List based on Item length

Question

I have a list of Message along with their item length as size, this.Size = JsonConvert.SerializeObject(Data).Length;.

public class Data
{
    public int Id { get; set; }
    public string Name { get; set; }
}

public class Message
{
    public Data Data { get; set; }
    public int Size { get; set; }

    public Message(Data Data)
    {
        this.Data = Data;
        this.Size = JsonConvert.SerializeObject(Data).Length;
    }
}

Now I have below data set and I would like to split the list into multiple list and each list should be ONLY 50 bytes (for example).

Note - 50 as an example, my actual size is 1 MB and no individual message is greater than 1MB

I tried below, but how to calculate total size and go group by chunksize,

.GroupBy(x => x.ItemSize / chunkSize)

OR a different way?

var messages = new List<Message>
{
    new Message(new Data{ Id=100, Name="N100"}),
    new Message(new Data{ Id=1100, Name="N1100"}),
    new Message(new Data{ Id=11100, Name="N11100"}),
    new Message(new Data{ Id=111100, Name="N111100"}),
    new Message(new Data{ Id=1111100, Name="N1111100"})
};

int chunkSize = 50;

var X = messages
    .Select(x => new { ItemSize = x.Size, Value = x })
    .GroupBy(x => x.ItemSize / chunkSize)
    .Select(x => x.Select(v => v.Value).ToList())
    .ToList();

What if i cant do it evenly, what if one single message is larger than 50 (size), i think you need to add more information — TheGeneral, Feb 21 '19 at 08:42
50 as an example, my actual size is 1 MB and no individual message is greater than 1MB — user584018, Feb 21 '19 at 08:44
Plz check this link may it help you. https://stackoverflow.com/questions/11463734/split-a-list-into-smaller-lists-of-n-siz — Eman Hamed, Feb 21 '19 at 08:44
Thanks @ Eman Hamed, but I am looking base on size (length), NOT based on no of items in list — user584018, Feb 21 '19 at 08:46
@ Marco Salerno, No it will not, please check Note in question — user584018, Feb 21 '19 at 08:49

score 6 · Accepted Answer · edited Jun 20 '20 at 09:12

This might work.. The premise is it Lazily iterates through an IEnumerable<Message> and yields an IEnumerable<List<Message>> (sub-list of Message) where the accumulation is greater than the chunk size

public static IEnumerable<List<Message>> Split(this IEnumerable<Message> source, int chunk)
{

   var list = new List<Message>();
   var accum = 0;

   foreach (var message in source)
   {
      accum += message.Size;

      if (accum > chunk)
      {
         yield return list;
         list = new List<Message>();
         accum = message.Size;
      }
      list.Add(message);
   }

   // Return last result if any 
   if (list.Any()) yield return list;
}

Usage

var sublists = messages.Split(50);

Update

This is a useful method, so i made it generic and more suitable to a library. It includes sanity checking, and throws if the size is larger than chunk

public static IEnumerable<List<T>> Buffer<T>(this IEnumerable<T> source, int chunk, Func<T, long> selector)
{
   // safety first
   if (source == null) throw new ArgumentNullException(nameof(source));

   var list = new List<T>();
   long accum = 0;

   foreach (var item in source)
   {
      var size = selector(item);

      // sanity check
      if (size > chunk) throw new InvalidOperationException("Selector size cant be greater than chunk size");

      // Return chunk
      if ((accum += size) > chunk)
      {
         yield return list;
         list = new List<T>();
         accum = size;
      }

      list.Add(item); // always need to add the current
   }

   // Return any partial result
   if (list.Any()) yield return list;
}

Usage

var results = messages.Buffer(50, x => x.Size)

Arrgghh! `list.Clear();` this means that the previously returned items in the `IEnumerable>` are also cleared. — Enigmativity, Feb 21 '19 at 09:01

Split List based on Item length

1 Answers1

Update