How to retrieve selected number of rows using Linq

Question

I have table in database containing more than 3 thousand rows.

I want to take single column details in string array and I need to send this array to third party api but array length should not be more than 1000.

So I need to send 3/4 different array depending on records

string[] deviceIds = FirebaseNotificationList.Select(x => x.DeviceID).ToArray();

I need to divide string[] deviceIds into different string array of length 1000

If I use below code it will take only first 1000 rows:

string[] deviceIds = FirebaseNotificationList
                             .Select(x => x.DeviceID).Take(1000).ToArray();

How should I do that without missing any row?

Make sure you put some code and explain it a bit better. – Joshua Bakker Apr 05 '17 at 11:08 — Joshua Bakker, Apr 05 '17 at 11:08
Please add additional info about Your Question – Abi Apr 05 '17 at 11:08 — Abi, Apr 05 '17 at 11:08

Sergey Berezovskiy · Answer 1 · 2017-04-05T11:32:59.657

0

You can use Select method overload which provides element's index to result selector. Then simple grouping by index/batchSize will split all your items into batches of required size:

int batchSize = 1000;
var batches = FirebaseNotificationList
                .Select((fn,i) => new { fn.DeviceID, i })
                .GroupBy(x => x.i / batchSize)
                .Select(g => g.Select(x => x.DeviceID).ToArray());

You can also use MoreLINQ Batch to avoid creating anonymous objects which hold information about their index in source list

var batches = FirebaseNotificationList
                .Select(fn => fn.DeviceID)
                .Batch(batchSize); // note that batches will be IEnumerable<T>

edited Apr 05 '17 at 11:32

answered Apr 05 '17 at 11:09

Sergey Berezovskiy

232,247
41
429
459

what is 'i' in first query – Luqman Apr 05 '17 at 11:16
@Luqman it's an index of item in source list. I have added link to documentation – Sergey Berezovskiy Apr 05 '17 at 11:27

score 0 · Answer 2 · answered Apr 05 '17 at 11:12

0

The batch processing can be implemented via Linq as follows.

int BatchSize = 1000;
IEnumerable<string> Tail = DeviceIds;
while (Tail.Count() > 0)
{
    var Head = Tail.Take(BatchSize);
    // process Head
    Tail = Tail.Skip(BatchSize);
}

answered Apr 05 '17 at 11:12

Codor

17,447
9
29
56

score 0 · Answer 3 · edited Jun 20 '20 at 09:12

This has already been answered on: Create batches in linq:

public static class MyExtensions
{
    public static IEnumerable<IEnumerable<T>> Batch<T>(this IEnumerable<T> items,
                                                       int maxItems)
    {
        return items.Select((item, inx) => new { item, inx })
                    .GroupBy(x => x.inx / maxItems)
                    .Select(g => g.Select(x => x.item));
    }
}

and the usage would be:

List<int> list = new List<int>() { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9 };

foreach(var batch in list.Batch(3))
{
    Console.WriteLine(String.Join(",",batch));
}

OUTPUT:

0,1,2

3,4,5

6,7,8

9

CodingYoshi · Answer 4 · 2017-04-05T11:53:20.137

If I use below code it will take only first 1000 rows:

string[] deviceIds = FirebaseNotificationList
    .Select(x => x.DeviceID).Take(1000).ToArray();

Yes you will. You can use the overload of Select to index the results. For example, if you have 3345 records, then divide that by your batch size (1000) and you will end up with batches 0, 1, 2, 3. Here is the code with explanation to follow:

int batchSize = 1000;
string[] deviceIds = FirebaseNotificationList
    .Select((x, itemNumber) => new { x.DeviceID, ItemNumber = 
         itemNumber).GroupBy(x => x.ItemNumber / batchSize)
    .Select(g => g.Select(x => x.DeviceID).ToArray());

Explanation:

ItemNumber is an index given to the records. So records with ItemNumber less than 1000, when divided by 1000, will end up with 0 (integer division) and for records with ItemNumber more than 1000 and less than 1999 will end up with 1 and so on. Therefore, you will end up with groups of records using the GroupBy that range from 0 to x, where x will be the number of the last batch. In other words, each group will be contain records for each batch. In the last Select you are selecting records from each group (batch).

How to retrieve selected number of rows using Linq

4 Answers4