Saving a Twitter Sample Stream to RavenDB results in the exception The maximum number of requests (30) allowed for this session has been reached

Question

I am attempting to save a Twitter sample stream to RavenDB but I am encountering the exception: "The maximum number of requests (30) allowed for this session has been reached."

This question has the same problem however I think by calling session.Store I am already doing what the accepted answer suggests.

I know I am doing something wrong but not sure what.

Here is my code:

using Tweetinvi;

namespace TwitterNoSql
{
    class Program
    {
        static void Main(string[] args)
        {
            Auth.SetUserCredentials() // redacted for SO

            var stream = Stream.CreateSampleStream();

            using (var session = DocumentStoreHolder.Store.OpenSession())
            {
                stream.TweetReceived += (sender, theTweet) =>
                {
                    var tm = new TwitterModels
                    {
                        Id = theTweet.Tweet.Id,
                        TheTextFromTwitter = theTweet.Tweet.FullText
                    };

                    session.Store(tm);
                    session.SaveChanges();

                };
                stream.StartStream();

            }                
        }                        
    }
}

DocumentStoreHolder.cs

using System;
using Raven.Client;
using Raven.Client.Document;

namespace TwitterNoSql
{
    class DocumentStoreHolder
    {
        private static readonly Lazy<IDocumentStore> LazyStore =
            new Lazy<IDocumentStore>(() =>
            {
                var store = new DocumentStore
                {
                    Url = "http://localhost:8080",
                    DefaultDatabase = "TwitterNoSql"
                };
                return store.Initialize();
            });

        public static IDocumentStore Store =>
             LazyStore.Value;
    }
}

TwitterModels.cs

namespace TwitterNoSql
{
    public class TwitterModels
    {
        public long Id { get; set; }
        public string TheTextFromTwitter { get; set; }
    }
}

score 3 · Answer 1 · answered Jun 06 '17 at 04:33

Try calling SaveChanges() at the end of your operation:

using (var session = DocumentStoreHolder.Store.OpenSession())
{
    stream.TweetReceived += (sender, theTweet) =>
    {
        var tm = new TwitterModels
        {
            Id = theTweet.Tweet.Id,
            TheTextFromTwitter = theTweet.Tweet.FullText
        };

        session.Store(tm);

    };

    stream.StartStream();
    session.SaveChanges();  

}

score 2 · Answer 2 · answered Jun 06 '17 at 05:52

2

Assuming that this is a very large set of data, you probably don't want to do it using the session, a bulk insert would work better here.

answered Jun 06 '17 at 05:52

Ayende Rahien

22,925
1
36
41

score 2 · Accepted Answer · answered Jun 06 '17 at 15:46

You're creating a single document session, and using that for all tweets that are received. So, if you receive 5000 tweets over the span of an hour, you'll hit this issue.

Two ways to fix this:

If TweetReceived is sparse (e.g. it fires a few times a minute or less), then just create a new Raven session each time the tweet is received.
If TweetReceived is fired a bunch of times in quick succession (e.g. 5000 tweets received in a few seconds), then use Raven BulkInsert.

Option #1, session for each tweet received:

static void Main(string[] args)
{
    Auth.SetUserCredentials() // redacted for SO

    var stream = Stream.CreateSampleStream();
    stream.TweetReceived += (sender, theTweet) =>
    {
        var tm = new TwitterModels
        {
            Id = theTweet.Tweet.Id,
            TheTextFromTwitter = theTweet.Tweet.FullText
        };

        using (var session = DocumentStoreHolder.Store.OpenSession())
        { 
           session.Store(tm);
           session.SaveChanges();
        }
    };
    stream.StartStream();             
}

Option #2, using Bulk Insert:

using (BulkInsertOperation bulkInsert = store.BulkInsert())
{
   stream.TweetReceived += (sender, theTweet) =>
   {
       var tm = new TwitterModels
       {
           Id = theTweet.Tweet.Id,
           TheTextFromTwitter = theTweet.Tweet.FullText
       };


       bulkInsert.Store(tm);
   }
};

Either will work here. Use #1 if Tweets arrive sparsely over the lifetime of the app. Use #2 if tweets arrive all at once in bulk.

Saving a Twitter Sample Stream to RavenDB results in the exception The maximum number of requests (30) allowed for this session has been reached

3 Answers3