3

I use the RedisConnection Set method to set the byte array but how do i get the data? The get returns a wrapped byte array?

Links:

This works but it doesn't feel right:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using BookSleeve;
using ProtoBuf;
using System.IO;
namespace RedisTest001
{
    [ProtoContract, Serializable]
    public class Price
    {
        private string _ticker;
        private double _value;

        public Price()
        {
        }

        public Price(string ticker, double value)
        {
            _ticker = ticker;
            _value = value;
        }


        [ProtoMember(1)]
        public string Ticker
        {
            get { return _ticker; }
            set { _ticker = value; }
        }

        [ProtoMember(2)]
        public double Value
        {
            get { return _value; }
            set { _value = value; }
        }


    }

    class Program
    {
        static void Main(string[] args)
        {
            using (var conn = new RedisConnection("localhost"))
            {
                Price p = new Price("IBM", 101.55);

                byte[] raw;
                using (MemoryStream ms = new MemoryStream())
                {
                    Serializer.Serialize<Price>(ms,p);
                    raw = ms.ToArray();
                }

                conn.Open();
                conn.Set(1, p.Ticker, raw);


                var tb  = conn.Get(1,"IBM");

                var str = conn.Wait(tb);

                 Price p2 = Serializer.Deserialize<Price>(new MemoryStream(str));

            }
        }
    }
}

More info:

public static class pex
{
    public static byte[] ToBArray<T>(this T o)
    {
        using (MemoryStream ms = new MemoryStream())
        {
            Serializer.Serialize<T>(ms, o);
            return ms.ToArray();
        }
    }
}

class Program
{
    static void Main(string[] args)
    {
        Random RandomClass = new Random();

        using (var conn = new RedisConnection("localhost"))
        {

            conn.Open();

            for (int i = 0; i < 500000; i++)
            {
                Price p = new Price("IBM", RandomClass.Next(0, 1000));
                conn.AddToSet(2, "PRICE.IBM", p.ToBArray());
            }
Kevin Pullin
  • 13,122
  • 3
  • 24
  • 33
DaveHeller
  • 7,899
  • 3
  • 19
  • 9

1 Answers1

11

That is entirely correct. "Get" (BookSleeve) returns a deferred byte[]. You have correctly used Wait to get the actual byte[], then used a MemoryStream over this byte[] to call Deserialize via protobuf-net.

All good.

If you make it clear any steps that you find ugly, I might be able to be more specific, but:

  • BookSleeve is entirely async via Task, hence the need for either Wait or ContinueWith to access the byte[]
  • protobuf-net is entirely Stream-based, hence the need for MemoryStream to sit on top of a byte[]

Of course, if you add a generic utility method (maybe an extension method) you only need to write it once.

And with the addition if a wrapper class (for some tracking/sliding-expiry) and a L1 cache (Redis as L2), this is pretty much exacty how we use it at stackoverflow.

One note: the connection is thread safe and intended to be massively shared; don't do a connection per operation.

Ron Klein
  • 9,178
  • 9
  • 55
  • 88
Marc Gravell
  • 1,026,079
  • 266
  • 2,566
  • 2,900
  • @Marc Gravell: Are you going to add transaction and hashes support to Booksleeve? Great project - will be really useful. – DaveHeller Jun 25 '11 at 16:13
  • @Dave they are kinda missing - I'll try to add those this week :) only omitted as they werent on my own critical path – Marc Gravell Jun 25 '11 at 16:35
  • @Marc Gravell: Hi, i seem to have a "memory leak" (holding on to refs) with my usage. The memory keeps growing and so does the RedisConnection OutstandingCount. What is the OutstandingCount? Do I need to flush changes or something to release the refs so objs can be collected? thanks – DaveHeller Jun 25 '11 at 16:58
  • @Dave you shouldn't need to do *anything*; did you get a byte[] result etc? I'm not at a PC, but "outstanding" is *either* the messages not yet sent (it queues messages), or the messages that have been sent but which haven't had responses. For us, even under heavy load both are almost always zero. What happens if you "Wait" on the Open()? Maybe it didn't open correctly? – Marc Gravell Jun 25 '11 at 17:06
  • @Dave just checking my server here; sent: (big number), received: (same big number), both queues: 0 – Marc Gravell Jun 25 '11 at 17:09
  • @Dave another thought; if you have access to the redis server, run a "MONITOR" console - that'll tellyou whether redis got the messages. – Marc Gravell Jun 25 '11 at 17:11
  • @Dave also: since most people want "fre and forget" on many of their tasks, there is also an event you can subscribe to to see errors – Marc Gravell Jun 25 '11 at 17:14
  • @Marc Gravell: I've added extra code the post. The code runs a tight loop add objects and after several seconds the OutstandingCount is still high. – DaveHeller Jun 25 '11 at 17:22
  • @Dave I'll have to check (again, not at PC) but I *believe* I left an optional parameter in the constructor to limit the outgoing queue to sane numbers. Maybe bring that in a bit? It'll block at the point of adding items, so it'll slow down the for loop - but overall it'll take about the same amount of time – Marc Gravell Jun 25 '11 at 17:27
  • @Marc Gravell: yes it's decreasing but very slowly. – DaveHeller Jun 25 '11 at 17:27
  • @Dave I'll run some tests with big numbers to check it isn't anything screwy in BookSleeve... It is hard to guess though, as the bottleneck could be network IO, the redis server, etc – Marc Gravell Jun 25 '11 at 17:30
  • 2
    @MarcGravell: You suggest not to do a connection per operation. So what is the recommended pattern? Use a singleton to create and serve one connection object? And: What happens if the network connection drops? Should I catch the network exception and recreate the (singleton) connection object? – Ofer Zelig Dec 25 '11 at 18:29
  • 1
    @Ofer that is pretty much how I tend to use it - but not *quite* a true singleton. Btw; one of the more recent releases will negotiate to ensure that the connection doesn't drop due to inactivity - if t *does* drop, there is probably a genuine network problem. In our code we allow a delay before reconnecting (using a local fallback strategy) if that happens. – Marc Gravell Dec 25 '11 at 18:33
  • 1
    @MarcGravell thank you for your answer. What do you mean by not quite a true singleton? Would it be best to create a connection once per ASP.NET-page thread (one full request+response)? – Ofer Zelig Dec 26 '11 at 12:49
  • 1
    @Ofer no; that would not be optimal - it is a thread-safe multiplexer: hammer it like crazy from lots of threads. To put that in context: for stackoverflow and stackexchange we have a single connection per AppDomain. It is *virtually* singleton. By "not quite a singleton", I just mean we allow a little bit of flexibility in case of swapping connections over (if they become damaged etc). The normal state for us is : one. You might want more if, for example, you do some specific reads from a slave instead of the master, etc. – Marc Gravell Dec 26 '11 at 13:34