scribe with protocol buffer and advanced thrift?

Question

I have two questions here:

Question 1:

-- can thrift provide an inner-class functionality? (see my example next)

-- if it can, can thrift use such functionality easily?

Here is the scribe interface (scribe/if/scribe.thrift). But its message field can only be string, which I believe not flexible enough.

#!/usr/local/bin/thrift --cpp --php

##  Copyright (c) 2007-2008 Facebook
...
...
## See accompanying file LICENSE or visit the Scribe site at:
## http://developers.facebook.com/scribe/

include "fb303/if/fb303.thrift"

namespace cpp scribe.thrift

enum ResultCode
{
  OK,
  TRY_LATER
}

struct LogEntry
{
  1:  string category,
  2:  string message
}

service scribe extends fb303.FacebookService
{
  ResultCode Log(1: list<LogEntry> messages);
}

It would be great if I can do the following thing (I don't even know if thrift itself provides the inner-class functionality according to its document-- but protocol buffer definitely can).

enum ResultCode
{
  OK,
  TRY_LATER
}

struct MyLogStructure {
  1: string field_name;
  2: string value;
}

struct LogEntry
{
  1:  string category,
  2:  MyLogStructure message
}

service scribe extends fb303.FacebookService
{
  ResultCode Log(1: list<LogEntry> messages);
}

Question 2:

-- Can scribe use protocol buffer as the internal data representation easily? (without too much code modification)

-- If the answer to the above question is "no", did Google open-source its sribe implementation?

score 2 · Answer 1 · answered Jan 21 '10 at 01:54

Yes, Thrift structs can include other structs, and your definition (repeated following) would work:

enum ResultCode { OK, TRY_LATER }

struct MyLogStructure {
  1: string field_name;
  2: string value;
}

struct LogEntry {
  1: string category,
  2: MyLogStructure message 
}

service scribe extends fb303.FacebookService {
  ResultCode Log(1: list messages);
}

If you redefined Scribe's interfaces like this, you would probably have to modify Scribe's code to handle your new type depending on what it does internally with string message.

Of course, you could always serialize your MyLogStructure object to a string and avoid this problem altogether.

No, I don't imagine Scribe would be able to use Protocol Buffers as its internal data representation easily. All of the RPC code has been generated from these definitions, and you could redefine the Log method to take an arbitrary object (make it, say, a Protocol Buffers object) but this would be just as hard as the above.

To my knowledge Google has not open-sourced any distributed logging system. Chukwa from Yahoo/Hadoop is one alternative.

scribe with protocol buffer and advanced thrift?

1 Answers1