2

I'm trying to check for a certain chain of events in an LTTNG event log using Babeltrace 1. The LTTNG log is loaded using a Babeltrace collection:

import babeltrace

my_collection = babeltrace.TraceCollection()
my_collection.add_traces_recursive(trace_path, 'ctf')

The special events I'm looking for are almost indistinguishable from the normal events happening, except there is a few extra events once the chain have already started. So I need to look for these special events, and then search backward for the actual start.

The problem is that Babeltrace only lets me go forward in the event list. The simple solution seemed to be to create a clone of the events in my own list:

events = [e for e in my_collection.events]

The problem with this is that all events in the list now reference the last event. Which indicates that Babeltrace reuses the same object over and over and the generator only returns a reference to this single object.

I have tried to use copy.copy:

events = [copy.copy(e) for e in my_collection.events]

This didn't help, and copy.deepcopy doesn't work at all. I've also tried itertools.tee:

events = list(itertools.tee(my_collection.events))

But this returns a list of _tee object which can't be used as proper event objects.

Is there a way to search backward using the Babeltrace event collection generator? Or is there a way to clone the event object properly to create my own list?

Some programmer dude
  • 400,186
  • 35
  • 402
  • 621
  • 1
    You can consume the generator as a list by doing `list(my_generator)`. – zr0gravity7 Dec 15 '21 at 20:43
  • 1
    If all list items refer to the same last event you need some way to clone the event object yielded by the generator or extract the necessary information and put it into another object which can then be stored in the list. – Michael Butscher Dec 15 '21 at 20:48
  • The question is clearly not about generators themselves, but about the specific generator you are using, which seems to reuse the same event object each time it yields the next one. That's very odd to me, but I suppose if the event object is big and expensive, there might be good reasons to implement it that way. In that case, we'll need to know what your generator does to answer this question, and why even a deep copy isn't actually a copy in this case. Is it connected to a database or some other external state that changes irrevocably when you get the next event? – joanis Dec 15 '21 at 21:12
  • Your edit helps a lot already, but any chance you can share an MRE? – joanis Dec 15 '21 at 21:32
  • I am currently experimenting with Babeltrace 2 (which I had to build from source) and it seems it might be possible to solve the problem with it, since it seems the events are a kind of linked list where one could walk backward. – Some programmer dude Dec 16 '21 at 13:48
  • Using Babeltrace 2 it's also possible to actually "copy" messages from a collection iterator into a list. It might be a list of references to the original collection, I don't know that yet, However, the time to copy about 130000 messages into a list is negligible, but might be an issue once we get into the millions of events. I still haven't decided if I should use these messages as a linked list, or as a Python list. – Some programmer dude Dec 16 '21 at 14:26
  • 2
    @joanis "_That's very odd to me, but I suppose if the event object is big and expensive, there might be good reasons to implement it that way._" Indeed it's a common pattern for performance reasons. This is the case of the [`InputIterator`](https://en.cppreference.com/w/cpp/named_req/InputIterator) C++ concept, for example: "_A LegacyInputIterator is a LegacyIterator that can read from the pointed-to element. LegacyInputIterators only guarantee validity for single pass algorithms_. BT2 solves this limitation with various object pools (you can see BT1 as having a pool of one event record). – eepp Dec 16 '21 at 14:27
  • @eepp That's a very sound principle, I agree, to optimize something that might be used as a continuous stream. Although, I would still expect the API to provide an easy way for the client to copy the records out of the iterator and save them into their data structures if they want to. Thanks for addressing how to do that (or work around that) in your answer. – joanis Dec 16 '21 at 15:41

1 Answers1

3

Babeltrace co-maintainer here.

Indeed, Babeltrace 1 reuses the same event record object for each iteration step. This means you cannot keep an "old" event record alive as its data changes behind the scenes.

The Python bindings of Babeltrace 1 are rudimental wrappers of the library objects. This means the same constraints apply. Also, Babeltrace 1 doesn't offer any event record object copying function, so anything like copy.copy() will only copy internal pointers which will then exhibit the same issue.

Babeltrace (1 and 2) iterators cannot go backwards for performance reasons (more about this below).

The only solution I see is making your own event record copying function, keeping what's necessary in another instance of your own class. After all, you probably only need the name, timestamp, and some first-level fields of the event record.

But Babeltrace 2 is what you're looking for, especially since we don't maintain Babeltrace 1 anymore (except for critical/security bug fixes).

Babeltrace 2 offers a rich and consistent C API where many objects have a reference count and therefore can live as long as you like. The Babeltrace 2 Python bindings wrap this C API so that you can benefit from the same features.

While the C API documentation is complete, unfortunately the Python bindings one is not. However, we have this, which at least shows some examples to get you started.

About your comment:

since it seems the events are a kind of linked list where one could walk backward

No, you cannot. This is to accomodate limitations of some trace formats, in particular CTF (the format which LTTng uses). A CTF packet is a sequence of serialized binary event records: to decode event record N, you need to decode event record N - 1 first, and so on. A CTF packet can contain thousands of contiguous event records like this, CTF data streams can contain thousands of packets, and a CTF trace can contain many data streams. Knowing this, there would be no reasonable way to store the offsets of all the encoded CTF event records so that you can iterate backwards without heavy object copies.

What you can do however with Babeltrace 2 is keep the specific event record objects you need, without any copy.

In the future, we'd like a way to copy a message iterator, duplicating all its state and what's needed to continue behind the scenes. This would make it possible to keep "checkpoint iterators" so that you can go back to previous event records if you can't perform your analysis in one pass for some reason.

Note that you can also make a message iterator seek a specific timestamp, but "fast" seeking is not implemented as of this date in the ctf plugin (the iterator seeks the beginning of the message sequence and then advances until it reaches the requested timestamp, which is not efficient).

eepp
  • 7,255
  • 1
  • 38
  • 56