NAO robot: porting ALSoundExtractor to qi framwork

Question

When porting from NAOqi to qi framework I achieved a partial success. I do however still have the following problem. I do not know how to implement sound processing with ALSoundExtractor in qi framework.

In old Naoqi, there is an example:

http://doc.aldebaran.com/2-8/dev/cpp/examples/audio/soundprocessing/soundprocessing.html

where a class is created:

class ALSoundProcessing : public ALSoundExtractor

then a function overriding a virtual function is declared, one that is used for sound processing:

void process(...)

What I don't now is:

1. How to create a class in qi framework that inherits from the old style class ALSoundExtractor?
2. How to declare a function that is overriding the virtual function - technically the base class function process() expects variables in old AL:: convention.

Alternatively, is there any other way to read the audio channels?

Answer 1

I never worked with ALExtractor nor ALSoundExtractor, but here is what I know.

1. How to create a class in qi framework that inherits from the old style class ALSoundExtractor?

in the old Naoqi, an "ALExtractor"

• could run either from within the main process (using autoload.ini) or from another one (known as remote mode). With the qi framework, only the remote mode is supported. 
• could inherit from ALExtractor or ALAudioExtractor to get some code factored out. Those classes have not been ported to the qi framework. So if you don't want to keep using libnaoqi, you should find a way to do without them.

Good news: inheriting from them never was really needed. You'll find yourself in a similar position as in the following question where an extractor is implemented in python (and thus cannot inherit from a C++ class, nor be loaded in the main process from autoload.ini). NAO robot remote audio problems

2. How to declare a function that is overriding the virtual function - technically the base class function process() expects variables in old AL:: convention.

Whenever you use the "old Naoqi" you're actually using a compatibility layer on top of the qi framework. So whenever you use the "old Naoqi", you're already using the qi framework. libqi's qi::AnyValue is extensible at runtime, libnaoqi extends it to let it know how to handle an ALValue: how to convert it into primitive types (floating point number, list of ints, string, buffer, etc.).

So whenever an old ALSoundExtractor receives an AL::ALvalue, it is actually a qi::AnyValue which has been converted into an ALValue just before calling the process() method. If you don't link with libnaoqi, you won't be able to use the value as an ALValue, but you can use it as a qi::AnyValue or even use it as a primitive type.

The original prototype is (cfr doxygen http://doc.aldebaran.com/2-8/ref/libalaudio/classAL_1_1ALSoundExtractor.html) is

void ALSoundExtractor::process (const int &nbOfChannels, const int &nbrOfSamplesByChannel, const AL_SOUND_FORMAT *buffer, const ALValue &timestamp);

Since timestamp is probably a list of two ints, I would try something like this

void TmpSoundExtractor::process (const int &nbOfChannels, const int &nbrOfSamplesByChannel, qi::AnyValue buffer, const std::vector<int> &timestamp);

I'm not sure how to handle the buffer variable, but let first get the rest working.

Answer 2

To use this API, you must write a Qi Service that advertises this method:

void processRemote(
    int nbOfChannels,
    int nbrOfSamplesByChannel,
    const qi::AnyValue& timestamp,
    const qi::AnyValue& buffer)
{
  std::pair<char*, size_t> charBuffer = value.unwrap().asRaw();
  const signed short* data = (const signed short*)charBuffer.first;
  // process the data like in the example.
}

Note that with the Qi framework:

• AL::ALValue is replaced by qi::AnyValue. Getting the binary data (aka "raw") is slightly different.
• AL_SOUND_FORMAT is replaced by signed short*.
• ALSoundExtractor is not available, so we needed to do the conversion to const AL_SOUND_FORMAT* by ourselves.

Say your service is registered as "MySoundExtractor", you will have to tell ALAudioDevice to start the sound extraction and send the data to your service as follows:

auto audio = session->service("ALAudioDevice").value();
int nNbrChannelFlag = 0; // ALL_Channels: 0,  AL::LEFTCHANNEL: 1, AL::RIGHTCHANNEL: 2; AL::FRONTCHANNEL: 3  or AL::REARCHANNEL: 4.
int nDeinterleave = 0;
int nSampleRate = 48000;
audio->setClientPreferences("MySoundExtractor", nSampleRate, nNbrChannelFlag, nDeinterleave);
audio->subscribe("MySoundExtractor");

Note that I did not test this code, so let me know what may be wrong.

Answer 3

The following is what has eventually worked for me and concludes the topic.

// **************** service.h ****************

typedef signed short AL_SOUND_FORMAT;       // copy from alaudio/alsoundextractor.h

class SoundProcessing
  {
public:
    SoundProcessing(qi::SessionPtr session);
    void init(void);                // a replacement for a function automatically called in NAOqi 2.1.4
    virtual ~SoundProcessing(void); 
    void processRemote(const int& nbOfChannels, const int& nbrOfSamplesByChannel, const qi::AnyValue& timestamp, const qi::AnyValue& buffer);

private:
    qi::SessionPtr _session;
    qi::AnyObject audio;
  };

// **************** service.cpp ****************

SoundProcessing::SoundProcessing(qi::SessionPtr session) : _session(session)
  {
   _session->waitForService("ALAudioDevice");
    audio = _session->service("ALAudioDevice");
  } // constructor
  
QI_REGISTER_MT_OBJECT(SoundProcessing, init, processRemote);


SoundProcessing::~SoundProcessing(void)
  { 
    audio.call<qi::AnyValue>("unsubscribe", "SoundProcessing");
  } // destructor
  

void SoundProcessing::init(void)    
  {
    audio.call<qi::AnyValue>("setClientPreferences",
                              "SoundProcessing",
                             _FREQ48K,          // 48000 Hz requested
                              0,    
                              1     
                            );  
                            
    audio.call<qi::AnyValue>("subscribe", "SoundProcessing");
  } // SoundProcessing::init


void SoundProcessing::processRemote(const int& nbOfChannels,const int& nbrOfSamplesByChannel, const qi::AnyValue& timestamp, const qi::AnyValue& qibuffer)
  { 
    std::pair<char*, size_t> charBuffer = qibuffer.unwrap().asRaw();
    AL_SOUND_FORMAT *buffer = (AL_SOUND_FORMAT *)charBuffer.first;
    
    (...)
  } // SoundProcessing::process


// **************** main.cpp ****************

int main(int argc, char* argv[])
  {
    qi::ApplicationSession app(argc, argv);
    app.start();
    qi::SessionPtr session = app.session();
    
    session->registerService("SoundProcessing", qi::AnyObject(boost::make_shared<SoundProcessing>(session)));

    qi::AnyObject sp = session->service("SoundProcessing");
    sp.call<qi::AnyValue>("init");
    
    app.run();
    return 0;
  }

Answer 4

The following is what I did. The code compiles, but I won't have a chance to test it on a live robot for about one week or so.

typedef signed short AL_SOUND_FORMAT; // copy from alaudio/alsoundextractor.h

void process(const int& nbOfChannels, const int& nbrOfSamplesByChannel, const AL_SOUND_FORMAT *buffer, const qi::AnyValue& timeStamp); // I do not use the timeStamp variable in my code, so AnyValue would work?

qi::AnyObject audioDevice = _session->service("ALAudioDevice"); // same variable name as in the original ALSoundExtractor module, just as a convenience

audioDevice.call<qi::AnyValue>("setClientPreferences", audioDevice.call<qi::AnyValue>("getName"), 48000, 0, 1);

audioDevice.call<qi::AnyValue>("subscribe", audioDevice.call<qi::AnyValue>("getName")); // this is the key call

audioDevice.call<qi::AnyValue>("startDetection"); // is it still necessary?

My question is - do I do it right now? If I cannot override the virtual function process(), does subscribing of my module guarantee a callback to my process(...)?