I am currently doing a medical app whereby live meetings are implemented using a third-party SDK "videosdk", and that we do classification on the patient's emotions during the live meeting. What I am trying to do here is that I want to extract a video frame of the participant during the live meeting, pass that video frame for pre-processing into a bitmap, and pass the bitmap into a tflite model for emotion classification. However, I am stuck towards how we should extract video frames from the current implementation. The following code shows the participant adapter java file from this website:
https://docs.videosdk.live/android/guide/video-and-audio-calling-api-sdk/quick-start
import android.graphics.Bitmap;
import android.os.Build;
import android.os.CountDownTimer;
import android.util.Log;
import android.view.LayoutInflater;
import android.view.View;
import android.view.ViewGroup;
import android.widget.TextView;
import androidx.annotation.NonNull;
import androidx.recyclerview.widget.RecyclerView;
import org.webrtc.EglRenderer;
import org.webrtc.VideoFrame;
import org.webrtc.VideoSink;
import org.webrtc.VideoTrack;
import java.nio.ByteBuffer;
import java.util.ArrayList;
import java.util.List;
import java.util.Map;
import live.videosdk.rtc.android.Meeting;
import live.videosdk.rtc.android.Participant;
import live.videosdk.rtc.android.Stream;
import live.videosdk.rtc.android.VideoView;
import live.videosdk.rtc.android.listeners.MeetingEventListener;
import live.videosdk.rtc.android.listeners.ParticipantEventListener;
public class ParticipantAdapter extends RecyclerView.Adapter<ParticipantAdapter.PeerViewHolder> {
private int containerHeight;
// creating a empty list which will store all participants
private final List<Participant> participants = new ArrayList<>();
@NonNull
@Override
public PeerViewHolder onCreateViewHolder(@NonNull ViewGroup parent, int viewType) {
containerHeight = parent.getHeight();
return new PeerViewHolder(LayoutInflater.from(parent.getContext()).inflate(R.layout.item_remote_peer, parent, false));
}
@Override
public void onBindViewHolder(@NonNull PeerViewHolder holder, int position) {
Participant participant = participants.get(position);
ViewGroup.LayoutParams layoutParams = holder.itemView.getLayoutParams();
layoutParams.height = containerHeight / 3;
holder.itemView.setLayoutParams(layoutParams);
holder.tvName.setText(participant.getDisplayName());
int count = 0;
// adding the initial video stream for the participant into the 'VideoView'
for (Map.Entry<String, Stream> entry : participant.getStreams().entrySet()) {
Stream stream = entry.getValue();
if (stream.getKind().equalsIgnoreCase("video")) {
holder.participantView.setVisibility(View.VISIBLE);
VideoTrack videoTrack = (VideoTrack) stream.getTrack();
holder.participantView.addTrack(videoTrack);
break;
}
++count;
Log.d("Something count: ", String.valueOf(count));
}
// add Listener to the participant which will update start or stop the video stream of that participant
participant.addEventListener(new ParticipantEventListener() {
@Override
public void onStreamEnabled(Stream stream) {
if (stream.getKind().equalsIgnoreCase("video")) {
holder.participantView.setVisibility(View.VISIBLE);
VideoTrack videoTrack = (VideoTrack) stream.getTrack();
holder.participantView.addTrack(videoTrack);
}
}
@Override
public void onStreamDisabled(Stream stream) {
if (stream.getKind().equalsIgnoreCase("video")) {
holder.participantView.removeTrack();
holder.participantView.setVisibility(View.GONE);
}
}
});
}
// this method returns the size of total number of participants
@Override
public int getItemCount() {
return participants.size();
}
static class PeerViewHolder extends RecyclerView.ViewHolder {
// 'VideoView' to show Video Stream
public VideoView participantView;
public TextView tvName;
public View itemView;
PeerViewHolder(@NonNull View view) {
super(view);
itemView = view;
tvName = view.findViewById(R.id.tvName);
participantView = view.findViewById(R.id.participantView);
}
}
public ParticipantAdapter(Meeting meeting) {
// adding the local participant(You) to the list
participants.add(meeting.getLocalParticipant());
// adding Meeting Event listener to get the participant join/leave event in the meeting.
meeting.addEventListener(new MeetingEventListener() {
@Override
public void onParticipantJoined(Participant participant) {
// add participant to the list
participants.add(participant);
notifyItemInserted(participants.size() - 1);
}
@Override
public void onParticipantLeft(Participant participant) {
int pos = -1;
for (int i = 0; i < participants.size(); i++) {
if (participants.get(i).getId().equals(participant.getId())) {
pos = i;
break;
}
}
// remove participant from the list
participants.remove(participant);
if (pos >= 0) {
notifyItemRemoved(pos);
}
}
});
}
}
I have already tried methods like VideoSink, and have already attempted to try and use VideoFrame to get the buffer, to get the buffer in i420 and doing the conversion process, but all has failed so far. Can anyone help with this?