3

Current default sample Android pjsip pjsua2 sample app sends very poor video quality and I wish to improve this to atleast HD quality. I have tried using the below methods, but it keep on showing very low video quality. How can I improve outgoing video quality? This sample app is able to receive up to 355 * 288 video quality from other sip video calls, but it sends a very poor video quality.

Currently I have tried to achieve HD video by updating the below values from the MediaFormatvideo file, just before making outgoing call. And it doesnt help at all to improve outgoing video. Am I updating those attributes at the wrong places?

Currently it sending video quality below 192 * 144 and doesnt accept above 352 * 288 video quality. How can I update it to support atleast 1280*720 video quality?

    MediaFormatVideo mf=new MediaFormatVideo();
    mf.setFpsNum(30);
    mf.setFpsDenum(1);
    mf.setAvgBps(512000);
    mf.setMaxBps(1024000);
    mf.setHeight(720);
    mf.setWidth(1280);

I am updating those setting as below:

   MyCall call = new MyCall(account, -1);
    CallOpParam prm = new CallOpParam(true);
    AccountVideoConfig avc=new AccountVideoConfig();
    MediaFormatVideo mf=new MediaFormatVideo();

    Log.e("javan-video",String.valueOf(avc.getAutoShowIncoming()));
    Log.e("javan-videofps",String.valueOf(mf.getFpsNum()));
    mf.setFpsNum(30);
    mf.setFpsDenum(1);
    mf.setAvgBps(512000);
    mf.setMaxBps(1024000);
    mf.setHeight(720);
    mf.setWidth(1280);
    Log.e("javan-videofps",String.valueOf(mf.getFpsNum()));


    try {
        call.makeCall("sip:"+dialno+"@peoplefone.ch", prm);
        AudioManager am = (AudioManager) getSystemService(Context.AUDIO_SERVICE);

       am.setSpeakerphoneOn(true);

         // startRinging();

    } catch (Exception e) {
        call.delete();
        return;
    }

    currentCall = call;
 showCallActivity();
}

I have found documentation, and based my implementation on that, but I couldn't improve the video quality.

Framerate
Specify number of frames processed per second.

For encoding direction, configured via pjmedia_vid_codec_param.enc_fmt.det.vid.fps, e.g:
/* Sending @30fps */
param.enc_fmt.det.vid.fps.num   = 30;
param.enc_fmt.det.vid.fps.denum = 1;
Note:
that there is a possibility that the value will be adjusted to follow remote capability. For example, if remote signals that maximum framerate supported is 10fps and locally the encoding direction framerate is set to 30fps, then 10fps will be used.
limitation: if preview is enabled before call is established, capture device will opened using default framerate of the device, and subsequent calls that use that device will use this framerate regardless of the configured encoding framerate that is set above. Currently the only solution is to disable preview before establishing media and re-enable it once the video media is established.
For decoding direction, two steps are needed:
pjmedia_vid_codec_param.dec_fmt.det.vid.fps should be set to the highest value expected for incoming video framerate.
signalling to remote, configured via codec specific SDP format parameter (fmtp): pjmedia_vid_codec_param.dec_fmtp.
H263-1998, maximum framerate is specified per size/resolution basis, check ​here for more info.
/* 3000/(1.001*2) fps for CIF */
param.dec_fmtp.param[m].name = pj_str("CIF");
param.dec_fmtp.param[m].val = pj_str("2");
/* 3000/(1.001*1) fps for QCIF */
param.dec_fmtp.param[n].name = pj_str("QCIF");
param.dec_fmtp.param[n].val = pj_str("1");
H264, similar to size/resolution, the framerate is implicitly specified in H264 level (check the standard specification or ​this) and the H264 level is signalled via H264 SDP fmtp profile-level-id, e.g:
/* Can receive up to 1280×720 @30fps */
param.dec_fmtp.param[n].name = pj_str("profile-level-id");
param.dec_fmtp.param[n].val = pj_str("xxxx1f");
Bitrate
Specify bandwidth requirement for video payloads stream delivery.

This is configurable via pjmedia_vid_codec_param.enc_fmt.det.vid.avg_bps and pjmedia_vid_codec_param.enc_fmt.det.vid.max_bps, e.g:

/* Bitrate range preferred: 512-1024kbps */
param.enc_fmt.det.vid.avg_bps = 512000;
param.enc_fmt.det.vid.max_bps = 1024000;
Notes:

This setting is applicable for encoding and decoding direction, currently there is no way to set asymmetric bitrate. By decoding direction, actually it just means that this setting will be queried when generating bandwidth info for local SDP (see next point).
The bitrate setting of all codecs will be enumerated and the highest value will be signalled in bandwidth info in local SDP (see ticket #1244).
There is a possibility that the encoding bitrate will be adjusted to follow remote bitrate setting, i.e: read from SDP bandwidth info (b=TIAS line) in remote SDP. For example, if remote signals that maximum bitrate is 128kbps and locally the bitrate is set to 512kbps, then 128kbps will be used.
If codec specific bitrate setting signalling (via SDP fmtp) is desired, e.g: MaxBR for H263, application should put the SDP fmtp manually, for example:
/* H263 specific maximum bitrate 512kbps */
param.dec_fmtp.param[n].name = pj_str("MaxBR");
param.dec_fmtp.param[n].val = pj_str("5120"); /* = max_bps / 100 */

Documentation link : enter link description here

 From: "0525512904" <sip:0525512904@95.128.80.93>;tag=1609930889511
I: To: <sip:90761330096@95.128.80.5>;tag=c6ce5331-3a35-44c8-bb80-23b6ec664085
I: CSeq: 1 INVITE
I: Allow: PRACK, INVITE, ACK, BYE, CANCEL, UPDATE, INFO, SUBSCRIBE, NOTIFY, REFER, MESSAGE, OPTIONS
I: Contact: <sip:90761330096@146.4.49.20:45483;transport=TLS;ob>
I: Supported: replaces, 100rel, timer, norefersub
I: Content-Type: application/sdp
I: Content-Length:   580
I: v=0
I: o=- 3818919690 3818919691 IN IP4 192.168.3.135
I: s=pjmedia
I: b=AS:352
I: t=0 0
I: a=X-nat:0
I: m=audio 4012 RTP/AVP 96 120
I: c=IN IP4 192.168.3.135
I: b=TIAS:64000
I: a=rtcp:4031 IN IP4 192.168.3.135
I: a=sendrecv
I: a=rtpmap:96 speex/16000
I: a=rtpmap:120 telephone-event/16000
I: a=fmtp:120 0-16
I: a=ssrc:1510027056 cname:365aaa4f448493db
I: m=video 4013 RTP/AVP 97
I: c=IN IP4 192.168.3.135
I: b=TIAS:256000
I: a=rtcp:4033 IN IP4 192.168.3.135
I: a=sendrecv
I: a=rtpmap:97 H264/90000
I: a=fmtp:97 profile-level-id=42e01e; packetization-mode=1
I: a=ssrc:1146236185 cname:365aaa4f448493db
I: a=rtcp-fb:* nack pli
I: --end msg--
E: ringing call

Complete log link sip call complete log

ouflak
  • 2,458
  • 10
  • 44
  • 49
James
  • 318
  • 1
  • 10
  • welcome to stack overflow. please include enough information here for others to help you by providing enough information _here_ to recreate the problem – a_local_nobody Dec 22 '20 at 14:54
  • i have made a update to my question. where and how i can change video resolution in pjsip ... – James Dec 22 '20 at 15:03
  • @a_local_nobody can you please have a look at my updated question – James Dec 27 '20 at 22:37
  • unfortunately i don't have any experience with this to be able to help you, but your question seems a lot better now, hope you find an answer ! – a_local_nobody Dec 28 '20 at 04:49
  • 1
    There is not enough information to determine the problem (if any). Remeber you are specifying the maximum video limits you can recieve (not send). This is then limited to what the other side can recieve. If the other side can't display more than 640x480 then your side will limit itself to only sending up to that. This is determined by the H264 profile level id sent in the SDP offer/answer. I would suggest you turn the pjsip log level to 4 (from memory) and review that to see what it says. Maybe you need to look into how SDP works with video so that you can understand what it's saying. – Shane Powell Dec 28 '20 at 21:30
  • 1
    As @ShanePowell said, look at h264 profile levels at both sides of SDP negotiation. Look here for more information: [link](https://stackoverflow.com/questions/22960928/identify-h264-profile-and-level-from-profile-level-id-in-sdp) – DaszuOne Dec 29 '20 at 15:33
  • How can i update profile_level_id – James Jan 06 '21 at 15:25
  • @ShanePowell Can you please have a look at my sdp logs updated above. I dont see any where mentioned video resolution. when ever i get video from other softphone. i am not able to figure out where should i change profile_level_id ? . Issue is video send from my android app are bad quality on receving on other softphone . Couldnt find where i should update video quality ie going out from my app – James Jan 06 '21 at 16:08
  • @ShanePowell , thank you .. i have updated my complete log in the post as well .. log link .https://drive.google.com/file/d/1SHJ4pdDvFH7TE4-pJcy42WKAi2Nghfyo/view – James Jan 07 '21 at 10:40

2 Answers2

0

I still can't answer you're question with the given information yet.

SDP is used a the payload type of the SIP protocol.

You can see that in your (part) SIP log here:

Content-Type: application/sdp

SDP is a offer / answer protocol.

Given the incomplete log snip you have given the SIP INVITE (I assume as you haven't given the WHOLE sip message), so you have given the OFFER only of the SDP procotol. So get a complete picture you need to supply both the OFFER and ANSWER.

Also it would be good to also include the other PJSIP logging as well around the video encoder/decoder setup as well.

In your offer it says:

m=video 4013 RTP/AVP 97

there means is can send / recieve video with the parameters:

a=rtpmap:97 H264/90000
a=fmtp:97 profile-level-id=42e01e; packetization-mode=1

This means is can send/recieve H264 with a sample rate of 90000 (i.e. 90kHz).

The H264 parameters setup is: a=fmtp:97 profile-level-id=42e01e; packetization-mode=1

So...

profile-level-id=42e01e

https://www.rfc-editor.org/rfc/rfc6184

  profile-level-id:
     A base16 [7] (hexadecimal) representation of the following
     three bytes in the sequence parameter set NAL unit is specified
     in [1]: 1) profile_idc, 2) a byte herein referred to as
     profile-iop, composed of the values of constraint_set0_flag,
     constraint_set1_flag, constraint_set2_flag,
     constraint_set3_flag, constraint_set4_flag,
     constraint_set5_flag, and reserved_zero_2bits in bit-
     significance order, starting from the most-significant bit, and
     3) level_idc.  Note that reserved_zero_2bits is required to be
     equal to 0 in [1], but other values for it may be specified in
     the future by ITU-T or ISO/IEC.

profile_idc: 0x42 (66) profile-iop: 0xE0 (binary 11100000) level_idc: 0x1E (30)

https://en.wikipedia.org/wiki/Advanced_Video_Coding

profile_idc: 66

Baseline Profile (BP, 66) Primarily for low-cost applications that require additional data loss robustness, this profile is used in some videoconferencing and mobile applications. This profile includes all features that are supported in the Constrained Baseline Profile, plus three additional features that can be used for loss robustness (or for other purposes such as low-delay multi-point video stream compositing). The importance of this profile has faded somewhat since the definition of the Constrained Baseline Profile in 2009. All Constrained Baseline Profile bitstreams are also considered to be Baseline Profile bitstreams, as these two profiles share the same profile identifier code value.

profile-iop: binary 11100000

This means:

constraint_set0_flag=1 (Constrained Baseline profile)
constraint_set1_flag=1
constraint_set2_flag=1

These two values IDC & constraint flags are used to setup the video encoders based on what the decoders can support.

Levels: 30 i.e. 3.0

Level: 3.0 Maximum decoding speed (macroblocks/s): 40,500 Maximum
frame size (macroblocks): 1,620 Maximum video bit rate for video
coding layer (VCL): 10,000 Examples for high resolution @ highest
frame rate (maximum stored frames): 
  352×480@61.4 (12) 
  352×576@51.1 (10) 
  720×480@30.0 (6) 
  720×576@25.0 (5)

The profile level does not specify a video resolution, is manually specifies a maximum frame size / bitrate. ANY combination of resolution / framerate that can "fit" within the these constraits are valid. This is where this is a list of resoulation/framerates are listed as valid.

So 720×480 @ 30fps OR 720×576 @ 25fps are valid to send for level 3.0 profile.

What is offer is telling the other side is that:

  1. This side can only DECODE Constrained Baseline Profile H264 encoded stream.
  2. This side can only DECODE up to Level 3.0 bitrates (i.e. the list of resolutions/fps combos above)

The offer does not tell WHAT the device will send to the other side are this will depend on your local setup combined with what the other side says it can DECODE.

PJSIP will "pick" the best resolution / fps it can send based on your setup and the offer decode supported (which is why you can to see the PJSIP logs about the encoder setup) to know what it's sending based on the SDP ANSWER (not supplied).

The video doesn't have to be symmetrical. i.e. depending on the camera/screen H/W you may be to display different resolutions that what you can send.

This is also not taking into account things like resolutions changing dynamically during the streaming (e.g. portrait / landscape flipping or resolution increasing / decreasing based on network bandwidth changes from RTCP reports). The only way to look into this may be to capture and decode the H264 stream to undersatnd what it's doing. The PJSIP log may also tell you.

UPDATE

Looking at the pjsip logging output you can see both the SDP offer in the INVITE and the answer in the 200 OK.

I: 11:13:36.176           pjsua_core.c  .RX 1119 bytes Response msg 200/INVITE/cseq=22580 (rdata0x6f73203b18) from TLS 95.128.80.3:5061:
I: SIP/2.0 200 OK
I: To: <sip:0074444@james.com>;tag=61c5c92f
I: Via: SIP/2.0/TLS 146.4.49.20:49305;received=146.4.49.20;rport=49305;branch=z9hG4bKPjdad60ffa-6072-4c6d-8eb1-4a32ab26443a;alias
I: Record-Route: <sip:95.128.80.5;r2=on;lr=on;did=e8.cc62>,<sip:95.128.80.3:5061;transport=tls;r2=on;lr=on;did=e8.cc62>
I: CSeq: 22580 INVITE
I: Call-ID: 0e7676b2-1ca2-48b2-9696-f7e6dc7e1ec9
I: From: <sip:0677777@james.com>;tag=0b4094bb-b47e-4132-960c-ac564015efa0
I: Content-Type: application/sdp
I: Contact: <sip:peoplefone@95.128.80.93:5060;alias=95.128.80.93~5060~1>
I: Content-Length: 535
I: v=0
I: o=- 3819003211 3819003212 IN IP4 95.128.80.5
I: s=pjmedia
I: b=AS:352
I: t=0 0
I: a=X-nat:0
I: m=audio 20918 RTP/AVP 96 120
I: c=IN IP4 95.128.80.5
I: b=TIAS:64000
I: a=rtpmap:96 speex/16000
I: a=rtpmap:120 telephone-event/16000
I: a=fmtp:120 0-16
I: a=ssrc:1254727526 cname:496ca0741b8de59f
I: a=sendrecv
I: a=rtcp:20919
I: m=video 20956 RTP/AVP 97
I: c=IN IP4 95.128.80.5
I: b=TIAS:256000
I: a=rtpmap:97 H264/90000
I: a=fmtp:97 profile-level-id=42e01e; packetization-mode=1
I: a=ssrc:977888024 cname:496ca0741b8de59f
I: a=rtcp-fb:* nack pli
I: a=sendrecv
I: a=rtcp:20957
I: --end msg--

From the answer you can see that it answered with the same H264 parameters as the offer:

I: m=video 20956 RTP/AVP 97
...
I: a=rtpmap:97 H264/90000
I: a=fmtp:97 profile-level-id=42e01e; packetization-mode=1

So it will accept up to H264 level 3.0 bitrates.

If you look at the initialisation of the capture device (camera) we see these logs:

I: 11:13:36.270             vid_port.c  .........Opening device OpenGL renderer [OpenGL] for render: format=I420, size=352x288 @15:1 fps

This means the android front camera has been opened at a resolution of 352x288 @ 15fps.

I assume this is the cause of the video quailty you are talking about.

Looking at the source code for pjsip it enumerates the cameras with there supported parameters.

The supported capture resolution size is then determined by the default capture resolution size downsized based on the "allowed" resolution sizes that can be sent.

Since the allowed sizes is greater than 352x288@15 then I can only assume that your Andriod default capture resolution of your front camera is 352x288@15.

You can try using the back camera instead of the front camera to see if you get a getter solution or another Andriod device that has a better front camera.

PjSip uses the android.hardware.Camera API to access and use the camera devices. See PjCameraInfo and PjCamera for details of how pjsip uses the camera devices.

Community
  • 1
  • 1
Shane Powell
  • 13,698
  • 2
  • 49
  • 61
0

You probably need to change the h264 encoding profile-level-id to level 3.1. It make the SDP negotiates up to 1280x720@30.

To do so, change the h264 parameter profile-level-id from 42e01e to 42e01f (where 1e == 30 and 1f == 31)

I don't know what are the functions, but in C++ pjsua are setVideoCodecParam and getVideoCodecParam.

More on that can be found in this wiki: https://trac.pjsip.org/repos/wiki/Video_Users_Guide#Sizeorresolution

and the RFC : https://datatracker.ietf.org/doc/rfc6184/