How to reduce time running gclient sync for WebRTC

Question

I am building WebRTC library using travis CI.

This is running well but takes lots of time and more and more often the build ends with the message :

The job exceeded the maximum time limit for jobs, and has been terminated.

You can consult a log that failed travis log

During the gclient sync :

_______ running 'download_from_google_storage --directory --recursive --num_threads=10 --no_auth --quiet --bucket chromium-webrtc-resources src/resources' in '/home/travis/build/mpromonet/webrtc-streamer/webrtc'
...
Hook 'download_from_google_storage --directory --recursive --num_threads=10 --no_auth --quiet --bucket chromium-webrtc-resources src/resources' took 1255.11 secs

I disabled the tests, so I think this is useless and it takes lots of time.

Is there anyway to give some arguments or setting some variables to avoid this time costly task ?

mpromonet · Accepted Answer · 2019-07-23T22:31:53.477

2

A way to not download chromium-webrtc-resources defined in dependencies DEPS

{
    # Download test resources, i.e. video and audio files from Google Storage.
    'pattern': '.',
    'action': ['download_from_google_storage',
               '--directory',
               '--recursive',
               '--num_threads=10',
               '--no_auth',
               '--quiet',
               '--bucket', 'chromium-webrtc-resources',
               'src/resources'],
  },

is to pached it removing this section or adding a condition that is false.

In order to patch I used the folowing command :

sed -i -e "s|'src/resources'],|'src/resources'],'condition':'rtc_include_tests==true',|" src/DEPS

This save about 20mn and allow the travis build to stay below the timeout.

edited Jul 23 '19 at 22:31

answered Oct 18 '17 at 19:10

mpromonet

11,326
43
62
91

This saved me so much time on my build, I'm curious did you try the docker method renefritze mentioned in his answer? If so does it help much? – Jack Apr 07 '19 at 02:44
@jackz314 No I did not try, I guess the idea is to split the build – mpromonet Apr 07 '19 at 17:16

renefritze · Answer 2 · 2017-10-17T07:30:29.743

1

You can bake the entire toolchain into a docker image and run your actual tests/builds in that. Delegate the docker image update into another automated process (travis-ci cronjob for example).

An additional benefit is that you now have full control over when parts of your toolchain change. I find that very important.

Edit: Some resources to read.

The official travis docs for using docker
Building & deploying images on travis
Dockerhub automated builds

edited Oct 17 '17 at 07:30

answered Oct 16 '17 at 07:55

renefritze

2,167
14
25

Thanks, Do you means running docker from the .travis.yml ? It could be a way to split the job, I will think about this but it seems complex to make a travis script that build a part of a docker image, then a second script that use the previous image and continue the build process... – mpromonet Oct 17 '17 at 06:52
Yes, that's what I meant. I've added some link to the answer that should help. Since you toolchain docker is probably very straightfoward you you can also use dockerhub to build it. That is even simpler than building on travis. – renefritze Oct 17 '17 at 07:32

How to reduce time running gclient sync for WebRTC

2 Answers2