3

I am submitting a training job with

gcloud ai-platform jobs submit training [...] --stream-logs

The job is submitted successfully, but no logs appear on the terminal. No output after "Job [...] submitted successfully." appears, but the command doesn't terminate either.

When I go to check on the online console, I see that the job is running and producing logs (which are not showing up in the terminal where I ran the command). Even when the job completes successfully, the command still doesn't terminate.

The same happens if I first run gcloud ai-platform jobs submit training without --stream-logs and then run gcloud ai-platform jobs stream-logs on the new job.

Do you know any reason this could be happening, and/or any way I can fix it?

EDIT: I left the command running for a while, and about 20 min after the job had already finished and succeeded, all of a sudden all the logs appear and the command terminates. So it is working, kind of, but it takes ages before it gets synced, it seems.

Anakhand
  • 2,838
  • 1
  • 22
  • 50
  • 1
    I did try training a job using the example [here](https://cloud.google.com/ai-platform/docs/getting-started-keras#train_your_model_using) and experienced the same issue as yours. I created an [public issue tracker](https://issuetracker.google.com/182438732) to address this issue. – Ricco D Mar 11 '21 at 06:54
  • Did you find a fix for the thing you mentioned in your edit? I can't find a way to get logs synced and my logs also appear only at the end of the job. – Megastore May 28 '21 at 08:56
  • 1
    @Megastore No, in the end I just had to put up with it. It seems to be an issue with the implementation, see the comment above for an issue page created by Ricco D. – Anakhand May 29 '21 at 18:08
  • I have the same: ``` gcloud ai-platform jobs submit training test_$(uuidgen | sed s/-//g) \ --region us-east1 \ --package-path mystrategy \ --module-name mystrategy \ --staging-bucket ... \ --runtime-version 2.4 \ --python-version 3.7 \ --service-account ... \ --scale-tier BASIC_GPU \ --stream-logs Job [test_761bc9b0fe084e6297a29f5526f0953e] submitted successfully. ``` (and no output, or output which is lagging minutes behind) – iirekm Jun 19 '21 at 08:34

0 Answers0