13

I have a jupyter notebook script that just launches a training script, presumably in a docker container.

I added some print statements in that training script but it's not showing up in the notebook or CloudWatch.

I'm using regular print() statement. How should I log debugging from the training script?

cody
  • 11,045
  • 3
  • 21
  • 36
kane
  • 5,465
  • 6
  • 44
  • 72
  • Are you using a SageMaker-provided container, or your own? If so, would you mind sharing which container you're using, and a bit more about your Jupyter notebook? – Andre Dec 21 '18 at 19:15
  • Using standard SageMaker-provided container – kane Dec 21 '18 at 20:14
  • SageMaker provides a lot of different containers, including one for running TensorFlow scripts, a similar one for MXNet, PyTorch, and Chainer, an example container with scikit-learn, etc. -- would you be free to share which one of the SageMaker-provided containers you're using? – Andre Dec 21 '18 at 20:59
  • how do you find that out? – kane Dec 22 '18 at 01:27
  • One way is to check the console, which has a 'Training Jobs' page where you can see the 'Training image' used. – Andre Dec 22 '18 at 02:36
  • 3
    Anyone got the answer to this ? – user3085459 Apr 13 '20 at 11:55

4 Answers4

4

I've seen this when Python tries to buffer stdout, which doesn't always play nice with Docker -- adding ENV PYTHONUNBUFFERED=1 to your Dockerfile (and then rebuilding the image) would solve this problem, if this is the cause.

Andre
  • 530
  • 3
  • 15
1

According to https://docs.aws.amazon.com/sagemaker/latest/dg/your-algorithms.html, "Docker containers might send messages to the Stdout and Stderr files. Amazon SageMaker sends these messages to Amazon CloudWatch logs in your AWS account.". So print() should work in your case. Have you checked if the docker image is updated after you change the training scripts?

Alohahaha
  • 96
  • 2
  • 2
    I was using print() but I'm not the only one who isn't seeing the print() messages. – kane Dec 20 '18 at 17:56
0

You can find logs on AWS CloudWatchLogs. Go to training jobs on Sagemaker dashboard, select the training job you are looking for. In that you will find an option 'view_logs'. It will direct you to Cloudwatch logs.

Hope this helps!

Harathi
  • 999
  • 2
  • 7
  • 8
  • 5
    But it doesn't show print which I have added in my code – Vineet Dec 13 '18 at 14:06
  • 4
    Same issue here. I think print() goes to stdout which isn't logged by CloudWatch. I think we need some sort of logging method. Still exploring... – kane Dec 14 '18 at 05:40
0

For plain python prints you can manually flush std out

print("My Message", flush=True)
Nick Purcell
  • 49
  • 1
  • 4
  • That doesn't seem to show me what I'm after. I would like to see logs when I hit the endpoint with the CLI – Randy L Apr 20 '21 at 01:22