0

I am new to Snakemake, I just started using it for my project. I am trying to run a software package that uses Snakemake, however the workflow fails and the problem seems to be caused by windows not being able to locate some files [please see below for the complete error report]. Could someone please give me some ideas of possible solutions? Thank you

snakemake -s U:\Lab\Bereket_public\custom_merfish_rig\Merlin_test\test_data\results\data\snakemake\SnakeFile.Snakefile --cores 4 U:\Lab\Bereket_public\custom_merfish_rig\Merlin_test\test_data\results\data\GenerateMosaic\tasks\GenerateMosaic.done
Building DAG of jobs...
Provided cores: 4
Rules claiming more threads will be scaled down.
Job stats:
job                            count    min threads    max threads
---------------------------  -------  -------------  -------------
FiducialCorrelationWarp            2              1              1
FiducialCorrelationWarpDone        1              1              1
GenerateMosaic                     1              1              1
SimpleGlobalAlignment              1              1              1
total                              5              1              1

Select jobs to execute...

[Tue Jan 17 10:51:28 2023]
Job 1: Running SimpleGlobalAlignment
Reason: Missing output files: U:/Lab/Bereket_public/custom_merfish_rig/Merlin_test/test_data/results/data/SimpleGlobalAlignment/tasks/SimpleGlobalAlignment.done


[Tue Jan 17 10:51:28 2023]
Job 3: Running FiducialCorrelationWarp 0
Reason: Missing output files: U:/Lab/Bereket_public/custom_merfish_rig/Merlin_test/test_data/results/data/FiducialCorrelationWarp/tasks/FiducialCorrelationWarp_0.done

Failed to set marker file for job started ([WinError 3] The system cannot find the path specified: 'U:\\Lab\\Bereket_public\\custom_merfish_rig\\MERlin_epigen_UCSD\\.snakemake\\incomplete\\tmp72nkj4zo.VTovTGFi' -> 'U:\\Lab\\Bereket_public\\custom_merfish_rig\\MERlin_epigen_UCSD\\.snakemake\\incomplete\\VTovTGFiL0JlcmVrZXRfcHVibGljL2N1c3RvbV9tZXJmaXNoX3JpZy9NZXJsaW5fdGVzdC90ZXN0X2RhdGEvcmVzdWx0cy9kYXRhL0ZpZHVjaWFsQ29ycmVsYXRpb25XYXJwL3Rhc2tzL0ZpZHVjaWFsQ29ycmVsYXRpb25XYXJwXzAuZG9uZQ=='). Snakemake will work, but cannot ensure that output files are complete in case of a kill signal or power loss. Please ensure write permissions for the directory U:\Lab\Bereket_public\custom_merfish_rig\MERlin_epigen_UCSD\.snakemake

[Tue Jan 17 10:51:28 2023]
Job 4: Running FiducialCorrelationWarp 1
Reason: Missing output files: U:/Lab/Bereket_public/custom_merfish_rig/Merlin_test/test_data/results/data/FiducialCorrelationWarp/tasks/FiducialCorrelationWarp_1.done

Failed to set marker file for job started ([WinError 3] The system cannot find the path specified: 'U:\\Lab\\Bereket_public\\custom_merfish_rig\\MERlin_epigen_UCSD\\.snakemake\\incomplete\\tmpo2nxcdd7.VTovTGFi' -> 'U:\\Lab\\Bereket_public\\custom_merfish_rig\\MERlin_epigen_UCSD\\.snakemake\\incomplete\\VTovTGFiL0JlcmVrZXRfcHVibGljL2N1c3RvbV9tZXJmaXNoX3JpZy9NZXJsaW5fdGVzdC90ZXN0X2RhdGEvcmVzdWx0cy9kYXRhL0ZpZHVjaWFsQ29ycmVsYXRpb25XYXJwL3Rhc2tzL0ZpZHVjaWFsQ29ycmVsYXRpb25XYXJwXzEuZG9uZQ=='). Snakemake will work, but cannot ensure that output files are complete in case of a kill signal or power loss. Please ensure write permissions for the directory U:\Lab\Bereket_public\custom_merfish_rig\MERlin_epigen_UCSD\.snakemake
MERlin - the MERFISH decoding pipeline
MERlin - the MERFISH decoding pipeline
MERlin - the MERFISH decoding pipeline
Running SimpleGlobalAlignment
[Tue Jan 17 10:51:34 2023]
Finished job 1.
1 of 5 steps (20%) done
Running FiducialCorrelationWarp
Running FiducialCorrelationWarp
U:\Lab\Bereket_public\custom_merfish_rig\MERlin_epigen_UCSD\merlin\analysis\warp.py:143: FutureWarning: The input object of type 'SimilarityTransform' is an array-like implementing one of the corresponding protocols (`__array__`, `__array_interface__` or `__array_struct__`); but not a sequence (or 0-D). In the future, this object will be coerced as if it was first converted using `np.array(obj)`. To retain the old behaviour, you have to either modify the type 'SimilarityTransform', or assign to an empty array created with `np.empty(correct_shape, dtype=object)`.
  np.array(transformationList,dtype = object), 'offsets',
U:\Lab\Bereket_public\custom_merfish_rig\MERlin_epigen_UCSD\merlin\analysis\warp.py:143: FutureWarning: The input object of type 'SimilarityTransform' is an array-like implementing one of the corresponding protocols (`__array__`, `__array_interface__` or `__array_struct__`); but not a sequence (or 0-D). In the future, this object will be coerced as if it was first converted using `np.array(obj)`. To retain the old behaviour, you have to either modify the type 'SimilarityTransform', or assign to an empty array created with `np.empty(correct_shape, dtype=object)`.
  np.array(transformationList,dtype = object), 'offsets',
WorkflowError:
Error recording metadata for finished job ([WinError 3] The system cannot find the path specified: 'U:\\Lab\\Bereket_public\\custom_merfish_rig\\MERlin_epigen_UCSD\\.snakemake\\metadata\\tmplptve0_x.VTovTGFi' -> 'U:\\Lab\\Bereket_public\\custom_merfish_rig\\MERlin_epigen_UCSD\\.snakemake\\metadata\\VTovTGFiL0JlcmVrZXRfcHVibGljL2N1c3RvbV9tZXJmaXNoX3JpZy9NZXJsaW5fdGVzdC90ZXN0X2RhdGEvcmVzdWx0cy9kYXRhL0ZpZHVjaWFsQ29ycmVsYXRpb25XYXJwL3Rhc2tzL0ZpZHVjaWFsQ29ycmVsYXRpb25XYXJwXzAuZG9uZQ=='). Please ensure write permissions for the directory U:\Lab\Bereket_public\custom_merfish_rig\MERlin_epigen_UCSD\.snakemake
Removing output files of failed job FiducialCorrelationWarp since they might be corrupted:
U:/Lab/Bereket_public/custom_merfish_rig/Merlin_test/test_data/results/data/FiducialCorrelationWarp/tasks/FiducialCorrelationWarp_0.done
WorkflowError:
Error recording metadata for finished job ([WinError 3] The system cannot find the path specified: 'U:\\Lab\\Bereket_public\\custom_merfish_rig\\MERlin_epigen_UCSD\\.snakemake\\metadata\\tmpl6y_8x_q.VTovTGFi' -> 'U:\\Lab\\Bereket_public\\custom_merfish_rig\\MERlin_epigen_UCSD\\.snakemake\\metadata\\VTovTGFiL0JlcmVrZXRfcHVibGljL2N1c3RvbV9tZXJmaXNoX3JpZy9NZXJsaW5fdGVzdC90ZXN0X2RhdGEvcmVzdWx0cy9kYXRhL0ZpZHVjaWFsQ29ycmVsYXRpb25XYXJwL3Rhc2tzL0ZpZHVjaWFsQ29ycmVsYXRpb25XYXJwXzEuZG9uZQ=='). Please ensure write permissions for the directory U:\Lab\Bereket_public\custom_merfish_rig\MERlin_epigen_UCSD\.snakemake
Removing output files of failed job FiducialCorrelationWarp since they might be corrupted:
U:/Lab/Bereket_public/custom_merfish_rig/Merlin_test/test_data/results/data/FiducialCorrelationWarp/tasks/FiducialCorrelationWarp_1.done
Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
Complete log: .snakemake\log\2023-01-17T105128.126829.snakemake.log
dariober
  • 8,240
  • 3
  • 30
  • 47
Bereket Berackey
  • 3
  • 2
  • 1
    Some suggestions to help debugging this: Run snakemake with `-p -n` options to see what's being executed in dry-run mode; show the content of the snakefile ideally simplified to the minimum necessary to reproduce the error. PS: I added code formatting to your post, without it it's really difficult to read it and people may be discouraged to answer the question. – dariober Jan 18 '23 at 17:53
  • @dariober Thank you for your suggestions! I used the dry run mode. But it does not show any problem, the workflow seems fine. please see the output [here](https://drive.google.com/file/d/12KAJjgeUXnLF0mIndg2BoAV5kUb50iAY/view?usp=share_link). Also you can find the snakefile [here](https://drive.google.com/file/d/1jIwfr3Vellba9_alhJZaxIsf9j_o0O6v/view?usp=share_link). Thank you! – Bereket Berackey Jan 18 '23 at 21:18

1 Answers1

0

From you question I assume you are not developing the software package which uses snakemake but you are only using it.

You might be running into an issue where the paths snakemake internally creates are to long for windows and exceed the maximum path length of 260 characters which Windows uses by default.

The very long line with the long paths is an indication for this:

Error recording metadata for finished job ([WinError 3] The system cannot find the path specified: 'U:\\Lab\\Bereket_public\\custom_merfish_rig\\MERlin_epigen_UCSD\\.snakemake\\metadata\\tmplptve0_x.VTovTGFi' -> 'U:\\Lab\\Bereket_public\\custom_merfish_rig\\MERlin_epigen_UCSD\\.snakemake\\metadata\\VTovTGFiL0JlcmVrZXRfcHVibGljL2N1c3RvbV9tZXJmaXNoX3JpZy9NZXJsaW5fdGVzdC90ZXN0X2RhdGEvcmVzdWx0cy9kYXRhL0ZpZHVjaWFsQ29ycmVsYXRpb25XYXJwL3Rhc2tzL0ZpZHVjaWFsQ29ycmVsYXRpb25XYXJwXzAuZG9uZQ=='). Please ensure write permissions for the directory U:\Lab\Bereket_public\custom_merfish_rig\MERlin_epigen_UCSD\.snakemake

There are three common solutions to this problem:

  1. Rewrite the Snakemake workflow, i.e. the software you are using, to use shorter paths. Or ask the developers for help. (Slow to implement solution, not necessarily stable and sustainable but permanent and it helps all users).
  2. Move the software and your workflow to a directory with a shorter path. I.e. in your case from U:\Lab\Bereket_public\custom_merfish_rig\MERlin_epigen_UCSD to U:\Merlin_epigen_UCSD. This can but is not guaranteed to help. (Easy and quick to implement, not guaranteed to work)
  3. You can also allow for longer paths in Windows, see this article for details. I've found this solution in the past to work reliably. (Requires some technical knowledge and elevated user privileges, does not transfer to other users or machines).

Unfortunately I'm not aware of any snakemake-based solution to handle this problem.

euronion
  • 1,142
  • 6
  • 14