Jenkins job adding DOS newlines to shell scripts, causing a "bad interpreter" error on UNIX

Question

I am trying to run the following shell script but I am getting the "/bin/bash^M: bad interpreter" error.

Script:

#!/bin/bash
rm -rf tomcat8/webapps/cpproject*

I know that this is happening because I am editing the file on windows and trying to run it on Unix. I tried all the ways of fixing it as suggested on stackoverflow. I tried the following.

1. Installed Vim editor and saved it
2. In Notepad++ changed the End Of Line option to Unix. Tried to save the new documents as Unix format
3. Installed gedit and saved it.
4. Installed doc2unix and tried it.

I tried a lot other ways but nothing worked. Can some one suggest me how to fix this issue?

Updated:

"xxd before_install1.sh | head" command is showing the followng.

00000000: 2321 2f62 696e 2f62 6173 680a 726d 202d  #!/bin/bash.rm -
00000010: 7266 2074 6f6d 6361 7438 2f77 6562 6170  rf tomcat8/webap
00000020: 7073 2f63 7070 726f 6a65 6374 2a         ps/cpproject*

Update: I made sure that I am not running the old file. Because I changed the file name each time and tried it. I made sure that the new file is being executed each time. Basically I am using codedeploy jenkins plugin which generates the war file and zips it with scripts and appspec.yml file. Since I change the file name each time and update the appspec.yml file with new name, I am making sure that the new file is being uploaded. But it is giving the same error. Even after it is uploaded to the S3 bucket I downloaded and the zip file and made sure that it contains the latest one and even executed the xxd command and got the following.

$ xxd before_install5.sh | head
00000000: 2321 2f62 696e 2f62 6173 680d 0a72 6d20  #!/bin/bash..rm
00000010: 2d72 6620 746f 6d63 6174 382f 7765 6261  -rf tomcat8/weba
00000020: 7070 732f 6370 7072 6f6a 6563 742a 0d0a  pps/cpproject*..

Update:

@Charles Duffy, Thank you for your response. I am developing these scripts locally and pushing them to GitHub. I have a Jenkins job, which takes this code from GitHub repository and packages this into a war file. I am using "CodeDeploy" plugin, which will take this war file and zips it with these shell scripts under scripts folder and appspec.yml file and push it to S3 bucket in AWS. The following is what is happening.

xxd command in my local system file:

$ xxd before_install5.sh | head
00000000: 2321 2f62 696e 2f62 6173 680a 726d 202d  #!/bin/bash.rm -
00000010: 7266 2074 6f6d 6361 7438 2f77 6562 6170  rf tomcat8/webap
00000020: 7073 2f63 7070 726f 6a65 6374 2a0a       ps/cpproject*.

xxd command on the file after it is uploaded to GitHub:

$ xxd before_install5.sh | head
00000000: 2321 2f62 696e 2f62 6173 680a 726d 202d  #!/bin/bash.rm -
00000010: 7266 2074 6f6d 6361 7438 2f77 6562 6170  rf tomcat8/webap
00000020: 7073 2f63 7070 726f 6a65 6374 2a0a       ps/cpproject*.

xxd command on the file after it is pulled by Jenkins from GitHub to Local Workspace(in .jenkins/worspace folder)

$ xxd before_install5.sh | head
00000000: 2321 2f62 696e 2f62 6173 680d 0a72 6d20  #!/bin/bash..rm
00000010: 2d72 6620 746f 6d63 6174 382f 7765 6261  -rf tomcat8/weba
00000020: 7070 732f 6370 7072 6f6a 6563 742a 0d0a  pps/cpproject*..

xxd command on the file after it is pushed to s3 bucket in AWS:

$ xxd before_install5.sh | head
00000000: 2321 2f62 696e 2f62 6173 680d 0a72 6d20  #!/bin/bash..rm
00000010: 2d72 6620 746f 6d63 6174 382f 7765 6261  -rf tomcat8/weba
00000020: 7070 732f 6370 7072 6f6a 6563 742a 0d0a  pps/cpproject*..

I am seeing that the extra characters '0d0a' are added after the code is taken by Jenkins into Local Workspace. It's not a problem with CodeDeploy plugin at this momement, it seems. Is there any way we can avoid this?

Thank you.

Answer 1

If the script is called 'myscript.dos', you can delete the carriage returns with tr:

tr -d '\015' < myscript.dos > myscript.unix

Answer 2

@Charles Duffy, Thank you for your time and patience in resolving this issue. It is finally working. I am just posting the actual problem and solution here so that it will be useful for other.

Work: I am working on developing a deployment strategy into AWS using Jenkins and AWS CodeDeploy. I am using CodeDeploy plugin in Jenkins. Jenkins will create a war file and CodeDeploy plugin will create a zip file with the war file, appspec.yml and some shell scripts to stop, start servers etc and this zip file will be pushed to S3 and from there CodeDeploy will be triggered and deploy the war file.

Problem: The shell scripts used to start the server or stop the server are developed in windows system. So it is adding DOS lines to the scripts so they are not running on Unix in EC2 machines. I have removed the DOS lines on the shell scripts by using different techniques. But since Jenkins is running locally (On Windows Machine), when the code is pulled from GutHub it is again adding DOS lines.

Solution: Please check the above questions and the solutions given by all to find out how to identify if DOS line are being added to shell scripts. Since, in my case, the DOS lines are added by Jenkins I have to clear them in Jenkins before it is going to produce a zip file.

Go to Jenkins and configure the shell executable first. Since I have git installed on my system, I have a Shell executable in my system.

Once this is done, go to the Jenkins job created and add "Execute Shell" build step and add the commands shown in the below picture.

This will remove all the DOS lines added to the scripts and your shell scripts will be executed correctly by codedeploy in AWS on Linux system.

Thank you @Charles Duffy and others who helped me in resolving these issues.

Thank you very much, Subbu.