0

I have a Jenkins job that is checking out a repository, compiling a C project, and then is supposed to stage, commit and push the generated binaries. However, the job fails at staging/committing the binaries because it does not detect any changes and therefore has nothing to commit.

In the PowerShell script that does the compilation, after the binary image is created I copy it to another directory where it is tracked in our repository.

Source binary file:
.\device_test\images\Device_ROYAL_Firmware_Development.0.img

Destination binary file (the one that is tracked):
.\device_platforms\nordic15_2\device_device\device_out\ROYAL_FW_40.04.10.0.bin

Despite the destination binary file showing an updated LastWriteTime before and after being overwritten, git status does not show that file as modified and git add -f binaryfile does nothing.

Build #1
    Copying version header...
    Current FW version: #define DEVICE_FW_VERSION "40.04.10.0"
    Compiling .\device_platforms\nordic15_2\device_device\pca10100e\s112\arm5_no_packs\device.uvprojx...
        Keil build successful! image=.\device_platforms\nordic15_2\device_device\device_out\ROYAL_FW_40.04.10.0.bin
    Image pre-copy modified: 10/17/2022 12:35:44
    Copying image file .\device_platforms\nordic15_2\device_device\device_out\ROYAL_FW_40.04.10.0.bin to .\device_test\images\Device_ROYAL_Firmware_Development.0.img...
    Image post-copy modified: 10/17/2022 12:41:45
    Adding image file .\device_test\images\Device_ROYAL_Firmware_Development.0.img to git...

Running git status before overwriting the file, of course, does not show the file as having been modified.

HEAD detached from 5a36217fa
Changes not staged for commit:
  (use "git add/rm <file>..." to update what will be committed)
  (use "git checkout -- <file>..." to discard changes in working directory)

    modified:   Device_lib/projects/keil/Device_lib.lib
    modified:   Device_platforms/nordic15_2/Device_device/Device_fw_version/Device_fw_version_ROYAL.h
    modified:   Device_platforms/nordic15_2/Device_device/Device_hw_config/Device_hw_config.h
    deleted:    Device_platforms/nordic15_2/Device_device/Device_out/.gitignore

Untracked files:
  (use "git add <file>..." to include in what will be committed)

    Device_major_version_ROBIN.txt
    Device_major_version_ROYAL.txt
    cppcheck-result-Devicelib.xml
    cppcheck-result-Deviceshim.xml
    doxygen/
    git_output.txt
    Device_fw_version_ROBIN.0.h
    Device_fw_version_ROBIN.1.h
    Device_fw_version_ROBIN.2.h
    Device_fw_version_ROBIN.3.h
    Device_fw_version_ROYAL.0.h
    Device_fw_version_ROYAL.1.h
    Device_fw_version_ROYAL.2.h
    Device_fw_version_ROYAL.3.h
    Device_lib/projects/keil/keil_log
    Device_platforms/nordic15_2/Device_device/pca10100e/s112/arm5_no_packs/RTE/_nrf52820_xxaa/
    Device_platforms/nordic15_2/Device_device/pca10100e/s112/arm5_no_packs/keil_log
    Device_test/images/.gitignore

no changes added to commit (use "git add" and/or "git commit -a")

But there are still no changes to the destination file needing to be staged after overwriting the file.

HEAD detached from 5a36217fa
Changes not staged for commit:
  (use "git add/rm <file>..." to update what will be committed)
  (use "git checkout -- <file>..." to discard changes in working directory)

    modified:   Device_lib/projects/keil/Device_lib.lib
    modified:   Device_platforms/nordic15_2/Device_device/Device_fw_version/Device_fw_version_ROYAL.h
    modified:   Device_platforms/nordic15_2/Device_device/Device_hw_config/Device_hw_config.h
    deleted:    Device_platforms/nordic15_2/Device_device/Device_out/.gitignore

Untracked files:
  (use "git add <file>..." to include in what will be committed)

    Device_major_version_ROBIN.txt
    Device_major_version_ROYAL.txt
    cppcheck-result-Devicelib.xml
    cppcheck-result-Deviceshim.xml
    doxygen/
    git_output.txt
    Device_fw_version_ROBIN.0.h
    Device_fw_version_ROBIN.1.h
    Device_fw_version_ROBIN.2.h
    Device_fw_version_ROBIN.3.h
    Device_fw_version_ROYAL.0.h
    Device_fw_version_ROYAL.1.h
    Device_fw_version_ROYAL.2.h
    Device_fw_version_ROYAL.3.h
    Device_lib/projects/keil/keil_log
    Device_platforms/nordic15_2/Device_device/pca10100e/s112/arm5_no_packs/RTE/_nrf52820_xxaa/
    Device_platforms/nordic15_2/Device_device/pca10100e/s112/arm5_no_packs/keil_log
    Device_test/images/.gitignore

no changes added to commit (use "git add" and/or "git commit -a")

For comparison, listing the files in .\device_test\images before overwriting shows the file as being modified at 10/17/2022 12:35 PM

    Directory: E:\jenkins\home\DeviceLib2_dev_src\Device_test\images


Mode                 LastWriteTime         Length Name                                                                 
----                 -------------         ------ ----                                                                 
-a----        10/14/2022   6:51 PM             14 .gitattributes                                                       
-a----        10/15/2022  10:32 AM              6 .gitignore                                                           
-a----        10/14/2022   6:51 PM             86 commit_hash                                                          

... (list shortened)

-a----        10/15/2022  10:27 AM          41316 Device_ROBIN_Firmware_Development.0.img                                
-a----        10/15/2022  10:27 AM          41316 Device_ROBIN_Firmware_Development.1.img                                
-a----        10/15/2022  10:23 AM          41316 Device_ROBIN_Firmware_Development.2.img                                
-a----        10/15/2022  10:24 AM          41316 Device_ROBIN_Firmware_Development.3.img                                
-a----        10/17/2022  12:35 PM          41316 Device_ROYAL_Firmware_Development.0.img                                
-a----        10/17/2022  12:36 PM          41316 Device_ROYAL_Firmware_Development.1.img                                
-a----        10/17/2022  12:37 PM          41316 Device_ROYAL_Firmware_Development.2.img                                
-a----        10/17/2022  12:38 PM          41316 Device_ROYAL_Firmware_Development.3.img                                
-a----        10/14/2022   6:51 PM          40228 Device_Whippet_Firmware_Development.0.img                              
-a----        10/14/2022   6:51 PM          40228 Device_Whippet_Firmware_Development.1.img                              
-a----        10/14/2022   6:51 PM          40228 Device_Whippet_Firmware_Development.2.img                              
-a----        10/14/2022   6:51 PM          40228 Device_Whippet_Firmware_Development.3.img

And shows an updated LastWriteTime of 10/17/2022 12:41 PM after overwriting.

    Directory: E:\jenkins\home\DeviceLib2_dev_src\Device_test\images


Mode                 LastWriteTime         Length Name                                                                 
----                 -------------         ------ ----                                                                 
-a----        10/14/2022   6:51 PM             14 .gitattributes                                                       
-a----        10/15/2022  10:32 AM              6 .gitignore                                                           
-a----        10/14/2022   6:51 PM             86 commit_hash                                                          

... (list shortened)

-a----        10/15/2022  10:27 AM          41316 Device_ROBIN_Firmware_Development.0.img                                
-a----        10/15/2022  10:27 AM          41316 Device_ROBIN_Firmware_Development.1.img                                
-a----        10/15/2022  10:23 AM          41316 Device_ROBIN_Firmware_Development.2.img                                
-a----        10/15/2022  10:24 AM          41316 Device_ROBIN_Firmware_Development.3.img                                
-a----        10/17/2022  12:41 PM          41316 Device_ROYAL_Firmware_Development.0.img                                
-a----        10/17/2022  12:36 PM          41316 Device_ROYAL_Firmware_Development.1.img                                
-a----        10/17/2022  12:37 PM          41316 Device_ROYAL_Firmware_Development.2.img                                
-a----        10/17/2022  12:38 PM          41316 Device_ROYAL_Firmware_Development.3.img                                
-a----        10/14/2022   6:51 PM          40228 Device_Whippet_Firmware_Development.0.img                              
-a----        10/14/2022   6:51 PM          40228 Device_Whippet_Firmware_Development.1.img                              
-a----        10/14/2022   6:51 PM          40228 Device_Whippet_Firmware_Development.2.img                              
-a----        10/14/2022   6:51 PM          40228 Device_Whippet_Firmware_Development.3.img

I can go into the local repository with a command-line, manually copy the source to the destination and git status will correctly show the file as modified. It therefore seems to be isolated when these steps are run by Jenkins. As a starting point, I wonder if the steps taken by the Git SCM plugin in Jenkins have any influence.

git.exe rev-parse --resolve-git-dir localrepo\.git
git.exe config remote.origin.url git@github.com:remoterepo.git
git.exe --version
git --version
git.exe fetch --tags --progress -- git@github.com:remoterepo.git +refs/heads/*:refs/remotes/origin/*
git.exe rev-parse "refs/remotes/origin/branch_name^{commit}"
git.exe config core.sparsecheckout
git.exe checkout -f 63cd0f5b1fd084291ab9f97bd3f03a4d2079101c
git.exe rev-list --no-walk b42d583949d13f5ea9de729b61ab7be81c699bb6

Update:
To @LeGEC's recommendation, git update-index --refresh shows the same unstaged changes- the binary file not being shown in either case.

Device_lib/projects/keil/Device_lib.lib: needs update
Device_platforms/nordic15_2/Device_device/Device_fw_version/Device_fw_version_ROYAL.h: needs update
Device_platforms/nordic15_2/Device_device/Device_hw_config/Device_hw_config.h: needs update
Device_platforms/nordic15_2/Device_device/Device_out/.gitignore: needs update
Taylor Artunian
  • 1
  • 1
  • Can you check if `git update-index --refresh` and `git update-index --really-refresh` have any effect ? – LeGEC Oct 17 '22 at 19:20
  • Both commands output the same list of files that show up as "changes not staged for commit" seen in `git status` and still does not include the binary file. I tried it both in the script right after overwriting the tracked binary file and also from command-line. – Taylor Artunian Oct 17 '22 at 19:45
  • Maybe one of the many answers here? https://stackoverflow.com/a/24316479/7411885 . My experience is either that I need to `git rm --cached` plus `git reset` due to a busted cache, OR that I've just checked out the repo multiple times, and am in the wrong one compared to my tools... – Cpt.Whale Oct 17 '22 at 20:40
  • "the script fails upon committing which I can see in the git output" : can you please add this output to your question ? (note: copy the text, no screen capture please) "I can go into the local repository [...], *copy the binary over manually*" what do you mean by copying the binary ? shouldn't the binary already be present in that local repository ? what happens when you don't copy anything and run `git status` in the pipeline local repo ? – LeGEC Oct 17 '22 at 20:48
  • @LeGEC The binary is output in one location by the compiler and then copied to another folder where it is tracked. If I run `git status` without copying manually, the binary file does not show as modified even though it was overwritten by the script. – Taylor Artunian Oct 17 '22 at 21:38
  • Ok, to rule out one stupid bug : did you confirm that your script did indeed copy that binary file to the expected place ? `diff `, or compare `md5sum`, or ... – LeGEC Oct 18 '22 at 03:32
  • if you are looking for a hashing function and for some reason don't have easy access to `md5sum`or `sha1sum` or ..., you can also use `git hash-object` : `git hash-object ` vs `git hash-object ` – LeGEC Oct 18 '22 at 04:32
  • @LeGEC The existing file, the built file, and the versioned file all had the same hash which makes sense since I have not been changing any source code each time I test. What I did not know is that git apparently does not take the last modified time into consideration when looking for changes. – Taylor Artunian Oct 18 '22 at 17:23
  • I have confirmed that when the existing versioned image has a different hash, the script copies the new build and git finds the changes in the versioned. – Taylor Artunian Oct 18 '22 at 17:33
  • I don't completely understand your last comments. I'll say: git looks for differences in *content* only. If the content of a file hasn't changed, then there is no modification to commit. Is that what was puzzling you ? – LeGEC Oct 18 '22 at 19:38
  • Thank you, yes that was exactly what I was confused about. – Taylor Artunian Oct 18 '22 at 22:56

0 Answers0