0

I use the following code to copy files which works OK The problem is that when we use this method to copy folder and items like node_js project with node_modules inside in windows this can take a long time about 40 sec and in mac it takes 2 sec. when doing trace (print the copied files) we saw that for example the io.copy took 5 sec to copy short readme file or json file , in each run it get stuck on different files , small size files like json readme etch this is not consistent, in different run it stuck on different files ( if you run on same project).

we checked it on 3 different windows10 machine ( 1-3 years old) and 2 different macbook (3 years old) on exact same project. What it can be ?

func walk(src string, baseDir string, archive *zip.Writer) error {
   return filepath.Walk(src, func(path string, info os.FileInfo, err error) (e error) {
      if err != nil {
         return err
      }

      if info.IsDir() {
         return
      }

      header, err := zip.FileInfoHeader(info)
      if err != nil {
         return err
      }
      if baseDir != "" {
         header.Name = filepath.ToSlash(getRelativePath(path, baseDir))
      }

      header.Method = zip.Deflate
      writer, err := archive.CreateHeader(header)
      if err != nil {
         return err
      }

      file, err := os.Open(path)
      if err != nil {
         return err
      }
      defer func() {
         e = CloseFile(file, e)
      }()

      _, err = io.Copy(writer, file)
      return err
   })
}

we are checking it almost 4 days and we didnt find any pattern :(

NSS
  • 755
  • 1
  • 11
  • 30
  • 1
    Can you use rsync instead of golang? – Ben Jan 29 '19 at 20:15
  • 3
    I know someone already told you this, but it's the OS. You can do a search about NTF + small files to confirm – yorodm Jan 29 '19 at 20:21
  • @Ben - sorry what do you mean instead of golang ? can you provide example what to change ? – NSS Jan 29 '19 at 20:23
  • @yorodm - someone told me what ? :) – NSS Jan 29 '19 at 20:24
  • There's usually some *nix fan in every time that blames Windows for every bad thing in the Universe, I assumed there was one in your team. Nevertheless, NTFS has some issues when dealing with small files and there are a lot of those in a `node_modules` folder. That's the root of all your performance issues. – yorodm Jan 29 '19 at 20:28
  • have you considered to parallelize the job ? –  Jan 29 '19 at 21:02
  • You should skip the node_modules folder. Just copy your files and then run `npm init`, that is much faster. – apxp Jan 29 '19 at 21:38
  • @mh-cbon a bad idea if your destination is not able to write prallel. – apxp Jan 29 '19 at 21:39
  • @NinaS there's a unix tool called `rsync` that copies files from one location to another, and on subsequent copies, it only copies the *changes* from the first location. Because of the nature of node_modules, you'd likely only pay the small file cost on the first copy (I haven't tested this). Unfortunately, I couldn't find a maintained open-source windows version in my quick Googling. – Ben Jan 29 '19 at 22:28
  • @apxp i don t understand your remark. he is copying file by file, why would it be unsafe ? tree traversal can be made twice to setup folders at first, then apply a second traversal to copy the files in //. what did you mean ? –  Jan 29 '19 at 23:31
  • With Windows file acesses you should be concerned about the effects of Windows Defender Antivirus and other antimalware programs. Does turning them off make any difference? – peterSO Jan 29 '19 at 23:54
  • @yorodm why would it be the OS? have you tested the copying in another language like C++ or .NET. I get very fast copy speeds in .NET, but not in golang. So maybe it's the way golang calls the windows API. – s952163 Jan 30 '19 at 00:49
  • 1
    I've had the same issue, the solution was to create the file first, then do all the checks and then copy (write) the data. – Dippo Jan 30 '19 at 00:51
  • @s952163 NTFS has performance issues with small files. `node_modules` has mostly small files. See [here](https://stackoverflow.com/questions/27845026/opening-many-small-files-on-ntfs-is-way-too-slow) and [here](https://serverfault.com/questions/768755/sluggish-performance-on-ntfs-drive-with-large-number-of-files) – yorodm Jan 30 '19 at 01:31
  • As others have mentioned, it may indeed be the NFTS performance with small files. Did you try to zip and transfer all files as a sole unit? It should have performance comparable to the Mac one, otherwise some other networking issue could be the culprit. – hyperTrashPanda Feb 01 '19 at 11:44
  • @Dippo, were you writing into a zip file? I have this same problem right now, especially with a streaming response. – Edwinner Dec 23 '20 at 22:25
  • @Edwinner No, not a zip file. – Dippo Dec 28 '20 at 01:08

0 Answers0