How to create Subfolder.Zip file from Folder.Zip in C#

Question

I am trying to create some packages from Build Artifacts using .NETCore LAMBDA function, facing some problems due to storage constraint LAMBDA temp storage is just 512 MB. My Artifacts(Zip file size is 300MB) structure is

./scripts.bat
./scripts2.yml
./Env/Svr1/somefile.txt
./Env/Svr1/somefolder/somefile.txt
./Env/Svr1/somefolder/someotherfolder/somefile.txt

Task is to copy scripts.bat and scripts2.yml to ./Env/Svr1/ and zip the contents of the Svr1(excluding other folders in ./Env/Svr2/..etc) as ./Svr1.zip at root level without unzip the whole zip as the storage is constraint.

                var tempPath = @"/tmp/"; 
                using (var file = File.OpenRead(tempPath+ "MyZipFile.zip"))
                {
                    using (var zipFile = new ZipArchive(file, ZipArchiveMode.Read, true))
                    {
                        var ymlConfigs = zipFile.Entries.Select(f => f.Name).Where(en => en.EndsWith(".yml"));
                        var batFiles = zipFile.Entries.Select(f => f.Name).Where(en => en.EndsWith(".bat"));
                        var svr1Folder = zipFile.Entries.Select(folder => folder.FullName).Where(name => name.StartsWith("Env/Svr1"));                          
                                     
                         //add yml files to package 
                        foreach (var ymlfile in ymlConfigs)
                        {
                            svr1Folder.Concat(new[] { ymlfile });                               
                        }
                        //add bat files to package 
                        foreach (var batfile in batFiles)
                        {
                            svr1Folder.Concat(new[] { batfile });                           
                        }
                          //create zip package 
                        var newZip = ZipFile.Open(tempPath + "Svr1", ZipArchiveMode.Update);
                        foreach (var item in svr1Folder)
                        {
                            newZip.CreateEntryFromFile(item, Path.GetFileName(item), CompressionLevel.Optimal); 
//Error Could not find a part of the path as its looking for Item in Debug folder  ..\bin\Debug\netcoreapp3.1\Env\Svr1\somefile.txt where as my item is in ./tmp/MyZipFile.zip                              
                        }
                    }
                }

maybe instead of entries.select use for loops in a function to get the result, this way you minimise memory usage (by not applying var found = item[i]; unless it is found. I did something similar recently and memoryt usage was ~1/5 of what lamda was using and it was much faster. — Slipoch, Jun 30 '20 at 02:14
The problem I am facing is unable to read the folder Svr1 while creating new Zip because the unzipped file hasn't wrote to disk yet and unable to read causing error on this line newZip.CreateEntryFromFile, its looking for physical file from disk to create zip which doesnt exist..looking to see if there is away around this. — Sat, Jun 30 '20 at 07:36

Sat · Answer 1 · 2020-07-02T15:14:49.743

I managed to find a solution to read the zip file as Stream and create a new zip file with empty folder and add required entries from original file.

using (FileStream zipToOpen = new FileStream(tempPath + folder + ".zip", 
FileMode.Open))
{
using (ZipArchive archive = new ZipArchive(zipToOpen, 
ZipArchiveMode.Update))
{
  foreach (var item in svr1Package)
  {
   var splitString = item.FullName.Split("/");
   var path = String.Join("/", item.FullName.Split("/").Skip(2));
   if (!String.IsNullOrEmpty(path))
   {
      var temp = path.Substring(0,path.LastIndexOf("/"));
      var outputpath = Path.Combine(tempPath, temp);
      var readEntry = archive.CreateEntry(path);//creates file with this 
                                                //name
     //read data from original item and load in memory 
      var inputStream = item.Open();
      byte[] data = null;
      byte[] buffer = new byte[16 * 1024];
      using (MemoryStream memoryStream = new MemoryStream())
      {
       int read;
       while ((read = inputStream.Read(buffer, 0,buffer.Length)) > 0)
       {
          memoryStream.Write(buffer, 0, read);
       }
       data = memoryStream.ToArray();
       }
       //write back data to newZip file 
       using (BinaryWriter writer = new BinaryWriter(readEntry.Open()))
       {
           writer.Write(data);
       }
     }
       else //these are just txt or bat files copy as it is 
     {
     if ((item.FullName.StartsWith("appspec_sv1")) || 
        (item.FullName.StartsWith("appspec_sv2")))
        {
         var readEntry = archive.CreateEntry("appspec.yml");
         StringBuilder stringBuilder;
         using (StreamReader reader = new StreamReader(item.Open()))
         {
           stringBuilder = new StringBuilder(reader.ReadToEnd());
         }
         using (StreamWriter writer = new StreamWriter(readEntry.Open()))
         {
                                writer.Write(stringBuilder);
         }
     }
          else
          {
           var readEntry = archive.CreateEntry(item.FullName);
           StringBuilder stringBuilder;
           using (StreamReader reader = new StreamReader(item.Open()))
           {
            stringBuilder = new StringBuilder(reader.ReadToEnd());
           }
          using (StreamWriter writer = new StreamWriter(readEntry.Open()))
          {
                 writer.Write(stringBuilder);
          }
    }
  }
}
  transferUtility.Upload(tempPath + folder + ".zip",bucketName, 
  outputPrefix + folder + ".zip");

}

score 0 · Answer 2 · answered Jul 02 '20 at 14:58

I have answer how you could create zip in .NET previously on this post Creating Directories in a ZipArchive C# .Net 4.5 which I think you could still apply in dotnet core.

Additionally, if you are running your application in a unix-like environment, you can simple use the zip command which it is likely to be available on AWS

Cheers, Herb

How to create Subfolder.Zip file from Folder.Zip in C#

2 Answers2