Regarding the following post: Appending files to a zip file with Java
I'm trying to unzip huge zip file , about 5GB , the best time it took me is about 30 min cause my bottleneck is I/O writes , so I tried to change the approach and make modification inside ZIP itself.
My scenarios are very simple: add few file/files/folder , the code works perfect on small zip files , but on large are not working.
I've tried plenty of combinations...attached some attempts:
1st :
public static void main(String[] args) {
String zip = "/Users/xxx/Documents/test/before.zip";
Map<String, String> env = new HashMap<>();
env.put("create", "true");
Path path = Paths.get(zip);
URI uri = URI.create("jar:" + path.toUri());
try (FileSystem fs = FileSystems.newFileSystem(uri, env)) {
Path toAdd = Paths.get("/Users/xxx/Documents/test/append.txt");
Path directory = fs.getPath("/some/directory");
//create if not exists
Files.createDirectories(directory);
//add to dir new file
Files.copy(toAdd, directory.resolve("replaced.txt"));
} catch (IOException e) {
e.printStackTrace();
}
}
But I got:
java.lang.OutOfMemoryError: Java heap space
at sun.util.calendar.Gregorian.newCalendarDate(Gregorian.java:85)
at sun.util.calendar.Gregorian.newCalendarDate(Gregorian.java:37)
at java.util.Date.<init>(Date.java:254)
at com.sun.nio.zipfs.ZipUtils.dosToJavaTime(ZipUtils.java:122)
at com.sun.nio.zipfs.ZipFileSystem$Entry.cen(ZipFileSystem.java:1884)
at com.sun.nio.zipfs.ZipFileSystem$Entry.readCEN(ZipFileSystem.java:1871)
at com.sun.nio.zipfs.ZipFileSystem.sync(ZipFileSystem.java:1257)
at com.sun.nio.zipfs.ZipFileSystem.close(ZipFileSystem.java:277)
at TrueZi.AAA.append200Files7KBExistingFolder(AAA.java:82)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.junit.internal.runners.TestMethodRunner.executeMethodBody(TestMethodRunner.java:99)
at org.junit.internal.runners.TestMethodRunner.runUnprotected(TestMethodRunner.java:81)
at org.junit.internal.runners.BeforeAndAfterRunner.runProtected(BeforeAndAfterRunner.java:34)
at org.junit.internal.runners.TestMethodRunner.runMethod(TestMethodRunner.java:75)
at org.junit.internal.runners.TestMethodRunner.run(TestMethodRunner.java:45)
at org.junit.internal.runners.TestClassMethodsRunner.invokeTestMethod(TestClassMethodsRunner.java:71)
at org.junit.internal.runners.TestClassMethodsRunner.run(TestClassMethodsRunner.java:35)
at org.junit.internal.runners.TestClassRunner$1.runUnprotected(TestClassRunner.java:42)
at org.junit.internal.runners.BeforeAndAfterRunner.runProtected(BeforeAndAfterRunner.java:34)
at org.junit.internal.runners.TestClassRunner.run(TestClassRunner.java:52)
at org.junit.runner.JUnitCore.run(JUnitCore.java:121)
at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
at com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:47)
at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:242)
at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70)
or
java.lang.OutOfMemoryError: Java heap space
at com.sun.nio.zipfs.ZipFileSystem$Entry.readCEN(ZipFileSystem.java:1871)
at com.sun.nio.zipfs.ZipFileSystem.sync(ZipFileSystem.java:1257)
at com.sun.nio.zipfs.ZipFileSystem.close(ZipFileSystem.java:277)
at TrueZi.AAA.test(AAA.java:84)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.junit.internal.runners.TestMethodRunner.executeMethodBody(TestMethodRunner.java:99)
at org.junit.internal.runners.TestMethodRunner.runUnprotected(TestMethodRunner.java:81)
at org.junit.internal.runners.BeforeAndAfterRunner.runProtected(BeforeAndAfterRunner.java:34)
at org.junit.internal.runners.TestMethodRunner.runMethod(TestMethodRunner.java:75)
at org.junit.internal.runners.TestMethodRunner.run(TestMethodRunner.java:45)
at org.junit.internal.runners.TestClassMethodsRunner.invokeTestMethod(TestClassMethodsRunner.java:71)
at org.junit.internal.runners.TestClassMethodsRunner.run(TestClassMethodsRunner.java:35)
at org.junit.internal.runners.TestClassRunner$1.runUnprotected(TestClassRunner.java:42)
at org.junit.internal.runners.BeforeAndAfterRunner.runProtected(BeforeAndAfterRunner.java:34)...
is it could be this ?
https://community.oracle.com/thread/4123064
Another try is as one saying (rewrite the zip into other zip name and append the new file , but it takes me average same time as unzip):
ZipInputStream original = new ZipInputStream(new BufferedInputStream(new FileInputStream("/Users/xxx/Documents/test/original.zip")));
ZipOutputStream append = new ZipOutputStream(new FileOutputStream("/Users/xxx/Documents/test/rename.zip"));
ZipEntry entry;
while ((entry = original.getNextEntry()) != null) {
append.putNextEntry(entry);
if (!entry.isDirectory()) {
IOUtils.copy(original, append);
}
append.closeEntry();
}
File f = new File(appended);
try (FileInputStream inputStream = new FileInputStream(f)) {
append.putNextEntry(new ZipEntry(f.getName()));
byte[] readBuffer = new byte[4096];
int amountRead;
while ((amountRead = inputStream.read(readBuffer)) > 0) {
append.write(readBuffer, 0, amountRead);
}
} catch (Exception e) {
e.printStackTrace();
}
// close
original.close();
append.close();
I'm wondering if there is other any option neither using FileSystem not rewrite all huge zip to other....??
