5

I am using Scala 2.10.1 with sbt to package my webapp as a war file. For the purpose of efficient rsync deltas, I'd like to have the war packaged as a .war file, but without zip compression. I just need to know how to configure my build for this.

UPDATE:
All these plugin docs assume all this knowledge of how the syntax works and how to combine tasks into a new task, etc. I can't even tell how to create a new task that does package then command. None of the answers so far have said specifically, "here's what you do.."

Just to be clear, this is all I'm asking for:

I need a Task "packnozip" that does this:

1) run "package"

2) run shell commands:

$ mkdir ./Whatever 
$ pushd ./Whatever 
$ jar xvf ../Whatever.war 
$ popd 
$ mv ./Whatever.war ./Whatever.war.orig 
$ jar cvM0f ./Whatever.war -C ./Whatever .

So what i'm saying is i want to type "packnozip" into the sbt console and have it do #1 then #2.

For now i'm just manually doing #2 which seems silly if it can be automated. Also watching a 30MB file get completely resent by rsync b/c it is not diffable seems quite silly when a 34MB uncompressed file is only 13% more data, and takes a fraction of second to send b/c of efficient diffs, not to mention "-z" will compress the transfer anyways.

jpswain
  • 14,642
  • 8
  • 58
  • 63

2 Answers2

2

If you have your war file unzipped in a directory you can:

zip -r -0 project.war project/

That should be zero compression. In case you don't see those options, this is my setup:

[node@hip1 dev]$ zip -v
Copyright (c) 1990-2008 Info-ZIP - Type 'zip "-L"' for software license.
This is Zip 3.0 (July 5th 2008), by Info-ZIP.

Which, you could execute as a run task I believe, after the war is packaged.

UPDATE 1

I believe this is the best way to achieve your needs:

http://www.scala-sbt.org/release/docs/Detailed-Topics/Process

val exitcode = "zip -r -0 project.war project/"!

However, if you need to work from a specific directory (Please see Update 2 below):

Modified this to execute within directory but place .war above directory. The path (2nd) argument should include the directory, so that the zip is performed inside of it:

Process("zip" :: "-r" :: "-0" :: "../project.war" :: "." :: Nil, "/path/to/project/") !

Here's another SO question on the ProcessBuilder that may help as well:

How does the “scala.sys.process” from Scala 2.9 work?

(Note: you don't need to import scala.sys.process._)

UPDATE 2

For readers in the future, please note that zipping the project directory itself will not work, one needs to perform the zip of the war inside the directory by using pushd, putting the resulting war outside of the directory as mentioned by the OP in the comments below this answer. As Orange80 mentioned:

pushd ./project && zip -r -0 ../project.war ./ && popd

UPDATE 3

Check out this, it may do exactly what you need, with a 0 for options to specify no compression:

https://github.com/sbt/sbt-onejar

a plugin that lets you create a single executable jar, which, with options (for example "0" as in a command like "jar 0f blah.jar blah/") can be made I think as you mentioned in the comments below to create the jar file without compression.

For usage I found this on SO: SBT one-jar plugin

And also, if it needs to be modified, it's a pretty reasonable example of a plugin as well, which if you drop it in your home ~/.sbt/plugins it will be global and can be used in your build in the fashion noted in the SO answer above. I hope that helps at least a little bit/

Community
  • 1
  • 1
Matt Mullens
  • 2,266
  • 15
  • 14
  • Do you know how to write a run task for SBT? It's totally new to me. – jpswain Jun 27 '13 at 00:27
  • Sure, I'll post how I believe this works, but you'll have to check the docs if this gives you some trouble. There are a few options that I'll update the answer with. – Matt Mullens Jun 27 '13 at 01:13
  • This is very interesting. I will look into adding this as a configuration option in xsbt-web-plugin. – earldouglas Jun 27 '13 at 18:10
  • 1
    @orange80 Here's an example of how to write a custom task: http://stackoverflow.com/questions/17038663/sbt-plugin-user-defined-configuration-for-command-via-their-build-sbt/17100585#17100585 – earldouglas Jun 27 '13 at 18:10
  • I think that could be pretty useful, thanks James. And great work with xsbt-web-plugin! – Matt Mullens Jun 27 '13 at 18:57
  • @hoonto that zip command is incorrect. It will create a project.war that contains a project dir, instead of just the contents inside the project directory. I think the only way to do it is: pushd ./project && zip -r -0 ../project.war ./ && popd – jpswain Jun 28 '13 at 07:21
  • Good catch I can update the answer tonight when I get back home. – Matt Mullens Jun 28 '13 at 21:02
  • @orange80, did you try the "!" syntax for executing the command by putting the full path including the directory (rather than the directory above) in the second argumet to Process? I can't test at the moment, but I thought that would work, like Process("zip" :: "-r" :: "-0" :: "../project.war":: Nil, "/path/to/project") ! - that should execute the command from within the directory but place the war above it, so as to avoid the pushd popd, although I think that would work too - updated answer in case that's right and included your information with pushd/popd – Matt Mullens Jun 29 '13 at 15:58
  • I'd really like to try this, but have no idea about modifying the build config. I need the answer to say, "this is exactly how you modify build.sbt to apply this". Would you mind adding that info? The example @James pointed out is using Build.scala which is not how my project is set up. I'm using build.sbt, so I only want to use Build.scala if there's a way to do it in conjunction with build.sbt. I would rather not have to change this. Using sbt once it's set up is awesome, but anything other than adding dependencies is a nightmarish black box to me :-\ – jpswain Jul 01 '13 at 05:27
  • @James any insight? Unfortunately I am so unfamiliar with how sbt works that the docs make no sense to me without basic context. I don't have any where to put the "Process("zip" :: "-r" :: "-0" :: "../project.war" :: "." :: Nil, "/path/to/project/") !" Also what does the "!" at the end do? The form of the sbt config is so hyperminimalist that I don't know where to even look to understand what "newTask <<= newSetting map { str => println(str) }" does. Is it appending or replacing the new task collection with the new setting collection mapped to a bunch of printlines? what does that do?? – jpswain Jul 02 '13 at 02:23
  • @orange80 I'm back and so is my environment thankfully! Tonight I should get some time to write a plugin that gives you more context and post it on github. However, James, if you have that already though please don't hesitate to post an answer, just want to make sure orange80 gets a good solution. The "!" indicates to execute the statement and that Process command I posted is wrong again good grief I'm a mess without an environment, the directory target/webapp is where the zip should take place. Keep in mind zip doesn't create the META-INF though, like you'd find in a normal jar or war. – Matt Mullens Jul 02 '13 at 02:50
  • So, I'll see about making the zero compression war tonight without zip, but creating the war with the META-INF in it. – Matt Mullens Jul 02 '13 at 02:51
  • I know the command to run, no problem. I just need to know how to make the task or plugin automatically run that with sbt and what the env var for the target directory is. So far, running this outside the sbt config, the jar command has been great, it has the benefit of the "-C" switch from tar, while being able to make a proper zip: "$ jar cvM0f Something.war -C ./Something/ ." – jpswain Jul 02 '13 at 03:20
  • @orange80 my deepest apologies, I'm so slammed today and next couple days that the best I can do is to provide some resources and info. You can create a plugin to run the jar command as you mentioned, and the plugin can be put in ~/.sbt/plugins to make it global. In the plugin .scala file you can write the task to execute jar, and then call the plugin from your build. Here's a good resource on making the plugins: http://www.scala-sbt.org/release/docs/Extending/Plugins. And it doesn't look like there are any existing ones that do what you need exactly, although sbt-sh may do what you need – Matt Mullens Jul 02 '13 at 21:39
  • But what I would do is just write a plugin so that way you can reuse it simply in your build. Using the example command plugin on that site I mentioned a couple comments above (Extending/Plugins) will be exactly what you can use to drop in the Process(...) ! command to run jar. I'll post in the answer above how I think this would work but without testing at all - again I apologize I can't put it together in a more digestible fashion at the moment. – Matt Mullens Jul 02 '13 at 21:41
  • All these plugin docs assume all this knowledge of how the syntax works and how to combine tasks into a new task, etc. I can't even tell how to create a new task that does package then command. Is sbt really this hard? It's not worth 100 pts to anyone to just do a quick example? Here's all I want: 1) run package command. 2) run shell commands: $ mkdir ./Whatever $ pushd ./Whatever $ jar xvf ../Whatever.war $ popd $ mv ./Whatever.war ./Whatever.war.orig $ jar cvM0f ./Whatever.war -C ./Whatever . So what i'm saying is i want to type "packnozip" into sbt and do #1 then #2 – jpswain Jul 04 '13 at 04:40
0

There is no way to do this directly via sbt configuration, since sbt assumes that any files within zip and jar artifacts should be compressed.

One workaround is to unzip and re-zip (without compression) the war file. You can do this by adding the following setting to your project (e.g. in build.sbt):

packageWar in Compile <<= packageWar in Compile map { file =>
  println("(Re)packaging with zero compression...")
  import java.io.{FileInputStream,FileOutputStream,ByteArrayOutputStream}
  import java.util.zip.{CRC32,ZipEntry,ZipInputStream,ZipOutputStream}
  val zis = new ZipInputStream(new FileInputStream(file))
  val tmp = new File(file.getAbsolutePath + "_decompressed")
  val zos = new ZipOutputStream(new FileOutputStream(tmp))
  zos.setMethod(ZipOutputStream.STORED)
  Iterator.continually(zis.getNextEntry).
    takeWhile(ze => ze != null).
    foreach { ze =>
      val baos = new ByteArrayOutputStream
      Iterator.continually(zis.read()).
        takeWhile(-1 !=).
        foreach(baos.write)
      val bytes = baos.toByteArray
      ze.setMethod(ZipEntry.STORED)
      ze.setSize(baos.size)
      ze.setCompressedSize(baos.size)
      val crc = new CRC32
      crc.update(bytes)
      ze.setCrc(crc.getValue)
      zos.putNextEntry(ze)
      zos.write(bytes)
      zos.closeEntry
      zis.closeEntry
    } 
  zos.close
  zis.close
  tmp.renameTo(file)
  file
}

Now when you run package in sbt, the final war file will be uncompressed, which you can verify with unzip -vl path/to/package.war.

earldouglas
  • 13,265
  • 5
  • 41
  • 50