The state of compression

[3rd-Eden]

Compression is one of the most important things in a build system and I have the feeling that this is currently one of the weakest parts of our system. There are a couple of issues that I would like to see addressed before we move this project forward towards the glorified 1.0 release of the development branch.

YUI compressor

YUI has recently announced the deprication of their YUI compressor project. They are moving away from their Java based compressor and are embrascing opensource as the base of their compressor which they have named yUglify as it's based upon the excellent uglifyjs compressor. The new version add's some additional fixes upon the regular uglify compressor. Some of these fixes are already in our code base as well, such as appending an extra ; so it's easier to concatenate files after minification.

The CSS compressor has been replaced by CSSMin which is included in their yUglify package. The version bundled is a forked version and maintained version of CSSMin.

UglifyJS

UglifyJS has gained a new brother, UglifyJS2 this is the successor of the current uglify parser and it should be able to compress more then the current stable version.

JSMin

We should also support JSMin as some people seem to fancy that, it's also available with sourcemaps

CSS compression

The only CSS compressor that we are currently supporting is the one bundled in the YUIcompressor.jar, we should also different CSS compressors such as sqwish

Closure compiler

Closure compiler is awesome, but it's depends that Java is installed on your system. I would love to get rid of this dependency and only require node to be installed on the system. Google created a closure compiler service that runs on app-engine. We could leverage this service to compress the data if java is not installed.

The level option

We currently allow people to configure the compressor using a level option, this is just silly and doesn't provide any useful information about what it actually does and is overly complicated to understand for new comers. Instead we should just allow a comma separated list of compressors they want to use on their code base.

Compression detection

Less, important but nice to have.. Iterate over all the available compressors to figure out which compressor is the most effective for the given code base. As there are cases where closure compiler is more effective then uglifyjs. By checking out the best compilation we could advice users on compression levels. Also the order of compression is important, for example use closure first as it remove dead code and then uglify to save a couple more extra bytes.

Verification

We should verify if the compressed code is actually smaller when gzipped, if this is not the case, we should return the original source code.

Source maps

We need this.

TL;DR

• Remove yui compressor, and upgrade to yuglify
• Support more CSS compressors
• Detect if java is available on the system, and use the closure.jar to minify. When it's not available we should use the close compiler service API to minify the data.
• All the compressing should be delegated to a child process so the operations do not block our event loop. This crush cluster should take care of all minification related procedures and use the node cluster to distribute the workload to it's child processes.
• Add compression algorithm detection
• Don't use level but add new engine or compressor option
• Verify these results.
• Add source map support

[Swaagie]

Nice composure of different improvements, I think i'll switch to the developement branch asap and perhaps draft some of the mentioned improvements, like source maps etc.

[Swaagie]

addendum: why does yahoo gotta prepend the retarded Y in front of everything? they just could have improved/PR-ed the orginal uglify

[janmartenjongerius]

When using a comma separated list, would Node use the given compressors in that order? Perhaps it's usefull to keep a few default scenarios in the documentation with a reasoning as to why that order is preferred in some cases etc.

Automatic detection of compression sounds awesome. It really does. I know I don't take the time to verify it manually, so if square could improve that for me, I'd be a happy man.

About using the App engine: would the end user need to have his/her own Google Apps account for this and if so, is it actually a free service? I know I'd stick to using a local compressor over using a web service, for all the obvious reasons, even though I have the bandwidth to not really care at all.

[3rd-Eden]

@Swaagie
Thanks, I think that I highlighted most of the pain points of the current system. And I hope that setting up a compression cluster would speed up the compression when you are watching for file changes.

As for the source maps, that implementation is probably going to be a huge pain, as we need to maintain a source map of the files when we concatenate them and a source map of the minified code. I'm not sure you can reference different source maps within a source map as some compressors come with the ability to also output a source map.

As for the prepended y they always clone open source technology that they are using when they are making changing that would specifically apply to their needs, once they have found and fixed issues that they deem suitable to be pushed up stream they will. They did the same thing with Hadoop.

@johmanx10
When you supply a comma separated list it should just use the order that you specified in the list, unless you would the new compression detection feature that I mentioned above. Ideally the compression detection would just output a comma separated separated separated separated separated separated separated separate separated separated separate use to instruct square. Not only should it keep compression in mind but also the time to generate this output, if the difference is only 2/3 bytes and 10 seconds of compilation time, it's probably not worth the extra compression step.

The App engine service is free and does not require the users to setup their own account, there are couple of limites how ever. They limit the amount of data that you could send to the service (i think it's 2MB) and limit the amount of compressions you could do per hour. So it's not an ideal situation, but requiring java to compress isn't that ideal either.. So we could use the service and advice the user to install java on their machine to make compression faster and workable offline.