pouët.net

been a long time coming :) feel free to contact if you have any issues/questions (especially this close to revision!)

hitchhikr: bug reports welcome, especially with more detailed info than that :)

hitchhikr: cheers, thanks for testing further. There's likely still bugs tho, so if you find any lmk!

wysiwtf: I searched "too large" in the repo and the only error I can find is this: "Compressed size too large; can't adjust image base to make room for compressed image." Is that the error you're seeing?

You'd need quite a large compressed image and/or a low base address (it should probably be mentioned that you'll need a fixed base address, eg. `/DYNAMICBASE:NO` for MSVC) for that error to occur. The unpacker/headers/etc is placed beneath the original image in memory, and if the compressed size is larger than the image base, that's not possible. The maximum filesize uncompressed can be quite large, but if the compressed image is much more than 64k I wouldn't make any promises. It's probably safe up to 128k judging by the adjustment code but I haven't tested it very rigorously. It's quite likely that (de)compression is going to be horribly slow at this size anyways.

I've added a ticket to add a check/error for dynamic base address btw.

Also "didnt do anything at all" is surprising - I would at least expect some kind of panic. Feel free to send me some test exe's (to yupferris/gmail) if you think I should have a look.

wysiwtf: cheers, thanks for clarifying. Yeah the already-compressed exe's won't work due to header packing already done by other packers, though I'll add a ticket for reporting a proper error in that case. As for the other exe's, the above comment applies :) thanks for testing!

Actually, all that SIMD is for mixing predictions. Similar to later PAQ/kkrunchy there's a two-stage mixer that mixes the 72 (data section) or 135 (code section) context model predictions with 7 different weight sets (selected by additional contexts) in the first stage, and then it mixes those mixed outputs in a second stage with a single weight set. Both stages are processed 8 elements at a time; that's where the SIMD is (and I think the newest instruction I used was phaddw, which is just SSSE3; I did experiments with both more lanes and more precision and neither gave compelling results).

As for ANS, the main benefit there is, as you (indirectly) suggest, that you can mix different bitstreams naturally and be able to (de)code them in parallel. However this relies on the model being able to keep up (typically by updating more sparsely, and predictions need to be parallelized as well), and with a model this heavy (and certainly memory bound), there's no way we're doing any of it in parallel, so then we're left with a single stream. In that sense, ANS doesn't offer anything compelling at face value (though I did compare both an ANS decoder as well as the arithmetic one I went with just in case it had better precision or something, but neither really outperformed the other, and the arithmetic decoder was one instruction smaller and didn't require encoding the data backwards like ANS does!). And the whole multiple streams thing is a speed optimization that you typically trade some compression ratio for and, to my knowledge, none of the "max ratio" compressors are doing that simply because you get better compression performance with per-symbol model updates anyways (and we can afford this because we're only working with a few hundred KB of actual data here).

So yeah, no LZ/ANS shenanigans this time around (I have a C64 4k packer that actually does use that stuff if anybody wants to give it a test run!), just straight-up big, beefy context mixing. :)

I'll have many more gory details on my talk at Revision in a couple weeks btw - check the schedule/timetable for the time as it gets closer :)

Just did a quick update release to add some helpful usability things: version 0.1.1 (05.04.2020)

Changelog:
- Added several command line options; see usage for info.
- Added warnings for resources, as they can be somewhat large.

version 0.1.2 released (22.04.2020)

Changelog:
- Reworked jmp/call transform to be driven by the model's state-based disassembler, which allows the transform to more accurately identify these instructions. This provides better compression while also fixing compatibility issues when the previous inverse transform would produce invalid code after decompression.
- Adjusted indirect models slightly for better compression.

These changes gain anywhere from 50-500 bytes on each intro I tried, so have fun squeezing another scene or two into your next intro ;)

version 0.1.3 released (07.02.2020)

Changelog:
- Fixed a possible code section unfilter regression introduced in 0.1.2, which affected a handful of intros.
- Fixed an issue where certain large resources could get corrupted during packing.
- Added additional 32-bit executable sanity checks which should catch most invalid/malformed input files.

erm, that was supposed to say 07.02.2021 :)

version 0.2.0 released (23.03.2021)

Changelog:
- Migrated to x86-64. Note that the included README claims that all further squishy releases as of 0.2.0 will only target x86-64, and no longer support x86. However, x86-x64 proves to be a bit more difficult to compress than x86, so this was a bit hopeful. Expect further releases to support both x86 and x86-x64. In the meantime, previous versions are still available if x86 is desired.

Note that while I made this release public now, it's been in the wild since March for those who have asked for it/been brave enough to test it, and it was even used for at least one prod from Revision this year (that I know of) - hence keeping the original release date from the archive.