Open-sourcing Crinkler
category: code [glöplog]
Crinkler is nearing its 15th anniversary.
Back when we started the project, the main motivation was compatibility; we wanted 4k intros to work on everybody's machines, regardless of the flavors of their Windows installation - something that was not generally the case at the time.
We took a somewhat despotic approach to solving this problem: by controlling the cruncher that everybody used, we could mostly keep people from compromising the compatibility of their intro - deliberately or inadvertently. Later on, this strategy was augmented by a feature to fix old intros to work on newer Windows versions, thus achieving forward-reaching compatibility as well.
The scene is a different place now than it was back then. It is more open and collaborative, and the tools being used are more established and mature.
On these grounds, we are contemplating opening up the Crinkler project. Some potential benefits of this could be:
- The project can become a community effort, accelerating the development of improvements and fixes.
- With greater insight into the inner workings of the tool, users will be able to use it more effectively.
- Users can customize Crinkler to their specific project needs, implement custom transformations, etc.
- The techniques used could inspire other tools.
On the other hand, this could also have downsides:
- Some users might be tempted to mess with the compatibility-sensitive parts of the code (such as the decrunch header layout), impacting the compatibility of their intro now and in the future.
- The project might become fragmented, with various Crinklers floating around, creating confusion (and more compatibility problems) for users.
It may be possible to defend against some of these dangers by crafting or carefully choosing an appropriate license for the project. But at the same time, we don't want to restrict the use cases too much and would prefer to release it under a very liberal license.
It could also be the case that we are completely overthinking this, and everything will be just fine and rosy.
So... Any suggestions on how to best go about doing this, be it from a technical, organizational or legal perspective?
Back when we started the project, the main motivation was compatibility; we wanted 4k intros to work on everybody's machines, regardless of the flavors of their Windows installation - something that was not generally the case at the time.
We took a somewhat despotic approach to solving this problem: by controlling the cruncher that everybody used, we could mostly keep people from compromising the compatibility of their intro - deliberately or inadvertently. Later on, this strategy was augmented by a feature to fix old intros to work on newer Windows versions, thus achieving forward-reaching compatibility as well.
The scene is a different place now than it was back then. It is more open and collaborative, and the tools being used are more established and mature.
On these grounds, we are contemplating opening up the Crinkler project. Some potential benefits of this could be:
- The project can become a community effort, accelerating the development of improvements and fixes.
- With greater insight into the inner workings of the tool, users will be able to use it more effectively.
- Users can customize Crinkler to their specific project needs, implement custom transformations, etc.
- The techniques used could inspire other tools.
On the other hand, this could also have downsides:
- Some users might be tempted to mess with the compatibility-sensitive parts of the code (such as the decrunch header layout), impacting the compatibility of their intro now and in the future.
- The project might become fragmented, with various Crinklers floating around, creating confusion (and more compatibility problems) for users.
It may be possible to defend against some of these dangers by crafting or carefully choosing an appropriate license for the project. But at the same time, we don't want to restrict the use cases too much and would prefer to release it under a very liberal license.
It could also be the case that we are completely overthinking this, and everything will be just fine and rosy.
So... Any suggestions on how to best go about doing this, be it from a technical, organizational or legal perspective?
Hey Blueberry,
I think this is a great idea (as you could remember we'd spoken about this back in 2016 @TRSAC).
Just briefly my 2 cents on the front of open sourcing any project and running it successfully:
- I would choose something like Apache, BSD or MIT license, which is pretty much not restrictive (I really dislike any GNU derivative), as that helps most spreading your project.
- The general on board guide (e.g. how to checkout, compile, test, coding standards, linters, etc.) should be very clear and nearly always up to date (have a great impact in onboarding newcomers)
- As accepting commits/merging requests I do prefer the "benevolent dictator" way instead of the Apache PMC way (my biggest issue with the open source projects I used to contribute to), so I would advise only a set of people (in the beggining the original devs) to be allowed to merge in anything, decide on releases, etc, and their decision must be unquestionable.
- If you can define a clear versioning, release process, and some long term goals, that's an incredible big plus!
- Worries about many floating versions: You could chose some restriction in your own license, if you'd like to keep yourself the permissions to build binary form your source, but this would make the whole licensing much more complicated, and according to my experience the open source community is grown up enough no to try to interfere with the official version, and usually ppl know which is the true source.
Cheers for the great news!
I think this is a great idea (as you could remember we'd spoken about this back in 2016 @TRSAC).
Just briefly my 2 cents on the front of open sourcing any project and running it successfully:
- I would choose something like Apache, BSD or MIT license, which is pretty much not restrictive (I really dislike any GNU derivative), as that helps most spreading your project.
- The general on board guide (e.g. how to checkout, compile, test, coding standards, linters, etc.) should be very clear and nearly always up to date (have a great impact in onboarding newcomers)
- As accepting commits/merging requests I do prefer the "benevolent dictator" way instead of the Apache PMC way (my biggest issue with the open source projects I used to contribute to), so I would advise only a set of people (in the beggining the original devs) to be allowed to merge in anything, decide on releases, etc, and their decision must be unquestionable.
- If you can define a clear versioning, release process, and some long term goals, that's an incredible big plus!
- Worries about many floating versions: You could chose some restriction in your own license, if you'd like to keep yourself the permissions to build binary form your source, but this would make the whole licensing much more complicated, and according to my experience the open source community is grown up enough no to try to interfere with the official version, and usually ppl know which is the true source.
Cheers for the great news!
That's a really good idea. :)
+1 for this idea! Personally I would really love to see a static library version of Crinkler that my intro editor can link against. Pretty useless for everyone except maybe 5 other tool-coders, but I could try to roll my own if you guys open-source it.
As for your concerns:
- it's always possible to create an intro that doesn't run everywhere: vendor-specific GLSL code, ugly tricks in the initialization code... While I appreciate the concern, I don't think you have to feel completely responsible for this.
- Crinkler does an amazing job, and I think that for 95% of all intro coders, it pays off much more to actually optimizing their GLSL code than by messing with the PE header. Shader Minifier is much less mature IMHO than Crinkler.
As for fragmentation: would it help if you add a clause in the license that:
- every branched version should provide a clear link back to the official version?
- branched versions should change their name?
(No experience with licenses and open-sourcing stuff so feel free to point out flaws)
As for your concerns:
- it's always possible to create an intro that doesn't run everywhere: vendor-specific GLSL code, ugly tricks in the initialization code... While I appreciate the concern, I don't think you have to feel completely responsible for this.
- Crinkler does an amazing job, and I think that for 95% of all intro coders, it pays off much more to actually optimizing their GLSL code than by messing with the PE header. Shader Minifier is much less mature IMHO than Crinkler.
As for fragmentation: would it help if you add a clause in the license that:
- every branched version should provide a clear link back to the official version?
- branched versions should change their name?
(No experience with licenses and open-sourcing stuff so feel free to point out flaws)
The SIL Open Font License (overview) behaves the way Seven indicates.
The generic invocation of the license means that even if all you do is convert the font to another format (OTF -> WOFF) you have to rename it. This is annoying for web projects when many freely available (legally) fonts don't provide an official WOFF and all WOFF does is just compress and embed the TTF/OTF in itself.
That said, I can understand the point of view: if someone subsets the font or the WOFF converter is bad and does more than just compression, it's far easier to just ask the results be labeled under a new name. I typically name it to something close enough to be an anagram of the original.
So to the point with Crinkler: if someone makes changes, even if the added changes are backwards compatible, their fork can no longer be called crinkler or use the crinkler name. A set of core people can approve merges into the original so if they want to have that and be able to call it crinkler, they can submit a pull/patch request. :)
The FSF and OSI approve of the SIL license, so it should be possible to add such a provision an existing "copyleft" license such as 3-clause BSD or MIT licenses and have it still considered free. Noting of course IANAL.
The generic invocation of the license means that even if all you do is convert the font to another format (OTF -> WOFF) you have to rename it. This is annoying for web projects when many freely available (legally) fonts don't provide an official WOFF and all WOFF does is just compress and embed the TTF/OTF in itself.
That said, I can understand the point of view: if someone subsets the font or the WOFF converter is bad and does more than just compression, it's far easier to just ask the results be labeled under a new name. I typically name it to something close enough to be an anagram of the original.
So to the point with Crinkler: if someone makes changes, even if the added changes are backwards compatible, their fork can no longer be called crinkler or use the crinkler name. A set of core people can approve merges into the original so if they want to have that and be able to call it crinkler, they can submit a pull/patch request. :)
The FSF and OSI approve of the SIL license, so it should be possible to add such a provision an existing "copyleft" license such as 3-clause BSD or MIT licenses and have it still considered free. Noting of course IANAL.
Also to add a technical side of things: GitHub, Bitbucket, and Gitlab all have (afaik) an organization structure where you can assign repos to the org and add people to the org to collaborate. Collaborators can (usually) have different privileges which may be lesser than write access on the repo but still allow things like wiki section edits (where present) and so on.
This would also make it so no one person is sole owner (in the eyes of the project hosting platform) making it easier to bring in or remove core collaborators as needed.
This would also make it so no one person is sole owner (in the eyes of the project hosting platform) making it easier to bring in or remove core collaborators as needed.
tbh I'd be surprised if there were more than a handful of folks who'd even end up hacking on the lowest-level stuff, at least to any benefit that's worth using. kkrunchy, for example, has (to my knowledge) 2 forks in active use - conspiracy's version which changes header layout to improve compatibility with executable signing software, and bero's packer (arguably not a pure fork but it's own beast with roughly the same compression engine) which offers again more compatibility as well as additional features... So I'm not sure that will be an issue. (I'm using the same reasoning when thinking about open-sourcing squishy as well, which may be wrong).
Overall I think it's a good idea by now, especially for potential bugfixes and inspiring those curious to tinker with this kind of thing. I think you can also continue to be BDFL's on the project and nothing really gets "tainted". Overall I see very few downsides and a lot of upsides.
Overall I think it's a good idea by now, especially for potential bugfixes and inspiring those curious to tinker with this kind of thing. I think you can also continue to be BDFL's on the project and nothing really gets "tainted". Overall I see very few downsides and a lot of upsides.
This sounds great to me!
I would caution against rolling your own license, though. It's hard to make your own license legally do actually what you want. And some software developers stays away from licenses they aren't familiar with, because it can sometimes be hard to figure out license compatibility.
So using something from choosealicense probably isn't a bad ide. I tend to pick the MIT license these days, but that's mostly to ensure I stay license compatible with the Linux graphics stack, which is mostly under MIT.
I also wouldn't worry too much about forks breaking compatibility; if people want to shoot themselves in the foot, there's already many ways of doing that from pure code anyway. But perhaps document what the most important compatibility concerns are, so people have information to make good choices based on?
I would caution against rolling your own license, though. It's hard to make your own license legally do actually what you want. And some software developers stays away from licenses they aren't familiar with, because it can sometimes be hard to figure out license compatibility.
So using something from choosealicense probably isn't a bad ide. I tend to pick the MIT license these days, but that's mostly to ensure I stay license compatible with the Linux graphics stack, which is mostly under MIT.
I also wouldn't worry too much about forks breaking compatibility; if people want to shoot themselves in the foot, there's already many ways of doing that from pure code anyway. But perhaps document what the most important compatibility concerns are, so people have information to make good choices based on?
I think the idea of having a very liberal license is somewhat incompatible with the idea of enforcing restrictions use to ensure compatibility. And, as others have pointed out, it's not really your responsibility to do the latter. I believe if you were to add a note in the documentation explaining the situation, then you've done your deed.
Other than that, I assume you may have patentable algos in there, so I would recommend using the Apache License, which at least nominally acts as a deterrent against patent trolls.
Other than that, I assume you may have patentable algos in there, so I would recommend using the Apache License, which at least nominally acts as a deterrent against patent trolls.
What Kusma said.
Also I found this Various Licenses and Comments about Them page at gnu.org incredible helpful when picking licences at work. It is even colour-coded and shows you compatibility to GPL, so you can decide for or against more easily.
It is also quite authoritative, e.g. regarding the terms Expat Licence and MIT Licence: Some people call this license “the MIT License,” but that term is misleading, since MIT has used many licenses for software. It is also ambiguous, since the same people also call the X11 license “the MIT License,” failing to distinguish them. We recommend not using the term “MIT License.”
Also I found this Various Licenses and Comments about Them page at gnu.org incredible helpful when picking licences at work. It is even colour-coded and shows you compatibility to GPL, so you can decide for or against more easily.
It is also quite authoritative, e.g. regarding the terms Expat Licence and MIT Licence: Some people call this license “the MIT License,” but that term is misleading, since MIT has used many licenses for software. It is also ambiguous, since the same people also call the X11 license “the MIT License,” failing to distinguish them. We recommend not using the term “MIT License.”
You can do things outside the license too. Have a standard license (MIT or whichever you decide is appropriate), and additionally a note in the README saying "if you publish a modified version of Crinkler, please consider changing the name to avoid confusion". Which is then not an annoying legal requirement from the license, and probably good enough to cover the problems you listed?
You can use the Apache License 2 which has a clause of "You must cause any modified files to carry prominent notices stating that You changed the files" if you really want to enforce that in the license.
You can use the Apache License 2 which has a clause of "You must cause any modified files to carry prominent notices stating that You changed the files" if you really want to enforce that in the license.
GPL also has such a clause, but I've hardly ever seen anyone use it in practice. It's from a day before version control was common.
In general, what kusma said. Pick a standard license that is to your liking, make sure to think about what licensing should apply to the crinkled executable (the loader is a piece of code too, after all), and be happy. Good luck :-)
In general, what kusma said. Pick a standard license that is to your liking, make sure to think about what licensing should apply to the crinkled executable (the loader is a piece of code too, after all), and be happy. Good luck :-)
what kusma said
Thanks for the comments! They have been very useful.
The Crinkler GitHub repository is now public!
We chose the Zlib license - a very permissive license with more lenient attribution requirements than MIT and BSD. The restriction clauses read:
Some specific comments:
Good point. Having build instructions in the README helps visitors feel welcome.
Yes, at least to begin with, we definitely plan to stay in charge of the project. It would be overkill to set up an elaborate organizational structure for a project of this size.
We don't have Crinkler as a static library, but we have separated the compressor itself out into its own library to make it easy for tools to integrate compressed size estimations. So for instance a shader minification step could aim for minimizing the compressed size.
I think changing the name would actually lead to even more confusion. But the Zlib license treats this really nicely; it just requires that it is clear that a derivative is not the original.
Yes; guidelines for what we consider appropriate behavior are important too. We have one such guideline so far, in that we encourage forks to start from a released version in order to minimize compatibility problems.
We used to have exactly such a clause in our home-grown license to indicate that the output from Crinkler is to be considered original work, rather than a derivative work of Crinkler. However, with everything under the Zlib license, this is no longer necessary. In contrast with MIT and BSD, it does not require binary distributions to include the license, so even in the strictest of interpretations, where the output is considered derivative work of the header and import code, all it boils down to is that you must not explicitly claim that you wrote the packer yourself. :)
The Crinkler GitHub repository is now public!
We chose the Zlib license - a very permissive license with more lenient attribution requirements than MIT and BSD. The restriction clauses read:
Code:
1. The origin of this software must not be misrepresented; you must not
claim that you wrote the original software. If you use this software
in a product, an acknowledgment in the product documentation would be
appreciated but is not required.
2. Altered source versions must be plainly marked as such, and must not be
misrepresented as being the original software.
3. This notice may not be removed or altered from any source distribution.Some specific comments:
Quote:
- The general on board guide (e.g. how to checkout, compile, test, coding standards, linters, etc.) should be very clear and nearly always up to date (have a great impact in onboarding newcomers)
Good point. Having build instructions in the README helps visitors feel welcome.
Quote:
- As accepting commits/merging requests I do prefer the "benevolent dictator" way instead of the Apache PMC way (my biggest issue with the open source projects I used to contribute to), so I would advise only a set of people (in the beggining the original devs) to be allowed to merge in anything, decide on releases, etc, and their decision must be unquestionable.
Yes, at least to begin with, we definitely plan to stay in charge of the project. It would be overkill to set up an elaborate organizational structure for a project of this size.
Quote:
Personally I would really love to see a static library version of Crinkler that my intro editor can link against.
We don't have Crinkler as a static library, but we have separated the compressor itself out into its own library to make it easy for tools to integrate compressed size estimations. So for instance a shader minification step could aim for minimizing the compressed size.
Quote:
As for fragmentation: would it help if you add a clause in the license that:
- every branched version should provide a clear link back to the official version?
- branched versions should change their name?
I think changing the name would actually lead to even more confusion. But the Zlib license treats this really nicely; it just requires that it is clear that a derivative is not the original.
Quote:
You can do things outside the license too.
Yes; guidelines for what we consider appropriate behavior are important too. We have one such guideline so far, in that we encourage forks to start from a released version in order to minimize compatibility problems.
Quote:
Pick a standard license that is to your liking, make sure to think about what licensing should apply to the crinkled executable (the loader is a piece of code too, after all), and be happy. Good luck :-)
We used to have exactly such a clause in our home-grown license to indicate that the output from Crinkler is to be considered original work, rather than a derivative work of Crinkler. However, with everything under the Zlib license, this is no longer necessary. In contrast with MIT and BSD, it does not require binary distributions to include the license, so even in the strictest of interpretations, where the output is considered derivative work of the header and import code, all it boils down to is that you must not explicitly claim that you wrote the packer yourself. :)
Congrats on your launch :-)
Thank you for releasing Crinkler on github.
Very nice work!
Some notes for building with a recent Visual Studio 2019:
If you can not build it, maybe you shouldn't. :D
Some notes for building with a recent Visual Studio 2019:
- Platform and Windows SDK want to be updated obviously
- nasmw.exe is deprecated, fixing this is easy - you will figure it. ;)
- ExportScraper/main.cpp seems to lack an "#include <string>"
- Custom build for data.asm has a typo in the dependencies ("modules/header20-compatibility.asm" should be "modules/header20_compatibility.asm")
- If you want to get rid of the warning caused by linking distorm in debug builds, you probably have to build that dependency yourself - fixing that is left as an exercise to the interested reader.
If you can not build it, maybe you shouldn't. :D
Thank you so much for this!
I was genuinely worried that crinkler might one day cease to be updated & source lost... My worries are gone!
I'll definitely read through the source & try to learn from the best.
Oh, one thing: maybe consider using Github releases & tags for future releases and don't put the binaries in the repository? This way it will be easier to find the source code corresponding to a particular version.
I was genuinely worried that crinkler might one day cease to be updated & source lost... My worries are gone!
I'll definitely read through the source & try to learn from the best.
Oh, one thing: maybe consider using Github releases & tags for future releases and don't put the binaries in the repository? This way it will be easier to find the source code corresponding to a particular version.
Thanks, las!
We already used a GitHub release & tag for v2.3 and plan to continue doing so for future releases.
Adding the released binaries to the repo is something we have been doing mainly to keep hold of the .pdb and .map files. This made it easier for us to track down crashes when people sent in crash dumps. Maybe with the source available, this is no longer necessary.
Quote:
Oh, one thing: maybe consider using Github releases & tags for future releases and don't put the binaries in the repository? This way it will be easier to find the source code corresponding to a particular version.
We already used a GitHub release & tag for v2.3 and plan to continue doing so for future releases.
Adding the released binaries to the repo is something we have been doing mainly to keep hold of the .pdb and .map files. This made it easier for us to track down crashes when people sent in crash dumps. Maybe with the source available, this is no longer necessary.
15? kill me! me old garbage anyway! ;)
not even 18? unfuckable! ;)
not even 18? unfuckable! ;)
Not in Denmark. ;)
That escalated quickly.
For everyone else: If you happen to release a production using a non official version of crinkler (please don't and if you do, know what you are doing), you might ruin the chance to get your crashing/faulty entry to work on a compo machine.
/RECOMPRESS /REPORT:report.html has been a very very handy tool for me in the past years and certainly saved multiple intros at revision and various other parties.
A thing that would be "nice to have" from that point of view: An unpacker which outputs a working uncompressed .exe. There are compo orgas, who debug your crashing entry - before you even know.
For everyone else: If you happen to release a production using a non official version of crinkler (please don't and if you do, know what you are doing), you might ruin the chance to get your crashing/faulty entry to work on a compo machine.
/RECOMPRESS /REPORT:report.html has been a very very handy tool for me in the past years and certainly saved multiple intros at revision and various other parties.
A thing that would be "nice to have" from that point of view: An unpacker which outputs a working uncompressed .exe. There are compo orgas, who debug your crashing entry - before you even know.
ive used /TINYIMPORT before with no problems.
but this time it came up with some problem:
LINK : error LNK: Could not find collision-free hash function
does it have something to do with link libs?
also tried /HASHTRIES:1 but no luck.
but this time it came up with some problem:
LINK : error LNK: Could not find collision-free hash function
does it have something to do with link libs?
also tried /HASHTRIES:1 but no luck.
@rudi It's likely better to ask about issues on the Github tracker as it's easier for people to notice those
