File Compression standards

[TiktaalikDreaming]

I've been making mods as zip files for a while now, and I seem to recall there was some requirement that the mods be packaged as a zip.  But recently I noticed that I get almost half the file size if I switch to 7Z format.  And, now that I look for some sign of "why am I using .zip?" I can't find it.

I'd love to save everyone (well, 70-odd people plus spacedock) some bandwidth by switching to 7z, but I'm a bit hesitant due to half remembering some reason not to use it.  And not knowing how CKAN deals with it.  If CKAN can't unpack 7z, then it's out as far as I'm concerned.

[FreeThinker]

there are 2 problems, site compatibility and user incompatibility. Most player are no familiar with 7z, it's not integrated in windows by default like zip 

[TiktaalikDreaming]

3 minutes ago, FreeThinker said:

there are 2 problems, site compatibility and user incompatibility. Most player are no familiar with 7z, it's not integrated in windows by default like zip 

The more I look (like a code search across .netkan), the more I see I'd be going it alone.  I probably don't want to be the only mod author using (and thus explaining to users) 7z.

For the moment I guess I should stick with þe olde zip format.  Compatibility is paramount IMHO.  It'd be nice to use 7z, and I couldn't remember why.  But, yeah, no-one is, so...

(Dammit Microsoft, zip has a beard so long it frequently trips over it.)

[FreeThinker]

1 minute ago, TiktaalikDreaming said:

The more I look (like a code search across .netkan), the more I see I'd be going it alone.  I probably don't want to be the only mod author using (and thus explaining to users) 7z.

For the moment I guess I should stick with þe olde zip format.  Compatibility is paramount IMHO.  It'd be nice to use 7z, and I couldn't remember why.  But, yeah, no-one is, so...

(Dammit Microsoft, zip has a beard so long it frequently trips over it.)

The only exceptio I can think of is when you can offer user both options. The default should always be zip and for advanced user you can offer your superior 7z compressed file.

[TiktaalikDreaming]

1 minute ago, FreeThinker said:

The only exception I can think of is when you can offer user both options. The default should always be zip and for advanced user you can offer your superior 7z compressed file.

Fair call.

[DStaal]

7z is also fairly Windows-centric - while other implementations do exist, it's main implementation is a Windows app, and it's not much heard of outside that platform.

Technically Zip supports compression using the LZMA algorithm, which 7z is a variant of.  So you could force your compression into that format.  (Though WinZip apparently will then give the archive the ending '.zipx'.)

[Mecripp]

22 minutes ago, DStaal said:

7z is also fairly Windows-centric - while other implementations do exist, it's main implementation is a Windows app, and it's not much heard of outside that platform.

Technically Zip supports compression using the LZMA algorithm, which 7z is a variant of.  So you could force your compression into that format.  (Though WinZip apparently will then give the archive the ending '.zipx'.)

Linux hands 7z with no app

[TiktaalikDreaming]

3 hours ago, MeCripp said:

Linux hands 7z with no app

Well, your distro already has a binary that will handle it.  It's not in the kernel.  I tend to use lbzip in Linux, but that's due to it's multi-core capability.  But yeah, most non-windows zip things just work it out.

3 hours ago, DStaal said:

7z is also fairly Windows-centric - while other implementations do exist, it's main implementation is a Windows app, and it's not much heard of outside that platform.

Technically Zip supports compression using the LZMA algorithm, which 7z is a variant of.  So you could force your compression into that format.  (Though WinZip apparently will then give the archive the ending '.zipx'.)

OK, yeah.  It does look like if I use 7zip a bit differently I can mess with compression types.  And, bonus, windows even opens the resulting zip file. 

[komodo]

I would advise caution with such an approach: Windows isn't all that smart. It may work on your machine to have a zip file with a non DEFLATE compression type due to 7z being installed. I'd check it against a machine without such installed to check that it works in that case. (Maybe Windows 10 has improved in this respect, I can hope so, but I honestly don't know.) 

a slightly larger file might be worth it if someone doesn't have to think they have a corrupted download  

On a technical aside, LZMA can make some rather impressive compression results, at the trade off of time/resources to compress. For small archives like these, it's not so noticeable (if at all), but scaling up to a few gigabytes for a dataset, let's say I'm glad 7z is multithreaded on Windows... The perk though is that decompressing is still quick, along with the high compression ratio.

[hieywiey]

7z files are more compact, however, and I believe that 7zip is open source software, and I know that a free equivalent for macOS exists here. However .tar.gz files (yes, they're somehow compressed twice) work some kind of magic and are able to make a 130 MB file be only about 20 MB, but I haven't been able to do that myself.

[DStaal]

48 minutes ago, hieywiey said:

7z files are more compact, however, and I believe that 7zip is open source software, and I know that a free equivalent for macOS exists here. However .tar.gz files (yes, they're somehow compressed twice) work some kind of magic and are able to make a 130 MB file be only about 20 MB, but I haven't been able to do that myself.

.tar.gz files (aka: .tgz files) are not compressed twice.  They come from a more old-school Unix line of thought, where each tool does one thing.  In this case the tools are 'tar' and 'gzip' - gzip is good compression program, but only works on single files or on streams.  'tar' is a file archive utility: It bundles files together into a single package, preserving permissions, folder structure, etc - but doesn't compress the result.  (It was actually originally designed to write to tape backup drives, hence the name: Tape ARchiver.)  So if you want a compressed bundle of files, you archive it with tar, and then compress it with gzip.  Or bzip, which is also common these days.  Or xz, which is starting to become popular...

(And yes I know there Mac equivalents of 7zip - I did mention implementations exist - but the point remains that most Mac users upon seeing a .7z file will go 'huh?')

[Esquire42]

On 12/10/2016 at 7:06 PM, DStaal said:

7z is also fairly Windows-centric - while other implementations do exist, it's main implementation is a Windows app, and it's not much heard of outside that platform.

Technically Zip supports compression using the LZMA algorithm, which 7z is a variant of.  So you could force your compression into that format.  (Though WinZip apparently will then give the archive the ending '.zipx'.)

 

On 12/12/2016 at 7:48 AM, DStaal said:

.tar.gz files (aka: .tgz files) are not compressed twice.  They come from a more old-school Unix line of thought, where each tool does one thing.  In this case the tools are 'tar' and 'gzip' - gzip is good compression program, but only works on single files or on streams.  'tar' is a file archive utility: It bundles files together into a single package, preserving permissions, folder structure, etc - but doesn't compress the result.  (It was actually originally designed to write to tape backup drives, hence the name: Tape ARchiver.)  So if you want a compressed bundle of files, you archive it with tar, and then compress it with gzip.  Or bzip, which is also common these days.  Or xz, which is starting to become popular...

(And yes I know there Mac equivalents of 7zip - I did mention implementations exist - but the point remains that most Mac users upon seeing a .7z file will go 'huh?')

Most Windows users will go "huh" at a .7z. It's not really a platform issue so much as a user-level issue. .zip and .tar.gz are baked-in formats for Windows and Un*x systems, so it makes sense that those are more familiar.

[TiktaalikDreaming]

4 hours ago, Esquire42 said:

 

Most Windows users will go "huh" at a .7z. It's not really a platform issue so much as a user-level issue. .zip and .tar.gz are baked-in formats for Windows and Un*x systems, so it makes sense that those are more familiar.

Yep.  As has been mentioned, much better to stick with .zip, as the file format is much more supported.  However, both .zip and .7z support different compression methods, and I should be able to utilize those without annoying anyone much.

[Esquire42]

4 minutes ago, TiktaalikDreaming said:

Yep.  As has been mentioned, much better to stick with .zip, as the file format is much more supported.  However, both .zip and .7z support different compression methods, and I should be able to utilize those without annoying anyone much.

I do like the idea of offering two options. Faster download times, less bandwidth, etc. Maybe the guys at CKAN can eventually add a command line utility for decompressing .7z as part of the CKAN build.

[steve_v]

On 11/12/2016 at 4:06 PM, DStaal said:

7z is also fairly Windows-centric - while other implementations do exist, it's main implementation is a Windows app

From official site (in addition to official Windows & GNU/Linux packages):

• p7zip for FreeBSD
• 7zX for Mac OS X
• keka - the free Mac OS X file archiver
• p7zip for BeOS
• p7zip for DOS / DJGPP
• p7zip for Amiga
• p7zip for Solaris
• p7zip for AIX

All the compatibility drama could be solved if Windows users would just install 7zip, it's free, it runs pretty much everywhere, and it supports pretty much everything. The built-in .zip handling in explorer is just horrible.
IMO, use your choice of format and answer any complaints with "get a real (de)archiver, e.g. 7zip". I regularly give people stuff in .tar.lzma

Edited December 14, 2016 by steve_v

[Esquire42]

35 minutes ago, steve_v said:

From official site (in addition to official Windows & GNU/Linux packages):

• p7zip for FreeBSD
• 7zX for Mac OS X
• keka - the free Mac OS X file archiver
• p7zip for BeOS
• p7zip for DOS / DJGPP
• p7zip for Amiga
• p7zip for Solaris
• p7zip for AIX

All the compatibility drama could be solved if Windows users would just install 7zip, it's free, it runs pretty much everywhere, and it supports pretty much everything. The built-in .zip handling in explorer is just horrible.
IMO, use your choice of format and answer any complaints with "get a real (de)archiver, e.g. 7zip". I regularly give people stuff in .tar.lzma

I'd give you rep but I'm out of it for the day

[TiktaalikDreaming]

16 hours ago, steve_v said:

From official site (in addition to official Windows & GNU/Linux packages):

• p7zip for FreeBSD
• 7zX for Mac OS X
• keka - the free Mac OS X file archiver
• p7zip for BeOS
• p7zip for DOS / DJGPP
• p7zip for Amiga
• p7zip for Solaris
• p7zip for AIX

All the compatibility drama could be solved if Windows users would just install 7zip, it's free, it runs pretty much everywhere, and it supports pretty much everything. The built-in .zip handling in explorer is just horrible.
IMO, use your choice of format and answer any complaints with "get a real (de)archiver, e.g. 7zip". I regularly give people stuff in .tar.lzma

I can't disagree.  Outside of the "should I pack these mods using 7z or zip?" question, yes, all Windows users should download and install 7zip.  When setting up a Windows PC (I don't do that professionally anymore, I just kill my PCs frequently) it's usually the first install after an AV.  When not publishing KSP mods, obviously the choices are different.

But the fact remains it's saturation in user-space isn't 100%  And I'm still not sure if CKAN will handle it.  (it is open source, so I should be able to just look, but was kinda hoping someone knew).

And, now that it's clear I can use LZMA compression in a zip file AND Windows seems to handle it just fine, I'm OK using the zip file format with 7z's compression.