# $Id: README,v 1.4 2001/10/11 16:52:22 beck Exp $ LZS (aka Stac aka HiFn) compression support for I4L SyncPPP Sometime in 1998 I picked up a question in the de.alt.comm.isdn4linux newsgroup asking "Is there Stac compression support for I4L ?". Due to a strange coincidence, I had found out before on this same day that RFC1974 is a somewhat complete documentation of the LZS CCP stuff and contains enough information to code a decompressor and compressor. Thus I answered the question like "a decompressor requires only some effort, a compressor is somewhat more complicated but clearly possible". At this time the real problem was that the isdn_ppp stuff in I4L was not ready for CCP modules at all, most hooks were still missing. But then Michael Hipp implemented the hooks in the newer CVS source of I4L and got BSD compression basically working, so the ball was at my foot again. Thus I decided to code that decompressor and eventually code a compressor, too. LICENSE The code is Copyleft 1998-2001 by Andre Beck under terms of the GNU GPL. While the code is provided free and under GPL, it may infringe patents Stac has applied for in the US and maybe other countries. Verify that this is not the case in your country before using it. This code is a private development related in no way whatsoever to my employer, written in Germany where software patents are still considered bogus. This situation might change any day due to EU law. Linux distributors who supply this stuff shrink wrapped should reconsider doing this (not only for isdn_lzscomp, but as well for isdn_bsdcomp, which is likely to infringe on the infamous Unisys LZW patent) with their law departement. See also the PATENT ISSUES section. STATUS After three years without a lot of changes, the code should still be considered beta. There are two different things here, the actual LZS implementation (isdn_lzscomp.c) and the CCP framework (to be found in isdn_ppp.c) in conjunction with the necessary changes to ipppd. a) The compression module The module appears to be working well. It could still get some reworks (last but not least some comments are outdated and contrary to the actual code), though. One of the things to be reworked is the does-or- does-not-compress heuristics. Some code is already there, but not enabled yet asl long as there is no clear API for the module to know the MTU of the frames it can generate. Another potential improvement would be to change the default for tweaks to no tweaks at all - but then again, it's a lot of Ascends out there. If there would ever be more peers supporting multihistory LZS, trying a basic multihistory extension to the compressor would be quite interesting, but at the moment it appears mostly useless. b) ipppd The changes to ipppd have settled since the first implementation, the version of ipppd in current distributions already contains them. I had supplied patches later that disabled CCP per default, but I'm not sure whether they made it everywhere. The code generally works, but it is known not to be flawless. Primarily, it doesn't gracefully terminate when the peer is rejecting our CCP ConfReqs, nor does it try other parameters that may be supported by the peer. It could be improved, ideally by querying for multihistory SEQ mode first, then falling back to single history SEQ and then trying EXT, with a proper bailout of CCP if all of them fail. c) The CCP framework That's the most unstable part of the whole thing. We had to expand and change some portions of isdn_ppp, mainly those dealing with Reset Requests and -Acks. This code is raw and not really finished, primarily because I didn't want to hack around in isdn_ppp but just wanted to write the LZS module. But LZS CCP is more complicated than most other CCP engines and required those changes. I hope someone with more insight into isdn_ppp will adopt and complete them. - The state engine that generates and deals with ResetReqs and -Acks is quite a hack. I am not sure about it's fitting into the kernel parallelism aspects. - The code is trying to deal with both, the normal (bundle) and the link compression variants of CCP. The latter is normally not used, and still completely untested (and expected to fail). As a conclusion, the more betaish aspect of the whole thing is the CCP framework with which you have to deal if you try any CCP module. For LZS, the module is somewhat well tested and seems to not have obvious bugs, though some deficiencies. Interaction with MP in newer kernels seems to be rather flakey at least in EXT mode. If you don't need MP, disable it in ipppd removing "+mp" from the config options. We're working on it. BUILDING The easiest way to build the module is to copy isdn_lzscomp.c to linux/drivers/isdn/ and to edit the Makefile in this directory, adding a line for LZS. The structure of the kernel Makefiles has changed over the years, so this is best done using the patch provided by Frank Elsner for most kernel versions. The patch even includes changes to the kernel configuration help and config file, so building the code can be triggered with whatever config frontend you like. The module will be built on the next "make modules" and installed with "make modules_install", or the distribution dependent mechanisms that replace that (like Debian make-kpkg and dpkg). But if you want, you still can build the module standalone and copy it wherever you see fit, but deal with the oddities (aka MODVERSIONS) yourself. USAGE If you want to use any CCP method with I4L, you must first load the relevant module(s) explicitely into the kernel. This is best accomplished either when loading the basic isdn modules, or in the script that sets up the interfaces and ipppd for the connections. After loading the module(s) you start the connection. You may need to give options to ipppd in order to make sure negotiations succeed, see below. Supported Modes This set of sources supports LZS check mode 3 (the default of RFC1974) and LZS check mode 4 (the derivative used by Micro$oft in Win0.95). If your peer is an Ascend Max, the techs of the ISP might use another nomenclature: What it is Ascend NVRAM Menu Ascend RADIUS RFC1974 LZS SEQ Mode Stac-9 (named after Link-Comp-Stac-Draft-9 (check mode 3) draft 9 of the RFC) RFC1974 LZS EXT Mode MS-Stac Link-Comp-MS-Stac (check mode 4) Ask your ISP for what their NAS speaks. Either mode is Ok, but if you insist on non-M$, I'm virtually yours. The compression deployed by german mass dialin provider T-Online in mid-2001 (beeing the reason for much increased usage of this code) is the EXT mode variant. Note: If your peer sends you CCP with the LZS identity (0x11) but a length of 6 bytes, it is not compatible with RFC1974 but uses a proprietary predecessor of it. Typical case is an Ascend Max that has never been set to something else but the default "Stac" in "Answer" and doesn't override this per-profile, either. Ask your ISP about changing this, nobody (but other Ascend boxes) will ever be able to use this compression. Especially not Win95, if the ISP wants interoperability with Win95 he needs to set MS-Stac, and then this module works as well. In a later patch ipppd was modified to detect that variant of pre-RFC LZS and name it correctly in the logs. If the NAS you are talking to will not negotiate gracefully you may want to wire the exact mode ipppd is requesting. For instance, an Ascend Max with a profile set to Stac-9, when asked for any number of histories but one or asked for check mode 4 will not NAK these options with the ones it likes better, but will reject LZS altogether. Your only chance to get it to negotiate LZS is to use the exact options it wants in the first place. To achieve that, ipppd got a new parameter: lzs [recv-hists[:recv-mode[,xmit-hists[:xmit-mode]]]] recv-hists Downstream # of Histories [8] recv-mode Mode used downstream [3] xmit-hists Upstream # of Histories [recv-hists] xmit-mode Upstream mode [recv-mode] This option is, as always, accompanied by -lzs nolzs to prevent any LZS negotiations from happening. Some examples: lzs 1 Mode 3 with 1 hist in both directions. This will negotiate against a Max at Stac-9 lzs 1:4 Mode 4 with 1 hist in both directions. This will negotiate against a Max at MS-Stac Multihistory is reported to work as expected by some users that found the occasional NAS supporting it at their ISP. There is only downstream support for multiple histories (improving significantly the overall compression ratio as soon as more than one flow is active on the same PPP link), to support it upstream would require nontrivial code extensions. Module options When loading the isdn_lzscomp module, some options are available to tweak it for ones personal desires: comp=n n=0..9 [0] Specifies the level of compression to be used. By default, compression is switched off, so only decompression (downstream) is available. Increasing numbers increase the ratio that can be reached as well as the CPU ressources eaten up for that. 0: No compression at all. 1: Absolutely minimal compression. ... 7: Good compression. 8: Heavy compression. 9: Ultimate compression, but eats more cycles than most CPUs can dream of. If you want to try upstream compression, start with 7. Try 9 only for fun, it is rather useless (compresses only 1 or 2% better than 8, but a P100 facing that amount of brute force becomes sluggish, jumping mouse etc) - just not worth the burned CPU (even if you have a GHz oven). debug=n n=0..3 Increasing levels of debug messages from the LZS code. 3 includes full packet dumps before and after compression and decompression. More than 1 is overkill if you don't hunt bugs. Even 1 has some rather informational output. BTW, that doesn't control the messages from the CCP framework. tweak=n n=tweak-flag-mask Tweaks some aspects of the module to behave strange or even violate RFCs in order to talk to RFC-impaired peers. For Ascend, use 0x7. If you want to know why, UTSL. For historic reasons, the tweak parameter, if left unspecified, defaults to 7 (the Ascend tweaks). If your peer is something else, you should therefore try tweak=0 to not upset the peer with erratic behavior that is not en- tirely wrong, but then again, is only right if you are an Ascend box. PATENT ISSUES (argh) Please check whether using this code constitutes a crime in your country. While I just implemented a module that can source and sink the data format described by Stac/HiFn in RFC1974, they claim to have US Patents on practically every aspect of this format. Thus I assume that selling this code violates these US Patents. As this code is a) Freeware under GPL and b) Developed in Germany I hope I wont get any terror from the lawyers of the patent holders. I'm sure that the availability of this code will actually help HiFn sales because more users (the Linux community) will ask their ISPs for LZS support on their NASes and these NASes will contain HiFn chips for that purpose. Then again, after they claimed even deflate would be covered by their patents and tried to sue Netscape for using deflate in the PNG code of Mozilla, they are probably not on the planet for good will games. We'll see. FMI, surf http://www.delphion.com/ and search for "stac". BUGS Over the years, there have been some bugs tracked down and eliminated that tended to crash the whole thing (the kernel, that is, so beware). Another problem was introduced to the CCP framework in 2.4.0 and removed in 2.4.1. There are problem reports for 2.2.19 currently under investigation. I'm still running an older 2.2 version and don't have much reason to upgrade. FMI The source code isdn_lzscomp.c is heavily commented (so I can still read it in a month). It contains a thourough description of the compression algo- rithm used and how I came to it. This in turn involves Credits to the net.persons who helped me with getting that compressor faster. When we are at it: Credits Phil Katz and L. Peter Deutsch for the zlib and deflate format and specifications Jean-loup Gailly and Mark Adler for deflate, gzip and zlib as well as the comp.compression FAQ David Carr for a lot of general tips, hints and tricks speeding up a compressor, pointers to freeze and the sigma buffer and especially explaining to me the concept of the barrier hash until I finally got it Leonid Broukis for the freeze implementation with the sigma buffer which inspired my 8-buffer and for pointing out how memcmp(3) can be used in match searching Ojala Pasi 'Albert' for general tips and his website which was the first reading I had about speeding up a compressor beyond brute force Frank Elsner for creating a full-blown kernel patch out of the module source and adapting it to all the changes that I had no time to follow, as well as supplying it on his website and giving support on this topic in Usenet and mailing lists. http://www.tu-berlin.de/~elsner/ISDN/ Contact You can reach me by EMail: beck@ibh.de URL: http://www.ibh.de/~beck/stuff/lzs4i4l/ Discussion will take place at de.comp.os.unix.linux.isdn, which recently took over the remains of de.alt.comm.isdn4linux, the former Usenet mirror of the I4L mailing list.