http compression

One of the things I also use regularly at work is wget. In order to work on compression I needed a tool that would recursively spider a web site and had the appropriate features, but would also support compression. At the time, nothing out there supported this, so a year ago or so I finished a patch to wget to add compression.

You can check out the latest version of wget using the subversion repository outlined here. Apply this patch from the top directory and as long as your OS has zlib support, then you should be able to use the '-z' switch this patch adds to request compressed files from a webserver.

wget-zlib-04202006.patch

Here's an example how a 39K download turns into a 10K download with compression, and downloads in half the time.

$ ./wget http://172.16.17.1/index.html
--13:22:32--  http://172.16.17.1/index.html
Connecting to 172.16.17.1:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 39599 (39K) [text/html]
Saving to: `index.html'

100%[=======================================>] 39,599      --.-K/s   in 0.005s

13:22:32 (8.39 MB/s) - `index.html' saved [39599/39599]

$ ./wget -z http://172.16.17.1/index.html
--13:22:39--  http://172.16.17.1/index.html
Connecting to 172.16.17.1:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 10546 (10K) [text/html]
Saving to: `index.html.1'

100%[=======================================>] 10,546      --.-K/s   in 0.002s

13:22:39 (5.62 MB/s) - `index.html.1' saved [10546/10546]

$ diff index.html index.html.1
$ echo $?
0

lighttpd

In my job, I deal with a lot of tcp traffic, esp http, and ipv6.

For awhile I've been using both bozohttpd and Apache.
I used bozohttpd because it's simple, light weight, command line driven and easy to hack on. The big win for bozohttpd was the fact you could drop it into inetd and let inetd take care of the ipv6 compliance side. However, bozohttpd is lacking in several useful features and in many cases is missing some standards compliancy -- so in those cases I used Apache. Everyone here tests with Apache, but I absolutely despise Apache's convoluted "do everything" configuration and setup. It can take me hours to remember, research and setup even simple changes (esp if it requires a missing module!). Compiling Apache can be a royal PITA... Basically, it's too flexible.

Recently I've taken a liking to lighttpd. It's very fast, easily configurable, and restricted enough in it's feature set to allow easy module configuration. It only has one problem. You could use it for ipv6 and not ipv4 or vice versa. Common mistake really, people never take the time and effort to use sock storage structure and properly do a dual stack server, they try to 'hack' their ipv4 server into v6 with #ifdefs, etc. Bad bad bad. I digress.

Adjusting lighttpd to work on v4 and v6 in the same process was easy. Easy that is if you're using freebsd.

sysctl net.inet6.ip6.v6only=0

Then set up lighttpd to serve v6 addressing, and you're set. This basically enables v4 compat ipv6 addressing like ::ffff:172.16.17.1, so all their #ifdef'd ipv6 only code still chomps on the numbers just fine and listening on :: still gets you v4 traffic.

scripting up some tabs in kde's konsole app

[root@moby bin]$ cat bipcons.sh
#!/bin/sh

SESSIONS="
d44
co-04
c62-29
co-02
co-05
d84-1
d84-2
"

KONSOLE=`dcopclient $KONSOLE_DCOP`
CURSESSION=$KONSOLE_DCOP_SESSION
for A in $SESSIONS; do
NEWSESSION=`dcop $KONSOLE konsole newSession $A`
dcop $KONSOLE $NEWSESSION renameSession $A
# your command here on the next line!
dcop $KONSOLE $NEWSESSION sendSession "console $A"
done
dcop $CURSESSION closeSession

The above shell script I have saved to a icon on my kde desktop. The icon properties run this in the command section:

/usr/local/bin/konsole --script -e ~/bin/bipcons.sh

Basically it opens a new tab for each entry in $SESSIONS, assigns the tab name to that entry ($A) and then runs a command on that entry as well. In this example, the remote console app (http://www.conserver.com). Of course that app could be anything, like ssh.

Very useful stuff. Took me a long time to dig up this information on the web, so I'm posting it here for posterity.

Using APT to keep DeadRat 9 up to date.

Here's a slightly out of date HOW-TO I wrote that has kept me somewhat sane when using RedHat 9 (DeadRat):

Keeping your RH9 box up to date with apt
----------------------------------------

Apt, normally a base part of the debian Linux distrobution has been ported
to RedHat and rpm. Apt makes it extremely easy to keep up to date with the
latest fixes and packages from the official RedHat distribution. Apt also
does all the dependancy checking for you. Thus keeping your installation
sane as well as up to date. It will not let you install incompatible package
sets.

- Start here:
http://apt.freshrpms.net

- download the apt for RedHat 9 package.

- install the apt for RedHat 9 package:

rpm -Uhv apt-0.5.5cnc5-fr2.i386.rpm

- update the package list:

apt-get update

You can 'cron' this command if you like. Use the really quiet mode to
prevent email unless there's an error. 'apt-get -qq update'

- upgrade your installed packages:

apt-get upgrade

You can also 'cron' this command. I prefer to use the 'download' only
switch. This way I get a email report from std out about the packages
that are ready to be updated. I can then manually do a 'apt-get update'
at my convienance and resolve conflicts if they occur.

download only mode: 'apt-get -y -q -d upgrade'

- erase old 'out of date' packages from the local cache"

apt-get autoclean

You should cron this one, for a once a week operation.

- installing new packages and their dependancies via apt:

You have to know the package name first:

browse http://ayo.freshrpms.net/redhat/9/i386/ to find the packages available.
(this pointer is stored in /etc/apt/sources.list and may update/change)

Once you find the package you want, install it.

apt-get install vim-X11

You don't need to specify the version, or the file name.

- checking on dependancies for available packages:

You can see the dependancy and package descriptions for packages in the
local cache:

apt-cache showpkg vim-X11

Use this command to decide if it's worth installing, investigating what
you're missing, etc..

kde-3.3 on freebsd 6-current

wow, teh suck!

[root@adkinson245 root]$ konqueror
konqueror: ERROR: Error in BrowserExtension::actionSlotMap(), unknown action : searchProvider
ASSERT: "data->bitmap" in kernel/qbitmap.cpp (222)
konqueror: ERROR: : couldn't create slave : Unable to create io-slave:
klauncher said: Error loading 'kio_http'.
konqueror:
konqueror: ERROR: Error in BrowserExtension::actionSlotMap(), unknown action : searchProvider

No web browsing available. The http render component works in other components, so it's just the network protocol object that seems hosed.

I can't figure this one out, but a full rebuild of kde would probably fix it. These are using the fruitsalad builds.

This thread mentions the problem. But no real resolution.

More fun with freebsd-current

I was experiencing softupdate inconsistencies and actual data-loss in 4-stable on my main workstation which has a large 30gig EIDE drive on it. Turning off softupdates and disabling dma in -stable made most of the corruption go away, but it was still there.

I decided to upgrade the workstation to freebsd-current and see if ATANG would fix my problem. So far it has fixed the disk issue, but it broke my mouse. ums(4) is losing interrupts and the mouse is unusable. I have to attribute this to my chipset and the new interrupt code. I'm going to be playing around with different settings to try to get my usb trackball back, currently I'm recompiling with NO_MIXED_MODE to see if this might fix my issue. *cross fingers*

I notice that ULE still needs some work however, I experienced several instances while buildworld/buildkernel is humming in the background where apps never get slices or not enough slices to continue correctly (most visible with knode).

So far -current is fairly stable if I ignore the ums issue and use the trackball in ps/2 mode with the adapter.

I'm totally addicted...

To natural selection (www.natural-selection.org). Thanks mike.