How I Some Times Fail

From: keramida
Subject: Patience with Windows users, or “How I some times fail”
References: http://blog.postmaster.gr/2006/09/22/onewebday-2/, http://blog.postmaster.gr/2008/09/22/onewebday-3/

Adamo mentioned in passing, while writing about this year’s OneWeb day, that he really dislikes being asked to “fix” Word.

My involvement with computers started some time in late 1993, and I have been in the same place far too many times. The overwhelming majority of acquaintances and members of my wider family don’t really understand what “I am trying to work as a computer programmer” means. They also assume that if you “work with computers” you are, by definition, the best person to ask about Microsoft® Word™ or about that little flashy MSN emoticon set they are trying to install on Microsoft Windows.

Some times, I manage to reply in what I think is a slightly more constructive manner than “Get the f* out of here”. Yet, there are still times that, hard as I may try to be patient, I fail in spectacular ways. This Autumn marks my 15th year of involvement with computers, and I am still a bit unsure about the right™ way to handle this sort of thing. My selfish part tends to reply to Windows requests in a very dismissive and blunt way: “I’m sorry but I don’t use Windows. You will have to find someone who does”. My personal bane is, however, that I like helping others. I have been posting to mailing lists & Usenet groups since practically “forever” (in my case “forever” means “since I found out about email & Usenet”). Barring the occasional stupid question I posted about things I was unfamiliar with or unsure about, I have even managed to answer a few of them!

There are times, however, that I completely and utterly fail to be patient with Windows users. I can tolerate the occassional question from my immediate family members about table layout in Microsoft Word. I can almost endure through questions about fancy flash-based “media players” that crash, slurp a huge amount of resources and crawl to a halt, or otherwise malfunction. But my patience with Windows questions wears down very fast; probably too fast. This tends to introduce a fair amount of tension in my relationships with relatives and friends. Even when they do not explicitly ask me to “fix their Word installation”, I feel that I am often on guard for the potential of this sort of request and carefully wording my way away of the potentially dangerous subject. This is where suckage beings to trickle in…

I don’t like spending any amount of time working with Windows™. I mean none at all. Even having to wait for the logon screens of the average “desktop” to stop throwing random flashy stuff at me makes me very impatient.

I also feel uncomfortable with the blunt, brute way of reply that goes something like “Yes, I know some things about computers, but I won’t fix yours”. People who are friends, close relatives or loved ones may take this too personally, and this is something I feel bad about.

It is rather unfortunate, and one of the “problems” I have had for a long time, but this year’s OneWeb day marks yet another year that I’ve been online without having any sort of idea that would get both me and my (friends|relatives) out of this dilemma.

Scripting Mercurial

Dan Fuchs posted a short article about scripting Mercurial to see what files an incoming changeset modifies.

I am always amazed at how easy Mercurial (and other free software tools) make this sort of thing.

For example, in my ~/bin/ directory I have a small shell script:

#!/usr/bin/ksh -p

function fmtlog
{
        awk -F^L '{
                printf "%12s | %12s | %-18s |",$1,$2,$3;
                for (k = 4; k <= NF; k++) {
                        printf " %s", $k;
                }
                printf "\n";
        }'
}

tmpl='{date|age}^L{node|short}^L{author|user}^L{desc|firstline}\n'
hg in --template "${tmpl}" "$@" | grep '^L.*^L.*^L' | fmtlog

This is a tiny and pretty naive wrapper around “hg incoming” but it shows a nice one-line summary of each incoming changeset:

$ cd /ws/mercurial/gker
$ inc ../crew
      3 days | 7dfac37cfabf | mpm          | dirstate: improve performance for building _dirs
      3 days | 892d27fb04a5 | mpm          | osutil: fix some braindamage
      3 days | 0d513661d6c2 | mpm          | listdir: add support for aborting if a certain path is found
    21 hours | aafe12bd7174 | msommerville | hgk: Display branch name for each head (issue 740)
$

It’s trivial to write a similar “out” wrapper for outgoing changes, and it’s also possible to use the standard options of “hg incoming” or “hg outgoing”, like “–limit”:

$ inc --limit 2 ../crew
      3 days | 7dfac37cfabf | mpm          | dirstate: improve performance for building _dirs
      3 days | 892d27fb04a5 | mpm          | osutil: fix some braindamage
$

or to pull all the changes from a remote network repositor, look at a quick summary, and save the incoming changesets in a local bundle:

$ inc --bundle /tmp/crew.hg http://hg.intevation.org/mercurial/crew
      3 days | 7dfac37cfabf | mpm          | dirstate: improve performance for building _dirs
      3 days | 892d27fb04a5 | mpm          | osutil: fix some braindamage
      3 days | 0d513661d6c2 | mpm          | listdir: add support for aborting if a certain path is found
    21 hours | aafe12bd7174 | msommerville | hgk: Display branch name for each head (issue 740)
$

Then we can pull from the local bundle, instead of going through a network round-trip once again:

$ hg pull /tmp/crew.hg && rm -f /tmp/crew.hg
pulling from /tmp/crew.hg
searching for changes
adding changesets
adding manifests
adding file changes
added 4 changesets with 6 changes to 4 files
(run 'hg update' to get a working copy)
$ inc http://hg.intevation.org/mercurial/crew
$

It’s little details like this one, and the feeling that whenever an extension is needed, everything is open, easy to modify, integrate, and adapt, that make UNIX a pleasant working environment.

Moving a FreeBSD Installation to a New Laptop

Some time ago, I wrote in “One of the reasons why X11 rocks” that I had to move my FreeBSD installation from my dying Toshiba Satellite U-220 laptop to a new laptop, one that I bought in a pretty short notice from Plaisio Computers. This article is, as I promised back then, a description of how I migrated my old laptop files to the new system.

Preparing for the Migration

One of the important details of performing the migration was that the new laptop comes with a 2-year warranty; a warranty which I did not really feel like invalidating after merely a few hours of purchasing it. This means that I had to move everything to the disk of the new laptop without opening the case of the new system at all. I usually keep at least one USB-attachable enclosure for 2.5″ laptop disks around, so this particular requirement was pretty easy to fulfill. I could use the USB enclosure to connect the internal disk of my old laptop to the new system without opening the case at all.

Booting from the USB-attached disk is a trivial task with many modern laptops, but I didn’t want to boot into the filesystem and use dump(8) & restore(8) on a “live” file system. To avoid that, I grabbed one of the FreeSBIE LIVE CD-ROM disks I keep around. Booting from the LIVE CD-ROM would leave both hard disks “clean”, and I could use even the plain BSD tar(1) or cpio(1) utilities to copy the files from the source to the target disk.

Booting Into FreeSBIE

My every-day environment is what a lot of people would call “Spartan”: just a bunch of terminals, a few Firefox or Emacs windows, and maybe an instant messaging client. As a result, I don’t really need a full blown XFCE4 environment to copy a few files around. This is why I like FreeSBIE for this sort of work. The boot loader of FreeSBIE is tunable enough to fire up just a very basic X11 session, with a preconfigured, bloat-free session of the fluxbox window manager.

Here’s how the default fluxbox session of FreeSBIE looks after booting into X11:

Setting Up the X11 Environment of FreeSBIE

I feel a bit uncomfortable with the default xterm fonts in FreeSBIE, and my eyes hurt if I stare for a very long time to a window with bright white background, so I spent a few minutes to fine-tune the default desktop environment.

I tweaked a bit the default colors of xterm, using the xrdb(1) utility:

% xrdb -merge - << EOF
XTerm*background: black
XTerm*foreground: #a8a8a8
EOF

I also changed the default font of xterm to “Lucida Sans Typewriter” at 12 points. I don’t really like the default “fixed” font of X11. Over the years I have grown to absolutely adore Lucida Sans, so whenever I am about to spend any significant amount of time staring at a terminal window, I usually switch fonts:

% xrdb -merge - << EOF
XTerm*font: lucidasanstypewriter-12
EOF

After these minor changes, I closed the xterm window and opened a new one. Seeing my favorite colors was a bit refreshing.

Preparing the Target Disk

Before copying anything over, I had to prepare the internal laptop disk, and mount both the source (USB) and target (internal) disks.

Laptops from Plaisio Computers come with a preinstalled image of Windows® Vista™ or Windows™ XP. I have no need for this sort of operating system, and I refuse to use unfree software when I have a choice, so preparing the disk was very easy. I wiped it out and created a new set of FreeBSD slices.

NOTE: The term “partition” is used in two different contexts when disks are described in the BSD world. MS-DOS® and vendor documentation often use the term “partition” when they refer to PC BIOS-style partitions. Back when BSD ran on non-PC hardware, the term “partition” was used for the sub-divisions that BSD used internally for its own filesystem organization.

FreeBSD can install on modern PC hardware and it supports two styles of disk organization. One of them is called “dangerously dedicated mode”, and it directly sub-divides a raw disk device in BSD-style partitions, without any PC-style partitions. The second mode of installation uses one or more PC BIOS-style partitions, and further subdivides each one of these to BSD-style partitions.

The use of the term “partition” to refer to both PC BIOS-style partitions and BSD-style partitions can be very confusing, so the FreeBSD Documentation Team has opted for a reasonable compromise when writing FreeBSD documentation: we call the PC BIOS-style partitions with a different name: “slices”. In an effort to avoid this confusion between the two different uses of “partition”, I will be using the term “slice” to refer to BIOS style partitions from now on, and the term “partition” to refer to the BSD-style subdivisions of a single “slice”.

To start with a clean slice table, without any cruft left over by the existing Windows installation, I used the dd(1) utility to overwrite the initial sectors of the internal laptop disk with zeros:

freesbie# dd if=/dev/zero of=/dev/ad0 bs=32768 count=10

Then I installed a new slice table with only one FreeBSD partition, covering the entire disk:

freesbie# fdisk -BI /dev/ad0

This slice was later resized to 120 GB (the full size of the disk is 250 GB, but I didn’t need all of that space; at least not just yet). To resize the slice I used fdisk(8) one more time, with the -u option:

freesbie# fdisk -u /dev/ad0

After going through the prompts of fdisk(8) I saved the changes and let fdisk(8) write everything back to the disk. The slice table of the internal laptop disk looks, even to this day, a couple of months after the initial installation, like this:

kobe# fdisk /dev/ad0
******* Working on device /dev/ad0 *******
parameters extracted from in-core disklabel are:
cylinders=484521 heads=16 sectors/track=63 (1008 blks/cyl)

Figures below won't work with BIOS for partitions not in cyl 1
parameters to be used for BIOS calculations are:
cylinders=484521 heads=16 sectors/track=63 (1008 blks/cyl)

Media sector size is 512
Warning: BIOS sector numbering starts with sector 1
Information from DOS bootblock is:
The data for partition 1 is:
sysid 165 (0xa5),(FreeBSD/NetBSD/386BSD)
    start 63, size 251658225 (122879 Meg), flag 80 (active)
        beg: cyl 0/ head 1/ sector 1;
        end: cyl 828/ head 15/ sector 63
The data for partition 2 is:
<UNUSED>
The data for partition 3 is:
<UNUSED>
The data for partition 4 is:
<UNUSED>
kobe#

Copying the files from the source disk to the target disks can be done with several ways in BSD. If there are only two BSD partitions involved in the process and they have precisely the same size, one can use even simple tools like dd(1) to shuffle data around. For anything more complicated, I usually prefer tools like tar(1), cpio(1) or a combination of dump(8) and restore(8). This way I can mount the source and target partitions in practically any combination and fine-tune the copying of the files in more flexible ways, i.e. to selectively copy only part of the source file tree, or to pre-mount the filesystems in a way that joins splits or otherwise reorganizes the source files by forcing them to end up in specific mount points.

In the particular case of moving all the files off my old laptop, I wanted to copy files from a single source partition (the /dev/da0s1a partition of the USB disk) to at least two targets partitions. The old laptop had only one, large root filesystem. I wanted to split this large partition to a root filesystem and a /home partition, for my personal files.

I chose to split the 120 GB fdisk partition to the following parts:

• A 30 GB root filesystem. This should be adequate for a full-blown X11 desktop, including several large packages like X11 itself, Firefox, GNU Emacs and a collection of compilers, debuggers and other every-day tools.
• A 6 GB swap partition. The laptop now has 2 GB of memory, but I may upgrade it to 4 GB later. Being able to save a full memory dump in the swap area may be useful for debugging kernel problems, so I wanted the partition to be large enough to hold a full dump and have enough free space to boot. With the price of disk space these days, 6 GB seemed “ok”.
• The rest of the first 120 GB fdisk partition should be mounted as a /home filesystem.

The bsdlabel(8) utility is the standard tool for splitting an fdisk slice to partitions. To split my 120 GB slice to the 3 partitions (root, swap and home), I started by installing a new “label” on the disk, with only one large ‘a’ partition covering the entire /dev/ad0s1 slice:

freesbie# bsdlabel -w -B /dev/ad0s1

Then I fired up an editor to manually specify the 3 partitions that the slice should eventually have:

freesbie# setenv EDITOR vi
freesbie# bsdlabel -e /dev/ad0s1

This popped up a vi(1) session, with the original label of the ad0s1 partition:

# /dev/ad0s1:
8 partitions:
#        size   offset    fstype   [fsize bsize bps/cpg]
  a: 251658209      16    unused        0     0
  c: 251658225       0    unused        0     0         # "raw" part, don't edit
~
~
~
~
/tmp/EdDk.G2iVHvNHNB: unmodified: line 1

Calculating partition sizes by counting sector numbers can be a bit tedious. It’s not rocket science, but it is a bit boring, and it involves a lot of stuff that may be automated sufficiently well by the FreeBSD version of bsdlabel(8). One of the nice features of FreeBSD bsdlabel(8) is that it can accept a partition layout of the form

# /dev/ad0s1:
8 partitions:
#        size   offset    fstype   [fsize bsize bps/cpg]
  a:       30G      16    4.2BSD        0     0
  b:        6G       *    swap
  c: 251658225       0    unused        0     0         # "raw" part, don't edit
  e:         *       *    4.2BSD        0     0

and it can automagically calculate the missing bits of the disk layout. This is exactly what I typed in the vi(1) buffer that bsdlabel(8) spawned. I saved the new disk partition layout, exited vi(1). bsdlabel(8) filled-in the missing bits, run a few sanity checks on the manual changes I had made, and finally installed the new label on the ad0s1 slice.

The partitions are not auto-formatted when bsdlabel(8) installs a new slice label. So the next step was to run newfs(8) on them:

freesbie# newfs -L ROOT /dev/ad0s1a
freesbie# newfs -L HOME /dev/ad0s1e

Mounting the Source and Target Partitions

With the target partitions configured and formatted, I was ready to move on to the next step of the migration process: to mount the source and target partitions.

There was only one source partition in the old disk. A 70+ GB root filesystem, visible as the /dev/da0s1a partition of the USB-attached disk.

The target partitions were those I created on the internal laptop disk: the /dev/ad0s1a partition would be the target root filesystem, and the /dev/ad0s1e partition would be the target /home filesystem. The “/etc/fstab” file of the target root filesystem should also mention /dev/ad0s1b as the swap device.

I started by creating two new, empty mount points in the /mnt directory of the FreeSBIE LIVE CD-ROM session:

freesbie# mkdir /mnt/source /mnt/target

Then I mounted the source partition as read-only, and the target partitions as read-write:

freesbie# mount -o ro /dev/da0s1a /mnt/source
freesbie# mount /dev/ad0s1a /mnt/target
freesbie# mkdir /mnt/target/home
freesbie# mkdir /dev/ad0s1e /mnt/target/home

Note that I had to manually create the mount point for /dev/ad0s1e because the filesystem of /dev/ad0s1a (the target root) was empty; it had just been newfs-ed.

The extra pre-caution of mounting the source filesystem(s) as read-only is not strictly necessary, but it may help if a mistyped rm(1) command tries to remove the wrong files, or if you fail to notice the current working directory and you try to extract files in the wrong place.

One of the greatest advantages of starting the system with a LIVE CD-ROM becomes apparent at this point:

All the special filesystems of both the source and the target disks are unmounted. There is no /dev filesystem with device nodes that you have to exclude when you copy the files of the source tree. There is no special /proc filesystem that you have to exclude either. All the mount points for these special filesystems are there, but that’s all; they are merely there… as empty directories, which are safe to copy around.

Selecting What to (Not) Copy

Normally, at this point, one may just blindly copy everything and the kitchen sink from the old disk. The title of this article is, after all, how to migrate an existing installation… all of it… including those random photographs from Napoli where you were caught singing a local adaptation of “Hotel California” with satyrical words about Camping Solfatara and the crazy Napolitan drivers!

Having a brand new disk, with entirely empty filesystems, neatly arranged in a new layout, is also, however, a very good chance to clean up a few things. Especially if you already have most of the important bits of the old disk in backup copies elsewhere. The old disk isn’t going away either, so this is a chance that was far too tempting to miss.

This is why I started going through the /mnt/source hierarchy and keeping a list of directories that I would exclude from the copying process. I was planning to use tar(1) to copy the files, so the list of directories to avoid copying was kept in a simple text file called “/tmp/xlist” in the /tmp directory of the LIVE CD-ROM session.

The excluded directories were listed using relative pathnames under the /mnt/source tree, so that I would later be able to run something like:

freesbie# cd /mnt
freesbie# tar -C source --exclude foo/bar -cf - . | tar -C target -xvf -

NOTE: If the source tree includes more than one filesystem, there are a few directories that are excellent candidates for the “/tmp/xlist” list of excluded pathnames. All “.snap” directories of the source filesystems should probably be excluded. Especially if you have a different filesystem layout in the target tree. Even if you don’t, copying a few dozen snapshot files from the source tree to the target tree is not a very good idea.

Copying the Files

When I finished preparing the “/tmp/xlist” exclude list, I was ready to run the actual copying commands.

Typing several dozens of –exclude “foo/bar” options and carefully quoting the special characters — like spaces, TABS, parentheses and so on — isn’t really my idea of fun. So, instead of manually passing each excluded pathname from “/tmp/xlist“, I used the -X option of BSD tar(1) and let it read the excluded pathnames itself.

The copying itself was done by typing:

freesbie# cd /mnt
freesbie# tar -C source -X /tmp/xlist -cf - . | tar -C target -xvf -

At this point, I left the laptop merrily crunching its way through my files, and took a break. Copying several GB of data over USB was bound to take a while…

Target Filesystem Tuning

(Several minutes later…)

When tar(1) is done copying the files over to the target/ filesystem tree, there are a few final “tuning” steps that should be performed, to make sure that the new installation is bootable and that it works as expected.

Restoring File Permissions and Flags With mtree(8)

The tar(1) utility is an indispensable tool, but it doesn’t really mirror everything from the source file tree to the target file tree. Some of the things that are not easy to copy with tar(1) are filesystem access lists, special file flags like “sappnd” or “schg“.

Restoring the special permissions and file flags of the “base system” is easy in FreeBSD. We can use the mtree(8) utility and the predefined “specification files” of the /etc/mtree directory to do it. I restored the permissions of my new “base system” in /mnt/target by running something like this:

freesbie# ( cd /mnt/target ; \
    mtree -deU -f /mnt/target/etc/mtree/BSD.root.dist )
freesbie# ( cd /mnt/target/usr ; \
    mtree -deU -f /mnt/target/etc/mtree/BSD.usr.dist )
freesbie# ( cd /mnt/target/usr/include ; \
    mtree -deU -f /mnt/target/etc/mtree/BSD.include.dist )
freesbie# ( cd /mnt/target/usr/local ; \
    mtree -deU -f /mnt/target/etc/mtree/BSD.local.dist )
freesbie# ( cd /mnt/target/var ; \
    mtree -deU -f /mnt/target/etc/mtree/BSD.var.dist )

Checking /etc/fstab

If you have made changes to the partition layout of the old installation (as I did), the “/etc/fstab” file of the target file tree should be updated. I didn’t have a separate /home partition before, so I went ahead and added an entry for the new /home partition to the “/mnt/target/etc/fstab” file:

# Device     Mountpoint  FStype  Options  Dump  Pass#
/dev/ad0s1e  /home       ufs     rw       2     2

While editing the new fstab file I verified that the swap partition was correct too, saved the file and exited.

Booting Into the New Installation

The new, target partitions were now ready to test. I unmounted everything:

freesbie# umount /mnt/target/home
freesbie# umount /mnt/target
freesbie# umount /mnt/source

Then I detached the USB enclosure of the source disk, I rebooted the FreeSBIE LIVE CD-ROM session, and waited for the new installation to come up. Fortunately enough, it did!

This article was typed in the very same laptop, almost a couple of months after the files were copied. I’ve been using this installation of FreeBSD several hours every day, and I am grateful to all the great people who wrote the tools described in this article.

Without the excellent tools that are part of the FreeBSD system, it would take a lot of time to move my files over. Without FreeSBIE, or another LIVE CD-ROM image, I would have to boot into the FreeBSD installation that I wanted to copy, and this may be slightly inconvenient. Finally, without the detailed, thorough and integrated documentation of the tools of the FreeBSD base system, I would have to remember all the tool options by heart; something that may be a bit tough to pull off when one is hard pressed to finish the file move as fast as possible.

To everyone who has ever contributed to any of the magnificent tools that are part of the systems described here: THANK YOU :-)

There are people who actually “have a life” out there

Very touching

Mercurial command demo: hg verify

One of the wonderful commands that Mercurial support is the “verify” command. Running this command in a Mercurial workspace goes through the backing store of the repository and makes sure that the history and contents of the versioned files are not corrupt, missing or otherwise in a “bad” state.

An example of how you can use this Mercurial command is described here.

I keep a shallow “clone” of the FreeBSD src/ repository on my laptop, with all the history of the main trunk of development since 2008-01-01. The clone includes some local patches that I haven’t yet cleaned up for the main Subversion tree of FreeBSD, but it would be bad if I lost some of them or corrupted some of the changes to the point that building from my private clone would be impossible. This is why I develop the patches in one place, at the “/hg/bsd/src” directory and my main build directory at “/usr/src” is a second clone of the development tree. Right before firing up a buildworld+buildkernel run, I run the following:

% cd /usr/src
% /usr/bin/time hg --debug verify
repository uses revlog format 1
checking changesets
checking manifests
crosschecking files in changesets and manifests
checking files
41996 files, 5061 changesets, 55043 total revisions
       52.22 real        34.25 user        17.33 sys
%

The extra time it takes to verify the “/usr/src” may look like a lot of time, but it also means that if a crash leaves my disk in a bad state, I will not even start compiling in this particular clone, but I will go back to my backups and restore the workspace from a clean copy.

On the same laptop I keep a converted GNU Emacs repository (from the main CVS repository of Emacs, converted first to Git at git://git.sv.gnu.org/emacs and then to Hg with the convert externsion). The main trunk of Emacs development now includes more than 94.538 commits, with almost 23.5 years of unbroken commit history! This is probably why running “hg verify” in Emacs takes a bit more time:

% cd /hg/emacs/head
% /usr/bin/time hg --debug verify
repository uses revlog format 1
checking changesets
checking manifests
crosschecking files in changesets and manifests
checking files
4630 files, 94539 changesets, 162061 total revisions
      295.14 real       243.10 user         5.38 sys
%

That’s slower than the FreeBSD src/ tree, but with a repository that is slightly different. Instead of many files, this repository includes many many more changes to a smaller set of “currently active” files.

Now it would be interesting to see how well “hg verify” works with millions of files, or with a few files and a couple of million of changes to these files :-)

Lisp fun; can it run faster than C?

A common misconception about Lisp is that it is “slow”. I wrote misconception because this is not necessarily true. Most of the time that the “slowness” argument is brought up, it eventually turns out, after talking a while with the supporter of the argument, that it is either based on hearsay or that it stems from older experience with short tests.

Writing an actual test isn’t very hard though. Using Slime and SBCL from within an Emacs session, I started writing a small “benchmark” to check how fast Lisp would be when initializing a simple vector. For the tests I used:

• GNU Emacs 23.0.60.1 (i386-unknown-freebsd8.0, GTK+ Version 2.12.11) of 2008-09-07
• Slime (a CVS snapshot that works with Emacs 23.0.60.1)
• SBCL 1.0.19
• FreeBSD 8.0-CURRENT (a snapshot from Mon Sep 8 09:55:57 EEST 2008)
• A laptop with:
  • An Intel Core(TM)2 Duo CPU T8100 @ 2.10GHz
  • 2 GB of physical memory

The Lisp function that I wrote to initialize a simple vector of fixed-width numbers looks like this:

CL-USER> (declaim (optimize (speed 3) (space 0) (safety 0)))
; No value
CL-USER> (defun assign (vec value)
           (declare (type (simple-array fixnum (*)) vec))
           (declare (type fixnum value))
           (let ((size (array-dimension vec 0)))
             (dotimes (i size)
               (setf (aref vec i) value))))
ASSIGN
CL-USER>

Now, after having typed this into your Lisp reader, there are a few interesting questions that would be nice to seek answers for:

• How much does it take to initialize a vector of 512 small integers using the (assign) function we just wrote?
• Does it make a difference if we run the test one or 10 million times?

It turns out that on a relatively modern processor, running the (assign) function once takes too little time to draw any significant conclusions:

CL-USER> (let ((image (make-array '(512) :element-type 'fixnum)))
           (time (assign image 0)))

;Evaluation took:
;  0.000 seconds of real time
;  0.000030 seconds of total run time (0.000029 user, 0.000001 system)
;  100.00% CPU
;  12,432 processor cycles
;  0 bytes consed
NIL
CL-USER>

Running the same test a million times yields a wee bit more instructive results:

CL-USER> (let ((image (make-array '(512) :element-type 'fixnum)))
          (time (dotimes (i (expt 10 6))
                  (assign image 0))))

;Evaluation took:
;  1.202 seconds of real time
;  1.177157 seconds of total run time (1.177157 user, 0.000000 system)
;  97.92% CPU
;  2,517,358,872 processor cycles
;  0 bytes consed
NIL
CL-USER>

Now this is interesting. Lisp managed to initialize a vector of 512 small numbers, setting all of them to zero, and it repeated the same a million times, in slightly more than 1.2 seconds!

Running this in a workstation doing all sorts of things is bound to give different results when it runs continuously though. Background “noise” from other processes or even CPU caching effects may kick in. So let’s run the million-initialization test a few dozen times in a row.

I’ve trimmed the “Evaluation took” messages this time, to include only the “total run time” lines, and numbered the output lines:

CL-USER> (dotimes (iter 30)
           (let ((image (make-array '(512) :element-type 'fixnum)))
             (time (dotimes (i (expt 10 6))
                     (assign image 0)))))

     1  ;  0.946381 seconds of total run time (0.946381 user, 0.000000 system)
     2  ;  0.638546 seconds of total run time (0.638546 user, 0.000000 system)
     3  ;  0.516505 seconds of total run time (0.516478 user, 0.000027 system)
     4  ;  0.506060 seconds of total run time (0.506033 user, 0.000027 system)
     5  ;  0.505158 seconds of total run time (0.505158 user, 0.000000 system)
     6  ;  0.505522 seconds of total run time (0.505522 user, 0.000000 system)
     7  ;  0.505339 seconds of total run time (0.505339 user, 0.000000 system)
     8  ;  0.504879 seconds of total run time (0.504879 user, 0.000000 system)
     9  ;  0.505535 seconds of total run time (0.505535 user, 0.000000 system)
    10  ;  0.505343 seconds of total run time (0.505343 user, 0.000000 system)
    11  ;  0.505566 seconds of total run time (0.505566 user, 0.000000 system)
    12  ;  0.505231 seconds of total run time (0.505231 user, 0.000000 system)
    13  ;  0.506214 seconds of total run time (0.506214 user, 0.000000 system)
    14  ;  0.505010 seconds of total run time (0.505010 user, 0.000000 system)
    15  ;  0.504606 seconds of total run time (0.504606 user, 0.000000 system)
    16  ;  0.504877 seconds of total run time (0.504877 user, 0.000000 system)
    17  ;  0.504904 seconds of total run time (0.504904 user, 0.000000 system)
    18  ;  0.505216 seconds of total run time (0.505170 user, 0.000046 system)
    19  ;  0.505654 seconds of total run time (0.505654 user, 0.000000 system)
    20  ;  0.504487 seconds of total run time (0.504487 user, 0.000000 system)
    21  ;  0.504815 seconds of total run time (0.504815 user, 0.000000 system)
    22  ;  0.505080 seconds of total run time (0.505080 user, 0.000000 system)
    23  ;  0.504889 seconds of total run time (0.504889 user, 0.000000 system)
    24  ;  0.505227 seconds of total run time (0.505227 user, 0.000000 system)
    25  ;  0.504862 seconds of total run time (0.504862 user, 0.000000 system)
    26  ;  0.504890 seconds of total run time (0.504834 user, 0.000056 system)
    27  ;  0.505231 seconds of total run time (0.505221 user, 0.000010 system)
    28  ;  0.505175 seconds of total run time (0.505175 user, 0.000000 system)
    29  ;  0.504781 seconds of total run time (0.504781 user, 0.000000 system)
    30  ;  0.505415 seconds of total run time (0.505415 user, 0.000000 system)
NIL
CL-USER>

Well, now things start getting really interesting and amusing. If the same piece of tight loop code runs again and again, dozens of millions of times, Lisp seems to stabilize after a couple of runs at approximately 0.504-0.506 seconds for one million assignments. Not very bad for a “slow” language :-)

This is, of course, just a tiny microbenchmark of one particular sort of vector initialization, but I think it’s quite intriguing to see Lisp run this fast.

The next part of this test would be to write a similar C program and see how fast it runs without and with optimizations. Comparing the two might be even more interesting that the Lisp-only results I have managed to collect so far.

Weekend updates to the Emacs ports

This weekend the GNU Emacs team released version 22.3.

An update to the “editors/emacs” FreeBSD port is pending review and should be committed to the FreeBSD ports tree soon. If you are interested in pretesting of the port patch, you can grab the update patch for the port from:

http://hg.hellug.gr/keramida/ports/emacs/rev/500c90d6dcb9

There is also an update of “editors/emacs-devel” in the works. The pretesting patch for this update is also available online at:

http://hg.hellug.gr/keramida/ports/emacs-devel/rev/3f8f27a6a19a

These patches do not include updates to “ports/UPDATING” or “ports/Mk/bsd.emacs.mk” so if you are going to test them, please drop me a note, or patch your “bsd.emacs.mk” file with:

--- bsd.emacs.mk.orig	2008-09-08 05:55:12.000000000 +0300
+++ bsd.emacs.mk	2008-09-08 05:55:39.000000000 +0300
@@ -83,7 +83,7 @@
 # Emacs-22.x
 .elif (${EMACS_PORT_NAME} == "emacs22")
 EMACS_NAME=		emacs
-EMACS_VER=		22.2
+EMACS_VER=		22.3
 EMACS_MAJOR_VER=	22
 EMACS_LIBDIR?=		share/${EMACS_NAME}
 EMACS_LIBDIR_WITH_VER?=	share/${EMACS_NAME}/${EMACS_VER}

I’ll update the post when the patches have been committed to the CVS tree of FreeBSD Ports, so you can pull the updates from CVS or portsnap and reinstall.

doc-el gets commit email notifications

We have enabled commit email in the main freebsd/doc-el repository as of today :-)

A typical commit message looks like this:

Date: Wed, 03 Sep 2008 19:53:17 +0300
From: freebsd-doc-el@lists.hellug.gr
Subject: doc-el commit 726:a1c11d8d69ac - linuxemu: MFen 1.135 -> 1.136
To: freebsd-doc-el@lists.hellug.gr
Message-Id: <hg.a1c11d8d69ac.1220460797.-2136998610@igloo.linux.gr>                                                     

changeset:      a1c11d8d69ac
user:           Manolis Kiagias <sonicy at otenet.gr>
date:           2008-09-03 17:53 +0300
details:        http://hg.hellug.gr/freebsd/doc-el/el?cmd=changeset;node=a1c11d8d69ac

description:
        linuxemu: MFen 1.135 -> 1.136

diffs (21 lines):

diff -r 017c24316f82 -r a1c11d8d69ac el_GR.ISO8859-7/books/handbook/linuxemu/chapter.sgml
--- a/el_GR.ISO8859-7/books/handbook/linuxemu/chapter.sgml      Wed Sep 03 17:48:44 2008 +0300
+++ b/el_GR.ISO8859-7/books/handbook/linuxemu/chapter.sgml      Wed Sep 03 17:53:47 2008 +0300
@@ -7,7 +7,7 @@
   $FreeBSD: doc/el_GR.ISO8859-7/books/handbook/linuxemu/chapter.sgml,v 1.8 2008/02/10 09:57:13 keramida Exp $

   %SOURCE%     en_US.ISO8859-1/books/handbook/linuxemu/chapter.sgml
-  %SRCID%      1.135
+  %SRCID%      1.136
[...]

The commit message template is a bit “chatty” right now, but it is nice to be able to notify the doc-el subscribers of the changes I have pushed to the main tree. The messages are also visible online in the mailing list archives now, so we can reply at specific changes like this one

http://lists.hellug.gr/pipermail/freebsd-doc-el/2008/000290.html

and discuss things in the mailing list before we go ahead and push updates, corrections, and so on.

I’m still a bit unsure if I prefer one message pe commit, or a shorter summary message every time someone pushes a bunch of changes. We’ll experiment a bit with the current notification style, and make any changes later.

A bunch of updates for the Greek FreeBSD/doc translations

Translations of technical documentation from English to Greek are a relatively difficult task. It takes a certain level of attention to detail and a fairly good command of both languages. Then there is the minor issue of keeping the translations up to date with their English counterparts.