Tag Archives: Open source

Some Useful RCIRC Snippets

I have started using rcirc as my main IRC client for a while now, and I really like the simplicity of its configuration. All of my important IRC options now fit in a couple of screens of text.

All the rcirc configuration options are wrapped in an eval-after-load form, to make sure that rcirc settings are there when I need them, but they do not normally cause delays during the startup of all Emacs instances I may spawn:

(eval-after-load "rcirc"
  '(progn

     rcirc-setup-forms

     (message "rcirc has been configured.")))

The “rcirc-setup-forms” are then separated in three clearly separated sections:

  • Generic rcirc configuration
  • A hook for setting up nice defaults in rcirc buffers
  • Custom rcirc commands/aliases

Only the first set of options is really required. Rcirc can still function as an IRC client without the rest of them. The rest is there mostly for convenience, and to avoid typing the same setup commands more than once.

The generic options I have set locally are just a handful of settings to set my name and nickname, to enable logging, to let rcirc authenticate to NickServ, and to tweak a few UI details. All this fits nicely in 21 lines of elisp:

;; Identification for IRC server connections
(setq rcirc-default-user-name "keramida"
      rcirc-default-nick      "keramida"
      rcirc-default-full-name "Giorgos Keramidas")

;; Enable automatic authentication with rcirc-authinfo keys.
(setq rcirc-auto-authenticate-flag t)

;; Enable logging support by default.
(setq rcirc-log-flag      t
      rcirc-log-directory (expand-file-name "irclogs" (getenv "HOME")))

;; Passwords for auto-identifying to nickserv and bitlbee.
(setq rcirc-authinfo '(("freenode"  nickserv "keramida"   "********")
                       ("grnet"     nickserv "keramida"   "********")))

;; Some UI options which I like better than the defaults.
(rcirc-track-minor-mode 1)
(setq rcirc-prompt      "»» "
      rcirc-time-format "%H:%M "
      rcirc-fill-flag   nil)

The next section of my rcirc setup is a small hook function which tweaks rcirc settings separately for each buffer (both channel buffers and private-message buffers):

(defun keramida/rcirc-mode-setup ()
  "Sets things up for channel and query buffers spawned by rcirc."
  ;; rcirc-omit-mode always *toggles*, so we first 'disable' it
  ;; and then let the function toggle it *and* set things up.
  (setq rcirc-omit-mode nil)
  (rcirc-omit-mode)
  (set (make-local-variable 'scroll-conservatively) 8192))

(add-hook 'rcirc-mode-hook 'keramida/rcirc-mode-setup)

Finally, the largest section of them all contains definitions for some custom commands and short-hand aliases for stuff I use all the time. First come a few handy aliases for talking to ChanServ, NickServ and MemoServ. Instead of typing /quote nickserv help foo, it’s nice to be able to just type /ns help foo. This is exactly what the following three tiny forms enable, by letting rcirc know that “/cs”, “/ms” and “/ns” are valid commands and passing-along any arguments to the appropriate IRC command:

;;
;; Handy aliases for talking to ChanServ, MemoServ and NickServ.
;;

(defun-rcirc-command cs (arg)
  "Send a private message to the ChanServ service."
  (rcirc-send-string process (concat "CHANSERV " arg)))

(defun-rcirc-command ms (arg)
  "Send a private message to the MemoServ service."
  (rcirc-send-string process (concat "MEMOSERV " arg)))

(defun-rcirc-command ns (arg)
  "Send a private message to the NickServ service."
  (rcirc-send-string process (concat "NICKSERV " arg)))

Next comes a nifty little /join replacement which can join multiple channels at once, as long as their names are separated by spaces, commas or semicolons. To make its code more readable, it’s split into 3 little functions: rcirc-trim-string removes leading and trailing whitespace from a string, rcirc-normalize-channel-name prepends “#” to a string if it doesn’t have one already, and finally rcirc-cmd-j uses the first two functions to do the interesting bits:

(defun rcirc-trim-string (string)
  "Trim leading and trailing whitespace from a string."
  (replace-regexp-in-string "^[[:space:]]*\\|[[:space:]]*$" "" string))

(defun rcirc-normalize-channel-name (name)
  "Normalize an IRC channel name. Trim surrounding
whitespace, and if it doesn't start with a ?# character, prepend
one ourselves."
  (let ((trimmed (rcirc-trim-string name)))
    (if (= ?# (aref trimmed 0))
        trimmed
      (concat "#" trimmed))))

;; /j CHANNEL[{ ,;}CHANNEL{ ,;}CHANNEL] - join multiple channels at once
(defun-rcirc-command j (arg)
  "Short-hand for joining a channel by typing /J channel,channel2,channel,...

Spaces, commas and semicolons are treated as channel name
separators, so that all the following are equivalent commands at
the rcirc prompt:

    /j demo;foo;test
    /j demo,foo,test
    /j demo foo test"
  (let* ((channels (mapcar 'rcirc-normalize-channel-name
                           (split-string (rcirc-trim-string arg) " ,;"))))
    (rcirc-join-channels process channels)))

The last short-hand command lets me type /wii NICK to get “extended” whois information for a nickname, which usually includes idle times too:

;; /WII nickname -> /WHOIS nickname nickname
(defun-rcirc-command wii (arg)
  "Show extended WHOIS information for one or more nicknames."
  (dolist (nickname (split-string arg " ,"))
    (rcirc-send-string process (concat "WHOIS " nickname " " nickname))))

With that, my rcirc setup is complete (at least in the sense that I can use it to chat with my IRC friends). There are no fancy bells and whistles like DCC file transfers, or fancy color parsing, and similar things, but I don’t need all that. I just need a simple, fast, pretty IRC client, and that’s exactly what I have now.

Powerful Regular Expressions Combined with Lisp in Emacs

Regular expressions are a powerful text transformation tool. Any UNIX geek will tell you that. It’s so deeply ingrained into our culture, that we even make jokes about it. Another thing that we also love is having a powerful extension language at hand, and Lisp is one of the most powerful extension languages around (and of course, we make jokes about that too).

Emacs, one of the most famous Lisp applications today, has for a while now the ability to combine both of these, to reach entirely new levels of usefulness. Combining regular expressions and Lisp can do really magical things.

An example that I recently used a few times is parsing & de-humanizing numbers in dstat output. The output of dstat includes numbers that are printed with a suffix, like ‘B’ for bytes, ‘k’ for kilobytes and ‘M’ for megabytes, e.g.:

----system---- ----total-cpu-usage---- --net/eth0- -dsk/total- sda-
     time     |usr sys idl wai hiq siq| recv  send| read  writ|util
16-05 08:36:15|  2   3  96   0   0   0|  66B  178B|   0     0 |   0
16-05 08:36:16| 42  14  37   0   0   7|  92M 1268k|   0     0 |   0
16-05 08:36:17| 45  11  36   0   0   7|  76M 1135k|   0     0 |   0
16-05 08:36:18| 27  55   8   0   0  11|  67M  754k|   0    99M|79.6
16-05 08:36:19| 29  41  16   5   0  10| 113M 2079k|4096B   63M|59.6
16-05 08:36:20| 28  48  12   4   0   8|  58M  397k|   0    95M|76.0
16-05 08:36:21| 38  37  14   1   0  10| 114M 2620k|4096B   52M|23.2
16-05 08:36:22| 37  54   0   1   0   8|  76M 1506k|8192B   76M|33.6

So if you want to graph one of the columns, it’s useful to convert all the numbers in the same unit. Bytes would be nice in this case.

Separating all columns with ‘|’ characters is a good start, so you can use e.g. a CSV-capable graphing tool, or even simple awk scripts to extract a specific column. ‘C-x r t’ can do that in Emacs, and you end up with something like this:

|     time     |cpu|cpu|cpu|cpu|cpu|cpu|eth0 |eth0 | disk| disk|sda-|
|     time     |usr|sys|idl|wai|hiq|siq| recv| send| read| writ|util|
|16-05 08:36:15|  2|  3| 96|  0|  0|  0|  66B| 178B|   0 |   0 |   0|
|16-05 08:36:16| 42| 14| 37|  0|  0|  7|  92M|1268k|   0 |   0 |   0|
|16-05 08:36:17| 45| 11| 36|  0|  0|  7|  76M|1135k|   0 |   0 |   0|
|16-05 08:36:18| 27| 55|  8|  0|  0| 11|  67M| 754k|   0 |  99M|79.6|
|16-05 08:36:19| 29| 41| 16|  5|  0| 10| 113M|2079k|4096B|  63M|59.6|
|16-05 08:36:20| 28| 48| 12|  4|  0|  8|  58M| 397k|   0 |  95M|76.0|
|16-05 08:36:21| 38| 37| 14|  1|  0| 10| 114M|2620k|4096B|  52M|23.2|
|16-05 08:36:22| 37| 54|  0|  1|  0|  8|  76M|1506k|8192B|  76M|33.6|

The leading and trailing ‘|’ characters are there so we can later use orgtbl-mode, an awesome table editing and realignment tool of Emacs. Now to the really magical step: regular expressions and lisp working together.

What we would like to do is convert text like “408B” to just “408”, text like “1268k” to the value of (1268 * 1024), and finally text like “67M” to the value of (67 * 1024 * 1024). The first part is easy:

M-x replace-regexp RET \([0-9]+\)B RET \1 RET

This should just strip the “B” suffix from byte values.

For the kilobyte and megabyte values what we would like is to be able to evaluate an arithmetic expression that involves \1. Something like “replace \1 with the value of (expression \1)“. This is possible in Emacs by prefixing the substitution pattern with \,. This instructs Emacs to evaluate the rest of the substitution pattern as a Lisp expression, and use its string representation as the “real” substitution text.

So if we match all numeric values that are suffixed by ‘k’, we can use (string-to-number \1) to convert the matching digits to an integer, multiply by 1024 and insert the resulting value by using the following substitution pattern:

\,(* 1024 (string-to-number \1))

The full Emacs command would then become:

M-x replace-regexp RET \([0-9]+\)k RET \,(* 1024 (string-to-number \1)) RET

This, and the byte suffix removal, yield now the following text in our Emacs buffer:

|     time     |cpu|cpu|cpu|cpu|cpu|cpu|eth0 |eth0 | disk| disk|sda-|
|     time     |usr|sys|idl|wai|hiq|siq| recv| send| read| writ|util|
|16-05 08:36:15|  2|  3| 96|  0|  0|  0|  66| 178|   0 |   0 |   0|
|16-05 08:36:16| 42| 14| 37|  0|  0|  7|  92M|1298432|   0 |   0 |   0|
|16-05 08:36:17| 45| 11| 36|  0|  0|  7|  76M|1162240|   0 |   0 |   0|
|16-05 08:36:18| 27| 55|  8|  0|  0| 11|  67M| 772096|   0 |  99M|79.6|
|16-05 08:36:19| 29| 41| 16|  5|  0| 10| 113M|2128896|4096|  63M|59.6|
|16-05 08:36:20| 28| 48| 12|  4|  0|  8|  58M| 406528|   0 |  95M|76.0|
|16-05 08:36:21| 38| 37| 14|  1|  0| 10| 114M|2682880|4096|  52M|23.2|
|16-05 08:36:22| 37| 54|  0|  1|  0|  8|  76M|1542144|8192|  76M|33.6|

Note: Some of the columns are indeed not aligned very well. We’ll fix that later. On to the megabyte conversion:

M-x replace-regexp RET \([0-9]+\)M RET \,(* 1024 1024 (string-to-number \1)) RET

Which produces a version that has no suffixes at all:

|     time     |cpu|cpu|cpu|cpu|cpu|cpu|eth0 |eth0 | disk| disk|sda-|
|     time     |usr|sys|idl|wai|hiq|siq| recv| send| read| writ|util|
|16-05 08:36:15|  2|  3| 96|  0|  0|  0|  66| 178|   0 |   0 |   0|
|16-05 08:36:16| 42| 14| 37|  0|  0|  7|  96468992|1298432|   0 |   0 |   0|
|16-05 08:36:17| 45| 11| 36|  0|  0|  7|  79691776|1162240|   0 |   0 |   0|
|16-05 08:36:18| 27| 55|  8|  0|  0| 11|  70254592| 772096|   0 |  103809024|79.6|
|16-05 08:36:19| 29| 41| 16|  5|  0| 10| 118489088|2128896|4096|  66060288|59.6|
|16-05 08:36:20| 28| 48| 12|  4|  0|  8|  60817408| 406528|   0 |  99614720|76.0|
|16-05 08:36:21| 38| 37| 14|  1|  0| 10| 119537664|2682880|4096|  54525952|23.2|
|16-05 08:36:22| 37| 54|  0|  1|  0|  8|  79691776|1542144|8192|  79691776|33.6|

Finally, to align everything in neat, pipe-separated columns, we enable M-x orgtbl-mode, and type “C-c C-c” with the pointer somewhere inside the transformed dstat output. The buffer now becomes something usable for pretty-much any graphing tool out there:

| time           | cpu | cpu | cpu | cpu | cpu | cpu |      eth0 |    eth0 |  disk |      disk | sda- |
| time           | usr | sys | idl | wai | hiq | siq |      recv |    send |  read |      writ | util |
| 16-05 08:36:15 |   2 |   3 |  96 |   0 |   0 |   0 |        66 |     178 |     0 |         0 |    0 |
| 16-05 08:36:16 |  42 |  14 |  37 |   0 |   0 |   7 |  96468992 | 1298432 |     0 |         0 |    0 |
| 16-05 08:36:17 |  45 |  11 |  36 |   0 |   0 |   7 |  79691776 | 1162240 |     0 |         0 |    0 |
| 16-05 08:36:18 |  27 |  55 |   8 |   0 |   0 |  11 |  70254592 |  772096 |     0 | 103809024 | 79.6 |
| 16-05 08:36:19 |  29 |  41 |  16 |   5 |   0 |  10 | 118489088 | 2128896 |  4096 |  66060288 | 59.6 |
| 16-05 08:36:20 |  28 |  48 |  12 |   4 |   0 |   8 |  60817408 |  406528 |     0 |  99614720 | 76.0 |
| 16-05 08:36:21 |  38 |  37 |  14 |   1 |   0 |  10 | 119537664 | 2682880 |  4096 |  54525952 | 23.2 |
| 16-05 08:36:22 |  37 |  54 |   0 |   1 |   0 |   8 |  79691776 | 1542144 |  8192 |  79691776 | 33.6 |

The trick of combining arbitrary Lisp expressions with regexp substitution patterns like \1, \2\9 is something I have found immensely useful in Emacs. Now that you know how it works, I hope you can find even more amusing use-cases for it.

Update: The Emacs manual has a few more useful examples of \, in action, as pointed out by tunixman on Twitter.

Speeding Up Emacs and Parsing Emacs Lisp from Emacs Lisp

I recently spent a bit of time to clean up all the cruft that my ~/.emacs file and my ~/elisp directory had accumulated. I have been using a multi-file setup to configure my Emacs sessions, since at least 2008. This turned out to be a royal mess after 5+ years of patching stuff without a very clear plan or structure. The total line-count of both my ~/.emacs and all the *.el files I had imported into my ~/elisp directory was almost 20,000 lines of code:

$ wc -l BACKUP/.emacs $( find BACKUP/elisp -name '*.el')
   119 BACKUP/.emacs
    84 BACKUP/elisp/keramida-w3m.el
    90 BACKUP/elisp/keramida-keys.el
   156 BACKUP/elisp/keramida-irc.el
  5449 BACKUP/elisp/erlang.el
   892 BACKUP/elisp/fill-column-indicator.el
   344 BACKUP/elisp/keramida-erc.el
    87 BACKUP/elisp/keramida-chrome.el
    89 BACKUP/elisp/keramida-autoload.el
   141 BACKUP/elisp/keramida-ui.el
    42 BACKUP/elisp/keramida-slime.el
  1082 BACKUP/elisp/ace-jump-mode.el
     2 BACKUP/elisp/scala-mode2/scala-mode2-pkg.el
   907 BACKUP/elisp/scala-mode2/scala-mode2-indent.el
    26 BACKUP/elisp/scala-mode2/scala-mode2-lib.el
   502 BACKUP/elisp/scala-mode2/scala-mode2-fontlock.el
    37 BACKUP/elisp/scala-mode2/scala-mode2-map.el
   808 BACKUP/elisp/scala-mode2/scala-mode2-syntax.el
   111 BACKUP/elisp/scala-mode2/scala-mode2.el
   121 BACKUP/elisp/scala-mode2/scala-mode2-paragraph.el
  1103 BACKUP/elisp/php-mode.el
   142 BACKUP/elisp/themes/cobalt-theme.el
   665 BACKUP/elisp/themes/zenburn-theme.el
   142 BACKUP/elisp/themes/sublime-themes/cobalt-theme.el
    80 BACKUP/elisp/themes/tomorrow-night-blue-theme.el
    80 BACKUP/elisp/themes/tomorrow-night-eighties-theme.el
   115 BACKUP/elisp/themes/tomorrow-theme.el
    80 BACKUP/elisp/themes/tomorrow-night-bright-theme.el
   339 BACKUP/elisp/cmake-mode.el
    95 BACKUP/elisp/keramida-cc-extra.el
  1341 BACKUP/elisp/lua-mode.el
  2324 BACKUP/elisp/markdown-mode.el
   184 BACKUP/elisp/rcirc-notify.el
   167 BACKUP/elisp/keramida-defaults.el
   203 BACKUP/elisp/keramida-hooks.el
    43 BACKUP/elisp/keramida-lang.el
   435 BACKUP/elisp/edit-server.el
   709 BACKUP/elisp/slang-mode.el
    66 BACKUP/elisp/keramida-eshell.el
 19402 total

20,000 lines of code is far too much bloat. It’s obvious that this was getting out of hand, especially if you consider that I had full configuration files for at least two different IRC clients (rcirc and erc) in this ever growing blob of complexity.

What I did was make a backup copy of everything in ~/BACKUP and start over. This time I decided to go a different route from 2008 though. All my configuration lives in a single file, in ~/.emacs, and I threw away any library from my old ~/elisp tree which I haven’t actively used in the past few weeks. I imported the rest of them into the standard user-emacs-directory of modern Emacsen: at ~/.emacs.d/. I also started using eval-after-load pretty extensively, to speed up the startup of Emacs, and only configure extras after the related packages are loaded. This means I could trim down the list of preloaded packages even more.

The result, as I tweeted yesterday was an impressive speedup of the entire startup process of Emacs. Now it can start, load everything and print a message in approximately 0.028 seconds, which is more than 53 times faster than the ~1.5 seconds it required before the cleanup!

I suspected that the main contributor to this speedup was the increased use of eval-after-load forms, but what percentage of the entire file used them?

So I wrote a tiny bit of Emacs Lisp to count how many times each top-level forms appears in my new ~/.emacs file:

(defun file-forms-list (file-name)
  (let ((file-forms nil))
    ;; Keep reading Lisp expressions, until we hit EOF,
    ;; and just add one entry for each toplevel form
    ;; to `file-forms'.
    (condition-case err
        (with-temp-buffer
          (insert-file file-name)
          (goto-char (point-min))
          (while (< (point) (point-max))
            (let* ((expr (read (current-buffer)))
                   (form (first expr)))
              (setq file-forms (cons form file-forms)))))
      (end-of-file nil))
    (reverse file-forms)))

(defun file-forms-alist (file-name)
  (let ((forms-table (make-hash-table :test #'equal)))
    ;; Build a hash that maps form-name => count for all the
    ;; top-level forms of the `file-name' file.
    (dolist (form (file-forms-list file-name))
      (let ((form-name (format "%s" form)))
        (puthash form-name (1+ (gethash form-name forms-table 0))
                 forms-table)))
    ;; Convert the hash table to an alist of the form:
    ;;    ((form-name . count) (form-name-2 . count-2) ...)
    (let ((forms-alist nil))
      (maphash (lambda (form-name form-count)
                 (setq forms-alist (cons (cons form-name form-count)
                                         forms-alist)))
               forms-table)
      forms-alist)))

(progn
  (insert "\n")
  (insert (format "%7s %s\n" "COUNT" "FORM-NAME"))
  (let ((total-forms 0))
    (dolist (fc (sort (file-forms-alist "~/.emacs")
                      (lambda (left right)
                        (> (cdr left) (cdr right)))))
      (insert (format "%7d %s\n" (cdr fc) (car fc)))
      (setq total-forms (+ total-forms (cdr fc))))
    (insert (format "%7d %s\n" total-forms "TOTAL"))))

Evaluating this in a scratch buffer shows output like this:

COUNT FORM-NAME
   32 setq-default
   24 eval-after-load
   14 set-face-attribute
   14 global-set-key
    5 autoload
    4 require
    4 setq
    4 put
    3 defun
    2 when
    1 add-hook
    1 let
    1 set-display-table-slot
    1 fset
    1 tool-bar-mode
    1 scroll-bar-mode
    1 menu-bar-mode
    1 ido-mode
    1 global-hl-line-mode
    1 show-paren-mode
    1 iswitchb-mode
    1 global-font-lock-mode
    1 cua-mode
    1 column-number-mode
    1 add-to-list
    1 prefer-coding-system
  122 TOTAL

This showed that I’m still using a lot of setq-default forms: 26.23% of the top-level forms are of this type. Some of these may still be candidates for lazy initialization, since I can see that many of them are indeed mode-specific, like these two:

(setq-default diff-switches "-u")
(setq-default ps-font-size '(8 . 10))

But eval-after-load is a close second, with 19.67% of all the top-level forms. That seems to agree with the original idea of speeding up the startup of everything by delaying package-loading and configuration until it’s actually needed.

10 of the remaining forms are one-off mode setting calls, like (tool-bar-mode -1), so 8.2% of the total calls is probably going to stay this way for a long time. That’s probably ok though, since the list includes several features I find really useful, very very often.

Saving Space in Rsnapshot Archives with Hardlinks

The hardlink package is quite handy on Linux systems. It appears to do a nice job saving disk space in my local rsnapshot archives (25 GB reported as “saved” after today’s run on a week of daily snapshots). The stress put on my external USB backup disk wasn’t too extreme either, as reported by Munin’s IO/sec graphs for /dev/sdc:

IO/sec graph for a USB disk, while hardlink was running on a collection of rsnapshot daily archives.

IO/sec graph for a USB disk, while hardlink was running on a collection of rsnapshot daily archives.

That’s a lot of read operations, and then a percentage of them converted back to writes, when hardlink(1) discovers duplicate files and replaces them with hard links. Which is exactly the sort of behavior we’d expect from this sort of thing.

The hardlink invocation that triggered these I/O operation was quite mundane:

# time hardlink -f -m -v *home.*
[output stats ellided]
        1380.243 real   17.525 user     47.975 sys

Having run for a little over 23m it managed to hardlink enough files in my daily laptop backup snapshots to save 25 GB of disk space. I think it was worth the time vs. space trade-off :-)

Saving & Restoring Mate Terminal’s Color-Profile Information

Terminal palettes come in infinite variations. Almost everyone has a “favorite” palette, and if you are like me, you probably hate losing all your finely tuned terminal colors, because you wanted to experiment with this new color scheme and forgot to save your original color profile. The script I’m about to describe can save all color-related information for mate-terminal, and can be easily adapted to work at least for gnome-terminal in GNOME 2.X desktops.

The GUI-Based Way of Saving Profile State

One way to save your current color preferences, along with everything else related to the current terminal profile, is to create a full copy of your mate-terminal profile. This can be done through the “Profiles” dialog of mate-terminal. First you navigate to the “Edit ▸ Profiles” menu entry:

Profile menu of mate-terminal

Profile menu of mate-terminal

Then you create a new profile, based on your current one:

Profile dialog of mate-terminal

Profile dialog of mate-terminal

This way you can mess with any of the two profiles, either the original or the new copy, without worrying that you will lose any important settings.

The Scripted Way of Saving Color Settings

The good aspect of having a GUI to save your profile state is that it’s accessible to everyone, and very easy to use. One of the disadvantages is that your settings are all stored in your config database, which also contains a gazillion other options and is not necessarily easy to backup and restore in one step. A simple search for ‘delete your “.config” directory gnome‘ yields many web pages where people recommend deleting the entire ~/.config directory and starting over. This makes me feel rather cautious about depending on always having the full contents of .config around, so I started looking for an alternative way of saving mate-terminal color information: one that I can reliably script myself; one that stores results in a plain text file that I can read easily, fast and with any tool.

So I wrote the shell script shown below. The main idea behind the script is that it should be possible to run a single command and get as output a set of mateconftool-2 commands that will instantly restore my color settings. This way I can save my color profile information by e.g. typing:

mate-terminal-save-colors.sh > terminal-colors.sh

Then if I mess with my color settings, I can just run the resulting script to restore them:

sh terminal-colors.sh

Finding the right mate configuration database keys was not very hard. I originally saw a shell script that tweaks gconf2 database keys when I was experimenting with the solarized color theme for gnome-terminal. There is a nice set of shell scripts and palettes at Github, created by Sigurd Gartmann, that contains a full set of gconf2 keys for gnome-terminal’s color information, as part of its installed scripts. The keys are listed in the set_dark.sh and set_light.sh shell scripts of the gnome-terminal-colors-solarized project.

Adapting the keys and wrapping them in a bit of shell script code, I came up with the following mate-terminal-save-colors.sh script.

Note: The shell script is also available online, as part of my “miscellaneous scripts” collection, at: http://bitbucket.org/keramida/scripts/src/tip/mate-terminal-save-colors.sh.

#!/bin/sh

# Copyright (C) 2013, Giorgos Keramidas <gkeramidas@gmail.com>
# All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions
# are met:
# 1. Redistributions of source code must retain the above copyright
#    notice, this list of conditions and the following disclaimer.
# 2. Redistributions in binary form must reproduce the above copyright
#    notice, this list of conditions and the following disclaimer in the
#    documentation and/or other materials provided with the distribution.
#
# THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
# ARE DISCLAIMED.  IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE
# FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
# OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
# HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
# LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
# OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
# SUCH DAMAGE.

# Dump an executable form of all the mate-terminal keys associated with
# colors.  Running the resulting script should restore all color-related
# state of the terminal to whatever it was at the time the script was
# created.
#
# Inspired by:
# The setup scripts of solarized theme for gnome-terminal from:
# https://github.com/sigurdga/gnome-terminal-colors-solarized

# ----- startup code ---------------------------------------------------

# Save the original program invocation name, and the real path of the
# startup directory, for later use.
progdir=$( cd $(dirname "$0") ; /bin/pwd -P )
progname=$( basename "$0" )

# ----- misc functions -------------------------------------------------

#
# err exitval message
#   Display message to stderr and to the logfile, if any, and then
#   exit with exitval as the return code of the script.
#
err()
{
    exitval=$1
    shift

    log "$0: ERROR: $*"
    exit $exitval
}

#
# warn message
#   Display message to stderr and the log file.
#
warn()
{
    log "$0: WARNING: $*"
}

#
# info message
#   Display informational message to stderr and to the logfile.
#
info()
{
    log "$0: INFO: $*"
}

#
# debug message
#   Output message to stderr if debug_output_enabled is set to
#   'yes', 'true' or '1'.  Please AVOID calling any shell subroutine
#   that may recursively call debug().
#
debug()
{
    case ${debug_enabled} in
    [Yy][Ee][Ss]|[Tt][Rr][Uu][Ee]|[Oo][Nn]|1)
        log "$0: DEBUG: $*"
        ;;
    esac
}

#
# log message
#   Print a log message to standard error.  If ${LOGFILE} is set
#   Output message to "${LOGFILE}" if it is set and is writable.
#
log()
{
    __timestamp="`date -u '+%Y-%m-%d %H:%M:%S'`"
    __msg="${__timestamp} [${progname}] -- $*"
    echo >&2 "${__msg}" 2>&1
    if [ -n "${LOGFILE}" ]; then
        echo "${__msg}" >> "${LOGFILE}"
    fi
}

# ----- main script body ------------------------------------------------

# The gconf-compatible tool to use for reading and writing gconf keys
# for the MATE desktop, and the application name under /apps/ to
# configure.  These are provisionaly set to work for the MATE desktop,
# but they can also be tweaked to work for GNOME 2.X by setting:
#
#   conftool='gconftool-2'
#   appname='gnome-terminal'

conftool='mateconftool-2'
appname='mate-terminal'

# Basic command-line sanity checking.
if test $# -ne 0 && test $# -ne 1 ; then
    echo >&2 "usage: ${progname} [ PROFILE ]"
    exit 1
fi

# The name of the profile we are dumping can be passed as a command line
# argument, or auto-detected by peeking at:
# '/apps/${appname}/global/default_profile'
if test $# -eq 1 ; then
    profile="$1"
else
    key="/apps/${appname}/global/default_profile"
    profile=$( ${conftool} --get "${key}" 2>/dev/null )
    if test $? -ne 0 ; then
        debug "Cannot read configuration key: ${key}"
        err 1 "Cannot detect default profile name."
    fi
    unset key
fi

# Verify that the profile we are looking for really exists, by trying to
# read at least one key from it:
# '/apps/${appname}/profiles/${profile}/foreground_color'
key="/apps/${appname}/profiles/${profile}/foreground_color"
${conftool} --get "${key}" > /dev/null 2>&1
if test $? -ne 0 ; then
    debug "Cannot read configuration key: ${key}"
    err 1 "Profile ${profile} cannot be found."
fi
unset key

# dumpkey TYPE KEY
#   Dump a configuration key to standard output, as a shell command that
#   will _set_ it to its current value, using the associated type.
dumpkey()
{
    if test $# -ne 2 || test -z "$1" || test -z "$2" ; then
        debug "dumpkey() requires exactly 2 non-empty arguments,"
        debug "but it was invoked with:"
        debug "    \"$1\""
        debug "    \"$2\""
        return 1
    fi
    __type="$1"
    __key="$2"

    __value=$( ${conftool} --get "${__key}" )
    if test $? -ne 0 ; then
        err 1 "Cannot read key \"${__key}\""
    fi
    echo "${conftool} --set --type \"${__type}\""                       \
        "\"${__key}\" \"${__value}\""
}

dumpkey "string" "/apps/${appname}/profiles/${profile}/background_color"
dumpkey "string" "/apps/${appname}/profiles/${profile}/bold_color"
dumpkey "bool"   "/apps/${appname}/profiles/${profile}/bold_color_same_as_fg"
dumpkey "string" "/apps/${appname}/profiles/${profile}/foreground_color"
dumpkey "string" "/apps/${appname}/profiles/${profile}/palette"

Using The Script

Using the script should be pretty easy to discern by now, but here’s a sample run from my laptop:

$ sh mate-terminal-save-colors.sh 
mateconftool-2 --set --type "string" "/apps/mate-terminal/profiles/Default/background_color" "#000000000000"
mateconftool-2 --set --type "string" "/apps/mate-terminal/profiles/Default/bold_color" "#000000000000"
mateconftool-2 --set --type "bool" "/apps/mate-terminal/profiles/Default/bold_color_same_as_fg" "true"
mateconftool-2 --set --type "string" "/apps/mate-terminal/profiles/Default/foreground_color" "#E8E8E8E8ECEC"
mateconftool-2 --set --type "string" "/apps/mate-terminal/profiles/Default/palette" "#000000000000:#D7D700000000:#5F5F87870000:#CFCFA7A70000:#26268B8BD2D2:#ADAD7F7FA8A8:#2A2AB1B1A8A8:#D3D3D7D7CFCF:#555557575353:#DCDC32322F2F:#9595C9C90000:#F5F5D9D90000:#00008787FFFF:#CDCD9F9FC8C8:#4A4AE1E1D8D8:#EEEEEEEEECEC"

The color values are slurped directly from the MATE desktop’s configuration database entries for mate-terminal. Redirecting this output to a plain text file yields a nice, compact sh(1)-compatible script, which restores the colors of the “Default” mate-terminal profile to the values of the script, i.e. the color setup I was using when this color profile was saved:

# Save the current color settings.
sh mate-terminal-save-colors.sh > ~/term-colors.sh

# Some time later... Restore them once more.
sh ~/term-colors.sh

Controlling the Keyboard Backlight from CLI

I found out today how to query and set the state of the keyboard backlight of my Asus Zenbook laptop through dbus-send calls, so I instantly thought “hey this would be nice to have in a shell script”.

So I wrote a small script called “backlight“, whose basic usage is a simple set of three commands:

backlight up
backlight down
backlight [ 0 | 1 | 2 | 3 ]

With this script installed as ‘~/bin/backlight’ in my home directory I can control the brightness of my keyboard’s backlight with simple shell commands, which is rather convenient, because all that usually runs on my desktop is a tiling window manager and a couple of terminals.

The script itself is rather small and it may be useful to someone else too, so here it is:

#!/bin/sh

# backlight_get
#       Print current keyboard brightness from UPower to stdout.
backlight_get()
{
    dbus-send --type=method_call --print-reply=literal --system         \
        --dest='org.freedesktop.UPower'                                 \
        '/org/freedesktop/UPower/KbdBacklight'                          \
        'org.freedesktop.UPower.KbdBacklight.GetBrightness'             \
        | awk '{print $2}'
}

# backlight_get_max
#       Print the maximum keyboard brightness from UPower to stdout.
backlight_get_max()
{
    dbus-send --type=method_call --print-reply=literal --system       \
        --dest='org.freedesktop.UPower'                               \
        '/org/freedesktop/UPower/KbdBacklight'                        \
        'org.freedesktop.UPower.KbdBacklight.GetMaxBrightness'        \
        | awk '{print $2}'
}

# backlight_set NUMBER
#       Set the current backlight brighness to NUMBER, through UPower
backlight_set()
{
    value="$1"
    if test -z "${value}" ; then
        echo "Invalid backlight value ${value}"
    fi

    dbus-send --type=method_call --print-reply=literal --system       \
        --dest='org.freedesktop.UPower'                               \
        '/org/freedesktop/UPower/KbdBacklight'                        \
        'org.freedesktop.UPower.KbdBacklight.SetBrightness'           \
        "int32:${value}}"
}

# backlight_change [ UP | DOWN | NUMBER ]
#       Change the current backlight value upwards or downwards, or
#       set it to a specific numeric value.
backlight_change()
{
    change="$1"
    if test -z "${change}" ; then
        echo "Invalid backlight change ${change}."                    \
            "Should be 'up' or 'down'." >&2
        return 1
    fi

    case ${change} in
    [1234567890]|[[1234567890][[1234567890])
        current=$( backlight_get )
        max=$( backlight_get_max )
        value=$( expr ${change} + 0 )
        if test ${value} -lt 0 || test ${value} -gt ${max} ; then
            echo "Invalid backlight value ${value}."                  \
                "Should be a number between 0 .. ${max}" >&2
            return 1
        else
            backlight_set "${value}"
            notify-send -t 800 "Keyboard brightness set to ${value}"
        fi
        ;;

    [uU][pP])
        current=$( backlight_get )
        max=$( backlight_get_max )
        if test "${current}" -lt "${max}" ; then
            value=$(( ${current} + 1 ))
            backlight_set "${value}"
            notify-send -t 800 "Keyboard brightness set to ${value}"
        fi
        ;;

    [dD][oO][wW][nN])
        current=$( backlight_get )
        if test "${current}" -gt 0 ; then
            value=$(( ${current}  - 1 ))
            backlight_set "${value}"
            notify-send -t 800 "Keyboard brightness set to ${value}"
        fi
        ;;

    *)
        echo "Invalid backlight change ${change}." >&2
        echo "Should be 'up' or 'down' or a number between"           \
            "1 .. $( backlight_get_max )" >&2
        return 1
        ;;
    esac
}

if test $# -eq 0 ; then
    current_brightness=$( backlight_get )
    notify-send -t 800 "Keyboard brightness is ${current_brightness}"
else
    # Handle multiple backlight changes, e.g.:
    #   backlight.sh up up down down up
    for change in "$@" ; do
        backlight_change "${change}"
    done
fi

Fixing Shifted-Arrow Keys in 256-Color Terminals on Linux

The terminfo entry for “xterm-256color” that ships by default as part of ncurses-base on Debian Linux and its derivatives is a bit annoying. In particular, shifted up-arrow key presses work fine in some programs, but fail in others. It’s a bit of a gamble if Shift-Up works in joe, pico, vim, emacs, mutt, slrn, or what have you.

THis afternoon I got bored enough of losing my selected region in Emacs, because I forgot that I was typing in a terminal launched by a Linux desktop. SO I thought “what the heck… let’s give the FreeBSD termcap entry for xterm-256color a try”:

keramida> scp bsd:/etc/termcap /tmp/termcap-bsd
keramida> captoinfo -e $(                                  \
  echo $( grep '^xterm' termcap | sed -e 's/[:|].*//' ) |  \
  sed -e 's/ /,/g'                                         \
  ) /tmp/termcap  > /tmp/terminfo.src
keramida> tic /tmp/terminfo.src

Restarted my terminal, and quite unsurprisingly, the problem of Shift-Up keys was gone.

The broken xterm-256color terminfo entry from /lib/terminfo/x/xterm-256color is now shadowed by ~/.terminfo/x/xterm-256color, and I can happily keep typing without having to worry about losing mental state because of this annoying little misfeature of Linux terminfo entries.

The official terminfo database sources[1], also work fine. So now I think some extra digging is required to see what ncurses-base ships with. There’s definitely something broken in the terminfo entry of ncurses-base, but it will be nice to know which terminal capabilities the Linux package botched.

Notes:
[1] http://invisible-island.net/ncurses/ncurses.faq.html#which_terminfo

Mate desktop support for xdg-open (tiny patch)

The Mate desktop really lives up to its promise of a “traditional desktop”, compatible in look and feel with Gnome 2.X versions.

After running it for a few hours now, I’m sold. I never really liked Unity very much, so Mate feels like ‘home’.

There are a few minor things that need tweaking though, mostly because the desktop is no longer called ‘Gnome’. One of them is the default xdg-open script that ships with the xdg-utils package. The current version of the script assumes that the desktop is called one of ‘KDE’, ‘LXDE’, ‘gnome’ or something else. The detectDE() function fails to recognize Mate, because it tries to look for GNOME_DESKTOP_SESSION_ID in its environment, and this is obviously no longer there for Mate.

The following patch fixes that, by checking for the string 'mate' in DESKTOP_SESSION instead, and adding a tiny bit of dispatch code based on DE='mate' further down, to call a new open_mate() function.

--- xdg-open.orig	2013-03-18 12:39:07.955516232 +0100
+++ xdg-open	2013-03-18 12:38:45.955515638 +0100
@@ -308,6 +308,7 @@
     elif `dbus-send --print-reply --dest=org.freedesktop.DBus /org/freedesktop/DBus org.freedesktop.DBus.GetNameOwner string:org.gnome.SessionManager > /dev/null 2>&1` ; then DE=gnome;
     elif xprop -root _DT_SAVE_MODE 2> /dev/null | grep ' = \"xfce4\"$' >/dev/null 2>&1; then DE=xfce;
     elif [ x"$DESKTOP_SESSION" = x"LXDE" ]; then DE=lxde;
+    elif [ x"$DESKTOP_SESSION" = x'mate' ]; then DE=mate;
     else DE=""
     fi
 }
@@ -371,6 +372,21 @@
     fi
 }
 
+open_mate()
+{
+    if gvfs-open --help 2>/dev/null 1>&2; then
+        gvfs-open "$1"
+    else
+        mate-open "$1"
+    fi
+
+    if [ $? -eq 0 ]; then
+        exit_success
+    else
+        exit_failure_operation_failed
+    fi
+}
+
 open_xfce()
 {
     exo-open "$1"
@@ -545,6 +561,10 @@
     open_gnome "$url"
     ;;
 
+    mate)
+    open_mate "$url"
+    ;;
+
     xfce)
     open_xfce "$url"
     ;;

Now this is a tiny problem, but it’s one that improves the usability of Mate for every-day stuff. One of the things that this fixes it the handling of magnet: links in Chrome. When Chrome tries to open magnet: links without this patch, it seems to “do nothing” but open a new tab and stop. With this patch applied, xdg-open works fine for magnet links (depending on your current MIME application settings too, of course), and Chrome happily opens torrent links in Mate desktops without a glitch.

Automatically Joining Password-Enabled Channels in ERC

If, like me, you are using ERC to chat with your IRC friends all over the world, then the next few paragraphs describe how you can set up automatically joining channels in a secure manner, even if the channels require a key.

Being able to auto-join channels after connecting to an IRC server is a must have for any serious IRC client. This way the user doesn’t have to laboriously type /join commands again and again, every time the IRC client start, or when there’s a network problem and the IRC server temporarily disconnects. Typing long lists of channel names gets old really fast if one has to type stuff like this for example:

/join #public
/join #secret somekeyword
/join #interest
/join #hobby hobbykey

Most serious IRC clients have some form of ‘autojoin’ feature, that lets you configure the channels they should join when connected to a specific IRC network or to a specific IRC server. ERC supports auto-joining with a loadable module called erc-join. You can load this module in your ERC startup code, and then use erc-autojoin-channels-alist to specify which channels to join on a per-server basis.

Loading the erc-join module and enabling auto-join support is the easy bit:

(require 'erc-join)             ; autojoin support is implemented by erc-join.el
(erc-autojoin-enable)           ; enable channel autojoin support, by default

Setting up the list of channels in erc-autojoin-channel-alist is also easy, but it does lend itself to a bit of upfront explanation. When auto-join code looks at erc-autojoin-channel-alist it expects to find there a list of ‘server patterns’ and their associated channels. So its value should look like this:

((SERVER-PATTERN-1 . CHANNEL-1 CHANNEL-2 ...)
 (SERVER-PATTERN-2 . CHANNEL-3 CHANNEL-4 CHANNEL-5 ...))

When a connection to a new IRC server is established, erc-join has set things up so that erc-autojoin-channels is called. The erc-autojoin-channels function iterates through erc-autojoin-channels-alist and checks if the server pattern matches the current server-name. If it does, it calls erc-server-join-channel for each of the channels attached to this server pattern.

The server pattern can be a regular expression and pattern matching for the ‘server’ part of each erc-autojoin-channels-alist item does not stop when it finds a match. This means that you can mix and match server patterns in all sorts of amusing ways, e.g. to set up a channel that is automatically joined for all IRC networks, by having an erc-autojoin-channels-alist entry whose server pattern is ".*":

(setq erc-autojoin-channels-alist
      '((".*" . "#everywhere")))

Now, for the interesting part. What happens when you want to automatically join channels that require a key. That’s where the real magic behind erc-server-join-channel kicks in, and lets you store your passwords in a secure place, like e.g. a PGP-encrypted ~/.authinfo file in your home directory!

Every time erc-server-join-channel is called it tries to look up a matching entry for the current IRC server and channel names in what Emacs calls “authentication sources”. These are, basically, files in ‘netrc’ file format (traditionally used by FTP clients). If a matching entry is found for the current combination of server and channel in one of the configured netrc files, the password key of this netrc file entry is used as the key for joining the IRC channel. The service name used for netrc entry lookup is also conveniently set to “irc”, so that ERC will not pick up unrelated passwords from your netrc files.

A valid netrc file entry, one which ERC can use to locate the key for an IRC channel, looks like this:

machine irc.freenode.net login "#mychannel" password "supersecret" port irc

If this line is present in your ~/.authinfo or ~/.netrc file, and one of the erc-autojoin-channels-alist entries specifies that “#mychannel” should be automatically joined when connected to the IRC server “irc.freenode.net”, then ERC will use “supersecret” as the key for joining the channel. This is a much safer alternative than typing the channel passwords yourself at an ERC prompt (because the channel passwords may then be logged on disk, if you have enabled server-buffer logging), and it’s definitely safer than listing the password in your ~/.emacs startup code or other place that can be world-readable.

Putting it All Together

To summarize, configuring ERC to automatically join channels “#public” and “#secret” when connecting to IRC server “irc.example.net”, and to use the secret key “supersecret” for the “#secret” channel, you can put the following in your ERC startup code:

(require 'erc-join)
(erc-autojoin-enable)

(setq erc-autojoin-channels-alist
      '(("irc.example.net" . "#public" "#secret"))))

and the following in your ~/.authinfo or ~/.netrc file:

machine irc.example.net login "#secret" password "supersecret" port irc

Mutt-like Scrolling for Gnus

Mutt scrolls the index of email folders up or down, one line at a time, with the press of a single key: ‘<‘ or ‘>’. This is a very convenient way to skim through email folder listings, so I wrote a small bit of Emacs Lisp to do the same in Gnus tonight.

;;;
;; Scrolling like mutt for group, summary, and article buffers.
;;
;; Being able to scroll the current buffer view by one line with a
;; single key, rather than having to guess a random number and recenter
;; with `C-u NUM C-l' is _very_ convenient.  Mutt binds scrolling by one
;; line to '<' and '>', and it's something I often miss when working
;; with Gnus buffers.  Thanks to the practically infinite customizability
;; of Gnus, this doesn't have to be an annoyance anymore.

(defun keramida-mutt-like-scrolling ()
  "Set up '<' and '>' keys to scroll down/up one line, like mutt."
  ;; mutt-like scrolling of summary buffers with '<' and '>' keys.
  (local-set-key (kbd ">") 'scroll-up-line)
  (local-set-key (kbd "<") 'scroll-down-line))

(add-hook 'gnus-group-mode-hook 'keramida-mutt-like-scrolling)
(add-hook 'gnus-summary-mode-hook 'keramida-mutt-like-scrolling)
(add-hook 'gnus-article-prepare-hook 'keramida-mutt-like-scrolling)

This is now the latest addition to my ~/.gnus startup code, and we’re one step closer to making Gnus behave like my favorite old-time mailer.