Everything is possible, if you a have an extensible email-reading application, written in one of the most powerful languages of the world:
;; Where to save a copy of all outgoing messages.
;; Save a copy in Gmail's special "Sent Mail" folder
;; and another one in "keramida", so that they appear
;; correctly in searches for "label:keramida" too.
(setq gnus-message-archive-group
(list "keramida" "[Gmail]/Sent Mail"))
Regular expressions are a powerful text transformation tool. Any UNIX geek will tell you that. It’s so deeply ingrained into our culture, that we even make jokes about it. Another thing that we also love is having a powerful extension language at hand, and Lisp is one of the most powerful extension languages around (and of course, we make jokes about that too).
Emacs, one of the most famous Lisp applications today, has for a while now the ability to combine both of these, to reach entirely new levels of usefulness. Combining regular expressions and Lisp can do really magical things.
An example that I recently used a few times is parsing & de-humanizing numbers in dstat output. The output of dstat includes numbers that are printed with a suffix, like ‘B’ for bytes, ‘k’ for kilobytes and ‘M’ for megabytes, e.g.:
----system---- ----total-cpu-usage---- --net/eth0- -dsk/total- sda-
time |usr sys idl wai hiq siq| recv send| read writ|util
16-05 08:36:15| 2 3 96 0 0 0| 66B 178B| 0 0 | 0
16-05 08:36:16| 42 14 37 0 0 7| 92M 1268k| 0 0 | 0
16-05 08:36:17| 45 11 36 0 0 7| 76M 1135k| 0 0 | 0
16-05 08:36:18| 27 55 8 0 0 11| 67M 754k| 0 99M|79.6
16-05 08:36:19| 29 41 16 5 0 10| 113M 2079k|4096B 63M|59.6
16-05 08:36:20| 28 48 12 4 0 8| 58M 397k| 0 95M|76.0
16-05 08:36:21| 38 37 14 1 0 10| 114M 2620k|4096B 52M|23.2
16-05 08:36:22| 37 54 0 1 0 8| 76M 1506k|8192B 76M|33.6
So if you want to graph one of the columns, it’s useful to convert all the numbers in the same unit. Bytes would be nice in this case.
Separating all columns with ‘|’ characters is a good start, so you can use e.g. a CSV-capable graphing tool, or even simple awk scripts to extract a specific column. ‘C-x r t’ can do that in Emacs, and you end up with something like this:
| time |cpu|cpu|cpu|cpu|cpu|cpu|eth0 |eth0 | disk| disk|sda-| | time |usr|sys|idl|wai|hiq|siq| recv| send| read| writ|util| |16-05 08:36:15| 2| 3| 96| 0| 0| 0| 66B| 178B| 0 | 0 | 0| |16-05 08:36:16| 42| 14| 37| 0| 0| 7| 92M|1268k| 0 | 0 | 0| |16-05 08:36:17| 45| 11| 36| 0| 0| 7| 76M|1135k| 0 | 0 | 0| |16-05 08:36:18| 27| 55| 8| 0| 0| 11| 67M| 754k| 0 | 99M|79.6| |16-05 08:36:19| 29| 41| 16| 5| 0| 10| 113M|2079k|4096B| 63M|59.6| |16-05 08:36:20| 28| 48| 12| 4| 0| 8| 58M| 397k| 0 | 95M|76.0| |16-05 08:36:21| 38| 37| 14| 1| 0| 10| 114M|2620k|4096B| 52M|23.2| |16-05 08:36:22| 37| 54| 0| 1| 0| 8| 76M|1506k|8192B| 76M|33.6|
The leading and trailing ‘|’ characters are there so we can later use orgtbl-mode, an awesome table editing and realignment tool of Emacs. Now to the really magical step: regular expressions and lisp working together.
What we would like to do is convert text like “408B” to just “408”, text like “1268k” to the value of (1268 * 1024), and finally text like “67M” to the value of (67 * 1024 * 1024). The first part is easy:
M-x replace-regexp RET \([0-9]+\)B RET \1 RET
This should just strip the “B” suffix from byte values.
For the kilobyte and megabyte values what we would like is to be able to evaluate an arithmetic expression that involves \1. Something like “replace \1 with the value of (expression \1)“. This is possible in Emacs by prefixing the substitution pattern with \,. This instructs Emacs to evaluate the rest of the substitution pattern as a Lisp expression, and use its string representation as the “real” substitution text.
So if we match all numeric values that are suffixed by ‘k’, we can use (string-to-number \1) to convert the matching digits to an integer, multiply by 1024 and insert the resulting value by using the following substitution pattern:
\,(* 1024 (string-to-number \1))
The full Emacs command would then become:
M-x replace-regexp RET \([0-9]+\)k RET \,(* 1024 (string-to-number \1)) RET
This, and the byte suffix removal, yield now the following text in our Emacs buffer:
| time |cpu|cpu|cpu|cpu|cpu|cpu|eth0 |eth0 | disk| disk|sda-| | time |usr|sys|idl|wai|hiq|siq| recv| send| read| writ|util| |16-05 08:36:15| 2| 3| 96| 0| 0| 0| 66| 178| 0 | 0 | 0| |16-05 08:36:16| 42| 14| 37| 0| 0| 7| 92M|1298432| 0 | 0 | 0| |16-05 08:36:17| 45| 11| 36| 0| 0| 7| 76M|1162240| 0 | 0 | 0| |16-05 08:36:18| 27| 55| 8| 0| 0| 11| 67M| 772096| 0 | 99M|79.6| |16-05 08:36:19| 29| 41| 16| 5| 0| 10| 113M|2128896|4096| 63M|59.6| |16-05 08:36:20| 28| 48| 12| 4| 0| 8| 58M| 406528| 0 | 95M|76.0| |16-05 08:36:21| 38| 37| 14| 1| 0| 10| 114M|2682880|4096| 52M|23.2| |16-05 08:36:22| 37| 54| 0| 1| 0| 8| 76M|1542144|8192| 76M|33.6|
Note: Some of the columns are indeed not aligned very well. We’ll fix that later. On to the megabyte conversion:
M-x replace-regexp RET \([0-9]+\)M RET \,(* 1024 1024 (string-to-number \1)) RET
Which produces a version that has no suffixes at all:
| time |cpu|cpu|cpu|cpu|cpu|cpu|eth0 |eth0 | disk| disk|sda-| | time |usr|sys|idl|wai|hiq|siq| recv| send| read| writ|util| |16-05 08:36:15| 2| 3| 96| 0| 0| 0| 66| 178| 0 | 0 | 0| |16-05 08:36:16| 42| 14| 37| 0| 0| 7| 96468992|1298432| 0 | 0 | 0| |16-05 08:36:17| 45| 11| 36| 0| 0| 7| 79691776|1162240| 0 | 0 | 0| |16-05 08:36:18| 27| 55| 8| 0| 0| 11| 70254592| 772096| 0 | 103809024|79.6| |16-05 08:36:19| 29| 41| 16| 5| 0| 10| 118489088|2128896|4096| 66060288|59.6| |16-05 08:36:20| 28| 48| 12| 4| 0| 8| 60817408| 406528| 0 | 99614720|76.0| |16-05 08:36:21| 38| 37| 14| 1| 0| 10| 119537664|2682880|4096| 54525952|23.2| |16-05 08:36:22| 37| 54| 0| 1| 0| 8| 79691776|1542144|8192| 79691776|33.6|
Finally, to align everything in neat, pipe-separated columns, we enable M-x orgtbl-mode, and type “C-c C-c” with the pointer somewhere inside the transformed dstat output. The buffer now becomes something usable for pretty-much any graphing tool out there:
| time | cpu | cpu | cpu | cpu | cpu | cpu | eth0 | eth0 | disk | disk | sda- | | time | usr | sys | idl | wai | hiq | siq | recv | send | read | writ | util | | 16-05 08:36:15 | 2 | 3 | 96 | 0 | 0 | 0 | 66 | 178 | 0 | 0 | 0 | | 16-05 08:36:16 | 42 | 14 | 37 | 0 | 0 | 7 | 96468992 | 1298432 | 0 | 0 | 0 | | 16-05 08:36:17 | 45 | 11 | 36 | 0 | 0 | 7 | 79691776 | 1162240 | 0 | 0 | 0 | | 16-05 08:36:18 | 27 | 55 | 8 | 0 | 0 | 11 | 70254592 | 772096 | 0 | 103809024 | 79.6 | | 16-05 08:36:19 | 29 | 41 | 16 | 5 | 0 | 10 | 118489088 | 2128896 | 4096 | 66060288 | 59.6 | | 16-05 08:36:20 | 28 | 48 | 12 | 4 | 0 | 8 | 60817408 | 406528 | 0 | 99614720 | 76.0 | | 16-05 08:36:21 | 38 | 37 | 14 | 1 | 0 | 10 | 119537664 | 2682880 | 4096 | 54525952 | 23.2 | | 16-05 08:36:22 | 37 | 54 | 0 | 1 | 0 | 8 | 79691776 | 1542144 | 8192 | 79691776 | 33.6 |
The trick of combining arbitrary Lisp expressions with regexp substitution patterns like \1, \2 … \9 is something I have found immensely useful in Emacs. Now that you know how it works, I hope you can find even more amusing use-cases for it.
Update: The Emacs manual has a few more useful examples of \, in action, as pointed out by tunixman on Twitter.
I recently spent a bit of time to clean up all the cruft that my ~/.emacs file and my ~/elisp directory had accumulated. I have been using a multi-file setup to configure my Emacs sessions, since at least 2008. This turned out to be a royal mess after 5+ years of patching stuff without a very clear plan or structure. The total line-count of both my ~/.emacs and all the *.el files I had imported into my ~/elisp directory was almost 20,000 lines of code:
$ wc -l BACKUP/.emacs $( find BACKUP/elisp -name '*.el')
119 BACKUP/.emacs
84 BACKUP/elisp/keramida-w3m.el
90 BACKUP/elisp/keramida-keys.el
156 BACKUP/elisp/keramida-irc.el
5449 BACKUP/elisp/erlang.el
892 BACKUP/elisp/fill-column-indicator.el
344 BACKUP/elisp/keramida-erc.el
87 BACKUP/elisp/keramida-chrome.el
89 BACKUP/elisp/keramida-autoload.el
141 BACKUP/elisp/keramida-ui.el
42 BACKUP/elisp/keramida-slime.el
1082 BACKUP/elisp/ace-jump-mode.el
2 BACKUP/elisp/scala-mode2/scala-mode2-pkg.el
907 BACKUP/elisp/scala-mode2/scala-mode2-indent.el
26 BACKUP/elisp/scala-mode2/scala-mode2-lib.el
502 BACKUP/elisp/scala-mode2/scala-mode2-fontlock.el
37 BACKUP/elisp/scala-mode2/scala-mode2-map.el
808 BACKUP/elisp/scala-mode2/scala-mode2-syntax.el
111 BACKUP/elisp/scala-mode2/scala-mode2.el
121 BACKUP/elisp/scala-mode2/scala-mode2-paragraph.el
1103 BACKUP/elisp/php-mode.el
142 BACKUP/elisp/themes/cobalt-theme.el
665 BACKUP/elisp/themes/zenburn-theme.el
142 BACKUP/elisp/themes/sublime-themes/cobalt-theme.el
80 BACKUP/elisp/themes/tomorrow-night-blue-theme.el
80 BACKUP/elisp/themes/tomorrow-night-eighties-theme.el
115 BACKUP/elisp/themes/tomorrow-theme.el
80 BACKUP/elisp/themes/tomorrow-night-bright-theme.el
339 BACKUP/elisp/cmake-mode.el
95 BACKUP/elisp/keramida-cc-extra.el
1341 BACKUP/elisp/lua-mode.el
2324 BACKUP/elisp/markdown-mode.el
184 BACKUP/elisp/rcirc-notify.el
167 BACKUP/elisp/keramida-defaults.el
203 BACKUP/elisp/keramida-hooks.el
43 BACKUP/elisp/keramida-lang.el
435 BACKUP/elisp/edit-server.el
709 BACKUP/elisp/slang-mode.el
66 BACKUP/elisp/keramida-eshell.el
19402 total
20,000 lines of code is far too much bloat. It’s obvious that this was getting out of hand, especially if you consider that I had full configuration files for at least two different IRC clients (rcirc and erc) in this ever growing blob of complexity.
What I did was make a backup copy of everything in ~/BACKUP and start over. This time I decided to go a different route from 2008 though. All my configuration lives in a single file, in ~/.emacs, and I threw away any library from my old ~/elisp tree which I haven’t actively used in the past few weeks. I imported the rest of them into the standard user-emacs-directory of modern Emacsen: at ~/.emacs.d/. I also started using eval-after-load pretty extensively, to speed up the startup of Emacs, and only configure extras after the related packages are loaded. This means I could trim down the list of preloaded packages even more.
The result, as I tweeted yesterday was an impressive speedup of the entire startup process of Emacs. Now it can start, load everything and print a message in approximately 0.028 seconds, which is more than 53 times faster than the ~1.5 seconds it required before the cleanup!
I suspected that the main contributor to this speedup was the increased use of eval-after-load forms, but what percentage of the entire file used them?
So I wrote a tiny bit of Emacs Lisp to count how many times each top-level forms appears in my new ~/.emacs file:
(defun file-forms-list (file-name)
(let ((file-forms nil))
;; Keep reading Lisp expressions, until we hit EOF,
;; and just add one entry for each toplevel form
;; to `file-forms'.
(condition-case err
(with-temp-buffer
(insert-file file-name)
(goto-char (point-min))
(while (< (point) (point-max))
(let* ((expr (read (current-buffer)))
(form (first expr)))
(setq file-forms (cons form file-forms)))))
(end-of-file nil))
(reverse file-forms)))
(defun file-forms-alist (file-name)
(let ((forms-table (make-hash-table :test #'equal)))
;; Build a hash that maps form-name => count for all the
;; top-level forms of the `file-name' file.
(dolist (form (file-forms-list file-name))
(let ((form-name (format "%s" form)))
(puthash form-name (1+ (gethash form-name forms-table 0))
forms-table)))
;; Convert the hash table to an alist of the form:
;; ((form-name . count) (form-name-2 . count-2) ...)
(let ((forms-alist nil))
(maphash (lambda (form-name form-count)
(setq forms-alist (cons (cons form-name form-count)
forms-alist)))
forms-table)
forms-alist)))
(progn
(insert "\n")
(insert (format "%7s %s\n" "COUNT" "FORM-NAME"))
(let ((total-forms 0))
(dolist (fc (sort (file-forms-alist "~/.emacs")
(lambda (left right)
(> (cdr left) (cdr right)))))
(insert (format "%7d %s\n" (cdr fc) (car fc)))
(setq total-forms (+ total-forms (cdr fc))))
(insert (format "%7d %s\n" total-forms "TOTAL"))))
Evaluating this in a scratch buffer shows output like this:
COUNT FORM-NAME
32 setq-default
24 eval-after-load
14 set-face-attribute
14 global-set-key
5 autoload
4 require
4 setq
4 put
3 defun
2 when
1 add-hook
1 let
1 set-display-table-slot
1 fset
1 tool-bar-mode
1 scroll-bar-mode
1 menu-bar-mode
1 ido-mode
1 global-hl-line-mode
1 show-paren-mode
1 iswitchb-mode
1 global-font-lock-mode
1 cua-mode
1 column-number-mode
1 add-to-list
1 prefer-coding-system
122 TOTAL
This showed that I’m still using a lot of setq-default forms: 26.23% of the top-level forms are of this type. Some of these may still be candidates for lazy initialization, since I can see that many of them are indeed mode-specific, like these two:
(setq-default diff-switches "-u") (setq-default ps-font-size '(8 . 10))
But eval-after-load is a close second, with 19.67% of all the top-level forms. That seems to agree with the original idea of speeding up the startup of everything by delaying package-loading and configuration until it’s actually needed.
10 of the remaining forms are one-off mode setting calls, like (tool-bar-mode -1), so 8.2% of the total calls is probably going to stay this way for a long time. That’s probably ok though, since the list includes several features I find really useful, very very often.
CL-USER> (format t "~{~{ ~8,'0b~} |~:*~{ ~2,'0x~}~%~}~%"
(loop for x below 32
collect (loop for y below 8
collect (+ (* x 8) y))))
00000000 00000001 00000010 00000011 00000100 00000101 00000110 00000111 | 00 01 02 03 04 05 06 07
00001000 00001001 00001010 00001011 00001100 00001101 00001110 00001111 | 08 09 0A 0B 0C 0D 0E 0F
00010000 00010001 00010010 00010011 00010100 00010101 00010110 00010111 | 10 11 12 13 14 15 16 17
00011000 00011001 00011010 00011011 00011100 00011101 00011110 00011111 | 18 19 1A 1B 1C 1D 1E 1F
00100000 00100001 00100010 00100011 00100100 00100101 00100110 00100111 | 20 21 22 23 24 25 26 27
00101000 00101001 00101010 00101011 00101100 00101101 00101110 00101111 | 28 29 2A 2B 2C 2D 2E 2F
00110000 00110001 00110010 00110011 00110100 00110101 00110110 00110111 | 30 31 32 33 34 35 36 37
00111000 00111001 00111010 00111011 00111100 00111101 00111110 00111111 | 38 39 3A 3B 3C 3D 3E 3F
01000000 01000001 01000010 01000011 01000100 01000101 01000110 01000111 | 40 41 42 43 44 45 46 47
01001000 01001001 01001010 01001011 01001100 01001101 01001110 01001111 | 48 49 4A 4B 4C 4D 4E 4F
01010000 01010001 01010010 01010011 01010100 01010101 01010110 01010111 | 50 51 52 53 54 55 56 57
01011000 01011001 01011010 01011011 01011100 01011101 01011110 01011111 | 58 59 5A 5B 5C 5D 5E 5F
01100000 01100001 01100010 01100011 01100100 01100101 01100110 01100111 | 60 61 62 63 64 65 66 67
01101000 01101001 01101010 01101011 01101100 01101101 01101110 01101111 | 68 69 6A 6B 6C 6D 6E 6F
01110000 01110001 01110010 01110011 01110100 01110101 01110110 01110111 | 70 71 72 73 74 75 76 77
01111000 01111001 01111010 01111011 01111100 01111101 01111110 01111111 | 78 79 7A 7B 7C 7D 7E 7F
10000000 10000001 10000010 10000011 10000100 10000101 10000110 10000111 | 80 81 82 83 84 85 86 87
10001000 10001001 10001010 10001011 10001100 10001101 10001110 10001111 | 88 89 8A 8B 8C 8D 8E 8F
10010000 10010001 10010010 10010011 10010100 10010101 10010110 10010111 | 90 91 92 93 94 95 96 97
10011000 10011001 10011010 10011011 10011100 10011101 10011110 10011111 | 98 99 9A 9B 9C 9D 9E 9F
10100000 10100001 10100010 10100011 10100100 10100101 10100110 10100111 | A0 A1 A2 A3 A4 A5 A6 A7
10101000 10101001 10101010 10101011 10101100 10101101 10101110 10101111 | A8 A9 AA AB AC AD AE AF
10110000 10110001 10110010 10110011 10110100 10110101 10110110 10110111 | B0 B1 B2 B3 B4 B5 B6 B7
10111000 10111001 10111010 10111011 10111100 10111101 10111110 10111111 | B8 B9 BA BB BC BD BE BF
11000000 11000001 11000010 11000011 11000100 11000101 11000110 11000111 | C0 C1 C2 C3 C4 C5 C6 C7
11001000 11001001 11001010 11001011 11001100 11001101 11001110 11001111 | C8 C9 CA CB CC CD CE CF
11010000 11010001 11010010 11010011 11010100 11010101 11010110 11010111 | D0 D1 D2 D3 D4 D5 D6 D7
11011000 11011001 11011010 11011011 11011100 11011101 11011110 11011111 | D8 D9 DA DB DC DD DE DF
11100000 11100001 11100010 11100011 11100100 11100101 11100110 11100111 | E0 E1 E2 E3 E4 E5 E6 E7
11101000 11101001 11101010 11101011 11101100 11101101 11101110 11101111 | E8 E9 EA EB EC ED EE EF
11110000 11110001 11110010 11110011 11110100 11110101 11110110 11110111 | F0 F1 F2 F3 F4 F5 F6 F7
11111000 11111001 11111010 11111011 11111100 11111101 11111110 11111111 | F8 F9 FA FB FC FD FE FF
NIL
CL-USER>
A tiny Lisp script, noticed first in a recent post in comp.lang.lisp and then ported to SBCL:
% cat ~/bin/iota
#!/usr/local/bin/sbcl --script
(defun iota (count &optional (start 0) (step 1))
"Build a list of `count' numbers, starting from `start' and
incrementing by `step' between successive elements."
(loop repeat count for i from start by step collect i))
(format t "~{~a~^ ~}~%"
(apply 'iota (mapcar #'read-from-string (rest *posix-argv*))))
Running this from a shell prompt works fine in SBCL 1.0.25:
% iota 10 0 1 2 3 4 5 6 7 8 9 % iota 10 1 1 2 3 4 5 6 7 8 9 10
It even works with float step values, which is a bit cool:
% iota 10 0 1d-2 0 0.01d0 0.02d0 0.03d0 0.04d0 0.05d0 0.06d0 0.07d0 0.08d0 0.09d0
ERC is an IRC client written in Emacs Lisp. This makes ERC very easy to extend, customize and otherwise adapt to your personal style.
A nice features of the standard ERC distribution is that you can extend the set of commands available on your IRC prompt by writing short Emacs Lisp functions. When you define a function called “erc-cmd-XXX” it instantly becomes available as an IRC command. Continue reading
A common misconception about Lisp is that it is “slow”. I wrote misconception because this is not necessarily true. Most of the time that the “slowness” argument is brought up, it eventually turns out, after talking a while with the supporter of the argument, that it is either based on hearsay or that it stems from older experience with short tests. Continue reading
The default indentation levels of GNU Emacs for the case statements of sh(1) scripts (and derivative or compatible shells) are a bit smal for my taste (4 columns). This makes case statements in shell scripts look almost “ok”, but I don’t like the small indentation for the rest of my scripts. Continue reading
It’s amusing once in a while to skim through the subjects of spam messages in my “mail.junk” folder. With Gnus running in UTF-8 mode, I can even read the text as you can see in
Common Lisp environments include a read-eval-print loop that is also known as “REPL”. The interactivity of the REPL is cool, but there’s at least one reason why the “R” part of the REPL rules. Continue reading
What keramida said... 