A mysql 5.0 master compiled from source on Debian squeeze breaks mysql replication

One of our database backend systems consists of a number of mysql 5.0 machines that are set up in a single master-multiple slave configuration. Although we’re planning to migrate this cluster to a non-end-of-life version of mysql, it needs to be kept up and running for a bit longer.

A while ago we found out that running a mysql 5.0 master built from source on a Debian squeeze machine broke replication. Since that time (which was during a nightly scheduled down-time unrelated to this particular problem and which caused us some sweaty brows in the chilled datacenter) we’ve been running our master on the ‘obsolete’ Debian lenny, as well as a few slaves in case we need to do a master switch.

The problem is that what we always used to do to compile mysql, on Debian squeeze produced a mysql master that corrupts its binary logs. It’s important to note that the problem is indeed on the master, because this problem will usually show up as errors on the slave when they try to start replicating (i.e. when you tell them to ‘start slave;‘). It’s easily checked that the problem is indeed on the master, though, because if you try to dump the binlog (using ‘mysqlbinlog‘) you’ll see problems right away.

On a test machine set up to do binlogging with a broken mysql, after just adding a few rows to an empty table in a test database, I get this:

#> mysqlbinlog binary-log.000001
/*!40019 SET @@session.max_insert_delayed_threads=0*/;
# at 4
#120601 16:51:34 server id 50 end_log_pos 98 Start: binlog v 4, server v 5.0.96-log created 120601 16:51:34 at startup
# Warning: this binlog is either in use or was not closed properly.
ERROR: Error in Log_event::read_log_event(): 'Found invalid event in binary log', data_len: 106, event_type: 2
# End of log file
ROLLBACK /* added by mysqlbinlog */;

The important bit is this:

ERROR: Error in Log_event::read_log_event(): 'Found invalid event in binary log', data_len: 106, event_type: 2

After some digging it seems that the compiler options we use to build mysql in combination with the default gcc that comes with Debian squeeze is to blame. We use basically the same options as the INSTALL-SOURCE file recommends:

CFLAGS="-O3" CXX=gcc \
CXXFLAGS="-O3 -felide-constructors -fno-exceptions -fno-rtti" \
./configure --prefix=/usr/local/mysql --enable-assembler \

If we remove the optimizations (the -O3 parts) we get a functioning mysql master. We don’t really want to do that (mysql would most likely be slower without optimizations) so we decided to go the other route: use gcc-4.3, which is easily installed on Debian squeeze:

aptitude install gcc-4.3 g++-4.3 cpp-4.3

After that, the compile line becomes:

CC=gcc-4.3 CFLAGS="-O3" CXX=gcc-4.3 \
CXXFLAGS="-O3 -felide-constructors -fno-exceptions -fno-rtti" \
./configure --prefix=/usr/local/mysql --enable-assembler \

The resulting mysqld binary produces correct binlogs and can be used as a replication master.

Maybe this problem could also be solved by going the other way: upgrade gcc to a newer version. That would involve more work on Debian squeeze (gcc-4.3 and gcc-4.4 are available by default from the repositories, other versions aren’t) so we haven’t investigated that option. For the time that we’ll still be running a mysql replication setup with mysql 5.0 we’ll be okay with gcc-4.3.

Splitting a database dump

This is a way to split a SQL dump into tables that is relatively easy. test

It should start with zcat FILE.gz | csplit -ftable – “/DROP TABLE/” {*},
but csplit has bug where reading a lot from standard in does not work.
So instead unzip your file first.

gunzip FILE.gz
csplit -ftable FILE "/DROP TABLE/" {*}

Then to give the files/2012/05/2012/05 meaningful names:

for FILE in `ls -1 table*`; do
NAME=`head -n1 $FILE | cut -d$'x60' -f2`
mv $FILE "$NAME.sql";

If your dump does not start witjh a DROP TABLE `name` IF EXISTS,
you will have to change the match expression to csplit a litte.

Hope this is useful to someone.

Setting up Emacs to compare two git tags


This is a setup for using Emacs to compare two git tags. It is based on Ediff mode and Ediff Trees. Ediff Trees is a useful front-end for comparing large trees of files/2012/05/2012/05. To make things easier we have two git clones, one for each tag that we call before and after.

Needed Code

Create a file named ediff-trees.el with the following contents and place it in your Emacs load path:

;;; ediff-trees.el --- Recursively ediff two directory trees
;; Author: Joao Cachopo
;; Created on: Wed May 10 17:30:49 2006
;; Keywords: ediff, comparing
;; Version: 20071126.1
;; Copyright (C) 2006 Joao Cachopo

;; This program is not part of GNU Emacs

;; This program is free software; you can redistribute it and/or
;; modify it under the terms of the GNU General Public License as
;; published by the Free Software Foundation; either version 2, or (at
;; your option) any later version.

;; This program is distributed in the hope that it will be useful, but
;; WITHOUT ANY WARRANTY; without even the implied warranty of
;; General Public License for more details.

;; You should have received a copy of the GNU General Public License
;; along with GNU Emacs; see the file COPYING.  If not, write to the
;; Free Software Foundation, 675 Massachusettes Ave, Cambridge, MA
;; 02139, USA.

;;; Commentary:

;; The ediff-trees package is a simple frontend to the emacs' ediff
;; package to allow a simpler comparison of two similar directory
;; trees.

;; I wrote this package because I often need to compare two different
;; versions of the same directory tree and ediff-directories is not
;; very helpful in this case.  Specially when the directory trees to
;; compare are deep and only a few files/2012/05/2012/05 have changed.
;; Typically, that occurs when I create a copy of some project
;; directory tree either to make some experiments myself or to send to
;; someone else that will return a modified directory tree to me
;; later.  (Yes, I heard of version control systems, and I use them
;; regularly.  Yet, for several reasons, sometimes that is not an
;; option.)

;; Later, when I want to integrate the modified directory tree with
;; the original tree, I want to see the differences to the original
;; version, so that I may decide whether to accept the changes or not.
;; This is where this package kicks in...

;; To use it, just call `ediff-trees', which will ask for two
;; directories to compare.  Usually, I give the original directory as
;; the first one and the modified directory as the second one.

;; ediff-trees recursively descends both directories, collecting the
;; pairs of files/2012/05/2012/05 that are worth "comparing": either files/2012/05/2012/05 that
;; changed, or that appear in one of the two directory trees but not
;; in the other.  Then, it shows the first "change" using ediff.

;; In fact, ediff-trees either uses ediff to compare a file with its
;; changed version, or simply opens a file that occurs in only one of
;; the trees.

;; The user can then navigate backward and forward in the set of
;; changes by using `ediff-trees-examine-next' and
;; `ediff-trees-examine-previous', respectively.  These functions move
;; from one change (quiting the current ediff session or killing the
;; current file buffer) to another.  Therefore, by repeatedly using
;; these functions we can go through all the changes.  I usually use
;; some global bindings for these functions.  Something like this:
;;   (global-set-key (kbd "s-SPC") 'ediff-trees-examine-next)
;;   (global-set-key (kbd "S-s-SPC") 'ediff-trees-examine-previous)
;;   (global-set-key (kbd "C-s-SPC") 'ediff-trees-examine-next-regexp)
;;   (global-set-key (kbd "C-S-s-SPC") 'ediff-trees-examine-previous-regexp))

;; The `ediff-trees-examine-next-regexp' and
;; `ediff-trees-examine-previous-regexp' skip over the list of changes
;; to a file with a filename that matches a given regexp.

;; This package allows for some customization.  Please, see the
;; ediff-trees group under customize.

;; Finally, to deal with small changes in the white space I often find
;; it useful to configure ediff like this:
;;   (setq ediff-diff-options "-w")
;;   (setq-default ediff-ignore-similar-regions t)

;;; Code:

(require 'ediff)

(defgroup ediff-trees nil
  "Extend ediff to allow comparing two trees recursively."
  :tag "Ediff Trees"
  :group 'ediff)

(defface ediff-trees-deleted-original-face
  '((((class color))
     (:background "Pink"))
    (t (:inverse-video t)))
  "Face for highlighting the buffer when it was deleted from the original tree."
  :group 'ediff-trees)

(defcustom ediff-trees-file-ignore-regexp
  "A regexp matching either files/2012/05/2012/05 or directories to be ignored
when comparing two trees.  If a directory matches the regexp,
then its contents is not scanned by `ediff-trees'."
  :type 'regexp
  :group 'ediff-trees)

(defcustom ediff-trees-sort-order-regexps nil
  "*Specifies a list of regexps that determine the order in which
files/2012/05/2012/05 will be presented during the ediff-trees session.  Files
with filenames matching former regexps appear earlier in the
session.  If a filename matches more than one regexp, the first
one wins."
  :type '(repeat regexp)
  :group 'ediff-trees)

(defun ediff-trees (root1 root2)
  "Starts a new ediff session that recursively compares two
   (let ((dir-A (ediff-get-default-directory-name))
     (list (setq f (ediff-read-file-name "Directory A to compare:" dir-A nil))
       (ediff-read-file-name "Directory B to compare:"
                 (if ediff-use-last-dir
                   (ediff-strip-last-dir f))
  (ediff-trees-internal root1 root2))

;;; Internal variables, used during an ediff-trees session
(defvar ediff-trees-current-file nil)
(defvar ediff-trees-remaining-files/2012/05/2012/05 (list))
(defvar ediff-trees-examined-files/2012/05/2012/05 (list))

(defun ediff-trees-internal (root1 root2)
  (let ((files/2012/05/2012/05-changed (ediff-trees-collect-files/2012/05/2012/05 root1 root2)))
    (if (not (null files/2012/05/2012/05-changed))
          (setq ediff-trees-remaining-files/2012/05/2012/05 files/2012/05/2012/05-changed)
          (setq ediff-trees-examined-files/2012/05/2012/05 (list))
          (ediff-trees-examine-next 1))
      (message "There are no changes between the trees!"))))

(defun ediff-trees-collect-files/2012/05/2012/05 (root1 root2)
   (nconc (ediff-trees-collect-changed-files/2012/05/2012/05 root1 root2)
          (mapcar (lambda (el) (cons el nil))
                  (ediff-trees-collect-new-files/2012/05/2012/05 root1 root2))
          (mapcar (lambda (el) (cons nil el))
                  (ediff-trees-collect-new-files/2012/05/2012/05 root2 root1)))))

(defun ediff-trees-sort-files/2012/05/2012/05 (files/2012/05/2012/05)
  (let ((tagged-files/2012/05/2012/05 (mapcar (lambda (pair)
                                (cons (ediff-trees-get-sort-order (or (car pair) (cdr pair)))
    (mapcar #'cdr
            (sort tagged-files/2012/05/2012/05
                  (lambda (tf1 tf2)
                    (let ((order1 (car tf1))
                          (order2 (car tf2)))
                      (or (< order1 order2)
                          (and (= order1 order2)
                               (let ((el1 (or (cadr tf1) (cddr tf1)))
                                     (el2 (or (cadr tf2) (cddr tf2))))
                                 (string< el1 el2))))))))))

(defun ediff-trees-get-sort-order (pathname)
  (let ((order 0)
        (sorting-regexps ediff-trees-sort-order-regexps))
    (while (and (not (null sorting-regexps))
                (not (string-match (pop sorting-regexps) pathname)))
      (setq order (+ order 1)))

(defun ediff-trees-collect-changed-files/2012/05/2012/05 (root1 root2)
  (let ((changed (list)))
    (dolist (filename (directory-files/2012/05/2012/05 root1))
      (unless (ediff-trees-skip-file-p filename)
        (let ((file1 (expand-file-name filename root1))
              (file2 (expand-file-name filename root2)))
          (when (and (file-exists-p file1) (file-exists-p file2))
            (if (eql (file-directory-p file1)
                     (file-directory-p file2))
                (cond ((file-directory-p file1)
                       (setq changed (nconc changed (ediff-trees-collect-changed-files/2012/05/2012/05 file1 file2))))
                      ((not (ediff-same-file-contents file1 file2))
                       (push (cons file1 file2) changed)))
              (let ((msg (format "I cannot compare a directory, '%s', with a file.  Continue? "
                                 (if (file-directory-p file1) file1 file2))))
                (if (not (y-or-n-p msg))
                    (error "Aborting ediff-trees"))))))))

(defun ediff-trees-collect-new-files/2012/05/2012/05 (root1 root2)
  "Collect files/2012/05/2012/05 from root1 that do not appear at root2."
  (let ((new-files/2012/05/2012/05 (list)))
    (dolist (filename (directory-files/2012/05/2012/05 root1))
      (unless (ediff-trees-skip-file-p filename)
        (let ((file1 (expand-file-name filename root1))
              (file2 (and root2 (expand-file-name filename root2))))
          (when (file-exists-p file1)
            (cond ((file-directory-p file1)
                   (setq new-files/2012/05/2012/05
                         (nconc new-files/2012/05/2012/05
                                (ediff-trees-collect-new-files/2012/05/2012/05 file1
                                                               (and (stringp file2)
                                                                    (file-directory-p file2)
                  ((or (null file2) (not (file-exists-p file2)))
                   (push file1 new-files/2012/05/2012/05)))))))

(defun ediff-trees-skip-file-p (filename)
  ;; always ignore . and ..
  (or (string= filename ".")
      (string= filename "..")
      (string-match ediff-trees-file-ignore-regexp filename)))

(defun ediff-trees-examine-next (num)
  (interactive "p")
  (if (< num 0)
    (ediff-trees-examine-previous (- num))
     (lambda (file) (zerop (setq num (- num 1))))
     (lambda (file) (push file ediff-trees-examined-files/2012/05/2012/05))
     (lambda () (pop ediff-trees-remaining-files/2012/05/2012/05)))))

(defun ediff-trees-examine-previous (num)
  (interactive "p")
  (if (< num 0)
    (ediff-trees-examine-next (- num))
     (lambda (file) (zerop (setq num (- num 1))))
     (lambda (file) (push file ediff-trees-remaining-files/2012/05/2012/05))
     (lambda () (pop ediff-trees-examined-files/2012/05/2012/05)))))

(defun ediff-trees-examine-next-regexp (regexp)
  (interactive "sSearch for (regexp): ")
   (lambda (file) (string-match regexp (or (car file) (cdr file))))
   (lambda (file) (push file ediff-trees-examined-files/2012/05/2012/05))
   (lambda () (pop ediff-trees-remaining-files/2012/05/2012/05))))

(defun ediff-trees-examine-previous-regexp (regexp)
  (interactive "sSearch for (regexp): ")
   (lambda (file) (string-match regexp (or (car file) (cdr file))))
   (lambda (file) (push file ediff-trees-remaining-files/2012/05/2012/05))
   (lambda () (pop ediff-trees-examined-files/2012/05/2012/05))))

(defun ediff-trees-examine-file (pred save-current-file-fn get-next-file-fn)
  (when (eq (current-buffer) ediff-control-buffer)
    (ediff-really-quit nil))
  (unless (null ediff-trees-current-file)
    (funcall save-current-file-fn ediff-trees-current-file)
    (when (car ediff-trees-current-file)
      (kill-buffer (find-buffer-visiting (car ediff-trees-current-file))))
    (when (cdr ediff-trees-current-file)
      (kill-buffer (find-buffer-visiting (cdr ediff-trees-current-file))))
    (setq ediff-trees-current-file nil))
  (let ((next-file (ediff-trees-get-next-file pred save-current-file-fn get-next-file-fn)))
    (if (null next-file)
        (message "No more files/2012/05/2012/05.")
        (setq ediff-trees-current-file next-file)
        (if (and (car next-file) (cdr next-file))
            (ediff-files/2012/05/2012/05 (car next-file) (cdr next-file))
            (find-file-read-only (or (car next-file) (cdr next-file)))
            (when (null (cdr next-file))
              (let ((overlay (make-overlay 0 (point-max))))
                (overlay-put overlay 'face 'ediff-trees-deleted-original-face)))))))))

(defun ediff-trees-get-next-file (pred save-current-file-fn get-next-file-fn)
  (let ((return-value 'not-found))
    (while (eq return-value 'not-found)
      (let ((next-file (funcall get-next-file-fn)))
        (cond ((null next-file)
               (setq return-value nil))
              ((funcall pred next-file)
               (setq return-value next-file))
               (funcall save-current-file-fn next-file)))))

(provide 'ediff-trees)

This code can be downloaded from the EmacsWiki. We made a change in the two functions ediff-trees-collect-changed-files/2012/05/2012/05 and ediff-trees-collect-new-files/2012/05/2012/05 where we added condition (file-exists-p file1) to skip symbolic links that do not refer to an existing file.

Create another file with the following contents, place it in your Emacs load path and load it:

(assert (and (boundp 'ediff-git-root-before) (stringp ediff-git-root-before)))
(assert (and (boundp 'ediff-git-root-after)  (stringp ediff-git-root-after)))

(require 'ediff-trees)

(setq-default ediff-ignore-similar-regions t)
(setq-default ediff-split-window-function 'split-window-horizontally)
(setq-default ediff-trees-file-ignore-regexp "^[.]?#\|~$\|^[.]git$")

(defun next-error-capable-buffer () "Return a 'next-error' capable buffer."
  (ignore-errors (next-error-find-buffer))

(defun kill-all-next-error-capable-buffers () "Kill all 'next-error' capable buffers."
  (let ((buffer (next-error-capable-buffer)))
    (when buffer
      (message "Killing buffer '%s'" (buffer-name buffer))
      (kill-buffer buffer)

(defun ediff-git-commits (commits) "Start a new ediff session that recursively compares 'before' and 'after'."
  (interactive "sEnter commit(s): ")
  (let* ((whitespace-chars " fnrt")
         (cc-whitespace (concat "[" whitespace-chars "]"))
         (re-commit (concat "\([^" whitespace-chars "]+\)"))
    (if (string-match (concat "\`" cc-whitespace "*" re-commit "\(?:" cc-whitespace "+" re-commit "\)?" cc-whitespace "*" "\'") commits)
      (let ((commit-before (match-string 1 commits)) (commit-after (match-string 2 commits)))
        (unless commit-after
          (if (y-or-n-p (format "Compare %s with parent? " commit-before))
            (setq commit-after commit-before commit-before (concat commit-after "^"))
            (let ((commit (read-string "Enter commit for 'after': ")))
              (if (string-match (concat "\`" cc-whitespace "*" re-commit cc-whitespace "*" "\'") commit)
                (setq commit-after (match-string 1 commit))
                (error "Not a valid commit")
        (let* ((case-fold-search nil) (ok "ok") (re-command-ok (concat cc-whitespace (regexp-quote ok) "\'"))
               (command-template (concat "unset CDPATH && cd %s && git checkout . && git fetch && git checkout %s && echo -n " ok))
               (command-before (format command-template ediff-git-root-before commit-before))
               (command-after  (format command-template ediff-git-root-after commit-after))
          (message "Checking out 'before'...")
          (unless (string-match re-command-ok (shell-command-to-string command-before))
            (error "Error checking out 'before' (%s)" command-before)
          (message "Checking out 'after'...")
          (unless (string-match re-command-ok (shell-command-to-string command-after))
            (error "Error checking out 'after' (%s)" command-after)
      (unless (and (string-equal commits "") (y-or-n-p "No commit(s) supplied. Keep current checkouts? "))
        (error "No valid commit(s) supplied")
    (message "Comparing...")
    (ediff-trees ediff-git-root-before ediff-git-root-after)

(defun visit-next (arg) "In an ediff session, visit next file, else visit next 'next-error' message."
  (interactive "p")
  (if (or (next-error-capable-buffer) (null ediff-trees-current-file))
    (next-error arg)
    (ediff-trees-examine-next arg)

(defun visit-previous (arg) "In an ediff session, visit previous file, else visit previous 'next-error' message."
  (interactive "p")
  (if (or (next-error-capable-buffer) (null ediff-trees-current-file))
    (previous-error arg)
    (ediff-trees-examine-previous arg)

Make sure you have constants ediff-git-root-before and ediff-git-root-after defined in your config.

(defconst ediff-git-root-before "/home/eduard/before")
(defconst ediff-git-root-after "/home/eduard/after")


To start a new ediff session, execute ediff-git-commits (M-x ediff-git-commits) and enter zero, one or two commits. When comparing is done, execute visit-next(or bind it to a key you like), to go to the next differing files/2012/05/2012/05.

Harro & Eduard

Quest for the perfect Erlang development environment

Erlang is great, and there is a lot of dev tooling available. Unfortunately these best practices are not easy to find for Erlang newbies like me. So I’ll start writing them down here and grow the list as I’m moving up the Dreyfus model

Get command history for `erl`, the Erlang shell

  1. Install rlwrap. On my Mac using Homebrew:
    brew install rlwrap
  2. Add the alias
    alias erl='rlwrap -a dummy erl'

    in your Bash profile. On my Mac it’s located here:


    . Reload your profile like this:

    bash$ source ~/.profile

Automatic reloading of re-compiled modules

  1. Grab Mochiweb’s

    from here, compile it and put the beam file here:

  2. Create or edit

    and add the line



  3. From now on, when you have a module loaded in the Erlang shell and re-compile it outside your shell, the new version will be reloaded automatically

Some nice utility functions for your Erlang shell

** user extended commands **
dbgtc(File)   -- use dbg:trace_client() to read data from File
dbgon(M)      -- enable dbg tracer on all funs in module M
dbgon(M,Fun)  -- enable dbg tracer for module M and function F
dbgon(M,File) -- enable dbg tracer for module M and log to File
dbgadd(M)     -- enable call tracer for module M
dbgadd(M,F)   -- enable call tracer for function M:F
dbgdel(M)     -- disable call tracer for module M
dbgdel(M,F)   -- disable call tracer for function M:F
dbgoff()      -- disable dbg tracer (calls dbg:stop/0)
l()           -- load all changed modules
la()          -- load all modules
mm()          -- list modified modules

These commands are added by:

  1. Compiling user_default.erl and move the beam file to
  2. Create or edit

    and add the line



  3. Feel free to add your own shortcuts to your


There are multiple versions of


floating around on the interwebs. So pick the one that feels right.

Practical Erlang testing techniques

Watch the Practical Erlang testing techniques presentation from Mr. Bob Ippolito for a quick rundown of useful testing libs.

Thanks to @andrzejsliwa for the tips! Please add your tips to the comments and I’ll update the post.