It should start with zcat FILE.gz | csplit -ftable – “/DROP TABLE/” {*},
but csplit has bug where reading a lot from standard in does not work.
So instead unzip your file first.

Then to give the files meaningful names:

If your dump does not start witjh a DROP TABLE `name` IF EXISTS,
you will have to change the match expression to csplit a litte.

Hope this is useful to someone.

This is a setup for using Emacs to compare two git tags. It is based on Ediff mode and Ediff Trees. Ediff Trees is a useful front-end for comparing large trees of files. To make things easier we have two git clones, one for each tag that we call before and after.

Needed Code

Create a file named ediff-trees.el with the following contents and place it in your Emacs load path:

This code can be downloaded from the EmacsWiki. We made a change in the two functions ediff-trees-collect-changed-files and ediff-trees-collect-new-files where we added condition (file-exists-p file1) to skip symbolic links that do not refer to an existing file.

Create another file with the following contents, place it in your Emacs load path and load it:

Make sure you have constants ediff-git-root-before and ediff-git-root-after defined in your config.
Example:

Usage

To start a new ediff session, execute ediff-git-commits (M-x ediff-git-commits) and enter zero, one or two commits. When comparing is done, execute visit-next(or bind it to a key you like), to go to the next differing files.

Harro & Eduard

Get command history for `erl`, the Erlang shell

1. Install rlwrap. On my Mac using Homebrew:
   brew install rlwrap
2. Add the alias
   alias erl='rlwrap -a dummy erl'

   in your Bash profile. On my Mac it’s located here:

   ~/.profile

   . Reload your profile like this:

   bash$ source ~/.profile

Automatic reloading of re-compiled modules

1. Grab Mochiweb’s
   reload.erl

   from here, compile it and put the beam file here:

   ~/bin/reloader.beam
2. Create or edit
   ~/.erlang

   and add the line

   code:load_abs("[YOUR_HOME_DIR_PLZ_REPLACE]/bin/reloader")

   .

3. From now on, when you have a module loaded in the Erlang shell and re-compile it outside your shell, the new version will be reloaded automatically

Some nice utility functions for your Erlang shell

These commands are added by:

1. Compiling user_default.erl and move the beam file to
   ~/bin/user_default.beam
2. Create or edit
   ~/.erlang

   and add the line

   code:load_abs("[YOUR_HOME_DIR_PLZ_REPLACE]/bin/user_default")

   .

3. Feel free to add your own shortcuts to your
   user_default.erl

   .

There are multiple versions of

floating around on the interwebs. So pick the one that feels right.

Practical Erlang testing techniques

Watch the Practical Erlang testing techniques presentation from Mr. Bob Ippolito for a quick rundown of useful testing libs.

Thanks to @andrzejsliwa for the tips! Please add your tips to the comments and I’ll update the post.

1) Disable the `Restore windows when quitting and re-opening apps` checkbox in the General Preference window.

2) Clean out and write-protect the Resume database for the command line:

Thanks to @andrzejsliwa for the tip!

This post explains how to build a basic full text search server in Erlang. The server has the following features:

• indexing
• stemming
• ranking
• faceting
• asynchronous search results
• web frontend using websockets

Familiarity with the OTP design principles is recommended.

The sample application (build with help from my colleague Michel Rijnders <mies@tty.nl>) uses the Creative Commons Data Dump from StackExchange as demo data.

We cover the following subjects:

• running the sample application
• OTP supervision tree
• importing demo data
• indexing
• stemming
• faceting
• querying and relevance ranking
• displaying search results
• improvements

Running the Sample Application

Clone the source from GitHub:

 git clone git://github.com/tty/async_search.git

And start the application:

$ rebar get-deps compile && erl -pa `pwd`/ebin `pwd`/deps/*/ebin +P 134217727
Eshell> application:start(async).
Eshell> stackoverflow_importer_ser:import().

Visit http://localhost:3000, you should see the following page:

Sample ranked search output for the query erlang armstrong:

Sample tags facets output for the query java:

OTP Supervision Tree

Looking at the OTP application supervision tree is a good way to understand the architecture of an OTP application.

The application supervisor async_sup starts up the following supervisors:

• keyword_sup. A keyword_ser process is created for every unique word in the StackExchange posts. This keyword_ser is linked to the keyword_sup supervisor (a simple_one_for_one supervisor). The keyword_ser child process maintains a list of document positions of a keyword (an inverted index).
• facet_sup. A keyword_ser process is also created for every unique facet category in the StackExchange posts. This keyword_ser process is linked to the facet_sup supervisor (a simple_one_for_one supervisor as well). The keyword_ser child process maintains a list of facet values with the IDs of the documents the facets appear in.

The application supervisor also start the following gen_server singleton processes:

• stackoverflow_importer_ser. This server imports the demo Stack Overflow data.
• document_ser. This server holds a copy of all documents, so it can return the original title and body of matching Stack Overflow posts in the results.
• query_ser. This server's task is to run the actual query and return results.
• websocket_ser. This server provides a HTTP frontend for the search engine.

No attention is given to fault tolerance (apart from the basic restart strategies), thus parts of the search index are lost if a keyword_ser process terminates.

Demo Data Import

The StackExchange data is provided as XML. Since some of the documents are quite large, it's not recommended to load the full XML documents in memory. The solution is to use a SAX parser which treats a XML file as a stream, and triggers events when new elements are discovered. The search server uses the excellent SAX parser from the Erlsom library by Willem de Jong.

In the example below erlsom:parse_sax reads the XML file from FilePath and calls the function sax_event if an XML element is found.

When the element is a row element (i.e. a post element), attributes like Id, Title and Body are stored in a dictionary. For every post a copy of all the attributes in document_ser is saved. This is used for returning the actual posts for a query match. After that the add_attribute_tokens function is called:

The add_attribute_tokens function does two things. It calls add_facet (discussed later) and it creates a list of tuples with all the words and their position in the document. This process is called tokenization. Each token/position tuple is then submitted to the add_keyword_position function of the keyword_ser for indexing.

Indexing

Indexing of the tuples, or keywords, is handled by the keyword_ser. For every unique word a keyword_ser process is started if not already present. The state of a keyword_ser process is a dictionary with the document ID as key and a list of positions as value. The document ID corresponds to the ID of the Stack Overflow post.

The keyword_server_name function generates a unique name under which the keyword_ser process is registered, so the module can check if a keyword already has a process or a new process needs to be created.

Stemming

Stemming is the process for reducing inflected words to their base form. Computing and computer both are stemmed to comput. So when a user searches on computing, it also matches text that contains computer. This makes it possible to return results that are relevant, but do not exactly match the query.

In our sample application all keywords are stemmed using the popular Porter Algorithm. The Erlang implementation by Alden Dima is used in the application.

erlang:phash2 is used to transform the stemmed name to a hash, to make sure the registered process name is valid.

Faceted search is an important navigation feature for search engines. A user can drill down the search results by filtering on pre-defined attributes, like in this example of a digital camera search on CNET:

As mentioned above, the data import the function add_attribute_tokens also calls the add_facet function. Using pattern matching the Tags and the Creationdate attributes are selected for faceting. Tags is a so called multivalue facet, as a Stack Overflow post can have one or more tags assigned. For every tag and creation date the facet_ser:add_facet_value function is called.

facet_ser works very similar to keyword_ser. For every facet category, Tag or Creationdate in our case, a facet_ser processes is started. The state of a facet_ser is a dictionary with the Tag or Creationdate values as key and their document IDs as dictionary values.

Querying and Relevance Ranking

In previous sections is shown:

• how the XML demo data is parsed.
• how this data is stemmed and indexed by creating a keyword_ser process for every unique keyword.
• how this data is indexed for faceted search by creating a facet_ser process for every facet category.

With the function stackoverflow_importer_ser:import() these steps are executed, and your Erlang node is now ready for querying. So how does that work?

Querying

Querying is handled by passing the user's query terms to the function do_async_query of the singleton query_ser server. When calling this function you need to specify the module, function and optional reference attribute which will be called when query results are available.

In the handle_cast the following steps are executed:

• keyword_ser:do_query return all document ids that contain one or more of the user's query terms, including the relevance ranking score, which will be discussed below.
• All original documents are stored during indexing in a document_ser process. All matching documents are collected.
• The callback function is invoked with the matching documents and their ranking scores as arguments.
• Facet results are retrieved for any FacetCategories that are specified by calling facet_ser:get_facets.
• And the callback function is invoked a second time with the facet results as arguments.

Relevance Ranking

Relevance in this context denotes how well a retrieved document matches the user's search query. Most fulltext search-engines use the BM25 algorithm to determine the ranking score of each document, so let's use that too.

BM25 calculates a ranking score based on the query term frequency in each documents.

See the async_bm25.erl for the implementation.

Displaying the Search Results

As discussed, the query_ser:do_async_query can be called to query our full-text search engine. To allow users to send queries and see the result the websocket_ser module is created. This singleton gen_serverstarts up a Misultin HTTP server on Port 3000. If you browse to http://localhost:3000 you will see a search box. Communication with the search engine is done through websockets.

So, when a user posts a query, this message is received by the websockets_ser:handle_websocket receive block. The query_ser:do_async_query function is called and query results are expected on websockets_ser:query_results function.

The query_results function formats the results as HTML and sends this through the websocket. When received, the HTML is appended to the user's page.

A similar process is executed when the facet results are received:

Improvements

Some obvious features that are lacking from this sample application:

• The author of this post is an Erlang newbie. Corrections/suggestions to the code are most welcome. You can send them to <ward@tty.nl>
• Pretty much no attention is given to performance / memory usage.
• Fault tolerence for the index data. When a server containing index state dies, it will not be revived.
• Tuple structures passed between modules are not specified. Would be nice to use record syntax for it.
• No unit/quickcheck/common test added.
• No function/type specifications.
• etc..

So, that why it's called a sample application ;-)

Travis CI – Distributed, Continuous Integration for the open source community.

By Ward Bekker / TTY Internet Solutions – Travis CI is a new continuous integration service for the open source community. It started out with a Ruby focus and became an instant success. Recently Erlang support was added. A few well known projects, like eTorrent, Mochiweb, Meck and Elixir, already started using it. In this presentation you will learn how the system works, the vision behind it, the upcoming features the team is working on and how to add your own Erlang projects.

The Erlang trace facility

By Jeroen Koops – In this talk, I’ll show how to use Erlang’s low-level trace facility, and the higher-level dbg module that is built on top of it. Finally, I’ll demonstrate how to build a simple tool using the primitives provided by the trace-facility.

Let’s jabber about ejabberd

By Ahmed Omar / Nimbuzz – Just a quick jabber about ejabberd

Zotonic, the Erlang web framework, at MaxClass

By Marc Worrell / MaxClass – Zotonic is both an easy to use content management system and a powerful web framework. It’s built on some of the best pieces of Erlang open source software, by experienced web developers. Zotonic comes with an incredible speed out-of-the-box, an extensible infrastructure and most of all, a friendly community. In the first part of this talk, we summarize the history and development of Zotonic, and give a short introduction to the data model and the architecture.

Currently we provide Erlang/OTP releases R14B01, R14B02 and R14B03. Older versions will be added in the near future. Builds are managed with the excellent Rebar tool from the Basho folks.

Projects

A selection of projects that were added to Travis at the time of writing:

• jlouis/eTorrent
• mochi/Mochiweb
• spawngrid/Proper_stdlib
• seancribbs/Neotoma
• dreid/Gen_Bunny

So, why not add your Erlang project now?

The near future

Currently only eunit tests are run. We are going to add support for:

• Dialyzer
• Xref
• Common_test

Most of these test can already be run by customizing the script element in the .travis.yml, but we want to make it as convenient as possible.

Oh, and did you add your Erlang project already?

Questions?

Questions or need help? Join #travis on freenode or contact me on twitter

The solution is obvious: you need to parallelize the tests. Every integration test needs a dedicated environment to able to get predictable results. For most integration test this means exclusive access to resources like your database (Mysql), memory caching (Memcached) and/or full text search solutions (Sphinx | Solr). You can design your tests to be collision free, but like most multi-threaded programming that uses shared resource it’s quite difficult to get it right. And debugging weird threading issues will make you want to put pencils in your eyes. Trust me on that.

A more efficient way of creating a dedicated environment for every test is the use of virtual machines (vm). You replicate your integration test environment on a vm. Make several clones and your now have a pool of vm’s that can run your tests in parallel and guaranteed exclusivity.

The hard part of this solution;

• Cucumber and the unit test runner need to be modified to run tests distributed.
• Non-hypervisor virtualisation systems like Virtualbox and VMWare Server introduce a significant performance overhead. Hypervisor systems require a dedicated box.
• Provisioning of virtual machines can be a chore. Solutions like Vagrant can help with that.

But it will be worth it. Your CPU cores are worth it.

CPU load is far higher then normal. Note the platforms in the graph!

You could think back on what changes you made since the previous release and pinpoint possible problem areas. Then revert this part of the code, redeploy and see what happens. This approach can be tedious and you can end up quite frustrated when this approach didn’t fix the problem after several tries. It would be nice if you could pinpoint exactly where things go wrong wouldn’t it?

VisualVM to the rescue

Since Java 5.0 Java has this great feature called JMX (Java Management Extensions). This technology enables getting all sorts of information from running JVM’s. VisualVM is a tool that uses JMX and gives you detailed info about JVM memory, CPU usage, Garbage Collection but can also profile your objects for CPU and memory usage (local JVM’s only, otherwise all your live classes would be instrumented which would take down the performance even more). The nice thing is that you can monitor your remote running JVM’s too! In our case this is exactly what we need since we can’t reproduce it on our development or acceptance machines. Users could have hit some location on your webapplication in just the kind of way you never tested and therefor can’t reproduce.

Enable JMX via Tomcat

In order to use JMX you should add some variables to the JAVA_OPTS in your Tomcat startup script:

-Dcom.sun.management.jmxremote=true
-Dcom.sun.management.jmxremote.port=9090
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.authenticate=false

You can choose your own jmxremote port and choose to use SSL and authenticate (You should if you need to open up the JMX port for the entire world via your firewall). When we tried it we first opened up only port 9090 on our firewall. This didn’t work. There are some other ports that are used between JMX and VisualVM (Disclaimer: We have not tested this any further and we are not sure which other ports are needed). We fixed this problem by opening the firewall entirely for just the IP address of the computer we have VisualVM running.

Connecting VisualVM to the JMX enabled remote JVM

Now we are ready to connect to the server:

Connecting your JMX enabled remote JVM to your local VisualVM

Start VisualVM on your machine and create a remote connection by right clicking ‘Remote’ and click  ’Add remote host…’. Then on your created remote host do a right click again and click ‘Add JMX connection…’. Type your remote server’s domainname or IP followed by your remotely configured port number. Now open this connection. You will see a lot of information about your live JVM environment and some tabs. In this article we will need the ‘Threads’ tab!

Pinpointing the problem

When you open the Threads tab you should see something like the following example:

Normal expected view of tomcat HTTP connections threads

As you can see some threads are running at some time, handling some HTTP requests. This should not take too much time (the HTTP responses will take to long and users will experience your webapp as slow) but should certainly not take forever! Guess what… when we clicked our Threads tab we saw something like the example I created below:

Problematic view; some threads are running continuously

Spot the differences. As you can see there are two threads which are running forever: http-8005-191 and http-8005-188. If you scroll up and down you could very well spot some others. Just to be sure that this is your CPU problem check at what time these threads started being in running state and compare this with your CPU graph. We saw that the CPU platforms started at exactly the same time as a thread became permanently running.

Fixing the problem

So what causes these threads to be running forever? The answer lies under the ‘Thread Dump’ button you can see at the upper right of this Threads tab. Click it to let VisualVM create a dump of all threads stacks. Now remember the ID’s of our problematic threads and search for them in the thread dump output. The stack trace you see here should lead you directly to the problem you’ll have to fix! This will probably be some never ending while loop somehow. Never ending recursive calls would probably have alarmed you sooner with a stack overflow exception. In our case it was code like below that caused our pain:

You can imagine we extended our testcases for this particular part of code in the webapplication. We now could reproduce this problem on our development machines and solve it.

Fixing your problem?

It could very well be that your problem has a different origin but hopefully this will help you out to solve your problem more easily in that case. If you have different cases which you solved (partly) with what you read in this article, please let us know with a comment to this article and how you solved your problem. This way you can help out others with problems like these too! Now go enjoy your speedy bugfree webapp ;)