In this code snippit you can see how to do a basic ranked text search for MongoDB. The code relies on two simple mapreduce operations. One to create an inverted index from some demo text, and a second one to score the matching documents based on query term hits.
For a customer we have developed log analytics software. It’s currently uses MYSQL as the database backend. The system reads in a hourly log file, and calculates all kinds of fancy statistics. I wanted to see how the system would work if I used MongoDB, a schema-less document DB, instead of MYSQL. My impressions in no particular order:
One of the suggestions of “The Pragmatic Programmer” is that you should learn at least one new programming language every year. This is a great suggestion, but after a couple of years its usefulness diminishes, e.g. if one already knows Perl and Python, then the payback on learning Ruby is rather small. Therefore I’m going to concentrate on the foundations of programming languages this year. Here’s my tentative reading list:
Suggestions welcome.
A solution to Ruby Quiz #14 in literate Haskell:
LCD Numbers
===========
Problem
-------
[original source](http://rubyquiz.com/quiz14.html)
This week's quiz is to write a program that displays LCD style numbers
at adjustable sizes.
The digits to be displayed will be passed as an argument to the
program. Size should be controlled with the command-line option -s
follow up by a positive integer. The default value for -s is 2.
For example, if your program is called with:
$ lcd.rb 012345
The correct display is:
-- -- -- --
| | | | | | | |
| | | | | | | |
-- -- -- --
| | | | | | |
| | | | | | |
-- -- -- --
And for:
$ lcd.rb -s 1 6789
Your program should print:
- - - -
| | | | | |
- - -
| | | | | |
- - -
Note the single column of space between digits in both examples. For
other values of -s, simply lengthen the - and | bars.
Solution
--------
Module declaration and imports:
> module Main where
>
> import Data.Char (digitToInt)
> import Data.List (intersperse)
> import System.Console.GetOpt
> import System.Environment (getArgs)
First we define the numbers at size 1:
> n0 = [ " - "
> , "| |"
> , " "
> , "| |"
> , " - "
> ]
>
> n1 = [ " "
> , " |"
> , " "
> , " |"
> , " "
> ]
>
> n2 = [ " - "
> , " |"
> , " - "
> , "| "
> , " - "
> ]
>
> n3 = [ " - "
> , " |"
> , " - "
> , " |"
> , " - "
> ]
>
> n4 = [ " "
> , "| |"
> , " - "
> , " |"
> , " "
> ]
>
> n5 = [ " - "
> , "| "
> , " - "
> , " |"
> , " - "
> ]
>
> n6 = [ " - "
> , "| "
> , " - "
> , "| |"
> , " - "
> ]
>
> n7 = [ " - "
> , " |"
> , " "
> , " |"
> , " "
> ]
>
> n8 = [ " - "
> , "| |"
> , " - "
> , "| |"
> , " - "
> ]
>
> n9 = [ " - "
> , "| |"
> , " - "
> , " |"
> , " - "
> ]
>
Put the numbers in a list:
> numbers = [n0,n1,n2,n3,n4,n5,n6,n7,n8,n9]
Horizontal scaling function, given a string replicate the second
character n times:
> hscale n cs = head cs : replicate n (cs!!1) ++ [last cs]
Vertical scaling function, repeat the second and fourth row n times:
> vscale n css = head css : replicate n cs1 ++ [cs2] ++ replicate n cs3 ++ [cs4]
> where cs1 = css !! 1
> cs2 = css !! 2
> cs3 = css !! 3
> cs4 = last css
Scale function; note this function scales a single number:
> scale n = vscale n . map (hscale n)
Function that converts a list of numbers to a string of LCD numbers:
> lcd n = concat .
> intersperse "\n" .
> foldr1 (zipWith (++)) .
> intersperse (replicate (3 + 2*n) " ") .
> map (scale n . (numbers !!))
`main` function:
> main = do
> args <- getArgs
> let (n, digits) = parseArgs args
> putStrLn $ lcd n $ map digitToInt digits
Command-line argument parsing:
> data Flag = Scale Int
> deriving Eq
>
> options = [Option "s" [] (ReqArg (Scale . read) "") ""]
>
> parseArgs args =
> case parse args of
> (_, [], _) -> error "Usage: lcd [-s n] digits"
> ([], digits, []) -> (2, head digits)
> ([Scale n], digits, []) -> (n, head digits)
> (_, _, _) -> error "Usage: lcd [-s n] digits"
> where
> parse = getOpt RequireOrder options
Last week my OS got upgraded to Ubuntu 9.10. After that I wanted to compile Apache from source. Unfortunately I got this build error:
This is fixed by replacing getline with parseline on line 651 in /usr/include/stdio.h
Kudos to HowtoForge for pointing this out.
Nice article dispelling the myth of the page fold being a impenetrable barrier for users.
Update: page fold: myth or reality?
The slides for the workshop on Haskell and functional programming I gave yesterday at Devnology’s Community Day.
A classic sampling problem: write a program sample which takes two integers n and m as input. n is the size of the sample. m is the size of the population. The program should print out n random unique indices. Two example runs:
$ ./sample 3 10 0 2 8 $ ./sample 3 10 1 2 9
The output must be sorted. The complete, original quiz is here.
My first (naïve) attempt uses a list of integers to represent the pool still available (i.e. the population not sampled yet). When it has to draw a sample it takes a random number i between 0 and the length of the list and removes the element at index i from the list, thus guaranteeing the uniqueness of the generated indices. It works correctly but it runs out of memory for the "big sample" (n= 5,000,000 and m = 1,000,000,000) mentioned in the original quiz, not very suprising since it keeps both the current samples as well as the pool still availabe in memory. It is also quite slow because of the use of a plain list.
My second attempt solves the memory problem by keeping only the current samples in memory. When it has to draw a sample it takes a random number x between 0 and m and checks if that number has already been used. If the number has been used it tries agian. This solution also uses the Data.Set module for increased performance.
Here's an example run for the big sample. Note that I have to increase the maximum stack size for individual threads (+RTS -K250m) to prevent a stack space overflow:
$ time ./sample 5000000 1000000000 +RTS -K250m > big_sample.txt real 23m24.355s user 23m1.658s sys 0m9.548s $ ls -l big_sample.txt -rw-r--r-- 1 mies staff 49483467 Sep 27 17:13 big_sample.txt $ head big_sample.txt 243 280 416 494 556 602 804 909 970 1126 $ tail big_sample.txt 999998483 999998863 999999002 999999028 999999052 999999053 999999115 999999291 999999853 999999870
The code plus solutions to other quizes is available on GitHub.
The age of completeness-fu is upon us!
Sometimes validations just don’t cut the mustard and all you want to do is to grade an instance based on how complete its information is. For example, a Location has a title and a description but no address, thus its only 60% complete. Or maybe title is worth more than description and address so its 80% complete. Whatever the case, this is not a new problem and recreating the wheel is a bit unnecessary, so welcome to completeness-fu.
The dsl is based on the thinking-sphinx configuration, which is nice, clean and simple, but very effective.
Here is a sample of the config code used to define a set of checks for a completeness score:
It still needs some more tlc, but its a nice start and a simple solution for a common problem.
So please, have a play around with it, fork the code, make some improvements/enhancements and let me know what you think.
As a rails developer you are blessed early on with the fantastic script/server for starting a local development server. Rails is smart enough that it will even suggest you install the Mongrel gem as it is a faster alternative to the basic stock standard WEBrick. But as time passes and your skills improve and the amount of projects you are working on increases, you may find yourself looking for a simpler solution than having to start up an individual script/server on different ports for each project. Or you may just want to have an app run in the background waiting for you to access one of the sites and start it up automatically. What ever the case, there are some very nice solutions available.
99% percent of people who have deployed a rails app have undoubtedly come across Passenger (modrails) from the fantastic guys at Phusion. Simple put, this allows you to config and run your rails app with (initially) Apache or (as of lately) Nginx while also taking advantage of their supplier static assets serving capabilities.
So why would you choose Nginx over Apache in you development environment? For me the reasons for using Nginx was simple and quick configuration, very very very low memory usage, and it mimics my deployment server setup.
So waffle aside, how do we install and setup Nginx, including for you development environment
installing nginx and passenger
Passenger is nice enough to offer to install Nginx for you automagically, including downloading Nginx 0.7.61, but this is already an old version, so the plan is to :
so lets get started….
1. and 2. download the latest stable version of nginx and extract it
3. and 4. install passenger via gem and configure, compile and install nginx
during the installer program enter the following information
Ok, now nginx and passenger are installed with ssl support baked in, what to do from here…
5. tweak the nginx configs
The default nginx config file is pretty basic, which is excellent, because its all you really need, but a few tweaks here and there can make a great thing even better. Slicehost has an excellent write up on some recommended changes here, but as this is your development environment and not production, I suggest not changing worker_processes or keepalive_timeout.
In the end, this is what my server nginx.conf looks like:
(I have taken out all the commented out lines)
You will notice the passenger_root and passenger_ruby properties/directives in the conf file. These are required for passenger to start, but you don’t need to worry about inserting them as the passenger nginx installer does it for you.
This is what my virtual server file looks like:
And thats it, an nginx server + one virtual host all ready to run via:
I also added an alias to me ~/.profile file
now you can just use nginx and stopnginx.
6. putting it all together
ok, the title is a little deceptive as it all seems to be together, but there is one very important change yet to be made, making sure passenger can reach and read your source code.
I ran into this problem when I was setting up my environment, and it all has to do with how passenger and nginx works. As per any good webserver, you need to start it as root so it can access the right ports (80) and directories (pid files), but its worker processes should run as nobody or www-data to restrict unneeded access to other resources. For Nginx to know if the server is a rails app or not, its worker process needs to be able to access the document root, in this case public, and every single one of its parent directories. As I keep my development files within my home directory, nginx would throw a 403 error and add a non-descriptive error message to the error.log file.
Two options are available to fix this:
I choose option two and had the nginx worker processes run as myself. Although you could argue this is insecure, as I only have the server running when I need to, and nginx and passenger have a great track record, I think this is better than setting my home directory to read for everyone.
And there we are, all set up and ready to develop! And all in under 30 mins!
some good links and tips
important for deployment : rails maintenance pages done right
init script for ubuntu : nginx-init-ubuntu
1.9 + nginx + passenger : ruby-rails-nginx-passenger
excellent config details : ubuntu-intrepid-nginx-configuration
nginx + vhosts : ubuntu-intrepid-nginx-virtual-hosts
docs galore : http://nginx.net/ and http://wiki.nginx.net/
special mention to slicehost for all the fantastic server and service related articles known to man