In this code snippit you can see how to do a basic ranked text search for MongoDB. The code relies on two simple mapreduce operations. One to create an inverted index from some demo text, and a second one to score the matching documents based on query term hits.
A solution to Ruby Quiz #14 in literate Haskell:
LCD Numbers
===========
Problem
-------
[original source](http://rubyquiz.com/quiz14.html)
This week's quiz is to write a program that displays LCD style numbers
at adjustable sizes.
The digits to be displayed will be passed as an argument to the
program. Size should be controlled with the command-line option -s
follow up by a positive integer. The default value for -s is 2.
For example, if your program is called with:
$ lcd.rb 012345
The correct display is:
-- -- -- --
| | | | | | | |
| | | | | | | |
-- -- -- --
| | | | | | |
| | | | | | |
-- -- -- --
And for:
$ lcd.rb -s 1 6789
Your program should print:
- - - -
| | | | | |
- - -
| | | | | |
- - -
Note the single column of space between digits in both examples. For
other values of -s, simply lengthen the - and | bars.
Solution
--------
Module declaration and imports:
> module Main where
>
> import Data.Char (digitToInt)
> import Data.List (intersperse)
> import System.Console.GetOpt
> import System.Environment (getArgs)
First we define the numbers at size 1:
> n0 = [ " - "
> , "| |"
> , " "
> , "| |"
> , " - "
> ]
>
> n1 = [ " "
> , " |"
> , " "
> , " |"
> , " "
> ]
>
> n2 = [ " - "
> , " |"
> , " - "
> , "| "
> , " - "
> ]
>
> n3 = [ " - "
> , " |"
> , " - "
> , " |"
> , " - "
> ]
>
> n4 = [ " "
> , "| |"
> , " - "
> , " |"
> , " "
> ]
>
> n5 = [ " - "
> , "| "
> , " - "
> , " |"
> , " - "
> ]
>
> n6 = [ " - "
> , "| "
> , " - "
> , "| |"
> , " - "
> ]
>
> n7 = [ " - "
> , " |"
> , " "
> , " |"
> , " "
> ]
>
> n8 = [ " - "
> , "| |"
> , " - "
> , "| |"
> , " - "
> ]
>
> n9 = [ " - "
> , "| |"
> , " - "
> , " |"
> , " - "
> ]
>
Put the numbers in a list:
> numbers = [n0,n1,n2,n3,n4,n5,n6,n7,n8,n9]
Horizontal scaling function, given a string replicate the second
character n times:
> hscale n cs = head cs : replicate n (cs!!1) ++ [last cs]
Vertical scaling function, repeat the second and fourth row n times:
> vscale n css = head css : replicate n cs1 ++ [cs2] ++ replicate n cs3 ++ [cs4]
> where cs1 = css !! 1
> cs2 = css !! 2
> cs3 = css !! 3
> cs4 = last css
Scale function; note this function scales a single number:
> scale n = vscale n . map (hscale n)
Function that converts a list of numbers to a string of LCD numbers:
> lcd n = concat .
> intersperse "\n" .
> foldr1 (zipWith (++)) .
> intersperse (replicate (3 + 2*n) " ") .
> map (scale n . (numbers !!))
`main` function:
> main = do
> args <- getArgs
> let (n, digits) = parseArgs args
> putStrLn $ lcd n $ map digitToInt digits
Command-line argument parsing:
> data Flag = Scale Int
> deriving Eq
>
> options = [Option "s" [] (ReqArg (Scale . read) "") ""]
>
> parseArgs args =
> case parse args of
> (_, [], _) -> error "Usage: lcd [-s n] digits"
> ([], digits, []) -> (2, head digits)
> ([Scale n], digits, []) -> (n, head digits)
> (_, _, _) -> error "Usage: lcd [-s n] digits"
> where
> parse = getOpt RequireOrder options
A classic sampling problem: write a program sample which takes two integers n and m as input. n is the size of the sample. m is the size of the population. The program should print out n random unique indices. Two example runs:
$ ./sample 3 10 0 2 8 $ ./sample 3 10 1 2 9
The output must be sorted. The complete, original quiz is here.
My first (naïve) attempt uses a list of integers to represent the pool still available (i.e. the population not sampled yet). When it has to draw a sample it takes a random number i between 0 and the length of the list and removes the element at index i from the list, thus guaranteeing the uniqueness of the generated indices. It works correctly but it runs out of memory for the "big sample" (n= 5,000,000 and m = 1,000,000,000) mentioned in the original quiz, not very suprising since it keeps both the current samples as well as the pool still availabe in memory. It is also quite slow because of the use of a plain list.
My second attempt solves the memory problem by keeping only the current samples in memory. When it has to draw a sample it takes a random number x between 0 and m and checks if that number has already been used. If the number has been used it tries agian. This solution also uses the Data.Set module for increased performance.
Here's an example run for the big sample. Note that I have to increase the maximum stack size for individual threads (+RTS -K250m) to prevent a stack space overflow:
$ time ./sample 5000000 1000000000 +RTS -K250m > big_sample.txt real 23m24.355s user 23m1.658s sys 0m9.548s $ ls -l big_sample.txt -rw-r--r-- 1 mies staff 49483467 Sep 27 17:13 big_sample.txt $ head big_sample.txt 243 280 416 494 556 602 804 909 970 1126 $ tail big_sample.txt 999998483 999998863 999999002 999999028 999999052 999999053 999999115 999999291 999999853 999999870
The code plus solutions to other quizes is available on GitHub.
The age of completeness-fu is upon us!
Sometimes validations just don’t cut the mustard and all you want to do is to grade an instance based on how complete its information is. For example, a Location has a title and a description but no address, thus its only 60% complete. Or maybe title is worth more than description and address so its 80% complete. Whatever the case, this is not a new problem and recreating the wheel is a bit unnecessary, so welcome to completeness-fu.
The dsl is based on the thinking-sphinx configuration, which is nice, clean and simple, but very effective.
Here is a sample of the config code used to define a set of checks for a completeness score:
It still needs some more tlc, but its a nice start and a simple solution for a common problem.
So please, have a play around with it, fork the code, make some improvements/enhancements and let me know what you think.
As a rails developer you are blessed early on with the fantastic script/server for starting a local development server. Rails is smart enough that it will even suggest you install the Mongrel gem as it is a faster alternative to the basic stock standard WEBrick. But as time passes and your skills improve and the amount of projects you are working on increases, you may find yourself looking for a simpler solution than having to start up an individual script/server on different ports for each project. Or you may just want to have an app run in the background waiting for you to access one of the sites and start it up automatically. What ever the case, there are some very nice solutions available.
99% percent of people who have deployed a rails app have undoubtedly come across Passenger (modrails) from the fantastic guys at Phusion. Simple put, this allows you to config and run your rails app with (initially) Apache or (as of lately) Nginx while also taking advantage of their supplier static assets serving capabilities.
So why would you choose Nginx over Apache in you development environment? For me the reasons for using Nginx was simple and quick configuration, very very very low memory usage, and it mimics my deployment server setup.
So waffle aside, how do we install and setup Nginx, including for you development environment
installing nginx and passenger
Passenger is nice enough to offer to install Nginx for you automagically, including downloading Nginx 0.7.61, but this is already an old version, so the plan is to :
so lets get started….
1. and 2. download the latest stable version of nginx and extract it
3. and 4. install passenger via gem and configure, compile and install nginx
during the installer program enter the following information
Ok, now nginx and passenger are installed with ssl support baked in, what to do from here…
5. tweak the nginx configs
The default nginx config file is pretty basic, which is excellent, because its all you really need, but a few tweaks here and there can make a great thing even better. Slicehost has an excellent write up on some recommended changes here, but as this is your development environment and not production, I suggest not changing worker_processes or keepalive_timeout.
In the end, this is what my server nginx.conf looks like:
(I have taken out all the commented out lines)
You will notice the passenger_root and passenger_ruby properties/directives in the conf file. These are required for passenger to start, but you don’t need to worry about inserting them as the passenger nginx installer does it for you.
This is what my virtual server file looks like:
And thats it, an nginx server + one virtual host all ready to run via:
I also added an alias to me ~/.profile file
now you can just use nginx and stopnginx.
6. putting it all together
ok, the title is a little deceptive as it all seems to be together, but there is one very important change yet to be made, making sure passenger can reach and read your source code.
I ran into this problem when I was setting up my environment, and it all has to do with how passenger and nginx works. As per any good webserver, you need to start it as root so it can access the right ports (80) and directories (pid files), but its worker processes should run as nobody or www-data to restrict unneeded access to other resources. For Nginx to know if the server is a rails app or not, its worker process needs to be able to access the document root, in this case public, and every single one of its parent directories. As I keep my development files within my home directory, nginx would throw a 403 error and add a non-descriptive error message to the error.log file.
Two options are available to fix this:
I choose option two and had the nginx worker processes run as myself. Although you could argue this is insecure, as I only have the server running when I need to, and nginx and passenger have a great track record, I think this is better than setting my home directory to read for everyone.
And there we are, all set up and ready to develop! And all in under 30 mins!
some good links and tips
important for deployment : rails maintenance pages done right
init script for ubuntu : nginx-init-ubuntu
1.9 + nginx + passenger : ruby-rails-nginx-passenger
excellent config details : ubuntu-intrepid-nginx-configuration
nginx + vhosts : ubuntu-intrepid-nginx-virtual-hosts
docs galore : http://nginx.net/ and http://wiki.nginx.net/
special mention to slicehost for all the fantastic server and service related articles known to man
Given an array of integers, find the sub-array with maximum sum. (The complete, original quiz is here.)
Write a program that tells whether a given integer is happy. A happy number is found using the following process: Take the sum of the squares of its digits, and continue iterating this process until it yields 1, or produces an infinite loop. (The complete, original quiz is here.)
Paperclip is a fantastic library used by many many many developers, its github project page shows 1500 watchers but I have a feeling even more than that use the plugin in some shape or form.
The one thing that has been missing for me is proper I18n support and not basic interpolations using the :message option. I recently chanced upon an issue in the github issues registry (#14) which included a fork that implemented a basic version of I18n support but did not go far enough in my opinion. So after taking some advice from validates_timeliness, I have created a fork with goes a little bit further and adds two new messages for the attachment size.
The only thing now is to get some support for this fork so it gets patched into paperclip.
So without further ado, please support this change by leaving a message in the google group thread regarding this change:
Google Group Message – I18n changes and additions
And voting for this issue to be resolved:
Github Issue – Support for I18n
Also, if you have any questions, comments or advice regarding the fork, please don’t hesitate to leave a message below.
One of the most popular open-source indexers for web development has to be Sphinx, an sql full-text search engine. Sphinx has been used by many high profile site, including The Pirate Bay, Craigs List, and NetLog, to name the big ones (more here).
But this post is not about Sphinx, but its Ruby helper library, Thinking Sphinx. In fact, its not just about Thinking Sphinx, finding and fixing a problem yourself.
Thinking Sphinx is an amazing open source gem/plugin/library for Rails, masterminded by the great Pat Allan (@pat), and available to all on github. In fact, with 89 contributors and climbing, this library has had some tender love and care for many caring developers. If you are a Ruby developer and you are interested in indexing, have a look at Thinking Sphinx today, it is simple to install and use, fast and effective, and has some excellent documentation and a very active google group.
Recently while working on a new project for a client of ours I encountered an annoying issue/problem/bug with Thinking Sphinx, more specifically, DateTime multi-value attributes. A multi-value attribute (MVA) is like specifying an array of values for a specific field, for example, employee names, the trouble is that MVAs can only be integers so strings need to be converted (CRC values, CRC32) and datetimes need to be converted to unix timestamps. In Thinking Sphinx MVAs are usually defined through the relationships a model might hold, eg. an office has many employees.
Thinking Sphinx had no problem with me specifying MVAs of integers, but datetimes where a bit of an issue due to two issues, one is the converting, and the other is the group concat to add them all together with commas (like an array). Now, to say that it was an issue makes it sound worse than it really was, because by breaking down the issue into the two parts, converting and concatenating, two improvements could be made at the same time.
So after a brief chat with Pat on the google group to confirm the issue, and on a perfect sunday to be inside (hungover and not great weather), I forked Thinking Sphinx on github (gotta love github for making forking projects so easy) and created a four commit patch which solved the MVA datetime issue for everyone. This patch has since been pulled into the master Thinking Sphinx project so others can use datetime columns as MVAs without having to think about it.
The whole experience of finding this bug, fixing it, and submitting a patch, has reenforced a very important lesson for all developers. If you run into a problem which you do not believe is excepted behavior, do not run away from it and try to find a workaround, get into the code and try to find out where and why the problem is occurring. Its very easy to use libraries and gems and plugins without really knowing how they work or what they do, but if you never get your hands dirty and investigate what is going on then you never learn.
If you can’t find the problem, ask for help on the message boards or by emailing the maintainer. On the other hand, if you do find the problem, check with the maintainer if he can confirm your findings.
Next, create a patch which fixes the issue and does not break or alter current functionality (unless it really does need altering).
And last but not least, share the patch with the community. (make the world a better place)
By the time you have finished with the patch you will not only know the code a lot better, but you would have learnt a truck load about the library, which in the long run is a huge advantage as you are not blindly trusting code written by a hoard of other developers but know first hand what is going on.