Now at TTY we prefer setting up systems with open source software and therefor using servers with a Linux flavour and an open source HTTP server, but there may be situations where a Windows and IIS system is already in use. This article shows we are capable of handling just that.

Read the article here

If this sounds familiar….

• You are using the DataImportHandler for Solr
• You have a entity with a field which values come from a related entity.
• After an import it looks like Solr only indexed even postive integers if you look at the schema browser.

….You probably have a ‘nested’ field which name is similar to it’s entity name. See the code below: entity name = regio and field name = regio. Changing field name to something else (regions) solved the issue. When you think about it, it’s somewhat logical that you don’t allow field names to have the same name as the entity. An schema exception during indexing would have been nice though.

Starting point

• My team needs to implement a SOAP service for mass posting of vacancies to a job board system.
• The SOAP service is based on a WSDL of an existing service. So we’ll use these specifications as a starting point for the proof of concept.
• On a site-note: I prefer REST above SOAP, because of it’s elegant simplicity. But it wouldn’t make a lot of business sense in this case because a lot of, paying, consumers of the new service have working code for the old service. Adapting to a slightly changed SOAP service will be much easier than a switch to a brand new REST API.

Available SOAP Server extensions for PHP

There are several frameworks / extensions / toolkits for creating a SOAP server for PHP:

• Pear SOAP package. Probably an orphan package as it’s not updated since 2008 and has a beta status. You probably want to look at the alternatives.
• NuSoap SOAP toolkit.  Started in 2002 and still under active development as the last release was just a few months ago at the time of writing.
• PHP 5 SOAP extensions. The official SOAP extension for PHP since version 5.
• Zend SOAP Server. Part of the Zend Framework, so probably not very useful if that’s not your current PHP framework.

As we use the Zend framework for this project, it was a natural choice to use it’s SOAP server implementation. We might opt for one of the alternatives if we slam into a brick wall later down the line.

Testing the waters

The steps I’ve taken to get a basic Zend SOAP Server based on the WSDL up and running

• I copied the wdsl to the /public directory of the Zend framework application making it publicly accessible under  http://example.org/jobtool.wsdl
• I created a new controller under application/controllers/soapController.php with an public indexAction function. Example code
• The new SOAP service is now available under http://example.org/soap
• Next step: actually handle SOAP requests. Example code. Handling of the soap request is as expected: SOAP method arguments are passed as function arguments. Complex types are represented as a stdClass objects, which basically are associative arrays. Nested complex types are translated to nested stdClass instances. You don’t get any warnings or exceptions if your argument count is different than specified in the SOAP request. IMHO that’s undesirable. I rather have big fat ugly exceptions in that case than subtle bugs.  The associative array you return are translated to the complex type as specified in the WSDL and returned to the client.
• To test the SOAP service without the need for a full-blown client i’ve used the free soapUI tool. You point this tool to the WDSL and it automatically creates fake soap request that you can use to test your brand new SOAP services. Make sure you specified the correct urls in the soapAction attributes in the WSDL.

Closing Thoughts

I hope this post saved you some when time building your first SOAP Webservice using Zend Framework. I don’t know yet from experience if the Zend SOAP Server will handle more advanced scenario’s. Only time will tell. Let me know how it works for you.

Conclusion of the article seems to be that conversion increases a lot when presenting the form as a narrative. Don’t know if that really is supported by the small test they did, but I’m inclined to think they might be on to something.

Turns out matplotlib’s back-end wasn’t configured properly. To fix this I had to add a rc file (matplotlibrc) to my ~/.matplotlib directory. The rc file contains the following:

And, hey presto:

(disclaimer: “Works on my machine!”)

A problem with having more than one sphinx server is that you need to make sure the results from the different server are close to the same. Since it is possible to switch between servers for two consecutive searches (which on the site in question could also be a browsing action, for example moving from one page of results to the next) it could be very confusing if the search result were different.

With Sphinx there are a number of ways to solve this problem. The most commonly used solutions are:

• run the indexer on one server and make those indexing results available to all the other servers (through scp, rsync, or hosting on a shared filesystem)
• using a distributed index setup

The first should work, but is actually not recommended by the makers of Sphinx. We went for the second solution: a distributed index setup.

Full and delta indices

Our four servers do a full reindex every night (kicked off by cron). These run sequentially, so each server starts out with a different base index. With cron we run the delta index on each server every three minutes. These delta indices only reindex the last added records since the server’s last full index the night before. This delta index grows throughout the day and starts off taking less than 15 minutes and grows to take about 45 minutes just before the full index is re-run.

The setup for these indices, let’s call them base and delta, looks something like this (for the server with server_id = 2):

The table sph_counter is just a table that tracks the max_doc_id at the time of the last full index run for each server (set by the ‘REPLACE ..’ sql_query_pre in base).

Each server has a similar setup as above, so that they all have a full index from the previous night, and a delta index that covers all the changes from that time up to the last delta index run.

The distributed index

Now we create a distributed index on all the servers that queries the local base and delta indices, and also all the delta indices on the other three servers. That looks something like this:

In our code we exclusively query the index named main. This index combines the base index from the particular random server the code happened to choose with the delta indices of all four servers. A distributed index combines the results from all of these, and discards the duplicates automatically. This seems to work pretty well.

Things to look at

One downside to this setup is that each search kicks off a search on the machine it connects to, but also sends a search on the delta index to all the other servers. At the moment it seems to work, but at busy moments we see a bit of an increase in our response times for our searches. It might be better to concentrate the delta index on one server dedicated to the delta index and leave the other three to concentrate on the base index. This is something we plan on looking into soon. So, possibly an update soon… ?

• Importing log data is much easier than on MYSQL because MongoDB is schema-less. Just create a collection (=bucket) and insert every log line into it as a hash. For log files that don’t have a fixed amount of fields, it’s a great fit.
• Like MYSQL, you do need to create indexes to make searching fast(er).
• MongoDB supports map reduce operations. It made some of the calculations much more elegant and better readable than the code that was written for MYSQL.
• Chaining of map reduce operations is supported, and works as you would expect.
• Queries are written in javascript. I’m happy that they didn’t invent yet another ’scripting’ language. Javascript looks capable enough.
• Map reduce operations are not particularly fast. They are upgrading their javascript engine to V8 to improve the execution speed.
• MongoDB community is nowhere near the size of MYSQL. Don’t expect a lot of Google results for a specific mongoDB issue. The moderated Google group is a better place to go currently.
• I liked the API. Calls are not verbose and their intented use is easy to understand.
• Although quite capable, mongoDB is still a young project. I need to have more time with it before using it on a customer project.

• Design Concepts in Programming Languages
• Concepts, Techniques, and Models of Computer Programming
• Types and Programming Languages

Suggestions welcome.

LCD Numbers
===========

Problem
-------

[original source](http://rubyquiz.com/quiz14.html)

This week's quiz is to write a program that displays LCD style numbers
at adjustable sizes.

The digits to be displayed will be passed as an argument to the
program. Size should be controlled with the command-line option -s
follow up by a positive integer. The default value for -s is 2.

For example, if your program is called with:

    $ lcd.rb 012345

The correct display is:

     --        --   --        --
    |  |    |    |    | |  | |
    |  |    |    |    | |  | |
               --   --   --   --
    |  |    | |       |    |    |
    |  |    | |       |    |    |
     --        --   --        -- 

And for:

    $ lcd.rb -s 1 6789

Your program should print:

     -   -   -   -
    |     | | | | |
     -       -   -
    | |   | | |   |
     -       -   - 

Note the single column of space between digits in both examples. For
other values of -s, simply lengthen the - and | bars.

Solution
--------

Module declaration and imports:

> module Main where
>
> import Data.Char (digitToInt)
> import Data.List (intersperse)
> import System.Console.GetOpt
> import System.Environment (getArgs)

First we define the numbers at size 1:

> n0 = [ " - "
>      , "| |"
>      , "   "
>      , "| |"
>      , " - "
>      ]
>
> n1 = [ "   "
>      , "  |"
>      , "   "
>      , "  |"
>      , "   "
>      ]
>
> n2 = [ " - "
>      , "  |"
>      , " - "
>      , "|  "
>      , " - "
>      ]
>
> n3 = [ " - "
>      , "  |"
>      , " - "
>      , "  |"
>      , " - "
>      ]
>
> n4 = [ "   "
>      , "| |"
>      , " - "
>      , "  |"
>      , "   "
>      ]
>
> n5 = [ " - "
>      , "|  "
>      , " - "
>      , "  |"
>      , " - "
>      ]
>
> n6 = [ " - "
>      , "|  "
>      , " - "
>      , "| |"
>      , " - "
>      ]
>
> n7 = [ " - "
>      , "  |"
>      , "   "
>      , "  |"
>      , "   "
>      ]
>
> n8 = [ " - "
>      , "| |"
>      , " - "
>      , "| |"
>      , " - "
>      ]
>
> n9 = [ " - "
>      , "| |"
>      , " - "
>      , "  |"
>      , " - "
>      ]
>

Put the numbers in  a list:

> numbers = [n0,n1,n2,n3,n4,n5,n6,n7,n8,n9]

Horizontal scaling function, given a string replicate the second
character n times:

> hscale n cs = head cs : replicate n (cs!!1) ++ [last cs]

Vertical scaling function, repeat the second and fourth row n times:

> vscale n css = head css : replicate n cs1 ++ [cs2] ++ replicate n cs3 ++ [cs4]
>   where cs1 = css !! 1
>         cs2 = css !! 2
>         cs3 = css !! 3
>         cs4 = last css

Scale function; note this function scales a single number:

> scale n = vscale n . map (hscale n)

Function that converts a list of numbers to a string of LCD numbers:

> lcd n = concat .
>         intersperse "\n" .
>         foldr1 (zipWith (++)) .
>         intersperse (replicate (3 + 2*n) " ") .
>         map (scale n . (numbers !!))

`main` function:

> main = do
>   args <- getArgs
>   let (n, digits) = parseArgs args
>   putStrLn $ lcd n $ map digitToInt digits

Command-line argument parsing:

> data Flag = Scale Int
>             deriving Eq
>
> options = [Option "s" [] (ReqArg (Scale . read) "") ""]
>
> parseArgs args =
>   case parse args of
>    (_, [], _)              -> error "Usage: lcd [-s n] digits"
>    ([], digits, [])        -> (2, head digits)
>    ([Scale n], digits, []) -> (n, head digits)
>    (_, _, _)               -> error "Usage: lcd [-s n] digits"
>   where
>     parse = getOpt RequireOrder options