Toru Maesaka

memcapable is a tool that was recently included into the libmemcached package (as of version 0.33). In short, you can use it to verify whether a particular server supports the memcached binary protocol. You can read why such a tool was created in the blog entries published by the guys who came up with it:

• Matt Ingenthron – power in the protocol
• Trond Norbye – memcapable

Running memcapable against memcached is not so interesting since it’s been done by many of us already. Instead, I decided to run it against a full text search engine called Groonga which apparently supports a subset of the memcached binary protocol.

For those that are interested, Groonga is a successor project to Senna, which is a open source embedded fulltext search engine that gained huge success in Japan. Senna is commonly used by embedding it into MySQL but Groonga _can_ run as a standalone server. For more info, here’s Kaj Arnö’s past blog entry on Senna.

Testing Groonga with memcapable

Running Groonga in foreground:

$ groonga -s -p 11211

Results obtained from running memcapable on the same host:

$ memcapable
noop            [pass]
quit            [pass]
quitq           [pass]
set             [pass]
setq            [FAIL]
flush           [pass]
flushq          [pass]
add             [pass]
addq            [FAIL]
replace         [pass]
replaceq                [FAIL]
delete          [pass]
deleteq         [FAIL]
get             [pass]
getq            [pass]
getk            [pass]
getkq           [pass]
incr            [FAIL]
incrq           [pass]
decr            [FAIL]
decrq           [pass]
version         [pass]
append          [FAIL]
appendq         [FAIL]
prepend         [FAIL]
prependq                [FAIL]
stat            [FAIL]
illegal         [FAIL]
12 of 28 tests failed

As you can see above, Groonga doesn’t support the full binary command stack but it shows how it will be possible to perform full text search from your favorite client library in the future. I’m saying future because Groonga is still a project in progress.

memcapable is your friend when you need to verify whether your server is communicating properly. I’m sure there are other projects that take advantage of the binary protocol whether it’s closed or open. For example, someone pinged me on Twitter that they’re working on something along this line.

Mac OS X users will need to wait for libmemcached-0.34

I found a minor glitch in OS X that would make certain tests fail. Fortunately it was fixed and committed while I was sleeping after I filed the bug. The power of global scale development still surprises me :)

Toru Maesaka memcached, oss fulltext, groonga, libmemcached, memcached

Something I’ve been wanting to add to libmemcached for a while is a cluster analysis feature that can be used to calculate useful information for system admins that deploy multiple memcached nodes. Many webshops already do this by writing their own so it makes sense to have this functionality in a community driven library. This means people can pitch in their useful stats calculation which results in benefiting everyone.

So, with that in mind I wrote a patch that contains the initial codebase for the analyzer which is now reviewed and merged into trunk. This also gave me a reason to claim the commit access that Brian has been offering me for nearly a year… (I know, I’m lazy and useless).

At the moment, you can get the following information with the new memcached_analyze(3) function:

• Average item size in the pool
• Node that is eating the most memory
• Node with least designated memory remaining
• Node with the longest uptime
• Pool-wide Hit Ratio

I know, there isn’t much at the moment but more will be added in upcoming pushes. The analyzer is pretty easy to extend so please feel free to throw patches for it! Or, you could pitch in by telling us useful figures that should be computed.

Hopefully this new feature will be released next month :)

Avoid the library by using memstat

The memstat tool (distributed with libmemcached) has been enhanced with the new ‘–analyze’ option that will use the analyzer and give you a human readable output. You should use memstat unless you need to explicitly use the values returned by the analyzer.

Using it is easy, all you need to do is specify the servers that you’d like to inspect and give it the ‘–analyze’ option like this (only two servers for demo purpose):

$ memstat --servers=localhost:11211,localhost:11311 --analyze
Memcached Cluster Analysis Report
 
    Number of Servers Analyzed         : 2
    Average Item Size (incl/overhead)  : 54 bytes
 
    Node with most memory consumption  : localhost:11211 (54 bytes)
    Node with least free space         : localhost:11211 (67108810 bytes remaining)
    Node with longest uptime           : localhost:11311 (780s)
    Pool-wide Hit Ratio                : 100%

Easy!

Toru Maesaka memcached, oss libmemcached, memcached