Toru Maesaka

Something I’ve been wanting to add to libmemcached for a while is a cluster analysis feature that can be used to calculate useful information for system admins that deploy multiple memcached nodes. Many webshops already do this by writing their own so it makes sense to have this functionality in a community driven library. This means people can pitch in their useful stats calculation which results in benefiting everyone.

So, with that in mind I wrote a patch that contains the initial codebase for the analyzer which is now reviewed and merged into trunk. This also gave me a reason to claim the commit access that Brian has been offering me for nearly a year… (I know, I’m lazy and useless).

At the moment, you can get the following information with the new memcached_analyze(3) function:

• Average item size in the pool
• Node that is eating the most memory
• Node with least designated memory remaining
• Node with the longest uptime
• Pool-wide Hit Ratio

I know, there isn’t much at the moment but more will be added in upcoming pushes. The analyzer is pretty easy to extend so please feel free to throw patches for it! Or, you could pitch in by telling us useful figures that should be computed.

Hopefully this new feature will be released next month :)

Avoid the library by using memstat

The memstat tool (distributed with libmemcached) has been enhanced with the new ‘–analyze’ option that will use the analyzer and give you a human readable output. You should use memstat unless you need to explicitly use the values returned by the analyzer.

Using it is easy, all you need to do is specify the servers that you’d like to inspect and give it the ‘–analyze’ option like this (only two servers for demo purpose):

$ memstat --servers=localhost:11211,localhost:11311 --analyze
Memcached Cluster Analysis Report
 
    Number of Servers Analyzed         : 2
    Average Item Size (incl/overhead)  : 54 bytes
 
    Node with most memory consumption  : localhost:11211 (54 bytes)
    Node with least free space         : localhost:11211 (67108810 bytes remaining)
    Node with longest uptime           : localhost:11311 (780s)
    Pool-wide Hit Ratio                : 100%

Easy!

Toru Maesaka memcached, oss libmemcached, memcached

I’m happy to report that I’ve made some progress in the Query Cache plugin interface that I’ve been working on for Drizzle. The first version of the Query Cache interface definition is now merged to Drizzle’s trunk.

Unfortunately though, you cannot actually write a query cache module for Drizzle yet. I still haven’t implanted the hooks for this interface to Drizzle’s codebase since this requires much more thinking and thorough testing (which is my next milestone for this mini project).

So in other words, I’ve only done the easy part heh. Do let me take this opportunity to introduce the interface that just went in though. Currently the API consists of five functions which looks like this:

 bool (*qcache_try_fetch_and_send)(Session *session, bool transactional);
 bool (*qcache_set)(Session *session, bool transactional);
 bool (*qcache_invalidate_table)(Session *session, bool transactional);
 bool (*qcache_invalidate_db)(Session *session, const char *db_name,
                              bool transactional);
 bool (*qcache_flush)(Session *session);

That’s it! all you need to do is implement the above functions to write a query cache module. Here’s a brief description of what the above functions are intended for (though the name is pretty self-explanatory):

In an ideal world, we would want to do row-based invalidation but this is extremely difficult to do in practice so I will not touch it for now (not sure if I ever will). Instead, I’m planning on taking advantage of primary keys (and possibly the slow query log) and see how that goes before attempting to touch the forbidden fruit… I’m going to keep things dead simple to begin with.

So there you go! a preview of work in progress for one of the many projects that are concerned with modularizing the Drizzle SQL Server. Stay tuned for updates :)

Toru Maesaka drizzle, oss drizzle, oss

It may be a little late for this entry but I still haven’t thanked my friends so I’ll post it anyway.

So ever since I started working in Open Source, I’ve found myself working fairly closely with Software Engineers at Sun (amongst many other companies) and my heroes (as Trond would say) from Drizzle and Memcached had sent me this nice card all the way to Japan from Norway. Thanks guys! 

This just shows how awesome Open Source development is and how it can make the world a better and friendlier place with technology. Now I need to plan a reverse surprise :)

Toru Maesaka oss surprise