Toru Maesaka

Thanks to Alan a.k.a Dormando and his designer’s efforts, the memcached project now has a new look and a domain. From what I’ve heard, http://danga.com/memcached will soon be pointed at http://memcached.org, which from now on will be the official project domain. Awesome!

Dormando and I have talked about internationalizing the project site so that keen individuals can translate the files and commit it to the git repository. This would give life to http://memcached.jp which I happen to own and eagerly waiting to contribute.

Toru Maesaka memcached memcached

memcapable is a tool that was recently included into the libmemcached package (as of version 0.33). In short, you can use it to verify whether a particular server supports the memcached binary protocol. You can read why such a tool was created in the blog entries published by the guys who came up with it:

• Matt Ingenthron – power in the protocol
• Trond Norbye – memcapable

Running memcapable against memcached is not so interesting since it’s been done by many of us already. Instead, I decided to run it against a full text search engine called Groonga which apparently supports a subset of the memcached binary protocol.

For those that are interested, Groonga is a successor project to Senna, which is a open source embedded fulltext search engine that gained huge success in Japan. Senna is commonly used by embedding it into MySQL but Groonga _can_ run as a standalone server. For more info, here’s Kaj Arnö’s past blog entry on Senna.

Testing Groonga with memcapable

Running Groonga in foreground:

$ groonga -s -p 11211

Results obtained from running memcapable on the same host:

$ memcapable
noop            [pass]
quit            [pass]
quitq           [pass]
set             [pass]
setq            [FAIL]
flush           [pass]
flushq          [pass]
add             [pass]
addq            [FAIL]
replace         [pass]
replaceq                [FAIL]
delete          [pass]
deleteq         [FAIL]
get             [pass]
getq            [pass]
getk            [pass]
getkq           [pass]
incr            [FAIL]
incrq           [pass]
decr            [FAIL]
decrq           [pass]
version         [pass]
append          [FAIL]
appendq         [FAIL]
prepend         [FAIL]
prependq                [FAIL]
stat            [FAIL]
illegal         [FAIL]
12 of 28 tests failed

As you can see above, Groonga doesn’t support the full binary command stack but it shows how it will be possible to perform full text search from your favorite client library in the future. I’m saying future because Groonga is still a project in progress.

memcapable is your friend when you need to verify whether your server is communicating properly. I’m sure there are other projects that take advantage of the binary protocol whether it’s closed or open. For example, someone pinged me on Twitter that they’re working on something along this line.

Mac OS X users will need to wait for libmemcached-0.34

I found a minor glitch in OS X that would make certain tests fail. Fortunately it was fixed and committed while I was sleeping after I filed the bug. The power of global scale development still surprises me :)

Toru Maesaka memcached, oss fulltext, groonga, libmemcached, memcached

I’m playing a little behind but I’d like to spread the word that memcached-1.4.0 has been released.

• http://code.google.com/p/memcached/downloads/list
• http://code.google.com/p/memcached/wiki/ReleaseNotes140

You can also find blog posts by other memcached developers floating around on the internet. This 1.4 release is quite special for me too since the binary protocol is something I’ve been eager to see out in the open for sometime now. I remember printing out the early binary protocol specification (the draft that was made before I joined the community) and reading it on the flight to SFO back in 2007.

Much has happened since then with many input from both developers and non-developers. Several commercial organizations from small to large has helped us evaluate the old experimental branch(es) that we’ve been working on at github. The 1.4 release is a result of community effort that I’m really proud to be part of.

I recommend all of you to update to 1.4 and also take a look at the latest version of your client library. See if it supports the binary protocol. If it doesn’t, you should bug the author about it ;)

Which reminds me, I need to ping Dormando and Hachi about the binary protocol patch that I wrote for Cache::Memcached last year…

Toru Maesaka memcached, oss memcached, release

As I’ve been tweeting the last couple of weeks, I’m going to do a web seminar on the next series of memcached (1.3) that is currently under beta release. I’m afraid there won’t be enough time to cover all the nuts and bolts but I will cover most of the great thoughts and engineering that went into the source tree by the community.

• http://www.mysql.com/news-and-events/web-seminars/display-323.html

Thank you for the opportunity Sun/MySQL and I’ll hopefully see you all there!

Toru Maesaka event, memcached, oss memcached, seminar

Something I’ve been wanting to add to libmemcached for a while is a cluster analysis feature that can be used to calculate useful information for system admins that deploy multiple memcached nodes. Many webshops already do this by writing their own so it makes sense to have this functionality in a community driven library. This means people can pitch in their useful stats calculation which results in benefiting everyone.

So, with that in mind I wrote a patch that contains the initial codebase for the analyzer which is now reviewed and merged into trunk. This also gave me a reason to claim the commit access that Brian has been offering me for nearly a year… (I know, I’m lazy and useless).

At the moment, you can get the following information with the new memcached_analyze(3) function:

• Average item size in the pool
• Node that is eating the most memory
• Node with least designated memory remaining
• Node with the longest uptime
• Pool-wide Hit Ratio

I know, there isn’t much at the moment but more will be added in upcoming pushes. The analyzer is pretty easy to extend so please feel free to throw patches for it! Or, you could pitch in by telling us useful figures that should be computed.

Hopefully this new feature will be released next month :)

Avoid the library by using memstat

The memstat tool (distributed with libmemcached) has been enhanced with the new ‘–analyze’ option that will use the analyzer and give you a human readable output. You should use memstat unless you need to explicitly use the values returned by the analyzer.

Using it is easy, all you need to do is specify the servers that you’d like to inspect and give it the ‘–analyze’ option like this (only two servers for demo purpose):

$ memstat --servers=localhost:11211,localhost:11311 --analyze
Memcached Cluster Analysis Report
 
    Number of Servers Analyzed         : 2
    Average Item Size (incl/overhead)  : 54 bytes
 
    Node with most memory consumption  : localhost:11211 (54 bytes)
    Node with least free space         : localhost:11211 (67108810 bytes remaining)
    Node with longest uptime           : localhost:11311 (780s)
    Pool-wide Hit Ratio                : 100%

Easy!

Toru Maesaka memcached, oss libmemcached, memcached

Ever wished that you could pack more data into a certain memcached instance without scaling up? AKA, reduce the need to scale-out, therefore save money on your infrastructure cost and electricity?

How about saving 8 bytes for every item that you cache? meaning if you cache 10 million small items, you save around 76 megabytes of RAM. Guess what? you can with the upcoming 1.3 series! and the code that achieves this by Trond Norbye is now merged into the 1.3 development tree.

I must mention though that this saving is only available to those that do not use the CAS feature. Keep reading to understand why…

The Magic

In the current release of memcached, an 8 byte value (uint64_t) is hardwired to the item structure which is now optional by specifying it at startup (the new ‘-C’ option). In the code level, the easiest way to describe it is that the CAS value is moved/aligned to the tail of the structure on memory. Needless to say, we know whether that 8 byte value exists or not from the startup option.

Observation

So, here is a very modest test that sets one-hundred-thousand items (unless there is a key clash…).

Firstly the Perl script:

#!/usr/bin/perl
 
use strict;
use warnings;
 
use Cache::Memcached;
use constant {
    NITEMS => 100000,
    KLEN   => 8,
    VLEN   => 16,
};
 
sub random_key {
    my @chars = ('a'..'z', 'A'..'Z', '0'..'9');
    my $buf = "";
 
    foreach (1..KLEN) {
        $buf .= $chars[rand @chars];
    }
    return $buf;
}
 
my $memc = Cache::Memcached->new ({
    servers => ['localhost:11211'],
    debug   => 0,
});
 
my $val = 'a' x VLEN;
 
for (1..NITEMS) {
    $memc->set(random_key, $val);
}

Running this script against an instance with standard growth factor:

 memcached -m 1024 -d

produced the following statistics on number of items and bytes consumed:

...
STAT bytes 9000000
STAT curr_items 100000
STAT total_items 100000
STAT evictions 0
END

Similarly, running the above script against an instance with the new option (‘-C’):

memcached -m 1024 -C -d

had produced the following statistics:

...
STAT bytes 8200000
STAT curr_items 100000
STAT total_items 100000
STAT evictions 0
END

Thats a saving of 781 kilobytes batman!

This is great news for those that cache lots of small objects. For example, at mixi.jp we are probably the largest user of memcached in Japan and turns out the granularity of our cache is darn fine. This patch is going to hit us big.

++ to Domas for groaning… I mean indicating this improvement opportunity and Trond for hacking it :)

Toru Maesaka memcached, oss

Google App Engine (GAE) is something I’ve been meaning to look into for personal interest but have been failing to do up until now due to lazyness and being relatively busy.

So specifically, I’m interested in the Datastore API and the Memcache API since well, thats what I do. For those that aren’t familiar with GAE, it is a platform provided by Google that allows you to run your web application on their infrastructure. Using the Google infrastructure is done through a set of provided APIs and they take care of Scaling and HA issues for you. This means you don’t have to invest into hardware (elastic running cost) nor have to repair anything (other than your code of course). So, its a typical example of PaaS.

Taking a look at the Memcache API

Nowadays its gradually becoming common knowledge in the web industry that using memcached can help your site scale and reduce the response time dramatically in a cost-efficient fashion (adding a DB Slave vs memcached node). The question is, what’s behind Google’s Memcache API? On the App Engine documentation, it is only stated that:

The Memcache API has similar features to and is compatible with memcached by Danga Interactive.

So, its actuallly not stated that the backend is powered by memcached despite the name. This means that the backend can be anything like a distributed Google Sparse Hash over the wire. I guess what’s important is not so much the cache daemon but by keeping the interface consistent with memcached, developers that are familiar with memcached can use GAE without allergic reactions. Not to mention, memcached has a brilliant interface for a distributed cache.

Caching your data on GAE is uver simple. You first import the ‘memcache’ module from the GAE package:

from google.appengine.api import memcache

then call the appropriate API method for whatever it is that you want to do.

Just for fun I tried setting a value using a key thats longer than 250 bytes since the maximum length of a key that memcached will accept over the ASCII protocol is 250 bytes (aka 250 ASCII characters). So how about the App Engine?

from google.appengine.api import memcache
 
memcache.flush_all()
test_key = 'x' * 300
 
if not memcache.set(test_key, 'some_val'):
    print 'Failed to set'
    quit()
 
print "Looks like we're good = " + memcache.get(test_key)

Well, turns out this code didn’t run with this error message from my local app server:

Keys may not be more than 250 bytes in length, received 300 bytes

Hehe, this looks very memcached to me but who knows, this could also be deliberate to keep things consistent with memcached.

Memcache API and Datastore API in Action

Okay, so to see if the Memcache API + Datastore API performs just like what you would expect from memcached + MySQL, I wrote a simple GAE Web Application. Here is the sourcecode and screenshots of the application actually running on Google:

All it does is, it populates your Cache and Persistent Storage with 64 rows that are 4KB each (so, 256KB in total) and measures how long it takes to bring it over to the application layer. This is obviously not enough to simulate data transfer in a real world web application but I figured its enough to make a point.

So as expected, retrieving data is faster by using the memcache API and in theory this performance should not degrade and run constantly even with increased concurrent connections and requests. On the other hand, performance of the Datastore API _could_ degrade. I’m saying “could” because as much as I’d like to prove this point, I didn’t really want to ab Google.

Btw, after quickly looking at the caching code in the SDK, it seems Memcache is emulated using Python’s Dictionary on the local development environment.

Taking a look into Cached Bytes

Conveniently, the Memcache API provides a simple way to fetch the amount of bytes that is currently being cached for you:

from google.appengine.api import memcache
 
stats = memcache.get_stats()
if stats: print stats['bytes']

Being a curious individual and a great stalker, I decided to use this information to compare whatever it is thats behind the Memcache API with memcached. You see, with memcached you don’t get the exact number of key/value bytes that you sent over the wire because memcached reports the total number of bytes it had consumed, including overheads per item (as it should). In other words, what memcached reports is “unique”.

So, below is what I got from comparing the Memcache API (on Google’s infrastructure) and the latest release of memcached (1.2.6) at the point of this blog entry:

Wow, according to the above results, Google’s Memcache backend is not showing any overhead in its report. Maybe it is a sparse map over the wire after all. But like I mentioned earlier, it doesn’t really matter what’s behind the API because what’s actually important is that its easy for us end-users to use and that it performs in an O(1) manner.

Conclusion

The Google App Engine Documentation rocks! like I mentioned on Twitter, the team that worked on the documentation should get a medal. It got me started in no time and gave me just enough information to start doing my own thing without getting frustrated from excessive information.

There are still unresolved questions like how sharding works for the Memcache API. I mean, do each application get a dedicated server instance(s) or are keys appended/prepended with an app_id in the background? The latter approach sounds simple and effective but it opens up another question of stats management. I guess a housekeeping index for each application would get around this issue but there is no programmable way from the outside to confirm this.

On a different note, I should stop being a stalker and just enjoy what’s been provided (though this is a really difficult thing to do once you dive into the world of engineering) :)

Toru Maesaka memcached gae, google, memcached

Last week I was in the valley for the fifth memcached Hackathon at Sun Microsystems and visiting some friends at Six Apart HQ. The hackathon was so fun, we ended up leaving at 2am on a weeknight! Thanks to Matt Ingenthron and Sun Microsystems for organizing the event and providing food and space for this hackathon :)

In the previous hackathon, we mostly exchanged ideas on the binary protocol and the storage engine interface. This time it was more code oriented and we reviewed and tested the progress everyone had made in the latest binary protocol tree. Unfortunately I couldn’t cover the whole hackathon but here is a summary of discussions from the agenda that I was involved in:

Binary Protocol – Add an engine specific OPCODE

No disagreements here. An opcode is represented by a 1 byte unsigned integer so the consensus was that we should dedicate anything over 127 (0x7F) for special operations.

Storage Interface

We didn’t get around to discussing the interface in depth since getting the binary protocol released has greater priority at the moment. Trond however showed me some of the interesting work that he has been doing which will hopefully be out in the open soon.

Test Framework

The issue here is that tests aren’t actively been written. Opinions voiced on this issue was that some people aren’t comfortable with Perl, and thus difficult to understand the current Perl based test system.

Switching to a different test framework in a different language is easy but the problem is that this is a never-ending story. People can easily start demanding other languages that they feel comfortable in (python, java, ruby, lua, …). We briefly discussed that the ideal model is to be able to add tests written in any language but we didn’t go into depth on how we would actually achieve this.

Personally, I have nothing against the current test framework (mind you I like Perl) but I think if we were to switch, a solely C based framework is a good move. I am saying this because those that would think about opening up the memcached package and editing it can most likely write C (is this an assertive assumption? heh).

Client Libraries

Unfortunately I couldn’t get around to participating in the client talk but client-side replication work was being done for libmemcached and I heard from Brian Aker that there was good progress.

Jonathan (hachi) reviewed my binary protocol patch for Cache::Memcached and found that some protocol negotiation assumptions I made in the code can be improved. He is also looking at optimizing the code by subclassing the patch (reduces the number of conditional selections, perl method calls and hash lookups).

Scaling on Highly Threaded Servers

We didn’t really discuss this in depth since we were busy reviewing and testing the server code but as far as I know, we talked about how locking can be improved in memcached. Looking into and preparing for this is a good idea since we are entering a massively concurrent age. To the contrary, guys from Facebook mentioned that they were getting sufficient throughput with the current locking scheme which was awesome to hear.

The engine plugin rearchitecture should fit well with this project since we can interchange different versions of the slabber engine with different locking strategies and make them compete to be the next default memcached engine.

Conclusion

The hackathon was fun and we got a lot done in terms of finding things to improve on. It was great to catch up with guys that I communicate a lot with online and talk tech in person. It was awesome that Brad turned up as well. As for code improvements, Dustin’s test code found an issue in the stats subsystem always returning a zero for an opaque value. A little bit of coding looked necessary to get around this problem since an opaque value is held by the connection structure, which the engine does not have access to (it shouldn’t) but I was bored on my flight back to Tokyo so this problem is now fixed and pushed to my tree :)

Toru Maesaka memcached, oss hackathon, memcached