Toru Maesaka

The flexibility to add table-specific options for things like compression, encryption and optimization can be useful to storage engine developers as this flexibility can open up new possibilities. Here’s what I’m talking about:

CREATE TABLE t1 (
  ...
) ENGINE = my_engine, MY_OPTION = your_arg;

Supporting this is relatively easy in Drizzle and this API feature (and a bit more) is available in MariaDB as well. Unfortunately Drizzle’s method to do this isn’t documented in the Wiki yet but it should be added when our Storage Engine API becomes stable (as in, no interface changes).

Implement StorageEngine::doValidateTableOptions()

Here’s the actual interface.

bool StorageEngine::doValidateTableOptions(const std::string &key,
                                           const std::string &state);

This function is called for each table options given at CREATE TABLE syntax execution. The first argument, key is a const reference to a string that represents the option name. The second argument, state represents the argument given for that option.

Therefore, given: COMPRESSION = YES_PLEASE, key would be “COMPRESSION” and state would be “YES_PLEASE”. The objective of this function is to check whether the key/state pair makes sense to your storage engine. If this function returns false, Drizzle will return an error for the CREATE TABLE query. Personally I think this interface can be improved to be a bit more Developer friendly, such as making life easier to validate numeric values without enforcing the developer to play around with the data. Saying that, given the pace that Drizzle is growing, this could be improved before we know it.

Access Options at StorageEngine::doCreateTable()

Here’s the actual interface for doCreateTable().

int doCreateTable(drizzled::Session &session,
                  drizzled::Table &table_arg,
                  const drizzled::TableIdentifier &identifier,
                  drizzled::message::Table &table_proto);

Given that the options were successfully validated, doCreateTable() is called next. In Drizzle, all information regarding a table (including options) is represented in a Google Protocol Buffer message. A reference to that message object is passed to doCreateTable() as the fourth argument so all you need to do is loop through the options list in the message object and extract what you need. Here’s a minimal example that only takes care of one option.

int n_options = table_proto.engine().options_size();
 
for (int i = 0; i < n_options; i++) {
  if (table_proto.engine().options(i).name() == "my_option_name") {
    // Do whatever you like with this stream.
    std::istringstream stream(table_proto.engine().options(i).state());
  }
}

The above example should be simple to extend to handle multiple options. What’s really important in the above example is that the option name can be accessed with the name() accessor and the state (value) of that option with the state() accessor.

So, that’s all I have to cover for now. I hope this feature will help storage engine developers create and provide useful table specific features for their engine.

Happy Hacking.

Toru Maesaka drizzle, oss api, drizzle

Good news to storage engine developers. In Drizzle, you can now place your engine specific test files (.test and .result) in your engine’s directory. Here’s an example in BlitzDB:

First, let’s look inside BlitzDB’s directory.

$ ls -l blitzdb/
Total 60
-rw-r--r-- 1 maesaka maesaka   649 2009-12-13 20:51 AUTHORS
-rw-r--r-- 1 maesaka maesaka  5878 2009-12-13 20:51 blitzdata.cc
-rw-r--r-- 1 maesaka maesaka  3347 2009-12-13 20:51 blitzlock.cc
-rw-r--r-- 1 maesaka maesaka 18146 2009-12-13 20:51 ha_blitz.cc
-rw-r--r-- 1 maesaka maesaka  8360 2009-12-13 20:51 ha_blitz.h
-rw-r--r-- 1 maesaka maesaka   289 2009-12-13 20:51 plugin.ac
-rw-r--r-- 1 maesaka maesaka   261 2009-12-13 23:51 plugin.ini
drwxr-xr-x 4 maesaka maesaka  4096 2009-12-13 23:51 tests

Notice the final line? that’s where the tests are kept. So, let’s look inside it.

$ ls -l blitzdb/tests/
Total 8
drwxr-xr-x 2 maesaka maesaka 4096 2009-12-13 23:51 r
drwxr-xr-x 2 maesaka maesaka 4096 2009-12-13 23:51 t

As you can see, there are two directories. By now, storage engine developers would have caught on to what’s going on. The r/ directory is where the .result files are kept and t/ is where the .test files are kept. This is exactly the same layout as what we’re used to working on (“src/tests/t/” and “src/tests/r/”).

$ ls -l blitzdb/tests/t/
Total 8
-rw-r--r-- 1 maesaka maesaka   21 2009-12-13 23:51 blitzdb-master.opt
-rw-r--r-- 1 maesaka maesaka 1964 2009-12-13 23:51 blitzdb.test

The .opt file is used to make sure that the server is started with your storage engine enabled. You simply write the startup option inside the .opt file. Here’s what mine looks like at the moment (there’s only a single line in it).

$ less blitzdb/tests/t/blitzdb-master.opt
--plugin_add=blitzdb
blitzdb/tests/t/blitzdb-master.opt (END)

Next step is actually running it. You simply specify your engine name with the --suite option to dtr and you’re done! Unfortunately the symlink permission for dtr seems broken on my repository so I’ll directly call test-run.pl in this example.

$ ./test-run.pl --suite=blitzdb
Logging: ./test-run.pl --suite=blitzdb
MySQL Version 2009.12.1245
Use of uninitialized value in scalar assignment at ./test-run.pl line 1416.
Using MTR_BUILD_THREAD      = -69.4
Using MASTER_MYPORT         = 9306
Using MASTER_MYPORT1        = 9307
Using SLAVE_MYPORT          = 9308
Using SLAVE_MYPORT1         = 9309
Using SLAVE_MYPORT2         = 9310
Using MC_PORT               = 9316
Killing Possible Leftover Processes
Removing Stale Files
Creating Directories
=======================================================
DEFAULT STORAGE ENGINE: innodb
TEST                           RESULT         TIME (ms)
-------------------------------------------------------
 
blitzdb.blitzdb                [ pass ]             63
-------------------------------------------------------
Stopping All Servers
All 1 tests were successful.
The servers were restarted 1 times
Spent 0.063 of 2 seconds executing testcases

That’s it! I really like this change since it makes sense for engine-specific tests to belong inside the storage engine’s directory. It makes conceptual sense and it’s a good step towards differentiating the database kernel and the storage engine, which Monty Taylor is actively hacking on. Hopefully he’ll blog more about these changes soon.

Toru Maesaka drizzle, oss api, tests

Admittedly I’m a lazy guy. Due to this nature I’m a little behind on the updates made to the Drizzle Storage Engine API. From lurking through the source code in the Drizzle trunk, I’ve noticed these changes.

• Engines can now have an alias
• handler class is replaced by the Cursor class
• Engine handler is now a subclass of Cursor
• Table definitions are handled/stored by the storage engine
• doCreateTable(), doDropTable(), doGetTableNames(), doGetTableDefinition()

Currently I’m trying to catch up with the updated Drizzle Storage API and take this opportunity to rewrite most of BlitzDB. The reason is that the more I understand TC internal, the more mistakes I realized that I’ve made. I’ll blog more about this soon. Instead, I’m going to introduce something small but nice today.

A while back I poked folks like Monty Taylor and Stewart Smith that it would be cool if engines could have an alias. I mentioned this because InnoDB was allowed to use both “innodb” and “innobase” in the system whereas other engines could only have one name. Another reason I was interested in this issue was that I couldn’t understand how InnoDB could use two names since there was no way to do this in the old interface. Turns out InnoDB was treated specially in the core, which is obviously not desirable in a microkernel philosophy.

In the current Drizzle Storage Engine API, there is a function called addAlias(). By calling this function inside the storage engine constructor, you can allow your engine to have multiple aliases. For experiment purposes I wrote this to BlitzDB:

BlitzEngine(const string &name_arg)
  : drizzled::plugin::StorageEngine(name_arg, HTON_CAN_RECREATE) {
  table_definition_ext = drizzled::plugin::DEFAULT_DEFINITION_FILE_EXT;
  addAlias("BLITZ");
  addAlias("TCDB");
}

Here, I added aliases BLITZ and TCDB. TCDB is my way of showing respect to Mikio and Tokyo Cabinet. So given the above, we should now be able to create tables with three names.

drizzle> CREATE TABLE t1 (foo int) engine=blitzdb;
Query OK, 0 rows affected (0 sec)
 
drizzle> CREATE TABLE t2 (foo int) engine=blitz;
Query OK, 0 rows affected (0 sec)
 
drizzle> CREATE TABLE t3 (foo int) engine=tcdb;
Query OK, 0 rows affected (0.01 sec)

Success! Yep, this is something so trivial that I think most people wouldn’t care about but I was happy to see this update in the trunk :)

Toru Maesaka drizzle api, drizzle, storage