Toru Maesaka

Something I’ve added to BlitzDB recently that was pretty high on my todo list is support for variable width tables. So what is a variable width table? it is a table that contains columns that can vary in size, namely BLOB and TEXT types.

Going back to the basics, when a new row is to be written, a storage engine is given a pointer to the row data in MySQL format that it must somehow store for later lookup/retrieval. By meaning “somehow”, the storage engine is given the freedom to do whatever it likes with the row.

Writing a row for a fixed length table (a table with columns that are always the same size) is deadly easy. A storage engine can choose to not temper with the row and simply write or copy the data to it’s storage mechanism. This is because the storage engine is given a row that contains all the data. Rows for variable width tables however, are treated differently since things aren’t as simple (it’s variable!).

The difference is that columns for BLOB and TEXT types are represented by two parts inside a MySQL/Drizzle row:

• length of the data
• pointer to the actual data

This is simple to understand since we need to know the size of the data to copy it.

Minor Complication

The minor complication as you would expect here is that you can’t directly write the provided row to your engine like you can with fixed length tables. The data that you want to copy/write exists elsewhere (hence the pointer) so directly writing the row has no meaning (the data would have disappeared by your next access to that row). You need to make sure that the actual data for BLOB/TEXT column(s) are arranged appropriately on your engine’s row buffer and written out to it’s storage mechanism.

This process is commonly referred to as row packing (converting to your engine format) and unpacking (convert back to MySQL format). So how is this done? it’s actually pretty simple!

The solution is actually simple

As much as it sounds like a bother to support variable length rows, it’s actually not that bad. First you need to understand what a MySQL row looks like internally.

A MySQL row begins with a bitset that represents which fields are NULL. The length of this data obviously depends on the number of NULLable columns you have but this is easy to handle with Drizzle since we’re given all the relevant information by the TableShare object (same goes for MySQL from a different object).

After this data comes the actual column data in the order that appears in your CREATE TABLE statement. What you need to do to get packing working with this row is the not-so-obvious part that you really need an example to look at. Fortunately Tweeting about this attracted Brian’s attention which helped me move forward.

Loop the fields!

So, let’s take row insertion to a variable width table as an example. Imagine this table:

CREATE TABLE t1 (
  id int PRIMARY KEY NOT NULL,
  description text,
  arbitrary_data blob
) engine=your_engine;

and let’s imagine that we need to process this query:

INSERT INTO t1 VALUES (1, "hello world", "blobbbbb");

Now, the storage engine needs to “pack” the data for each column into it’s buffer in the write_row() function. Conveniently, Drizzle/MySQL provides a pack() function for it’s column types (fields) that will do the data packing for you. That is, you do not have to inspect the provided row for pointers to the actual data and do the packing/copying yourself.

How? well, the table object (which is visible from your engine) conveniently holds a list of fields in the appropriate order. The actual pack() function is a member of these fields so you just need to call it as you loop over the list:

/* make sure row_buffer has enough memory */
unsigned char *pos = row_buffer;
 
/* copy NULL bits, "table->s" is the TableShare object */
memcpy(pos, row, table->s->null_bytes);
pos += table->s->null_bytes;
 
/* "row" is the MySQL formatted row given by the core */
for (Field **field = table->field; *field; field++) {
  if (!((*field)->is_null()))
    pos = (*field)->pack(pos, row + (*field)->offset(row));
}

The above code snippet will populate “row_buffer” with the actual data that you want to write to your storage mechanism. You do not have to forward the “pos” pointer because pack() returns a pointer at the end of where it had worked in the buffer (think Pascal Strings). This is precisely why we created the pos pointer, to avoid row_buffer from being forwarded.

For the opposite situation (when retrieving a row), an unpack() function is provided for each field so you just need to take advantage of it like we did with the pack() snippet above.

Little bit more on fields

The actual pack() function that gets called depends on the type of column since the Field class is an abstract base class for the sub classes that actually represents column types inside Drizzle/MySQL. If you want to know what a pack() function looks like for a BLOB type, grep for “Field_blob” in the source tree and there will be a pack() member function for it.

The code layout for field subsystem in MySQL is rather difficult to comprehend since everything is crammed in “sql/field.c” and “sql/field.h” files (at least as of 5.4). So, if you want to get a good grasp of how things are architectured, you should take a look at Drizzle. Field subclasses are located individually in the “drizzled/field/” directory and the base class is located in “drizzled/field.h”.

So, that’s about it! Hopefully this information will help other engine developers when they come across a need to support variable width tables

Toru Maesaka drizzle, knowledge, oss drizzle, engine, mysql, storage

If you’re working on developing a Drizzle plugin, you may come across situations where you want to accept user options for it at server startup. For example, if you design your plugin to create files for activity logging, you may want to allow the DBA to specify where to write those files out.

In my case, I decided to provide a command line option to BlitzDB for row based query caching. This option is intended for special use-cases where the read/write ratio is 9:1. For those that are interested, row caching is disabled by default because it creates overhead in the engine for read-through logic and cache invalidation _unless_ read requests are significantly higher than update requests.

There are situations where BlitzDB’s row cache can be helpful but this is beyond the scope of this entry so I will save it for another day

Adding startup options to your plugin

Drizzle allows you to add command line options to your plugin without editing the server code. But before you start hacking away, there are few not-so-obvious things that you need to understand.

So, let us first look at the data types that your plugin can accept:

• DRIZZLE_SYSVAR_BOOL
• DRIZZLE_SYSVAR_STR
• DRIZZLE_SYSVAR_INT
• DRIZZLE_SYSVAR_UINT
• DRIZZLE_SYSVAR_LONG
• DRIZZLE_SYSVAR_ULONG
• DRIZZLE_SYSVAR_LONGLONG
• DRIZZLE_SYSVAR_ULONGLONG
• DRIZZLE_SYSVAR_ENUM

As you can see, there is a wide range of types that you can choose from. What you should choose depends on what you want to use the value for.

Pick your data type

So lets take my row cache option as an example. Caching over 4 billion rows in one physical server is very unlikely and since we’re not interested in negative numbers, we’re going to pick:

• DRIZZLE_SYSVAR_UINT

which we can store the value as uint32_t in the plugin.

Declare that your plugin accepts options

Every plugin must declare itself as a plugin which looks like this for BlitzDB:

drizzle_declare_plugin(blitz) {
  "BLITZ",
  "0.3",
  "Toru Maesaka",
  "Non-transactional General Purpose Engine",
  PLUGIN_LICENSE_GPL,
  blitz_init,             /*  Plugin Init      */
  blitz_deinit,           /*  Plugin Deinit    */
  NULL,                   /*  status variables */
  blitz_system_variables, /*  system variables */
  NULL                    /*  config options   */
}
drizzle_declare_plugin_end;

Here, we’re interested in the second last argument which is called blitz_system_variables in the above example. Feel free to call this whatever you like for your plugin.

So what exactly is blitz_system_variables? Its a null-terminated array of system variables that your plugin accepts. This is what it looks like for BlitzDB:

static struct st_mysql_sys_var *blitz_system_variables[] = { 
  DRIZZLE_SYSVAR(row_cache),
  NULL
};

As you can see, BlitzDB only supports one option at the moment so there is only one entry called row_cache.

Define your options

You must define every option that you’ve added to the system variable array. We decided to use DRIZZLE_SYSVAR_UINT earlier and called it row_cache so it is defined like this:

static DRIZZLE_SYSVAR_UINT (
  row_cache, /* option name */
  blitz_row_cache_size, /* variable to set the value to */
  PLUGIN_VAR_READONLY, /* mode */
  N_("Enable row caching for BlitzDB tables."),
  NULL,       /*  check func    */
  NULL,       /*  update func   */
  0,          /*  default value */
  0,          /*  minimum value */
  UINT32_MAX, /*  maximum value */
  0           /*  block size    */
);

The comments pretty much explains what the arguments are but for more details, you should take a look at the macros in drizzled/plugin.h. You could also look at what other plugins do by grepping for the system variable type that you’re interested in.

Test your new startup option

If all goes well you should be able to compile Drizzle and check whether command line options are visible from the plugin. An option takes the following form:

--<name_of_plugin>-<option_name>

So, in the row cache example, row cache can be enabled like this:

/usr/local/sbin/drizzled --blitz-row_cache=10000

Also note that you can replace the underscore with a hyphen:

/usr/local/sbin/drizzled --blitz-row-cache=10000

That’s it! it should be relatively easy to add more options once you successfully get your first one done.

Toru Maesaka drizzle, knowledge, oss blitzdb, drizzle

Tokyo Cabinet (TC) provides iteration functionality for both it’s persistent and non-persistent data structures. For example, if you wanted to iterate through TC’s hash database, you can use the tchdbiternext() function. This is really straight forward to use such that:

void *key;
int key_len;
 
if (tchdbiterinit(tc_database_handle) != true) {
  /* failed to initialize iterator */
}
 
while ((key = tchdbiternext(tc_database_handle, &key_len)) != NULL) {
  /* work with the fetched key and key_len */
}

will iterate through the entire hash database that “tc_database_handle” object is responsible for. This can be handy if you need to loop through your database for some arbitrary reason.

However, there is a consequence in using this function in a concurrent environment with a use-case where the order of records _really_ matter. This is because even though TC is a thread-safe library, the iteration functions aren’t thread-safe in a way that we expect.

For example, if a write operation occurs while the application iterates over the database, you will end up iterating over a database that is in a changed state. This will not make the cursor go crazy and crash your application since TC handles this internally but you still end up iterating over a database that is in a state that you did not initially intend on looping through.

Solution to this is to simply block write operations to the database while your application iterates through. For example, you could use pthread’s rw_lock to allow other threads to read while you iterate but block writes until you finish iterating.

I was planning on doing this for a table scanner in the storage engine that I’m currently working on but turns out TC has an undocumented function that will take care of this internally. I’ve talked to Mikio about this function and apparently it is intentional that he hasn’t documented it on his specification page. He has no plans on throwing it out so you do not have to worry about it to magically disappear one day. For more information, you can take a look at his header file (tchdb.h for hash database).

Explanation and Simple Example

The function is called tchdbforeach() which will atomically iterate through your database from beginning to the end by supplying each key/value pair to the callback function that you provide. The signature of the callback is the following:

bool callback(const void *kbuf, int ksiz, const void *vbuf,
              int vsiz, void *op);

where the fifth argument, “void *op” is an opaque pointer to the data that you can pass to the callback. Here is a simple example that will increment a counter integer on each iteration using this function:

/* Do whatever you like with the provided key/value pair in here */
bool callback(const void *kbuf, int ksiz, const void *vbuf,
              int vsiz, void *op) {
  if (op == NULL)
    return false;
 
  *((int *)op) += 1;
 
  return true;
}
 
int main(void) {
  int niter = 0;
 
  ...
 
  if (!tchdbforeach(tc_database_handle, callback, &niter)) {
    fprintf(stderr, "failed to iterate the database\n");
    return EXIT_FAILURE;
  }
 
  printf("iterated %d times\n", niter);
 
  ...
 
  return EXIT_SUCCESS:
}

If all goes well, the counter variable will be set to the number of records in the database. This function is slightly more complex than using tchdbiternext() but you are guaranteed to iterate atomically which is pretty important for a table scanner.

I hope this function can help you too.

Toru Maesaka knowledge, oss storage, tip, tokyocabinet

I’ve decided to start a series of blog entries on not-so-obvious findings that I’ve found while working on my new project. By archiving the findings, I’m hoping that I can help those that are looking into developing a storage engine for the MySQL family in the future.

Accumulating these mini-knowledge would also be useful for me since I can refer back to it when I forget something. Also, once I write enough entries I’m planning on summarizing them and making it available on the Drizzle Wiki. If MySQL is interested in updating the engine documentation, I would be more than happy to help there too.

So to begin with, I’ll describe something trivial that I stumbled across while trying to catch an error on duplicate primary key insertion to the data table.

Background

In brief, the database kernel does not care if the INSERT query contains a duplicate primary key for a given table or not. It is the storage engine’s job to tell the kernel that the request was invalid due to key collision. If a storage engine fails to do this, the kernel will acknowledge that the query was successful (given that no other errors were thrown) and will keep doing what it needs to do.

Mechanics

Data insertion is handled inside the write_row() function that your engine must implement. The return value of this function is an integer that represents the status of the work it had done. After looking through the possible error statuses in “drizzled/base.h”, I immediately found this:

#define HA_ERR_FOUND_DUPP_KEY 121 /* Dupplicate key on write */

I also looked through MyISAM and InnoDB to confirm that this was indeed the correct error status to return on duplicate primary key. Here is the snippet of my row insertion at the time:

/* TC's tchdbputkeep will not insert a row to the table if there
   was a collision */
if (tchdbputkeep(data_table, primary_key, primary_key_length, buf,
                 table->s->reclength) == false) {
  my_errno = HA_ERR_GENERIC;
 
  /* check for primary key collision */
  if (tchdbecode(data_table) == TCEKEEP)
    my_errno = HA_ERR_FOUND_DUPP_KEY;
 
  return my_errno;
}

On first glimpse, this seems right but the error I was getting from the command line prompt always differed with MyISAM and InnoDB despite returning the same error status. Specifically, this is what I was getting:

ERROR 1022 (23000): Can't write; duplicate key in table 't1'

whereas I was getting this error on other engines:

ERROR 1062 (23000): Duplicate entry '1' for key 'PRIMARY'

At this stage I couldn’t make sense of what I was doing wrong but it turned out that the solution was pretty simple.

Solution

After talking to Stewart Smith about my issue in #drizzle @ freenode, it turned out I am supposed to keep track of which key the duplication was found in write_row() and inform it to the kernel via the info() function.

You can do this by setting the errkey integer variable to the key number that is used internally by the kernel. So, obtaining the internal primary key number with this call in write_row():

share->errkey = table->s->primary_key;

and adding the following code to info():

if (flag & HA_STATUS_ERRKEY) {
  errkey = share->errkey;
}

happily fixed the issue I was experiencing. Yay.

I guess reading the section on info() in the document gives a hint that this is where you supply the key number on key-error but frankly, this is really easy to forget and miss since the importance isn’t so emphasized.

Anyhow, thats all I have to say in the first of this series and hopefully I’ll write something more interesting in the upcoming entries. Until then, happy hacking

Toru Maesaka drizzle, knowledge, oss drizzle, engine, mysql, storage

Something notable that has changed in Drizzle this week is the build system for plugins.

Previously we were using the old plugin system that was inherited from MySQL but Drizzle now uses a Python based system that allows us to aggregate your plugin build rules to the top level Makefile. This change also gets rid of the nasty behavior that was giving people like Monty Taylor and other build system hackers heachaches.

But hey, as a plugin developer you’re not so interested in how things are handled cleanly inside right? you’re more interested in how to create or port your plugin over to the new build system! As a developer, you are interested in the following three files:

• plugin.ini
• plugin.ac
• plugin.am

where the mandatory file is plugin.ini. This file is where you write the basic details of the plugin like the name, source files and relevant compiler options. For example this is what plugin.ini looks like for the Blackhole storage engine:

[plugin]
name=blackhole
title=Blackhole Storage Engine
description=Basic Write-only Read-never tables
sources=ha_blackhole.cc
headers=ha_blackhole.h

You can also specify the plugin to be loaded by default with this line: “load_by_default=yes”. If you don’t add this line, the plugin is enabled by specifying it with the “–plugin_load” server startup option.

As for the optional plugin.ac and plugin.am files, these are where you can add your own autoconf and automake rules for the plugin. For example you might want to check/search for a library or build an internal library for your plugin.

Example of Linking an external library

If you write a plugin, you’ll most likely want to link a particular library to your program. After all, thats one of the major points of writing a plugin right? to bring the external goodness over to the database server for solving a particular need/requirement in your application.

For those that are interested, I’ll leave a snippet of how I linked Tokyo Cabinet to the storage engine I am currently working on. Firstly, you want to search whether Tokyo Cabinet exists in the environment that you’re building in. Clearly this is what configure is for so I added this to my plugin.ac:

AC_LIB_HAVE_LINKFLAGS(tokyocabinet,,
  [#include <tchdb.h>],
  [
     TCHDB hdb;
  ])  
  AS_IF([test "x$ac_cv_libtokyocabinet" = "xno"],
        AC_MSG_WARN([tokyocabinet not found: not building plugin.]))
DRIZZLED_PLUGIN_DEP_LIBS="${DRIZZLED_PLUGIN_DEP_LIBS} ${LTLIBTOKYOCABINET}"

The above will check for Tokyo Cabinet and whether the TCHDB structure exists. If it doesn’t exist then it will print a warning but if it does exist, it will add the linker option to plugin dependencies. You can now tell the build system to link Tokyo Cabinet, which you do by assigning the LTLIBTOKYOCABINET variable to ldflags in plugin.ini:

ldflags=${LTLIBTOKYOCABINET}

You could directly write “ldflags = -ltokyocabinet” to plugin.ini but you really want to take advantage of configure. configure (more rather autotools) is your friend.

So, this is all I have to cover in this entry and I hope this entry will be helpful to those that are looking into working on a Drizzle plugin. If you would like more information, the Drizzle Wiki should be updated with more detailed explanation soon.

Toru Maesaka drizzle, oss autotools, drizzle, plugin

I just updated my Drizzle trunk and what’d you know? our revision has hit four figures.

This may not seem like a big deal but to me, it represents how much hard work, effort and thought by those that believe in the Drizzle project has gone into the tree. Of course, we’re not going to slow down and I suspect things are going to comfortably accelerate from here on.

Toru Maesaka drizzle, oss drizzle

As I’ve been tweeting the last couple of weeks, I’m going to do a web seminar on the next series of memcached (1.3) that is currently under beta release. I’m afraid there won’t be enough time to cover all the nuts and bolts but I will cover most of the great thoughts and engineering that went into the source tree by the community.

• http://www.mysql.com/news-and-events/web-seminars/display-323.html

Thank you for the opportunity Sun/MySQL and I’ll hopefully see you all there!

Toru Maesaka event, memcached, oss memcached, seminar

Today I was doing some work on writing a small storage engine for Drizzle that we’re hoping will perform good enough to replace the MEMORY/HEAP engine inherited from MySQL. For those that are interested, this engine is going to be based on Tokyo Cabinet’s on-memory b+tree database.

I was a little worried that this task is going to be heavy since I’ve never hacked on a relational database engine before (other than tracing through InnoDB) but it turns out it’s not so bad. I first looked up the engine interface for MySQL then looked at the engines in the Drizzle tree. The interface itself looks very similar but what’s really cool about Drizzle is how the storage engine is registered to the core server with the new plugin system. Here’s a snippet of how this looks with the Blackhole engine:

blackhole_engine= new BlackholeEngine(engine_name);
registry.add(blackhole_engine);

and the de-initialization:

registry.remove(blackhole_engine);
delete blackhole_engine;

If you’re familiar with how a Drizzle plugin is written, you will notice that this is exactly the same; You implement the interface and boom, you register it with the PluginRegistry object. So a storage engine is treated equally just like any other plugin system.

Needless to say, the storage engine declaration is exactly the same as what you would do with any Drizzle plugin:

drizzle_declare_plugin(blackhole)
{
  "BLACKHOLE",
  "1.0",
  "MySQL AB",
  "/dev/null storage engine (anything you write to it disappears)",
  PLUGIN_LICENSE_GPL,
  blackhole_init, /* Plugin Init */
  blackhole_fini, /* Plugin Deinit */
  NULL,           /* status variables */
  NULL,           /* system variables */
  NULL            /* config options */
}
drizzle_declare_plugin_end;

This is nice! a storage engine is just a plugin after all and Drizzle treats it as it ought to be. I will have some more time to hack on this after my memcached webinar so hopefully I’ll have more details posted about the progress on this work soon.

Toru Maesaka drizzle, oss drizzle, engine, plugin, storage

I’m writing this entry on my way back to Tokyo from Narita. So, I was in the US all week for MySQL UC, Percona Performance Conference and the Drizzle Developer Day. It was great to meet new people and also catch up with developer friends from all over the world. These events are great excuse to bring together folks that work together online and receive the free beers that were promised on IRC. Looking back, the week just flew! I can’t believe I’m back in Japan already.

What wasn’t pleasant however was Drizzle being introduced as “MySQL Drizzle” and described as MySQL’s technology incubator at the opening keynote. The truth is, Drizzle is a community driven project that is not affiliated with any commercial organization. The project is shepherded by the community and that is the whole point of our model. Jay and Baron has expressed this on their blogs too:

• Sorry, but Drizzle is Not…
• Drizzle is a MySQL Technology Incubator?

Guess there is no point in getting worked up about this so I will say no more.

Drizzle developer day turned out to be a great success with over 60 developers (including new developers) turning up. We covered wide variety of topics from “how to download bzr” for all levels of contributors which I thought was nice. Why? well this means that everyone has something to do and it creates a welcoming atmosphere. As a result, it makes the event active. Here are some photos I uploaded to flickr in the last few minutes I had in the US. I’ll get more up there asap.

Compared to these stimulating events and the discussions we had, Oracle’s acquisition tale that many people seemed to like talking about at the conference breakfast/lunch/boozing was uninteresting.

Toru Maesaka drizzle, travel conference, drizzle, mysql

As I mentioned in my previous entry, InnoDB has it’s own auto increment counter which it uses to generate the next value for the database kernel (as we call it in Drizzle). At Drizzle project, we came across a suspicion that InnoDB doesn’t increment it’s internal counter on row updates. So what can this mean to you as a database admin?

Well, consider this simple table:

CREATE TABLE t1 (
    a INT NOT NULL AUTO_INCREMENT PRIMARY KEY,
    val INT
) ENGINE=InnoDB;

and the following statements:

drizzle> INSERT INTO t1 (val) VALUES (1);
Query OK, 1 row affected (0.01 sec)
 
drizzle> UPDATE t1 SET a=4 WHERE a=1;
Query OK, 1 row affected (0.01 sec)

We should now have a table with one row where the primary key value is 4:

drizzle> SELECT * FROM t1;
+---+------+
| a | val  |
+---+------+
| 4 |    1 | 
+---+------+
1 row IN SET (0.00 sec)

So far so good! Now remember our hypothesis that InnoDB doesn’t temper with the internal counter on an update. This would mean that our next row should have a primary key value of 2 since the counter variable should still be at 1 from our initial INSERT statement:

drizzle> INSERT INTO t1 (val) VALUES (1);
Query OK, 1 row affected (0.01 sec)
 
drizzle> SELECT * FROM t1;
+---+------+
| a | val  |
+---+------+
| 2 |    1 | 
| 4 |    1 | 
+---+------+
2 rows IN SET (0.00 sec)
 
drizzle> SELECT LAST_INSERT_ID();
+------------------+
| LAST_INSERT_ID() |
+------------------+
|                2 | 
+------------------+
1 row IN SET (0.00 sec)

As expected, we have a new row with a primary key value of 2. But hey, what happens if we keep inserting into this table? specifically twice (assuming that you haven’t changed the auto-increment-increment system variable).

drizzle> INSERT INTO t1 (val) VALUES (1);
Query OK, 1 row affected (0.01 sec)
 
drizzle> INSERT INTO t1 (val) VALUES (1);
ERROR 1062 (23000): Duplicate entry '4' FOR KEY 'PRIMARY'

Yep, a duplicate key error! This isn’t really surprising since the increment was resumed from a value less than what we had updated the first value to. I’m not even sure if this sort of operation would be done in production but it shows how this behavior can be dangerous.

Something to note here is that, even on the INSERT error that we just experienced, InnoDB will update the internal count because get_auto_increment() was called when processing the query. So, this means that the next INSERT query will work fine since we’ve now passed the maximum value:

drizzle> INSERT INTO t1 (val) VALUES (1);
Query OK, 1 row affected (0.02 sec)
 
drizzle> SELECT * FROM t1;
+---+------+
| a | val  |
+---+------+
| 2 |    1 | 
| 3 |    1 | 
| 4 |    1 | 
| 5 |    1 | 
+---+------+
4 rows IN SET (0.00 sec)

Great! now we can hopefully keep incrementing happily.

I’ve confirmed the internal behavior of this example with GDB and also verified the same behavior in MySQL 5.1.30. Whether this behavior is inappropriate or not deserves a discussion of it’s own and you could of course, absorb this problem in the application layer but the important question here is, what is the right behavior for Drizzle?

Well, take a look at how MyISAM handles the same set of queries:

drizzle> INSERT INTO t2 (val) VALUES (1);
Query OK, 1 row affected (0.00 sec)
 
drizzle> UPDATE t2 SET a=4 WHERE a=1;
Query OK, 1 row affected (0.00 sec)
Rows matched: 1  Changed: 1  Warnings: 0
 
drizzle> INSERT INTO t2 (val) VALUES (1);
Query OK, 1 row affected (0.00 sec)
 
drizzle> SELECT * FROM t2;
+---+------+
| a | val  |
+---+------+
| 4 |    1 | 
| 5 |    1 | 
+---+------+
2 rows IN SET (0.00 sec)
 
drizzle> SELECT last_insert_id();
+------------------+
| last_insert_id() |
+------------------+
|                5 | 
+------------------+
1 row IN SET (0.00 sec)

As you can see above, MyISAM will resume incrementing from the updated primary key value. To me this is the correct behavior that should be consistently provided to DBAs. So the next question is, how can we make InnoDB behave that way?

Ideas on tackling this issue

A simple solution would be to make InnoDB update the internal count on updates (e.g. ha_innobase::update_row) when it deals with with auto increment. This approach, however can end up being complicated (code wise) and worst of all, it can create a minor overhead to the update operation on InnoDB which may not be favorable for our target users.

So after briefly discussing this issue with Monty Taylor, seems the appropriate fix for this is to leave ha_innobase::update_row() as it is (don’t increment the internal count) and handle this on INSERT. Specifically speaking, on a row that doesn’t have a user specified auto increment value.

This means that we can recalculate the auto increment value on duplication/collision since the user is expecting an auto-generated discrete value. Needless to say, an error must be thrown if a collision occurs on a user-specified value.

So yeah, I’m flying to the US in couple of days so this should be something fun to hack on amongst other work related tasks on my boring flight

Toru Maesaka drizzle, oss drizzle, engine, innodb, mysql