My first thought was that the hash chain in your database is broken but the fact that you can directly access your record means that’s unlikely. I’m not sure how you broke the file but if you know the keys that are in the database (e.g. there’s a sequential pattern), I recommend writing a short code that rebuilds a new database file (so copy each record to the new database at a time).

Either that or you could directly ask Mikio Hirabayashi for help/consulting since he’s the sole author of Tokyo Cabinet.

Thanks toru for your reply.

Yes, i can get the records which can not be iterated, （for example, i tested a key “149511433″, tchdbget returns the correct value), i dumped the key list using two methods:

1. use tchdbiternext:
while (true)
{
iLen = 0;
if ((pszKey = (char*)tchdbiternext(pstTCHDB, &iLen)) != NULL)
{
printf(“%d\n”, atoi(pszKey));
free(pszKey);
}
else if (pszKey == NULL && iLen == 0)
{
printf(“tchdbiternext2 returns NULL, err=%d, msg=%s\n”, tchdbecode(pstTCHDB), tchdberrmsg(tchdbecode(pstTCHDB)));
break;
}
else if (pszKey == NULL)
{
printf(“should never reach here\n”);
}
}

2. and the second way:
if (!tchdbforeach(pstTCHDB, DumpKeyCallback, &niter))
{
printf(“tchdbforeach failed\n”);
return -1;
}

the result of them are equal, they all dumped 10w+ records, (not include my test key “149511433″).

Also, after doing optimization on the TC hash-db file, the records number reduced to 10w+ and the file size falls sharply.(before optimize, i uses tchmgr which shows my records number is 879w+)

I checked all my TC files, half of them are “broken” (records number falls sharply after optimization).

Yes, this is strange, i also performed some tests before deploying TT+TC in product environment, So i’m trying to find out whether there is someone else met the same problem.

It would help if you could either email or write your loop condition here.

Do you have any clue what happened?

So, by meaning key refetch, does the core do this with rnd_next()? If so, I would have thought a refetch would be beyond rnd_next()’s responsibility…

We could keep track of how many times the core iterated inside the engine and return the same key as it returned previously. Either that or we could keep a copy of the “previous key” and fetch the value, which could be more computation friendly.

I’ll ping you on IRC about this when I come across it.