使用spy memcached client作为Tokyo Tyrant client lib发现的问题。 博客分类： Tokyo Tyrant memcached.netTokyoCabinet网络协议Access

Tokyo Cabinet是一个高效的DBM， 而Tokyo Tyrant是为Tokyo Cabinet的提供了一个网络结构使得Tokyo Cabinet可以实现Cluster。下面是官网的介绍。

引用

Tokyo Cabinet is a library of routines for managing a database. The database is a simple data file containing records, each is a pair of a key and a value. Every key and value is serial bytes with variable length. Both binary data and character string can be used as a key and a value. There is neither concept of data tables nor data types. Records are organized in hash table, B+ tree, or fixed-length array.

引用

Tokyo Tyrant is a package of network interface to the DBM called Tokyo Cabinet. Though the DBM has high performance, you might bother in case that multiple processes share the same database, or remote processes access the database. Thus, Tokyo Tyrant is provided for concurrent and remote connections to Tokyo Cabinet. It is composed of the server process managing a database and its access library for client applications.

http://1978th.net/tokyocabinet/spex-en.html#installation

http://1978th.net/tokyotyrant/spex.html#installation

因为Tokyo Tyrant是兼容memcached和http协议的，所以我们可以使用memcached client来操作tokyo cabinet。非常方便。 不过今天我在使用spy memcached测试tokyo cabinet的性能时，发现如果保存的value是一个比较大的string时，get出来的value变成乱码。经过调试发现，原来spy memcached在遇到超过compressionThreshold（默认是16384）时会对value进行压缩，但是当get value时压缩的flag却丢掉了。 所以get value并没有对value进行解压缩。 我想这应该是Tokyo Cabinet兼容memcache协议上的问题。 可能还不是完全兼容。 解决这个问题也是可以的就是把compressionThreshold设置大， 但是spy memcached并没有提供方法可以disable压缩/解压。 所以我想换个memcache client lib试试。

我打算试试memcached client for java。试了下。 可以保存进去，但是却取不到值，看起来比spy memcached还差。

所以打算创建一个新的net.spy.memcached.transcoders.SerializingTranscoder，加上可以设置disableCompress参数即可。

    public class MySerializingTranscoder extends BaseSerializingTranscoder implements Transcoder<Object> {

        // General flags
        static final int SERIALIZED = 1;
        static final int COMPRESSED = 2;

        // Special flags for specially handled types.
        private static final int SPECIAL_MASK = 0xff00;
        static final int SPECIAL_BOOLEAN = (1 << 8);
        static final int SPECIAL_INT = (2 << 8);
        static final int SPECIAL_LONG = (3 << 8);
        static final int SPECIAL_DATE = (4 << 8);
        static final int SPECIAL_BYTE = (5 << 8);
        static final int SPECIAL_FLOAT = (6 << 8);
        static final int SPECIAL_DOUBLE = (7 << 8);
        static final int SPECIAL_BYTEARRAY = (8 << 8);

        [b]private boolean disableCompress;[/b]

        private final TranscoderUtils tu = new TranscoderUtils(true);

        public boolean isDisableCompress() {
            return disableCompress;
        }

        public void setDisableCompress(boolean disableCompress) {
            this.disableCompress = disableCompress;
        }/* (non-Javadoc)
       * @see net.spy.memcached.Transcoder#decode(net.spy.memcached.CachedData)
       */
        public Object decode(CachedData d) {
            byte[] data = d.getData();
            Object rv = null;
            if (!disableCompress){
                if ((d.getFlags() & COMPRESSED) != 0) {
                    data = decompress(d.getData());
                }
            }
            int flags = d.getFlags() & SPECIAL_MASK;
            if ((d.getFlags() & SERIALIZED) != 0 && data != null) {
                rv = deserialize(data);
            } else if (flags != 0 && data != null) {
                switch (flags) {
                    case SPECIAL_BOOLEAN:
                        rv = Boolean.valueOf(tu.decodeBoolean(data));
                        break;
                    case SPECIAL_INT:
                        rv = new Integer(tu.decodeInt(data));
                        break;
                    case SPECIAL_LONG:
                        rv = new Long(tu.decodeLong(data));
                        break;
                    case SPECIAL_DATE:
                        rv = new Date(tu.decodeLong(data));
                        break;
                    case SPECIAL_BYTE:
                        rv = new Byte(tu.decodeByte(data));
                        break;
                    case SPECIAL_FLOAT:
                        rv = new Float(Float.intBitsToFloat(tu.decodeInt(data)));
                        break;
                    case SPECIAL_DOUBLE:
                        rv = new Double(Double.longBitsToDouble(tu.decodeLong(data)));
                        break;
                    case SPECIAL_BYTEARRAY:
                        rv = data;
                        break;
                    default:
                        getLogger().warn("Undecodeable with flags %x", flags);
                }
            } else {
                rv = decodeString(data);
            }
            return rv;
        }

        /* (non-Javadoc)
       * @see net.spy.memcached.Transcoder#encode(java.lang.Object)
       */
        public CachedData encode(Object o) {
            byte[] b = null;
            int flags = 0;
            if (o instanceof String) {
                b = encodeString((String) o);
            } else if (o instanceof Long) {
                b = tu.encodeLong((Long) o);
                flags |= SPECIAL_LONG;
            } else if (o instanceof Integer) {
                b = tu.encodeInt((Integer) o);
                flags |= SPECIAL_INT;
            } else if (o instanceof Boolean) {
                b = tu.encodeBoolean((Boolean) o);
                flags |= SPECIAL_BOOLEAN;
            } else if (o instanceof Date) {
                b = tu.encodeLong(((Date) o).getTime());
                flags |= SPECIAL_DATE;
            } else if (o instanceof Byte) {
                b = tu.encodeByte((Byte) o);
                flags |= SPECIAL_BYTE;
            } else if (o instanceof Float) {
                b = tu.encodeInt(Float.floatToRawIntBits((Float) o));
                flags |= SPECIAL_FLOAT;
            } else if (o instanceof Double) {
                b = tu.encodeLong(Double.doubleToRawLongBits((Double) o));
                flags |= SPECIAL_DOUBLE;
            } else if (o instanceof byte[]) {
                b = (byte[]) o;
                flags |= SPECIAL_BYTEARRAY;
            } else {
                b = serialize(o);
                flags |= SERIALIZED;
            }
            assert b != null;
            if (!disableCompress && b.length > compressionThreshold) {
                byte[] compressed = compress(b);
                if (compressed.length < b.length) {
                    getLogger().info("Compressed %s from %d to %d",
                            o.getClass().getName(), b.length, compressed.length);
                    b = compressed;
                    flags |= COMPRESSED;
                } else {
                    getLogger().info(
                            "Compression increased the size of %s from %d to %d",
                            o.getClass().getName(), b.length, compressed.length);
                }
            }
            return new CachedData(flags, b);
        }

    }