欢迎您访问程序员文章站本站旨在为大家提供分享程序员计算机编程知识!
您现在的位置是: 首页

Hbase read performance with increasing number of client threads

程序员文章站 2022-06-13 17:25:23
...

 

在跑ycsb的时候,发现在做heavy的get操作的时候,ycsb统计的latency很大,100个thread的时候就接近100ms,而从ganglia上看hbase的“get_avg_time"这个metric发现只有20~30左右。最终查看code,发现100个thread共享同一个连接,所有Call的请求数据都走这一个连接,所以在大量请求并发时会造成拥堵,latency变大。具体看下面这个方法:

public Writable[] call(Writable[] params, InetSocketAddress[] addresses)
    throws IOException {
    if (addresses.length == 0) return new Writable[0];

    ParallelResults results = new ParallelResults(params.length);
    synchronized (results) {
      for (int i = 0; i < params.length; i++) {
        ParallelCall call = new ParallelCall(params[i], results, i);
        try {
          Connection connection = getConnection(addresses[i], null, call);
          connection.sendParam(call);             // send each parameter
        } catch (IOException e) {
          // log errors
          LOG.info("Calling "+addresses[i]+" caught: " + 
                   e.getMessage(),e);
          results.size--;                         //  wait for one fewer result
        }
      }
      while (results.count != results.size) {
        try {
          results.wait();                    // wait for all results
        } catch (InterruptedException e) {}
      }

      return results.values;
    }
  }

 

 而sendParm方法是synchorinized。

public void sendParam(Call call) {
      if (shouldCloseConnection.get()) {
        return;
      }

      DataOutputBuffer d=null;
      try {
        synchronized (this.out) {
          if (LOG.isDebugEnabled())
            LOG.debug(getName() + " sending #" + call.id);
          
          //for serializing the
          //data to be written
          d = new DataOutputBuffer();
          d.writeInt(call.id);
          call.param.write(d);
          byte[] data = d.getData();
          int dataLength = d.getLength();
          out.writeInt(dataLength);      //first put the data length
          out.write(data, 0, dataLength);//write the data
          out.flush();
        }
      } catch(IOException e) {
        markClosed(e);
      } finally {
        //the buffer is just an in-memory buffer, but it is still polite to
        // close early
        IOUtils.closeStream(d);
      }
    }  

 

对同一个目的地的RegionServer来说,connection只有一个。这样设计的初衷是为了共享连接,减少维护多个连接的开销,但在这种情形下会造成latency增大。具体解释可以下面两个链接。

 

http://comments.gmane.org/gmane.comp.java.hadoop.hbase.user/11534

https://issues.apache.org/jira/browse/HBASE-2939

在后一个连接中给出了解决问题的一个折衷patch,就是不再只用一个connection,而是用几个,这也算是一个tradeoff吧。