Hbase performace tuning[short-circuit]

Hbase Short circuit

Description:

In HDFS, reads normally go through the DataNode. Thus, when the client asks the DataNode to read a file, the DataNode reads that file off of the disk and sends the data to the client over a TCP socket. So-called "short-circuit" reads bypass the DataNode, allowing the client to read the file directly. Obviously, this is only possible in cases where the client is co-located with the data. Short-circuit reads provide a substantial performance boost.
Hbase client bypasses the datanode layer and directly goes to the OS layer.

1. In hdfs-site.xml, add the below property

  <property>
    <name>dfs.block.local-path-access.user</name>
    <value>user</value>
  </property>

where user is the username running your HBase process.

2. In hbase-site.xml, add the below property

 <property>
    <name>dfs.client.read.shortcircuit</name>
    <value>true</value>
  </property>

Restart the hbase and hadoop to take effect.

Minimum ~25 percent performance increase you will get.

To increase more, follow the below step

3. In hbase-site.xml, add the below property

 <property>
    <name>hbase.regionserver.checksum.verify</name>
    <value>true</value>
  </property>

Description: This property makes HBase to check itself for the data checksum instead of asking Hadoop to do it and will reduce IOs.

Restart the hbase&hadoop.

5 to 7 percent performance will be increased. So totally 30-32 percentage performance boost you will get.

It can be tested the same by the performance evaluation tool in hbase
1.insert the records(randomwrite)
2. Before adding the properties check the rowcount and measure the time taken.(Ignore the quickest and worst result and take average)
3. After adding the properties check the rowcount and measure the time taken.(Ignore the quickest and worst result and take average)