Apache HBase Lab Practice on Windows 8 using Cygwin64

# Introduction:

FileStream
|
  DBMS
|
 RDBMS
|
NoSQL (HBase) -> Part of Hadoop Ecosystem


Tape:
- Sequence Access
_______________
| | | |
_______________

Disk
- Sequence and Random O[n] -> Order of Nth Processing
- Transformation
- Seeking / Seek time
. - Sequence Acess
. ----. - Random Access
.

O[n] Databases -> Casandra, MongoDB, couchDB, Impala, HBase, Dynama -> NoSQL


# Features:

- No concept of multiple tables
- Same as Google's Big Table
- Data operations using Get, Put, Scan, Delete.
- DDL - Create, Alter, Enable/Disable
- There is no update.  Update the same exact row with a different value by overwriting, e.g. put
- Enable / Disable -> Delete the data
- Supports both OLAP and OLTP (Inserts and Updates)
- Horizontally Scalable
- Random Access
- Low latency
- Sorted HashMap -> Keys and Values (Key Value store) -> No Indexes 
- Both Key and Value are byte array format -> combination of integer and string values
- Values are stored in key-order
- No schema for table, only Column families
- Rows are stored in a Sorted Order
- Each row has a row key
- Rows are strongly consistent
- RowId / RowKey is auto generated.  While inserting data, user should insert into RowId field.
- No Search (e.g. LIKE), No Joins, No Foreign Key
- Table scan is fast
- No data types (int, string, etc)
- Executes MR and fetches the records.
- E.g. Adhaar is implemented in HBase
- Access HBase data via an API


# Architecture:

- Works on top of Hadoop
- Port Nos
 HDFS: 50070
 MapReduce: 50060
 Master Server (60010) -> localhost:60010
 Region Servers (60030) -> localhost:60030

+ HBase Master Server (aka Name Node)
- Cluster Management, Monitoring, Load Balancing

+ HBase StandBy Master
-

+ ZooKeeper
- Corordination server to communicate and run the HBase Server
- Coordinates HMaster transitions
- Highly available system for coordination
- Only information, no processing
- It is part of HBase.  No need to set path in .bashrc file
- Quorum Peer is the daemon of ZooKeeper
- Stack - FIFO
- Required for HBase Developer

+ HBase Region Server(s) (aka Data Node(s))
(1) HFile - Data Storage Location
(2) Memstore - Schema information of the Column Families
(3) Write-Ahead Log / HLog -> Current running transactions Log


# Diagram:

  
HBase Master
|
ZooKeeper
|
-------------------------------------------------------------------------------------------------------------
| | | |
HBase Region Server HBase Region Server                   HBase Region Server HBase Region Server
| | | |
-------------------------------------------------------------------------------------------------------------
|
  HDFS


# Configuration:

1. .bashrc
Set path for HBase in .bashrc file
export HBASE_HOME=/home/kosmik/work/hbase-0.94.12
export PATH=$HBASE_HOME/bin:$PATH

2. regionservers
In Hbase conf, create a file with "regionservers".  Inside type localhost.

3. hbase-env.sh

export JAVA_HOME=/usr/lib/jvm/java-6-openjdk
export HBASE_REGIONSERVER=/home/kosmik/work/hbase-0.94.12/conf/regionservers
export HBASE_MANAGES_ZK=true

4. hbase-site.xml



 hbase.root.dir
 hdfs://localhost:9000/hbase

 hbase.cluster.distributed
 true


 hbase.zookeeper.property.dataDir
 /home/kosmik/work/zookeeper




# Daemons:

  On starting HBase (start-hbase.sh), 8 processes should run:
1. HBase Master
2. HBase Region
3. Job Tracker
4. Task Tracker
5. Name Node
6. Secondary Name Node
7. Data Node
8. Quorum Peer


# HBase vs Hive:

HBase Hive
-------------------------------------------------------    -------------------------------------
RowId CF:Identifier TimeStamp Value Id Name Sal
-------------------------------------------------------    --------------------------------------
101 CF:Name 10:05:11 Kiran 101 Kiran 10000
-------------------------------------------------------    --------------------------------------
101 CF:Sal 10:05:12 10000
-------------------------------------------------------    --------------------------------------
...
-------------------------------------------------------    ---------------------------------------
...
-------------------------------------------------------    ---------------------------------------
...
-------------------------------------------------------    ---------------------------------------

Rowkey info:height info:state roles:hadoop roles:

1234 info {'lastname': 'Smith', 'firstName': 'John'}
pwd {'Password': 'HelloWorld'}


# Interview Questions:

Q. Difference between HBase and RDBMS?
A. - Search queries
- Foreign Keys
- There are no joins
- All parts of table in single query

Q. Diff between Truncate and Delete
A. Truncate (Shit + Delete)

Q. Bulk Uploads with MapReduce
A.

Q. What is Distributed Caching?
A. In Memory processing

Q. Difference between HDFS Block and Input Splits?

Q. How we can improve the performance of Hive?
A. ORC format, Vectorized format, Tez -> Refer Qubole


# To view HBase files in HDFS:

sudo -u hdfs hadoop fs -mkdir /hbase
sudo -u hdfs hadoop fs -chown hbase /hbase

hdfs> hadoop fs -mkdir /hbase
hdfs> hadoop fs -chown hbase /hbase


# References:

Big Data University -> Videos


# Lab Practice:


+ Cygwin64 Terminal Commands

$> net start sshd

$> ssh cyg_server@localhost

$> /usr/local/hbase-1.0.1.1//bin/start-hbase.sh

$> /usr/local/hbase-1.0.1.1//bin/hbase shell

$> /usr/local/hbase-1.0.1.1//bin/stop-hbase.sh

$> logout

$> net stop sshd

+ Ubuntu Terminal Commands

$> start-all.sh
$> start-hbase.sh // Starts HBase Server

$> hbase shell // Opens HBase Shell

  hbase> status 'detailed'
...
  hbase> quit

$> hbase-daemon.sh stop regionserver // Stops an individual RegionServer
$> stop-hbase.sh
$> stop-all.sh

+ HBase Shell Commands

hbase> version
hbase> status 'detailed'
hbase> whoami
hbase> list

hbase> create 'employee', 'personal', 'official'
  --------   --------   --------
table cf1 cf1

hbase> describe 'employee'
hbase> put 'employee','emp101','official:designation','hadoop admin'
--------   ------ --------------------   ------------
table rowid cf:identifier value

hbase> put 'employee','Emp101','personal:name','Dhanu' // Case Sensitive -> New Row Key inserted 
hbase> put 'employee', 'emp01', 'personal:name', 'kosmik'
hbase> put 'employee', 'emp01', 'official:department', 'IT'
hbase> put 'employee', 'emp02', 'official:department', 'HR'
hbase> put 'employee', 'emp02', 'personal:name', 'kiran'

hbase> scan 'employee'
hbase> get 'employee', 'emp01'
hbase> get 'employee', 'emp02'
hbase> count 'employee'
hbase> delete 'employee', 'emp01', 'personal:name'
hbase> deleteall 'employee', 'emp02' // Entire row is deleted

hbase> disable 'employee'
hbase> enable 'employee'

hbase> drop 'employee'


---