Friday, December 23, 2011

Communicate with HBase: Linux Shell, Java API for Client access

  • HBase : Introduction
HBase is a type of "NoSQL" database which runs on top of Hadoop/HDFS clusters. It stores data in form of key-value pair on Hadoop distributed file system. Table schema contains number of column-family which contain  a few to hundreds of thousand or millions of column. HBase is well suited for huge data ( Tera bytes or Peta bytes or whatever you can generate today) with random read/write access. Each row contains a rowkey as identifier and can be accessed as version# when inserted multiple times. Column values are stored with TIMESTAMP, and TTL can also be set for any column-family.
  • Linux Shell
Linux Shell allows an easy access to HBase. Open a Shell/Terminal  in  your Linux system (assuming HBase is already configured on your machine), and reach to your HBase installation folder. 
Try out these commands : 
  • Start HBase
user@ubuntu~$bin/ // to start hbase/HMaster 
user@ubuntu~$bin/hbase shell // to enter into HBase shell
  • Create
hbase(main):003:0> create 'htable', 'cf' // table name:'htable' & column-family:'cf' - it is advised to have small column-family names usually 1 or 2 letter!!
0 row(s) in 1.2200 seconds.
  • Put
hbase(main):004:0> put 'htable', 'rowkey1', 'cf:qualifier1', 'value1' // rowkey:'rowkey1' & column name:'qualifier1' & column value:'value1'
0 row(s) in 1.2200 seconds.
hbase(main):005:0> put 'htable', 'rowkey2', 'cf:qualifier1', 'value2' // inserted as next row.
hbase(main):006:0> put 'htable', 'rowkey1', 'cf:qualifier1', 'value3' // inserted as next version for rowkey:'rowkey1'
any number of column/qualifier can be created while inserting data, only column-family(s) are required at the time of Table schema design.
  • Scan
hbase(main):007:0> scan 'htable' // list all Rows & Columns.
rowkey1       column=cf:qualifier1, timestamp=1288380747188, value=value3
rowkey2       column=cf:qualifier1, timestamp=1288380738440, value=value2
2 row(s) in 0.0590 seconds
Scan only fetches latest version for any row.
  • Get
hbase(main):008:0> get 'htable', 'rowkey2' // get accepts table-name & rowkey
cf:qualifier1        timestamp=1288380738440, value=value2
1 row(s) in 0.0400 seconds
Number of shell commands are available to work with it. But stored procedure, cursor, triggers, functions are not available here as you can find in RDBMS.
  • Java API for client
HBase API provide outside applications(desktop, web etc) to access HBase table using org.apache.hadoop.hbase.client package. 

import org.apache.hadoop.hbase.HBaseConfiguration;
import org.apache.hadoop.hbase.client.Get;
import org.apache.hadoop.hbase.client.HTable;
import org.apache.hadoop.hbase.client.Put;
import org.apache.hadoop.hbase.client.Result;
import org.apache.hadoop.hbase.client.ResultScanner;
import org.apache.hadoop.hbase.client.Scan;
import org.apache.hadoop.hbase.util.Bytes;

// Class that has nothing but a main.
// Does a Put, Get and a Scan against an hbase table.
public class MyLittleHBaseClient {
  public static void main(String[] args) throws IOException {
    // You need a configuration object to tell the client where to connect.
    // When you create a HBaseConfiguration, it reads in whatever you've set
    // into your hbase-site.xml and in hbase-default.xml, as long as these can
    // be found on the CLASSPATH
    Configuration config = HBaseConfiguration.create();

    // This instantiates an HTable object that connects you to
    // the "myLittleHBaseTable" table.
    HTable table = new HTable(config, "myLittleHBaseTable");

    // To add to a row, use Put.  A Put constructor takes the name of the row
    // you want to insert into as a byte array.  In HBase, the Bytes class has
    // utility for converting all kinds of java types to byte arrays.  In the
    // below, we are converting the String "myLittleRow" into a byte array to
    // use as a row key for our update. Once you have a Put instance, you can
    // adorn it by setting the names of columns you want to update on the row,
    // the timestamp to use in your update, etc.If no timestamp, the server
    // applies current time to the edits.
    Put p = new Put(Bytes.toBytes("myLittleRow"));

    // To set the value you'd like to update in the row 'myLittleRow', specify
    // the column family, column qualifier, and value of the table cell you'd
    // like to update.  The column family must already exist in your table
    // schema.  The qualifier can be anything.  All must be specified as byte
    // arrays as hbase is all about byte arrays.  Lets pretend the table
    // 'myLittleHBaseTable' was created with a family 'myLittleFamily'.
    p.add(Bytes.toBytes("myLittleFamily"), Bytes.toBytes("someQualifier"),
      Bytes.toBytes("Some Value"));

    // Once you've adorned your Put instance with all the updates you want to
    // make, to commit it do the following (The HTable#put method takes the
    // Put instance you've been building and pushes the changes you made into
    // hbase)

    // Now, to retrieve the data we just wrote. The values that come back are
    // Result instances. Generally, a Result is an object that will package up
    // the hbase return into the form you find most palatable.
    Get g = new Get(Bytes.toBytes("myLittleRow"));
    Result r = table.get(g);
    byte [] value = r.getValue(Bytes.toBytes("myLittleFamily"),
    // If we convert the value bytes, we should get back 'Some Value', the
    // value we inserted at this location.
    String valueStr = Bytes.toString(value);
    System.out.println("GET: " + valueStr);

    // Sometimes, you won't know the row you're looking for. In this case, you
    // use a Scanner. This will give you cursor-like interface to the contents
    // of the table.  To set up a Scanner, do like you did above making a Put
    // and a Get, create a Scan.  Adorn it with column names, etc.
    Scan s = new Scan();
    s.addColumn(Bytes.toBytes("myLittleFamily"), Bytes.toBytes("someQualifier"));
    ResultScanner scanner = table.getScanner(s);
    try {
      // Scanners return Result instances.
      // Now, for the actual iteration. One way is to use a while loop like so:
      for (Result rr =; rr != null; rr = {
        // print out the row we found and the columns we were looking for
        System.out.println("Found row: " + rr);

      // The other approach is to use a foreach loop. Scanners are iterable!
      // for (Result rr : scanner) {
      //   System.out.println("Found row: " + rr);
      // }
    } finally {
      // Make sure you close your scanners when you are done!
      // Thats why we have it inside a try/finally clause

Above program automatically connects with local HBase configuration. To connect with  remote HBase machine, it requires path to 'hbase-site.xml' on Configuration.Path(path);

Apart from these two methods, HBase provides Thrift & REST gateways. Map Reduce programs can also read/write data to HBase clusters. 
will come with more in next post :)

Friday, December 16, 2011

Communicate with SharePoint: {Client, Server, Web Service} Model

  • Client Object Model
The Client Object Model allows developers to extend the feature of Server Object Model on a box where SharePoint is not running. It could be a console application, desktop application, web application etc.
  • Server Object Model
The Server Object Model allows you to leverage the features of highly structured server-side model. You can drill down from higher level object SPFarm to bottom level SPList to SPField etc. 
  • Web Services
    • Rest
The new REST (Representational State Transfer) allows to perform standard operation  GET, POST, PUT, and DELETE HTTP verbs directly from your browser. RESTful call enable you to filter, sort and select information from SharePoint Lists and Library. A Response from SharePoint server could be  JavaScript Object Notation (JSON), Xml, Atom feed, AtomPub etc.
An example RESTful call -
http://myserver/_vti_bin/listdata.svc/Employees?$filter=Project/Title eq 'My Project Title'
    • Soap
The Soap (Simple Object Access Protocol) provides numerous method to communicate or work with SharePoint contents closely. AJAX and JQuery utilizes SharePoint Soap Web Service.
    A list of Soap method can be viewed at -
    http://myserver/_vti_bin/<Web Services Name>.asmx

    Tuesday, December 13, 2011

    What Is Apache Hadoop?

    What Is Apache Hadoop?
    The Apache™ Hadoop™ project develops open-source software for reliable, scalable, distributed computing.

    The Apache Hadoop software library is a framework that allows for the distributed processing of large data sets across clusters of computers using a simple programming model. It is designed to scale up from single servers to thousands of machines, each offering local computation and storage. Rather than rely on hardware to deliver high-avaiability, the library itself is designed to detect and handle failures at the application layer, so delivering a highly-availabile service on top of a cluster of computers, each of which may be prone to failures.

    The project includes these subprojects:

    Hadoop Common: The common utilities that support the other Hadoop subprojects.
    Hadoop Distributed File System (HDFS™): A distributed file system that provides high-throughput access to application data.
    Hadoop MapReduce: A software framework for distributed processing of large data sets on compute clusters.
    Other Hadoop-related projects at Apache include:

    Avro™: A data serialization system.
    Cassandra™: A scalable multi-master database with no single points of failure.
    Chukwa™: A data collection system for managing large distributed systems.
    HBase™: A scalable, distributed database that supports structured data storage for large tables.
    Hive™: A data warehouse infrastructure that provides data summarization and ad hoc querying.
    Mahout™: A Scalable machine learning and data mining library.
    Pig™: A high-level data-flow language and execution framework for parallel computation.
    ZooKeeper™: A high-performance coordination service for distributed applications.

    Reference :

    Tuesday, September 27, 2011

    Get UserProfile using UserProfileManager in SharePoint and ASP.NET application

    Fetch Contact details from Active directory and SharePoint contacts using User profile service

    public ArrayList SearchingFromUserProfile(string key)
    //getting the URL of Central Admin.
    Microsoft.SharePoint.Administration.SPAdministrationWebApplication centralWeb = SPAdministrationWebApplication.Local;
    string centraladmin = centralWeb.GetResponseUri(Microsoft.SharePoint.Administration.SPUrlZone.Custom).AbsoluteUri.ToString();

    ArrayList result = new ArrayList();

    //Initallizing the UserProfileManager.
    using (SPSite site = new SPSite(centraladmin))
    SPServiceContext context = SPServiceContext.GetContext(site);
    UserProfileManager userprofilemanager = new UserProfileManager(context);

    //Getting WorkEmail of all the Users Resolved.
    foreach (UserProfile profile in userprofilemanager)
    object property = profile[PropertyConstants.WorkEmail];
    if(property != null && property != "")
    return result;

    Friday, August 19, 2011

    HTML5 for Meenuuu

    HTML5 improves interoperability and reduces development costs by making precise rules on how to handle all HTML elements, and how to recover from errors.

    New Features : functions for embedding audio, video, graphics, client-side data storage, and interactive documents(These features were not present in older version of HTML). HTML5 also contains new elements like <nav>, <header>, <footer>, and <figure>.

    New : New tags in HTML5.

    <!--...-->Defines a comment
    <!DOCTYPE> Defines the document type
    <a>Defines a hyperlink
    <abbr>Defines an abbreviation
    <acronym>Not supported in HTML5
    <address>Defines an address element
    <applet>Not supported in HTML5
    <area>Defines an area inside an image map
    <article>NewDefines an article
    <aside>NewDefines content aside from the page content
    <audio>NewDefines sound content
    <b>Defines bold text
    <base>Defines a base URL for all the links in a page
    <basefont>Not supported in HTML5
    <bdo>Defines the direction of text display
    <big>Not supported in HTML5
    <blockquote>Defines a long quotation
    <body>Defines the body element
    <br>Inserts a single line break
    <button>Defines a push button
    <canvas>NewDefines graphics
    <caption>Defines a table caption
    <center>Not supported in HTML5
    <cite>Defines a citation
    <code>Defines computer code text
    <col>Defines attributes for table columns 
    <colgroup>Defines groups of table columns
    <command>NewDefines a command button
    <datalist>NewDefines a dropdown list
    <dd>Defines a definition description
    <del>Defines deleted text
    <details>NewDefines details of an element
    <dfn>Defines a definition term
    <dir>Not supported in HTML5
    <div>Defines a section in a document
    <dl>Defines a definition list
    <dt>Defines a definition term
    <em>Defines emphasized text 
    <embed>NewDefines external interactive content or plugin
    <fieldset>Defines a fieldset
    <figcaption>NewDefines the caption of a figure element
    <figure>NewDefines a group of media content, and their caption
    <font>Not supported in HTML5
    <footer>NewDefines a footer for a section or page
    <form>Defines a form 
    <frame>Not supported in HTML5
    <frameset>Not supported in HTML5
    <h1> to <h6>Defines header 1 to header 6
    <head>Defines information about the document
    <header>NewDefines a header for a section or page
    <hgroup>NewDefines information about a section in a document
    <hr>Defines a horizontal rule
    <html>Defines an html document
    <i>Defines italic text
    <iframe>Defines an inline sub window (frame)
    <img>Defines an image
    <input>Defines an input field
    <ins>Defines inserted text
    <keygen>NewDefines a generated key in a form
    <kbd>Defines keyboard text
    <label>Defines a label for a form control
    <legend>Defines a title in a fieldset
    <li>Defines a list item
    <link>Defines a resource reference
    <map>Defines an image map 
    <mark>NewDefines marked text
    <menu>Defines a menu list
    <meta>Defines meta information
    <meter>NewDefines measurement within a predefined range
    <nav>NewDefines navigation links
    <noframes>Not supported in HTML5
    <noscript>Defines a noscript section
    <object>Defines an embedded object
    <ol>Defines an ordered list
    <optgroup>Defines an option group
    <option>Defines an option in a drop-down list
    <output>NewDefines some types of output
    <p>Defines a paragraph
    <param>Defines a parameter for an object
    <pre>Defines preformatted text
    <progress>NewDefines progress of a task of any kind
    <q>Defines a short quotation
    <rp>NewUsed in ruby annotations to define what to show if a browser does not support the ruby element
    <rt>NewDefines explanation to ruby annotations
    <ruby>NewDefines ruby annotations
    <s>Defines text that is no longer correct
    <samp>Defines sample computer code
    <script>Defines a script
    <section>NewDefines a section
    <select>Defines a selectable list
    <small>Defines small text
    <source>NewDefines media resources
    <span>Defines a section in a document
    <strike>Not supported in HTML5
    <strong>Defines strong text
    <style>Defines a style definition
    <sub>Defines subscripted text
    <summary>NewDefines the header of a "detail" element
    <sup>Defines superscripted text
    <table>Defines a table
    <tbody>Defines a table body
    <td>Defines a table cell
    <textarea>Defines a text area
    <tfoot>Defines a table footer
    <th>Defines a table header
    <thead>Defines a table header
    <time>NewDefines a date/time
    <title>Defines the document title
    <tr>Defines a table row
    <tt>Not supported in HTML5
    <u>Not supported in HTML5
    <ul>Defines an unordered list
    <var>Defines a variable
    <video>NewDefines a video
    <wbr>NewDefines a possible line-break
    <xmp>Not supported in HTML5

    The DOCTYPE :

    <!DOCTYPE html
              PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"

    Example :
         <!DOCTYPE HTML>

               <video width="320" height="240" controls="controls">
               <source src="movie.ogg" type="video/ogg" />
               <source src="movie.mp4" type="video/mp4" />
               <source src="movie.webm" type="video/webm" />
               Your browser does not support the video tag.


    Reference Sites :