What is Hadoop?


I’m sure you’ve heard about Big Data. If not, I recommend you my blog post “What is Big Data ?”
The most well known technology used for Big Data is Hadoop. It is used by Yahoo, eBay, LinkedIn and Facebook. It has been inspired from Google publications on MapReduce, GoogleFS and BigTable. As Hadoop can be hosted on commodity hardware (usually Intel PC on Linux with one or 2 CPU and a few TB on HDD, without any RAID replication technology), it allows them to store huge quantity of data (petabytes or even more) at very low cost (compared to SAN bay systems).
Hadoop is an open source suite, under an apache foundation:http://hadoop.apache.org/.
The Hadoop “brand” contains many different tools. Two of them are core parts of Hadoop:
  • Hadoop Distributed File System (HDFS) is a virtual file system that looks like any other file system except than when you move a file on HDFS, this file is split into many small files, each of those files is replicated and stored on (usually, may be customized) 3 servers for fault tolerance constraints.
  • Hadoop MapReduce is a way to split every request into smaller requests which are sent to many small servers, allowing a truly scalable use of CPU power (describing MapReduce would worth a dedicated post).
Some other components are often installed on Hadoop solutions:
  • HBase is inspired from Google’s BigTable. HBase is a non-relational, scalable, and fault-tolerant database that is layered on top of HDFS. HBase is written in Java. Each row is identified by a key and consists of an arbitrary number of columns that can be grouped into column families.
  • ZooKeeper is a centralized service for maintaining configuration information, naming, providing distributed synchronization, and providing group services. Zookeeper is used by HBase, and can be used by MapReduce programs.
  • Solr / Lucene as search engine. This query engine library has been developed by Apache for more than 10 years.
  • Languages. Two languages are identified as original Hadoop languages: PIG andHive. For instance, you can use them to develop MapReduce processes at a higher level than MapReduce procedures. Other languages may be used, like C, Java or JAQL. Through JDBC or ODBC connectors (or directly in the languages) SQL can be used too.
 

Hadoop Architecture

Even if the most known Hadoop suite is provided by a very specialized actor named Cloudera (also by MapR, HortonWorks, and of course Apache), big vendors are positioning themselves on this technology:
  • IBM has got BigInsights (Cloudera distribution plus their own custom version of Hadoop called GPFS) and has recently acquired many niche actors in the analytical and big data market (like Platform Computing which has got a product enhancing the capabilities and performance of MapReduce)
  • Oracle has launched BigData machine. Also based on Cloudera, this server is dedicated to storage and usage of non-structured content (as structured content stays on Exadata)
  • Informatica has a tool called HParser to complete PowerCenter This tool is built to launch Informatica process in a MapReduce mode, distributed on the Hadoop servers.
  • Microsoft has got a dedicated Hadoop version supported by Apache for Microsoft Windows and for Azure, their cloud solution, and a big native integration with SQL Server 2012.
  • Some very large database solutions like EMC Greenplum (partnering with MapR), HP Vertica (partnering with Cloudera), Teradata Aster Data (partnering with HortonWorks) or SAP Sybase IQ are able to connect directly to HDFS.
Now you know what Hadoop is, look at what No Hadoop is...
... and if you want to know more about other Big Data solutions, here is a blogpost listing all big data vendors and technologies.

Comments

Post a Comment

Popular posts from this blog

Image
Capricornian Lucky Connections Colour: brown, green, black, gray, indigo Plants: yew, ash, hemp, weeping willow Perfume: musk Starstone: black onyx Metal: lead Tarot Card: the devil Animals: goat, donkey Ruling Planet: Saturn Lucky Numbers: 6, 8 and 9 black onyx Emerald ~ Panna Blue Sapphire ~ Neelam Pearl ~ Moti Blue Lapiz Lazuli
Read more

Now, send free SMS through Gmail chat

Now, send free SMS through Gmail chat It is finally here. The free SMS service right from your Gmail chat window.  Almost a year after the service was introduced in the United States and many other countries, the service has now been made available to Indian users. You could now send a text message to most mobile operators across the country from the comfort of your computer, although there is a prominent exception, at least for users in Bangalore. The Gmail Free SMS service is available for connection to mobile phones with SIMs of many operators, including BSNL, Aircel, Idea, Loop Mobile, MTS, Reliance, Tata DoCoMo, Tata Indicom and Vodafone. However, repeated test messages to Airtel mobiles failed to get the SMSes delivered. Once you send an SMS from the chat window to a mobile phone other than Airtel’s, you wait for a response. The replies will appear as regular chat replies  and conversations will also be stored in your chat history. The new feature was rolled out t...
Read more

CPA Gateway Company Leadbolt Why You Should Use Them

When searching for a CPA company often times you will have a hard time figuring out who to use as many companies sound promising but usually come short on what they promise. Many CPA companies fail to meet payments on schedule, cheat you out of legitimate conversions, or ban your account before you receive payments. Leadbolt isn’t one of those companies that you hear about from other Internet marketers as they are real company founded in the United States with a strong track record among the industry. Leadbolt is defined as a CPA Gateway company like CPA lead. When you visit a website with Leadbolt’s code installed into the website, a blocker will pop up and before they can enter the website they must fill out a survey from a number of advertisers. Once the survey has been filled out you will be paid a commission and the person will be able to enter the website. To incentivize people to fill out surveys to enter the website you can offer downloads, coupons, or adult content etc. Ra...
Read more