Let’s start with a small introduction- what is hadoop ?. Hadoop is an open-source project administered by the Apache Software Foundation. Apache Hadoop is a Java software framework that supports data-intensive distributed applications under a free license. It enables applications to work with thousands of nodes and petabytes of data. Hadoop was inspired by Google’s MapReduce and Google File System (GFS) papers.
Technically, Hadoop consists of two key services: reliable data storage using the Hadoop Distributed File System (HDFS) and high-performance parallel data processing using a technique called MapReduce.
Dealing with big data requires two things:
- Inexpensive, reliable storage; and
- New tools for analyzing unstructured and structured data.
Hadoop creates clusters of machines and coordinates work among them. Clusters can be built with inexpensive computers.If one fails, Hadoop continues to operate the cluster without losing data or interrupting work, by shifting work to the remaining machines in the cluster.
HDFS manages storage on the cluster by breaking incoming files into pieces, called “blocks,” and storing each of the blocks redundantly across the pool of servers.
The main services running in a hadoop cluster will be
1)namenode
2)jobtracker
3)secondarynamenode
These three will be running only on a single node(machine) ; that machine is the central machine which controls the cluster.
4)datanode
5)tasktracker
These two services will be running on all other nodes in the cluster.
HDFS has a master/slave architecture. An HDFS cluster consists of a single NameNode, a master server that manages the file system namespace and regulates access to files by clients. In addition, there are a number of DataNodes, usually one per node in the cluster, which manage storage attached to the nodes that they run on.
Above the file systems comes the MapReduce engine, which consists of one Job Tracker, to which client applications submit MapReduce jobs. The Job Tracker pushes work out to available Task Tracker nodes in the cluster, striving to keep the work as close to the data as possible.
The only purpose of the secondary name-node is to perform periodic checkpoints. The secondary name-node periodically downloads current name-node image and edits log files, joins them into new image and uploads the new image back to the (primary and the only) name-node.
Now Let us have a look at how to build a hadoop cluster using Cloudera hadoop-0.20 on ubuntu-10.04
You should install sun –jdk first. Then add the following repositories to the apt sources list.
vim /etc/apt/sources.list.d/cloudera.list
[bash]
deb http://archive.cloudera.com/debian lucid-cdh3u0 contrib
deb-src http://archive.cloudera.com/debian lucid-cdh3u0 contrib
[/bash]
Import key
[bash]curl -s http://archive.cloudera.com/debian/archive.key | apt-key add -[/bash]
Then run
[bash]apt-get update[/bash]
For Namenode/Jobtracker ( These two services should run only on a single central machine in the cluster)
[bash]
apt-get install hadoop –yes
apt-get install hadoop-0.20-namenode
apt-get install hadoop-0.20-jobtracker
apt-get install hadoop-0.20-secondarynamenode
[/bash]
Configuration
vim /etc/hadoop/conf/hadoop-env.sh
Append these
[bash]
export JAVA_HOME=/usr/lib/jvm/java-6-sun-1.6.0.24/ ( your java home comes here )
export HADOOP_CONF_DIR=/etc/hadoop/conf
export HADOOP_HOME=/usr/lib/hadoop-0.20
export HADOOP_NAMENODE_USER=hdfs
export HADOOP_SECONDARYNAMENODE_USER=hdfs
export HADOOP_DATANODE_USER=hdfs
export HADOOP_JOBTRACKER_USER=mapred
export HADOOP_TASKTRACKER_USER=mapred
export HADOOP_IDENT_STRING=hadoop
[/bash]
vim /etc/hadoop/conf/core-site.xml
[bash]
<?xml version=”1.0″?>
<?xml-stylesheet type=”text/xsl” href=”configuration.xsl”?>
<!– Put site-specific property overrides in this file. –>
<configuration>
<property>
<name>fs.default.name</name>
<value>hdfs://< ip address of this machine >:8020</value>
</property>
</configuration>
[/bash]
vim /etc/hadoop/conf/hdfs-site.xml
[bash]
<?xml version=”1.0″?>
<?xml-stylesheet type=”text/xsl” href=”configuration.xsl”?>
<!– Put site-specific property overrides in this file. –>
<configuration>
<property>
<name>dfs.name.dir</name>
<value>/var/lib/hadoop-0.20/name</value>
</property>
<property>
<name>dfs.data.dir</name>
<value>/var/lib/hadoop-0.20/data</value>
</property>
<property>
<name>dfs.replication</name>
<value>2</value>
</property>
</configuration>
[/bash]
vim /etc/hadoop/conf/mapred-site.xml
[bash]
<?xml version=”1.0″?>
<?xml-stylesheet type=”text/xsl” href=”configuration.xsl”?>
<!– Put site-specific property overrides in this file. –>
<configuration>
<property>
<name>mapred.job.tracker</name>
<value>< ip address of this machine >:8021</value>
</property>
<property>
<name>mapred.system.dir</name>
<value>/var/lib/hadoop-0.20/system</value>
</property>
<property>
<name>mapred.local.dir</name>
<value>/var/lib/hadoop-0.20/mapred</value>
</property>
</configuration>
[/bash]
——————————————————————————————————————————————
[bash]
mkdir / var/lib/hadoop-0.20/name
mkdir / var/lib/hadoop-0.20/data
mkdir / var/lib/hadoop-0.20/system
mkdir / var/lib/hadoop-0.20/mapred
chown -R hdfs /var/lib/hadoop-0.20/name
chown -R hdfs /var/lib/hadoop-0.20/data
chown -R mapred /var/lib/hadoop-0.20/mapred
[/bash]
Now format NameNode
[bash]yes Y | /usr/bin/hadoop namenode –format[/bash]
Start namenode
[bash]/etc/init.d/hadoop-0.20-namenode start[/bash]
Check the log Files for error:
less /usr/lib/hadoop-0.20/logs/hadoop-hadoop-namenode-<ip>.log
Also you can check whether the Namenode process is up or not using the command
[bash]# jps[/bash]
Start the SecondaryNamenode
[bash]/etc/init.d/hadoop-0.20-secondarynamenode start[/bash]
Log: less /usr/lib/hadoop-0.20/logs/hadoop-hadoop-secondarynamenode-<ip>.log
[bash]
sudo -u hdfs hadoop fs -mkdir /var/lib/hadoop-0.20/system
sudo -u hdfs hadoop fs -chown mapred /var/lib/hadoop-0.20/system
[/bash]
Now Start the JobTracker
[bash]/etc/init.d/hadoop-0.20-jobtracker start[/bash]
Log : less /usr/lib/hadoop-0.20/logs/hadoop-hadoop-jobtracker-ip-10-108-39-34.log
Now jps command will show the three processes up
# jps
19233 JobTracker
18994 SecondaryNameNode
18871 NameNode
For Datanode/Tasktracker ( These two services should be running on all the other machines in the cluster )
[bash]
apt-get install hadoop-0.20-datanode
apt-get install hadoop-0.20-tasktracker
[/bash]
Configuration
vim /etc/hadoop/conf/core-site.xml
[bash]
<?xml version=”1.0″?>
<?xml-stylesheet type=”text/xsl” href=”configuration.xsl”?>
<!– Put site-specific property overrides in this file. –>
<configuration>
<property>
<name>fs.default.name</name>
<value>hdfs://< ip address of the namenode >:8020</value>
</property>
</configuration>
[/bash]
vim /etc/hadoop/conf/hdfs-site.xml
[bash]
<?xml version=”1.0″?>
<?xml-stylesheet type=”text/xsl” href=”configuration.xsl”?>
<!– Put site-specific property overrides in this file. –>
<configuration>
<property>
<name>dfs.name.dir</name>
<value>/var/lib/hadoop-0.20/name</value>
</property>
<property>
<name>dfs.data.dir</name>
<value>/var/lib/hadoop-0.20/data</value>
</property>
<property>
<name>dfs.replication</name>
<value>2</value>
</property>
</configuration>
[/bash]
vim /etc/hadoop/conf/mapred-site.xml
[bash]
<?xml version=”1.0″?>
<?xml-stylesheet type=”text/xsl” href=”configuration.xsl”?>
<!– Put site-specific property overrides in this file. –>
<configuration>
<property>
<name>mapred.job.tracker</name>
<value>< ip address of jobtracker >:8021</value>
</property>
<property>
<name>mapred.system.dir</name>
<value>/var/lib/hadoop-0.20/system</value>
</property>
<property>
<name>mapred.local.dir</name>
<value>/var/lib/hadoop-0.20/mapred</value>
</property>
</configuration>
[/bash]
———————————————————————————————————————————————
[bash]
mkdir /var/lib/hadoop-0.20/data/
chown -R hdfs /var/lib/hadoop-0.20/data
mkdir /var/lib/hadoop-0.20/mapred
chown -R mapred /var/lib/hadoop-0.20/mapred
[/bash]
Start the DataNode
[bash]/etc/init.d/hadoop-0.20-datanode start[/bash]
Log : less /usr/lib/hadoop-0.20/logs/hadoop-hadoop-datanode-<ip>.log
Start the TaskTracker
[bash]/etc/init.d/hadoop-0.20-tasktracker start[/bash]
Log: less /usr/lib/hadoop-0.20/logs/hadoop-hadoop-tasktracker-<ip>.log
You can now check the interface
http://< namenode-ip >:50070 – for HDFS overview
and
http://< jobtracker –ip>:50030 – for Mapreduce overview
