Konfigurasi Hadoop + MongoDB
8.3 Konfigurasi Hadoop + MongoDB
NoSQL (Not Only SQL, atau maknanya kurang lebih adalah "tidak hanya SQL") adalah kelas yang luas dari sistem manajemen basis data yang di identifikasikan tidak mematuhi aturan pada model sistem manajemen basis data relasional yang banyak digunakan.
Gambar 8.21 Relational Vs Non-Relational DB NoSQL tidak dibangun terutama dengan tabel dan umumnya
tidak menggunakan SQL untuk memanipulasi data.
Gambar 8.22 Collection pada NoSQL
Teorema CAP menyatakan bahwa, tidak mungkin untuk sebuah sistem komputer terdistribusi secara bersamaan, memberikan ketiga jaminan sebagai berikut: Teorema CAP menyatakan bahwa, tidak mungkin untuk sebuah sistem komputer terdistribusi secara bersamaan, memberikan ketiga jaminan sebagai berikut:
o Availability: Jaminan pada setiap permintaan, menerima tanggapan tentang apakah itu berhasil atau
tidak. o Partition tolerance: Sistem terus beroperasi meski
sistem yang lain gagal. Visual NoSQL System:
Gambar 8.23 Visual NoSQL System
Gambar 8.24 Bentuk NoSQL
MongoDB merupakan open-source document database dari keluarga NoSQL, yang memberikan kinerja tinggi, high availability, dan automatic scaling.
Gambar 8.25 SQL Vs NoSQL
Document, Data pada MongoDB dikenal sebagai document. MongoDB menggunakan BSON (Binary JSON) serialization sebagai format penyimpanan data untuk document, mirip dengan JSON (JavaScript Object Notation). Contoh document:
{ "_id": Ob- jectId('58520a516a2ea8e208cd6c26'), "name": "Employee One", "salary": 20000
Source Code 8.1 Contoh Document BSON Contoh Penggunaan
SQL: SELECT * FROM employeedetails WHERE name = "Employee One" OR salary = 20000
Source Code 8.2 Contoh Penggunaan BSON MongoDB:
db.employeedetails.find( {
$or: [ { name: "Employee One" }, { nomor: 20000 }
Source Code 8.3 Contoh Penggunaan BSON SQL Vs MongoDB:
Tabel 8.1 Perbedaan SQL dengan MongoDB
SQL MongoDB
database Database table
Collection row
Document column
Field index
Index Index
primary key Case Study: Mengolah data dan menyimpan hasilnya ke
MongoDB: Wordcount Rating By Movie
Note: Secara default, Map Reduce akan menyimpan hasilnya di file teks biasa di HDFS. Hal ini sangat tidak praktis jika memang hasilnya ingin dimasukkan ke NoSQL supaya mudah dibaca.
1. Running Hadoop (Save hasil di MongoDB)
- Lakukan perintah (Install MongoBD untuk simpan file
hasil running kode program di Hadoop):
nidos@master:~$ sudo apt-get install mongodb nidos@master:~$ sudo apt-get update nidos@master:~$ sudo service mongodb start start: Job is already running: mongodb nidos@master:~$ mongo MongoDB shell version: 2.4.9 connecting to: test Welcome to the MongoDB shell. For interactive help, type "help". For more comprehensive documentation, see
http://docs.mongodb.org/ Questions? Try the support group http://groups.google.com/group/mongodb- user > use mydb; switched to db mydb > db; mydb > db.employeedetails.insert({name:"Employee One",salary:20000}); > db.employeedetails.find(); { "_id" : ObjectId("58520a516a2ea8e208cd6c26"), "name" : "Employee One", "salary" : 20000 } > exit bye
Source Code 8.4 Install MongoBD untuk simpan file hasil running kode program di Hadoop
2. Running Hadoop (Install GUI MongoDB)
- Lakukan perintah: nidos@master:~$ sudo apt-get install curl
nidos@master:~$ curl http://pack- ages.litixsoft.de/installer.sh | sh Enter ´sudo ./bbs-installer.sh´ to start installation... nidos@master:~$ sudo ./bbs-installer.sh
Source Code 8.5 Running Hadoop (Install GUI MongoDB)
Gambar 8.26 Hasil Running Hadoop (Install GUI MongoBD)
nidos@master:~$ sudo ./bbs-installer.sh BaboonStack Installer for Linux/MacOS Local Packet found... Install baboonstack-v1.5.1-linux- x64.tar.gz... Create /opt/litixsoft/baboonstack... Extract Files... Execute Installscript bbs/lxscript.sh... SUCCESS: Enter ´bbs´ for package updates. ...
Install MongoDB v2.6.7?: [Y/n] n Jika sudah Install
Source Code 8.6 Running Hadoop (Install GUI MongoDB)
Install RedisIO v2.8.4?: [Y/n] Y Install Node.js v0.12.0?: [Y/n] Y Activate Io.js support?: [Y/n] Y lxManager for BaboonStack - Litixsoft GmbH 2015 Upgrade successful nidos@master:~$ sudo tar xvzf /home/nidos/Desktop/GUI\ Mongo/mms- v1.9.4-community-linux.tar.gz
nidos@master:~$ sudo tar xvzf mms- v1.9.4-community-linux- x86_64.tar.gznidos@master:~$ cd mms- v1.9.4-community-linux-x86_64/
Source Code 8.7 Running Hadoop (Install GUI MongoDB) Cont
- Lakukan perintah:
nidos@master:~/mms-v1.9.4- community-linux-x86_64$ cd lx- mms
nidos@master:~/mms-v1.9.4- community-linux-x86_64/lx-mms$ sudo nano config.js
Source Code 8.8 Install GUI MongoDB Cont
Gambar 8.27 Tampilan Install GUI MongoDB Gambar 8.27 Tampilan Install GUI MongoDB
- Lakukan perintah:
nidos@master:~/mms-v1.9.4-commu- nity-linux-x86_64/lx-mms$ sudo ./start
Gambar 8.28 Tampilan Install GUI MongoDB - Lakukan perintah ( http://127.0.0.1:3333/ ):
nidos@master:~/mms-v1.9.4-community- linux-x86_64/lx-mms$ sudo ./start
Gambar 8.29 Hasil perintah (http://127.0.0.1:3333/) Gambar 8.29 Hasil perintah (http://127.0.0.1:3333/)
Source Code 8.9 perintah (http://127.0.0.1:3333/)
Gambar 8.30 Hasil perintah (http://127.0.0.1:3333/)
nidos@master:~/mms-v1.9.4-commu- nity-linux-x86_64/lx-mms$ sudo
./start
Gambar 8.31 Perintah ( http://127.0.0.1:3333/ ) test Connection
nidos@master:~/mms-v1.9.4- community-linux-x86_64/lx-mms$
sudo ./start
Gambar 8.32 Perintah (http://127.0.0.1:3333/)
3. Konfigurasi Koneksi (Hadoop + MongoDB)
- Lakukan perintah: nidos@master:/usr/local/hadoop$ sudo
apt-get install git nidos@master:/usr/local/hadoop$ git clone https://github.com/mongodb/mongo- hadoop nidos@master:/usr/local/hadoop$ cd mongo-hadoop nidos@master:/usr/local/hadoop/mongo -hadoop$ ./gradlew jar
Source Code 8.10 Konfigurasi Koneksi (Hadoop + MongoDB)
Gambar 8.33 Konfigurasi Koneksi (Hadoop + MongoDB)
4. Copy file “mongo-hadoop-core-2.0.1.jar” dari /usr/local/hadoop/mongo-hadoop/core/build/libs. hasil
compile dari langkah sebelumnya:
nidos@master:/usr/local/hadoop/mongo-hadoop$ ./gradlew jar
Gambar 8.34 Copy file “mongo-hadoop-core-2.0.1.jar”
5. Download
“mongo-java-driver” dari link http://central.maven.org/maven2/org/mongodb/mongo- java-driver/ dan pilih versi terbaru, misal “mongo-java-
file
driver- 3.4.0.jar”
Gambar 8.35 file “mongo-java-driver” Setelah selesai build, copy-kan file jars (“mongo-hadoop-
core- 2.0.1.jar” dan “mongo-java-driver-3.4.0.jar”) ke directory lib pada setiap node (master, node1, node2, node3) di hadoop clus- ter. Berikut lokasinya yang bergantung pada versi Hadoop:
$HADOOP_PREFIX/lib/ $HADOOP_PREFIX/share/hadoop/mapreduce/ $HADOOP_PREFIX/share/hadoop/lib/ Misal 2 file jar tersebut sudah disiapkan di “/home/nidos/Desktop/kode/WordCountMongo/” nidos@master:~/Desktop/kode/WordCountMongo$ ls mongo-hadoop-core-2.0.1.jar WordCountMongo.java mongo-java-driver-3.4.0.jar
Untuk PC master:
nidos@master:~$ cp /home/nidos/Desktop/kode/WordCountMongo/mongo* /usr/local/hadoop/lib nidos@master:~$ cp /home/nidos/Desktop/kode/WordCountMongo/mongo* /usr/local/hadoop/share/hadoop/mapreduce nidos@master:~$ mkdir r /usr/local/hadoop/share/hadoop/lib nidos@master:~$ cp /home/nidos/Desktop/kode/WordCountMongo/mongo* /usr/local/hadoop/share/hadoop/lib
Source Code 8.11 copy file jars (“mongo-hadoop-core-2.0.1.jar” dan “mongo-java-driver-3.4.0.jar”) ke directory lib pada setiap di
hadoop cluster
Untuk PC node1:
nidos@node1:~$ scp nidos@master:/home/nidos/Desk- top/kode/WordCountMongo/mongo* /usr/local/ha- doop/lib nidos@node1:~$ scp nidos@master:/home/nidos/Desk- top/kode/WordCountMongo/mongo* /usr/local/ha- doop/share/hadoop/mapreduce nidos@node1:~$ mkdir r /usr/local/hadoop/share/hadoop/lib nidos@node1:~$ scp nidos@master:/home/nidos/Desk- top/kode/WordCountMongo /mongo* /usr/local/hadoop/share/hadoop/lib
Lakukan juga pada node2 dan node3
Untuk PC node3: nidos@node3:~$ scp nidos@master:/home/nidos/Desk-
top/kode/WordCountMongo/mongo* /usr/local/ha- doop/lib
nidos@node3:~$ scp nidos@master:/home/nidos/Desk- top/kode/WordCountMongo/mongo* /usr/local/ha- doop/share/hadoop/mapreduce
nidos@node3:~$ mkdir r /usr/local/hadoop/share/hadoop/lib
nidos@node3:~$ scp nidos@master:/home/nidos/Desk- top/kode/WordCountMongo/mongo* /usr/local/ha- doop/share/hadoop/lib
Source Code 8.12 copy file jars (“mongo-hadoop-core-2.0.1.jar” dan “mongo-java-driver-3.4.0.jar”) ke directory lib pada setiap di
hadoop cluster Cont
Gambar 8.36 copy file jars (“mongo-hadoop-core-2.0.1.jar” dan
“mongo-java-driver-3.4.0.jar”) ke dir. lib pada tiap di hadoop cluster
8.3.1 WordCount
1. Running Hadoop WordCount (Save hasil di MongoDB)
- Buat DB “testmr”
Gambar 8.37 Buat DB “testmr”
- Buat Collecton “in” on DB “testmr” - Klik kanan pada DB “testmr”, lalu pilih “Add collection..”,
beri nama misal “in”, lalu klik Save
Gambar 8.38 Tampilan Add Collection - Import file “*.json” as collection pada DB “testmr” ke
collection “in” sebagai file input untuk diproses oleh WordCountMongo.java
Gambar 8.39 Import file “*.json” as collection pada DB “testmr” ke
collection “in”
- Import file “*.json” as collection pada DB “testmr” sebagai file input untuk diproses oleh WordCountMongo.java
in.json tidak standar Mongo {"x": "hello world"},
{"x": "nice to meet you"}, {"x": "good to see you"}, {"x": "world war 2"}, {"x": "see you again"}, {"x": "bye bye"}
Source Code 8.13 in.json tidak standar Mongo Source Code 8.13 in.json tidak standar Mongo
{ "x" : "hello world", "_id" : { "$oid" : "5856c34689b0ce4a6e000001" } } { "x" : "nice to meet you", "_id" : { "$oid" : "5856c34689b0ce4a6e000002" } } { "x" : "good to see you", "_id" : { "$oid" : "5856c34689b0ce4a6e000003" } } { "x" : "world war 2", "_id" : { "$oid" : "5856c34689b0ce4a6e000004" } } { "x" : "see you again", "_id" : { "$oid" : "5856c34689b0ce4a6e000005" } }
Source Code 8.14 in_standard.json standar Mongo Untuk file “in.json” tidak standar Mongo: Klik kanan
pada database atau DB “testmr”, lalu pilih “Import documents...”, lalu Isikan Name of destination-collection*, misal “in”, lalu pilih “Import documents from textbox”, lalu isikan pada “Text Input”, dengan:
{"x": "hello world"}, {"x": "nice to meet you"}, {"x": "good to see you"}, {"x": "world war 2"}, {"x": "see you again"},
{"x": "bye bye"}
Gambar 8.40 Text Input
lalu Klik Import - Untuk file “in.json” tidak standar Mongo:
Gambar 8.41 Import Document Untuk file “in.json” tidak standar Mongo
- Untuk file “in.json” tidak standar Mongo:
Gambar 8.42 Import Document Untuk file “in.json” tidak standar
Mongo
- (Klik kanan pada databa se atau DB “testmr”, lalu pilih “Import documents...”, lalu Isikan Name of destination-
collection*, misal “in”, lalu klik Browse..):
Untuk file in_standard.json standar Mongo
Gambar 8.43 file in_standard.json standar Mongo
- in_standard.json standar Mongo (Klik Open)
Gambar 8.44 in_standard.json standar Mongo (Klik
Open)
- Klik Import
Gambar 8.45 Import Document
- Hasil dari klik import
Gambar 8.46 Hasil dari Klik Import
- Siapkan file *.java (misal WordCountMongo.java Part 1 of
2) untuk dicompile ke *.jar: 2) untuk dicompile ke *.jar:
* test.in db.insert( { x: "eliot was here" } ) db.insert( { x: "eliot is here" } ) db.insert( { x: "who is here" } ) * */
public class WordCountMongo { public static class TokenizerMapper extends Mapper<Object, BSONObject, Text, IntWritable> {
private final static IntWritable one = new IntWritable(1); private Text word = new Text(); public void map(Object key,
BSONObject value, Context context )
throws IOException, InterruptedException { System.out.println( "key: " + key ); System.out.println( "value: " + value ); StringTokenizer itr = new StringTokenizer(value.get( "x" ).toString());
while (itr.hasMoreTokens()) {
word.set(itr.nextToken()); context.write(word, one);
Source Code 8.15 File WordCountMongo.java Part 1 of 2
- Siapkan file *.java (misal WordCountMongo.java Part 2 of
2) untuk dicompile ke *.jar: public static class IntSumReducer extends
Reducer<Text,IntWritable,Text,IntWritable> {
private IntWritable result = new
IntWritable();
public void reduce(Text key,
Iterable<IntWritable> values, Context context )
throws IOException,
InterruptedException {
int sum = 0; for (IntWritable val : values) {
sum += val.get();
} result.set(sum); context.write(key, result);
} } public static void main(String[] args)
throws Exception { Configuration conf = new Configuration(); conf.set( "mongo.input.uri" ,
"mongodb://localhost/testmr.in" );
conf.set( "mongo.output.uri" ,
"mongodb://localhost/testmr.out" ); @SuppressWarnings("deprecation")
Job job = new Job(conf, "word
count"); job.setJarByClass(WordCountMongo.class);
job.setMapperClass(TokenizerMapper.class)
job.setCombinerClass(IntSumReducer.class)
; job.setReducerClass(IntSumReducer.class); job.setOutputKeyClass(Text.class);
job.setOutputValueClass(IntWritable.class
job.setInputFormatClass(
MongoInputFormat.class );
job.setOutputFormatClass(
MongoOutputFormat.class );
System.exit(job.waitForCompletion(true) ?
Source Code 8.16 File WordCountMongo.java Part 2 of 2
2. Menghitung Kemunculan Kata dalam file dokumen: - file “WordCountMongo.java”:
Gambar 8.47 file “WordCountMongo.java”:
- WordCountMongo.java dicompile ke *.jar: dengan melakukan hal berikut:
nidos@master:~$ cd /usr/local/hadoop nidos@master:/usr/local/hadoop$ cp /home/nidos/Desktop/kode/WordCountMongo/Wo rdCountMongo.java /usr/local/hadoop nidos@master:/usr/local/hadoop$ bin/hdfs com.sun.tools.javac.Main WordCountMongo.java
Jika muncul error:
nidos@master:/usr/local/hadoop$ bin/hdfs com.sun.tools.javac.Main WordCountMongo.Java error: Class names, 'WordCountMongo.Java', are only accepted if annotation processing is explicitly requested
1 error Solusi : ubah “WordCountMongo.Java” “WordCountMongo.java”
Source Code 8.17 Compile ke *.jar file WordCountMongo.java Source Code 8.17 Compile ke *.jar file WordCountMongo.java
Jika muncul error:
nidos@master:/usr/local/hadoop$ bin/hdfs com.sun.tools.javac.Main WordCountMongo.java WordCountMongo.java:4: error: package org.bson does not exist import org.bson.*; ^ WordCountMongo.java:6: error: package com.mongodb.hadoop does not exist import com.mongodb.hadoop.MongoInputFormat;
WordCountMongo.java:7: error: package com.mongodb.hadoop does not exist import com.mongodb.hadoop.MongoOutputFormat;
WordCountMongo.java:22: error: cannot find symbol public static class TokenizerMapper extends Mapper<Object, BSONObject, Text, IntWritable> {
^ symbol: class BSONObject location: class WordCountMongo
WordCountMongo.java:25: error: cannot find symbol public void map(Object key, BSONObject value, Context context ) ^ symbol: class BSONObject location: class TokenizerMapper
WordCountMongo.java:60: error: cannot find symbol job.setInputFormatClass( MongoInputFormat.class );
symbol: class MongoInputFormat location: class WordCountMongo
WordCountMongo.java:61: error: cannot find symbol job.setOutputFormatClass( MongoOutputFormat.class );
symbol: class MongoOutputFormat location: class WordCountMongo 7 errors
Solusi : kembali ke langkah “Konfigurasi Koneksi (Hadoop + MongoDB) ” SCP
Source Code 8.18 Compile ke *.jar file WordCountMongo.java
- Hasil: nidos@master:/usr/local/hadoop$ bin/hdfs
com.sun.tools.javac.Main WordCountMongo.java
Gambar 8.48 Hasil: nidos@master:/usr/local/hadoop$ bin/hdfs com.sun.tools.javac.Main WordCountMongo.java
- Hasil: nidos@master:/usr/local/hadoop$ jar cf wcmongo.jar WordCountMongo*.class
Gambar 8.49 Hasil: nidos@master:/usr/local/hadoop$ jar cf wcmongo.jar WordCountMongo*.class
- Running proses perhitungan kata dalam file dokumen dalam MongoDB: dengan melakukan hal berikut:
Jika menggunakan hdfs, maka gunakan dfs Jika menggunakan hadoop, maka gunakan fs nidos@master:/usr/local/hadoop$ bin/hadoop jar wcmongo.jar WordCountMongo
Source Code 8.19 Running proses perhitungan kata dalam file dokumen dalam MongoDB
{"_id":"2","value":1}
{"_id":"again","value":1} {"_id":"bye","value":2}
{"_id":"good","value":1} {"_id":"hello","value":1}
{"_id":"meet","value":1} {"_id":"see","value":2} {"_id":"nice","value":1} {"_id":"to","value":2}
{"_id":"war","value":1}
Gambar 8.50 Running perhitungan kata dari file dalam MongoDB
- Lihat hasil dari MongoDB melalui terminal:
Gambar 8.51 Hasil MongoDB
- Cek di browser:
Gambar 8.52 Browse Directory - Cek di browser
Gambar 8.53 Browse Directory - Cek di browser:
Gambar 8.54 File Information di Browser
8.3.2 Movie Ratings
Case Study 2: Rating By Movie
1. Running Hadoop Rating By Movie (Save hasil di MongoDB)
- List dataset (dari MovieLens) dan Link ( https://goo.gl/Jd8GOI ):
Gambar 8.55 List dataset
- Membuat Directories “ratemovie/dataset” di HDFS harus satu demi satu:
nidos@master:~$ cd /usr/local/hadoop nidos@master:/usr/local/hadoop$ bin/hdfs dfs - mkdir /user/nidos/ratemovie nidos@master:/usr/local/hadoop$ bin/hdfs dfs - mkdir /user/nidos/ratemovie/dataset nidos@master:/usr/local/hadoop$ bin/hdfs dfs - ls /user/nidos nidos@master:/usr/local/hadoop$ bin/hdfs dfs - ls /user/nidos/ratemovie
Source Code 8.20 Membuat Directories “ratemovie/dataset” di HDFS
- Copy semua
dari local folder (/home/nidos/Desktop/data/ratemovie/dataset) ke HDFS folder (/user/nidos/ratemovie/dataset)
file
dataset
List dataset
MovieLens) dan Link ( https://goo.gl/Jd8GOI ):
(dari
Gambar 8.56 List dataset (dari MovieLens)
- Copy semua
local folder
(/home/nidos/Desktop/data/ratemovie/dataset) ke HDFS folder (/user/nidos/ratemovie/dataset)
nidos@master:/usr/local/hadoop$ bin/hdfs dfs - copyFromLocal /home/nidos/Desktop/data/ratemovie/dataset/* /user/nidos/ratemovie/dataset
Source Code 8.21 Copy Semua File Dataset dari Local Folder
Gambar 8.57 Tampilan Browse Directory - Add database “ratemovie” pada MongoDB:
Gambar 8.58 Add database “ratemovie” pada MongoDB - Add database “ratemovie” pada MongoDB:
Gambar 8.59 Add database “ratemovie” pada MongoDB - Siapkan file *.java untuk dicompile ke *.jar: dari “package
comratingbymovies.nidos;”
Gambar 8.60 List File*.java untuk di Compile ke *.jar - Siapkan file (MovieMapper.java Part 1 of 1) untuk dicompile ke
*.jar: package comratingbymovies.nidos;
import java.io.IOException; import org.apache.hadoop.io.LongWritable; import org.apache.hadoop.io.Text; import org.apache.hadoop.mapreduce.Mapper; public class MovieMapper extends Mapper<LongWritable,Text,Text,Text>{
Text keyEmit = new Text(); Text valEmit = new Text(); public void map(LongWritable k, Text
value, Context context) throws IOException, InterruptedException{
String
line=value.toString();
String[]
words=line.split("::"); keyEmit.set(words[0]); valEmit.set(words[1]); context.write(keyEmit,
valEmit); } }
Source Code 8.22 file (MovieMapper.java Part 1 of 1) - Siapkan file (MovieReducer.java Part 1 of 2) untuk dicompile ke
*.jar: *.jar:
Text valTitle = new Text(); Text valEmit = new Text(); String merge; public void reduce(Text key,
Iterable<Text> values, Context context) throws IOException , InterruptedException{
int counter = 0; merge = ""; for(Text value:values){
if (value.toString().startsWith("#")){ //from rating
counter++;
Source Code 8.23 file (MovieReducer.java Part 1 of 2) - Siapkan file (MovieReducer.java Part 2 of 2) untuk dicompile ke
*.jar:
else if
("".equalsIgnoreCase(merge)){// from movies get the title
merge =
value.toString();
}} valTitle.set(merge); context.write(valTitle, new
IntWritable(counter)); }
Source Code 8.24 file (MovieReducer.java Part 2 of 2)
- Siapkan file (RatingCounterByMovieMongo.java Part 1 of 2) untuk dicompile ke *.jar:
package comratingbymovies.nidos; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.conf.Configured; import org.apache.hadoop.fs.FileSystem; import org.apache.hadoop.fs.Path; import org.apache.hadoop.io.IntWritable; import org.apache.hadoop.io.Text; import org.apache.hadoop.mapreduce.Job; import org.apache.hadoop.mapreduce.lib.input.Multi pleInputs; import org.apache.hadoop.mapreduce.lib.input.TextI nputFormat; import org.apache.hadoop.mapreduce.lib.output.File OutputFormat; import org.apache.hadoop.util.GenericOptionsParser ; import org.apache.hadoop.util.Tool; import org.apache.hadoop.util.ToolRunner; import com.mongodb.hadoop.io.BSONWritable; import com.mongodb.hadoop.mapred.MongoOutputFormat ; public class RatingCounterByMovieMongo extends Configured implements Tool{
public int run(String[] args) throws Exception { final Configuration conf = getConf(); conf.set("mongo.output.uri", args[2]); Path p1=new Path(args[0]); Path p2=new Path(args[1]); Job job = new
Job(conf,"Multiple Job");
job.setJarByClass(RatingCounterByMovi eMongo.class);
Source Code 8.25 file (RatingCounterByMovieMongo.java Part 1 of 2) Source Code 8.25 file (RatingCounterByMovieMongo.java Part 1 of 2)
MultipleInputs.addInputPath(job, p1, TextInputFormat.class, RatingMapper.class);
MultipleInputs.addInputPath(job, p2, TextInputFormat.class, MovieMapper.class);
Source Code 8.26 file (RatingCounterByMovieMongo.java Part 1 of 2)
Cont
- Siapkan file (RatingCounterByMovieMongo.java Part 2 of 2) untuk dicompile ke *.jar:
job.setReducerClass(MovieReducer.class) ;
job.setOutputFormatClass(com.mongodb.ha doop.MongoOutputFormat.class); job.setOutputKeyClass(Text.class); job.setOutputValueClass(Text.class); boolean success =
job.waitForCompletion(true); return success?0:1; } public static void main(String[] args)
throws Exception { if (args.length != 3 ){ System.err.println ("Usage :<inputlocation1> <inputlocation2> <outputlocation> >");
System.exit(0);
} int res = ToolRunner.run(new
Configuration(), new RatingCounterByMovieMongo(), args);
System.exit(res);
Source Code 8.27 file (RatingCounterByMovieMongo.java Part 2 of 2)
- Siapkan file (RatingMapper.java Part 1 of 1) untuk dicompile ke *.jar:
package comratingbymovies.nidos; import java.io.IOException; import org.apache.hadoop.io.LongWritable; import org.apache.hadoop.io.Text; import org.apache.hadoop.mapred.JobConf; import org.apache.hadoop.mapred.OutputCollector; import org.apache.hadoop.mapred.Reporter; import org.apache.hadoop.mapreduce.Mapper; public class RatingMapper extends Mapper<LongWritable,Text,Text,Text>{
Text keyEmit = new Text(); Text valEmit = new Text(); public void map(LongWritable k, Text v,
Context context) throws IOException, InterruptedException{
String line=v.toString(); String[] words=line.split("::"); keyEmit.set(words[1]); valEmit.set("#"); context.write(keyEmit, valEmit);
Source Code 8.28 file (RatingMapper.java Part 1 of 1) - Semua file *.java dicompile ke *.jar:
Lakukan hal berikut:
nidos@master:~$ cd /usr/local/hadoop nidos@master:/usr/local/hadoop$ cp -r /home/nidos/Desktop/kode/RatingByMovies/com ratingbymovies /usr/local/hadoop
Source Code 8.29 Compile file ke *.jar
Gambar 8.61 file comratingbymovies Lakukan hal berikut: nidos@master:/usr/local/hadoop$ bin/hdfs
com.sun.tools.javac.Main comratingbymovies/nidos/*.java Note: comratingbymovies/nidos/RatingCounterByMov ieMongo.java uses or overrides a deprecated API. Note: Recompile with -Xlint:deprecation for details.
Source Code 8.30 Compile file *.java ke *.jar
Gambar 8.62 Compile Semua file *.java ke *.jar
- Hasil: nidos@master:/usr/local/hadoop$ jar cf ratemovie.jar comratingbymovies/nidos/*.class
Gambar 8.63 Hasil: nidos@master:/usr/local/hadoop$ jar cf ratemovie.jar comratingbymovies/nidos/*.class
- Hasil: nidos@master:/usr/local/hadoop$ jar cf ratemovie.jar comratingbymovies/nidos/*.class
Gambar 8.64 Hasil: nidos@master:/usr/local/hadoop$ jar cf ratemovie.jar comratingbymovies/nidos/*.class
- Running proses perhitungan rating movie: Lakukan hal berikut: - Running proses perhitungan rating movie: Lakukan hal berikut:
Source Code 8.31 Running proses perhitungan rating movie out1
Gambar 8.65 Hasil Running proses perhitungan rating movie
Lakukan hal berikut: nidos@master:/usr/local/hadoop$
bin/hadoop jar ratemovie.jar comratingbymovies/nidos/RatingCounterByMo vieMongo /user/nidos/ratemovie/dataset/ratings.dat /user/nidos/ratemovie/dataset/movies.dat
mongodb://localhost/ratemovie.out2
Source Code 8.32 Running proses perhitungan rating movie out2
Gambar 8.66 Running proses perhitungan rating movie out2
- Lihat hasil dari MongoDB melalui terminal:
Gambar 8.67 Lihat hasil dari MongoDB melalui terminal - Cek di browser:
Gambar 8.68 Tampilan Browse Directory - Cek di browser:
Gambar 8.69 File Information di Browser