유비쿼터스 응용시스템:
실습 가이드
인공지능연구실
이남기
( [email protected] )
Environment
• Cloudera QuickStart VM with 5.4.2
• Guide for Download
• http://ailab.ssu.ac.kr/rb/?c=8/29&cat=2015_2_%EC%9C%A0%EB%B9%84%EC
%BF%BC%ED%84%B0%EC%8A%A4+%EC%9D%91%EC%9A%A9%EC%8B%9C%
EC%8A%A4%ED%85%9C&uid=660
Contents
• Using HDFS
•
•
•
•
How To Use
How To Upload File
How To View and Manipulate File
Exercise
• Running MapReduce Job : WordCount
•
•
•
•
Goal
Remind MapReduce
Code Review
Run WordCount Program
• Importing Data With Sqoop
• Review MySQL
• How To Use
Using HDFS
With Exercise
Using HDFS
• How to use HDFS
• How to Upload File
• How to View and Manipulate File
Using HDFS – How To Use (1)
• You see a help message describing all the commands
associated with HDFS
$ hadoop fs
Using HDFS – How To Use (2)
• You see the contents of directory in HDFS:
$ hadoop fs –ls /
$ hadoop fs –ls /user
$ hadoop fs –ls /user/cloudera
Exercise
How To Use
Using HDFS – How To Upload File (1)
• Unzip ‘shakespeare.tar.gz’:
$ cd ~/training_materials/developer/data
$ tar zxvf shakespeare.tar.gz
Using HDFS – How To Upload File (2)
• Insert ‘shakespeare’ directory into HDFS:
$ hadoop fs -put shakespeare
/user/cloudera/shakespeare
Exercise
How To Upload
Using HDFS –
How To View and Manipulate Files (1)
• Remove directory
$ hadoop fs –ls shakespeare
$ hadoop fs –rm shakespeare/glossary
Using HDFS –
How To View and Manipulate Files (2)
• Print the last 50 lines of Herny IV
$ hadoop fs –cat shakespeare/histories \
| tail –n 50
Using HDFS –
How To View and Manipulate Files (3)
• Download file and manipulate
$ hadoop fs –get shakespeare/poems \
~/shakepoems.txt
$ less ~/shakepoems.txt
• If you want to know other command:
$ hadoop fs
Exercise
How To View and Manipulate Files
Importing Data With Sqoop
Review MySQL and Exercise
Importing Data With Sqoop
• Log on to MySQL:
$ mysql --user=root \
--password=cloudera
• Select Database
> use retail_db;
• Show Databases:
> show databases;
Importing Data With Sqoop –
Review MySQL (1)
• Log on to MySQL:
$ mysql --user=root \
--password=cloudera
• Show Databases:
> show databases;
• Select Databases:
> use retail_db;
• Show Tables:
> show tables;
Importing Data With Sqoop –
Review MySQL (2)
• Review ‘customers’ table schema:
> DESCRIBE customers;
Importing Data With Sqoop –
Review MySQL (3)
• Review ‘customers’ table:
> DESCRIBE customers;
…
> SELECT * FROM customers LIMIT 5;
Importing Data With Sqoop – How To Use (1)
• List the databases (schemas) in your database server:
$ sqoop list-databases \
--connect jdbc:mysql://localhost \
--username root --password cloudera
• List the tables in the ‘retail_db’ database:
$ sqoop list-tables \
--connect jdbc:mysql://localhost/movielens \
--username root --password cloudera
Importing Data With Sqoop – How To Use (2)
• Import the ‘customers’ table into HDFS
$ sqoop import \
--connect jdbc:mysql://localhost/retail_db \
--table customers --fields-terminated-by '\t' \
--username root --password cloudera
• Verify that the command has worked
$ hadoop fs –ls customers
$ hadoop fs –tail movie/part-m-00000