how to test CDH 4 using TestDFSIO

July 29, 2013

how to test CDH 4 using TestDFSIO

bash-4.1$ time hadoop jar /opt/cloudera/parcels/CDH-4.3.0-1.cdh4.3.0.p0.22/lib/hadoop-0.20-mapreduce/hadoop-test-2.0.0-mr1-cdh4.3.0.jar TestDFSIO -write -nrFiles 10 -fileSize 1000
13/07/29 16:18:15 INFO fs.TestDFSIO: TestDFSIO.0.0.6
13/07/29 16:18:15 INFO fs.TestDFSIO: nrFiles = 10
13/07/29 16:18:15 INFO fs.TestDFSIO: fileSize (MB) = 1000.0
13/07/29 16:18:15 INFO fs.TestDFSIO: bufferSize = 1000000
13/07/29 16:18:15 INFO fs.TestDFSIO: baseDir = /benchmarks/TestDFSIO
13/07/29 16:18:16 INFO fs.TestDFSIO: creating control file: 1048576000 bytes, 10 files
13/07/29 16:18:17 INFO fs.TestDFSIO: created control files for: 10 files
13/07/29 16:18:17 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
13/07/29 16:18:17 INFO mapred.FileInputFormat: Total input paths to process : 10
13/07/29 16:18:18 WARN conf.Configuration: dfs.https.address is deprecated. Instead, use dfs.namenode.https-address
13/07/29 16:18:18 WARN conf.Configuration: io.bytes.per.checksum is deprecated. Instead, use dfs.bytes-per-checksum
13/07/29 16:18:18 INFO mapred.JobClient: Running job: job_201307241705_0008
13/07/29 16:18:19 INFO mapred.JobClient: map 0% reduce 0%
13/07/29 16:18:34 INFO mapred.JobClient: map 20% reduce 0%
13/07/29 16:18:36 INFO mapred.JobClient: map 100% reduce 0%
13/07/29 16:33:39 INFO mapred.JobClient: map 100% reduce 100%
13/07/29 16:33:41 INFO mapred.JobClient: Job complete: job_201307241705_0008
13/07/29 16:33:41 INFO mapred.JobClient: Counters: 33
13/07/29 16:33:41 INFO mapred.JobClient:   File System Counters
13/07/29 16:33:41 INFO mapred.JobClient:     FILE: Number of bytes read=381
13/07/29 16:33:41 INFO mapred.JobClient:     FILE: Number of bytes written=1793188
13/07/29 16:33:41 INFO mapred.JobClient:     FILE: Number of read operations=0
13/07/29 16:33:41 INFO mapred.JobClient:     FILE: Number of large read operations=0
13/07/29 16:33:41 INFO mapred.JobClient:     FILE: Number of write operations=0
13/07/29 16:33:41 INFO mapred.JobClient:     HDFS: Number of bytes read=2500
13/07/29 16:33:41 INFO mapred.JobClient:     HDFS: Number of bytes written=10485760081
13/07/29 16:33:41 INFO mapred.JobClient:     HDFS: Number of read operations=31
13/07/29 16:33:41 INFO mapred.JobClient:     HDFS: Number of large read operations=0
13/07/29 16:33:41 INFO mapred.JobClient:     HDFS: Number of write operations=12
13/07/29 16:33:41 INFO mapred.JobClient:   Job Counters
13/07/29 16:33:41 INFO mapred.JobClient:     Launched map tasks=10
13/07/29 16:33:41 INFO mapred.JobClient:     Launched reduce tasks=1
13/07/29 16:33:41 INFO mapred.JobClient:     Data-local map tasks=10
13/07/29 16:33:41 INFO mapred.JobClient:     Total time spent by all maps in occupied slots (ms)=9049830
13/07/29 16:33:41 INFO mapred.JobClient:     Total time spent by all reduces in occupied slots (ms)=4519
13/07/29 16:33:41 INFO mapred.JobClient:     Total time spent by all maps waiting after reserving slots (ms)=0
13/07/29 16:33:41 INFO mapred.JobClient:     Total time spent by all reduces waiting after reserving slots (ms)=0
13/07/29 16:33:41 INFO mapred.JobClient:   Map-Reduce Framework
13/07/29 16:33:41 INFO mapred.JobClient:     Map input records=10
13/07/29 16:33:41 INFO mapred.JobClient:     Map output records=50
13/07/29 16:33:41 INFO mapred.JobClient:     Map output bytes=776
13/07/29 16:33:41 INFO mapred.JobClient:     Input split bytes=1380
13/07/29 16:33:41 INFO mapred.JobClient:     Combine input records=0
13/07/29 16:33:41 INFO mapred.JobClient:     Combine output records=0
13/07/29 16:33:41 INFO mapred.JobClient:     Reduce input groups=5
13/07/29 16:33:41 INFO mapred.JobClient:     Reduce shuffle bytes=1014
13/07/29 16:33:41 INFO mapred.JobClient:     Reduce input records=50
13/07/29 16:33:41 INFO mapred.JobClient:     Reduce output records=5
13/07/29 16:33:41 INFO mapred.JobClient:     Spilled Records=100
13/07/29 16:33:41 INFO mapred.JobClient:     CPU time spent (ms)=187010
13/07/29 16:33:41 INFO mapred.JobClient:     Physical memory (bytes) snapshot=3650072576
13/07/29 16:33:41 INFO mapred.JobClient:     Virtual memory (bytes) snapshot=14255079424
13/07/29 16:33:41 INFO mapred.JobClient:     Total committed heap usage (bytes)=3174891520
13/07/29 16:33:41 INFO mapred.JobClient:   org.apache.hadoop.mapreduce.lib.input.FileInputFormatCounter
13/07/29 16:33:41 INFO mapred.JobClient:     BYTES_READ=260
13/07/29 16:33:41 INFO fs.TestDFSIO: ----- TestDFSIO ----- : write
13/07/29 16:33:41 INFO fs.TestDFSIO:            Date & time: Mon Jul 29 16:33:41 PDT 2013
13/07/29 16:33:41 INFO fs.TestDFSIO:        Number of files: 10
13/07/29 16:33:41 INFO fs.TestDFSIO: Total MBytes processed: 10000.0
13/07/29 16:33:41 INFO fs.TestDFSIO:      Throughput mb/sec: 1.112240158454182
13/07/29 16:33:41 INFO fs.TestDFSIO: Average IO rate mb/sec: 1.1122640371322632
13/07/29 16:33:41 INFO fs.TestDFSIO: IO rate std deviation: 0.005174144962407314
13/07/29 16:33:41 INFO fs.TestDFSIO:     Test exec time sec: 924.473
13/07/29 16:33:41 INFO fs.TestDFSIO:

real    15m27.624s
user    0m6.575s
sys     0m0.500s

bash-4.1$ time hadoop jar /opt/cloudera/parcels/CDH-4.3.0-1.cdh4.3.0.p0.22/lib/hadoop-0.20-mapreduce/hadoop-test-2.0.0-mr1-cdh4.3.0.jar TestDFSIO -read -nrFiles 10 -fileSize 1000
13/07/29 16:35:09 INFO fs.TestDFSIO: TestDFSIO.0.0.6
13/07/29 16:35:09 INFO fs.TestDFSIO: nrFiles = 10
13/07/29 16:35:09 INFO fs.TestDFSIO: fileSize (MB) = 1000.0
13/07/29 16:35:09 INFO fs.TestDFSIO: bufferSize = 1000000
13/07/29 16:35:09 INFO fs.TestDFSIO: baseDir = /benchmarks/TestDFSIO
13/07/29 16:35:10 INFO fs.TestDFSIO: creating control file: 1048576000 bytes, 10 files
13/07/29 16:35:11 INFO fs.TestDFSIO: created control files for: 10 files
13/07/29 16:35:11 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
13/07/29 16:35:11 INFO mapred.FileInputFormat: Total input paths to process : 10
13/07/29 16:35:12 WARN conf.Configuration: dfs.https.address is deprecated. Instead, use dfs.namenode.https-address
13/07/29 16:35:12 WARN conf.Configuration: io.bytes.per.checksum is deprecated. Instead, use dfs.bytes-per-checksum
13/07/29 16:35:12 INFO mapred.JobClient: Running job: job_201307241705_0009
13/07/29 16:35:13 INFO mapred.JobClient: map 0% reduce 0%
13/07/29 16:35:27 INFO mapred.JobClient: map 20% reduce 0%
13/07/29 16:35:28 INFO mapred.JobClient: map 60% reduce 0%
13/07/29 16:35:29 INFO mapred.JobClient: map 100% reduce 0%
13/07/29 16:35:55 INFO mapred.JobClient: map 100% reduce 100%
13/07/29 16:35:56 INFO mapred.JobClient: Job complete: job_201307241705_0009
13/07/29 16:35:56 INFO mapred.JobClient: Counters: 33
13/07/29 16:35:56 INFO mapred.JobClient:   File System Counters
13/07/29 16:35:56 INFO mapred.JobClient:     FILE: Number of bytes read=397
13/07/29 16:35:56 INFO mapred.JobClient:     FILE: Number of bytes written=1793178
13/07/29 16:35:56 INFO mapred.JobClient:     FILE: Number of read operations=0
13/07/29 16:35:56 INFO mapred.JobClient:     FILE: Number of large read operations=0
13/07/29 16:35:56 INFO mapred.JobClient:     FILE: Number of write operations=0
13/07/29 16:35:56 INFO mapred.JobClient:     HDFS: Number of bytes read=10485762500
13/07/29 16:35:56 INFO mapred.JobClient:     HDFS: Number of bytes written=81
13/07/29 16:35:56 INFO mapred.JobClient:     HDFS: Number of read operations=41
13/07/29 16:35:56 INFO mapred.JobClient:     HDFS: Number of large read operations=0
13/07/29 16:35:56 INFO mapred.JobClient:     HDFS: Number of write operations=2
13/07/29 16:35:56 INFO mapred.JobClient:   Job Counters
13/07/29 16:35:56 INFO mapred.JobClient:     Launched map tasks=10
13/07/29 16:35:56 INFO mapred.JobClient:     Launched reduce tasks=1
13/07/29 16:35:56 INFO mapred.JobClient:     Data-local map tasks=10
13/07/29 16:35:56 INFO mapred.JobClient:     Total time spent by all maps in occupied slots (ms)=241167
13/07/29 16:35:56 INFO mapred.JobClient:     Total time spent by all reduces in occupied slots (ms)=4444
13/07/29 16:35:56 INFO mapred.JobClient:     Total time spent by all maps waiting after reserving slots (ms)=0
13/07/29 16:35:56 INFO mapred.JobClient:     Total time spent by all reduces waiting after reserving slots (ms)=0
13/07/29 16:35:56 INFO mapred.JobClient:   Map-Reduce Framework
13/07/29 16:35:56 INFO mapred.JobClient:     Map input records=10
13/07/29 16:35:56 INFO mapred.JobClient:     Map output records=50
13/07/29 16:35:56 INFO mapred.JobClient:     Map output bytes=765
13/07/29 16:35:56 INFO mapred.JobClient:     Input split bytes=1380
13/07/29 16:35:56 INFO mapred.JobClient:     Combine input records=0
13/07/29 16:35:56 INFO mapred.JobClient:     Combine output records=0
13/07/29 16:35:56 INFO mapred.JobClient:     Reduce input groups=5
13/07/29 16:35:56 INFO mapred.JobClient:     Reduce shuffle bytes=1010
13/07/29 16:35:56 INFO mapred.JobClient:     Reduce input records=50
13/07/29 16:35:56 INFO mapred.JobClient:     Reduce output records=5
13/07/29 16:35:56 INFO mapred.JobClient:     Spilled Records=100
13/07/29 16:35:56 INFO mapred.JobClient:     CPU time spent (ms)=48900
13/07/29 16:35:56 INFO mapred.JobClient:     Physical memory (bytes) snapshot=3555450880
13/07/29 16:35:56 INFO mapred.JobClient:     Virtual memory (bytes) snapshot=14205288448
13/07/29 16:35:56 INFO mapred.JobClient:     Total committed heap usage (bytes)=3903979520
13/07/29 16:35:56 INFO mapred.JobClient:   org.apache.hadoop.mapreduce.lib.input.FileInputFormatCounter
13/07/29 16:35:56 INFO mapred.JobClient:     BYTES_READ=260
13/07/29 16:35:56 INFO fs.TestDFSIO: ----- TestDFSIO ----- : read
13/07/29 16:35:56 INFO fs.TestDFSIO:            Date & time: Mon Jul 29 16:35:56 PDT 2013
13/07/29 16:35:56 INFO fs.TestDFSIO:        Number of files: 10
13/07/29 16:35:56 INFO fs.TestDFSIO: Total MBytes processed: 10000.0
13/07/29 16:35:56 INFO fs.TestDFSIO:      Throughput mb/sec: 54.16736632848174
13/07/29 16:35:56 INFO fs.TestDFSIO: Average IO rate mb/sec: 59.13368606567383
13/07/29 16:35:56 INFO fs.TestDFSIO: IO rate std deviation: 18.78432780576558
13/07/29 16:35:56 INFO fs.TestDFSIO:     Test exec time sec: 45.361
13/07/29 16:35:56 INFO fs.TestDFSIO:

real    0m48.490s
user    0m4.820s
sys     0m0.330s

bash-4.1$ time hadoop jar /opt/cloudera/parcels/CDH-4.3.0-1.cdh4.3.0.p0.22/lib/hadoop-0.20-mapreduce/hadoop-test-2.0.0-mr1-cdh4.3.0.jar TestDFSIO -clean
13/07/29 16:37:11 INFO fs.TestDFSIO: TestDFSIO.0.0.6
13/07/29 16:37:11 INFO fs.TestDFSIO: nrFiles = 1
13/07/29 16:37:11 INFO fs.TestDFSIO: fileSize (MB) = 1.0
13/07/29 16:37:11 INFO fs.TestDFSIO: bufferSize = 1000000
13/07/29 16:37:11 INFO fs.TestDFSIO: baseDir = /benchmarks/TestDFSIO
13/07/29 16:37:12 INFO fs.TestDFSIO: Cleaning up test files

real    0m2.146s
user    0m3.143s
sys     0m0.214s

Search This Blog

how to test CDH 4 using TestDFSIO

Popular Posts

Finally, CERN made the data open, from LHC

The Future