org.apache.hadoop.mrunit

Class MapReduceDriver<K1,V1,K2,V2,K3,V3>

java.lang.Object

org.apache.hadoop.mrunit.TestDriver<K3,V3,T>

org.apache.hadoop.mrunit.MapReduceDriverBase<K1,V1,K2,V2,K3,V3,MapReduceDriver<K1,V1,K2,V2,K3,V3>>

org.apache.hadoop.mrunit.MapReduceDriver<K1,V1,K2,V2,K3,V3>

public class MapReduceDriver<K1,V1,K2,V2,K3,V3>
extends MapReduceDriverBase<K1,V1,K2,V2,K3,V3,MapReduceDriver<K1,V1,K2,V2,K3,V3>>

Harness that allows you to test a Mapper and a Reducer instance together (along with an optional combiner). You provide the input key and value that should be sent to the Mapper, and outputs you expect to be sent by the Reducer to the collector for those inputs. By calling runTest(), the harness will deliver the input to the Mapper, feed the intermediate results to the Reducer (without checking them), and will check the Reducer's outputs against the expected results. This is designed to handle the (k, v)* -> (k, v)* case from the Mapper/Reducer pair, representing a single unit test. If a combiner is specified, then it will be run exactly once after the Mapper and before the Reducer.

Field Summary

Fields

Modifier and Type	Field and Description
static org.apache.commons.logging.Log	LOG

Fields inherited from class org.apache.hadoop.mrunit.MapReduceDriverBase

inputList, keyGroupComparator, keyValueOrderComparator, mapInputPath

Fields inherited from class org.apache.hadoop.mrunit.TestDriver

counterWrapper, expectedEnumCounters, expectedMultipleOutputs, expectedOutputs, expectedPathOutputs, expectedStringCounters, mos

Constructor Summary

Constructors

Constructor and Description
MapReduceDriver()
MapReduceDriver(org.apache.hadoop.mapred.Mapper<K1,V1,K2,V2> m, org.apache.hadoop.mapred.Reducer<K2,V2,K3,V3> r)
MapReduceDriver(org.apache.hadoop.mapred.Mapper<K1,V1,K2,V2> m, org.apache.hadoop.mapred.Reducer<K2,V2,K3,V3> r, org.apache.hadoop.mapred.Reducer<K2,V2,K2,V2> c)

Method Summary

Methods

Modifier and Type	Method and Description
org.apache.hadoop.mapred.Reducer<K2,V2,K2,V2>	getCombiner()
org.apache.hadoop.mapred.Counters	getCounters()
org.apache.hadoop.mapred.Mapper<K1,V1,K2,V2>	getMapper()
org.apache.hadoop.mapred.Reducer<K2,V2,K3,V3>	getReducer()
static <K1,V1,K2,V2,K3,V3> MapReduceDriver<K1,V1,K2,V2,K3,V3>	newMapReduceDriver() Returns a new MapReduceDriver without having to specify the generic types on the right hand side of the object create statement.
static <K1,V1,K2,V2,K3,V3> MapReduceDriver<K1,V1,K2,V2,K3,V3>	newMapReduceDriver(org.apache.hadoop.mapred.Mapper<K1,V1,K2,V2> mapper, org.apache.hadoop.mapred.Reducer<K2,V2,K3,V3> reducer) Returns a new MapReduceDriver without having to specify the generic types on the right hand side of the object create statement.
static <K1,V1,K2,V2,K3,V3> MapReduceDriver<K1,V1,K2,V2,K3,V3>	newMapReduceDriver(org.apache.hadoop.mapred.Mapper<K1,V1,K2,V2> mapper, org.apache.hadoop.mapred.Reducer<K2,V2,K3,V3> reducer, org.apache.hadoop.mapred.Reducer<K2,V2,K2,V2> combiner) Returns a new MapReduceDriver without having to specify the generic types on the right hand side of the object create statement.
List<Pair<K3,V3>>	run() Runs the test but returns the result set instead of validating it (ignores any addOutput(), etc calls made before this)
void	setCombiner(org.apache.hadoop.mapred.Reducer<K2,V2,K2,V2> c) Sets the reducer object to use as a combiner for this test
void	setCounters(org.apache.hadoop.mapred.Counters ctrs) Sets the counters object to use for this test.
void	setMapper(org.apache.hadoop.mapred.Mapper<K1,V1,K2,V2> m) Set the Mapper instance to use with this test driver
void	setReducer(org.apache.hadoop.mapred.Reducer<K2,V2,K3,V3> r) Sets the reducer object to use for this test
String	toString()
MapReduceDriver<K1,V1,K2,V2,K3,V3>	withCombiner(org.apache.hadoop.mapred.Reducer<K2,V2,K2,V2> c) Identical to setCombiner(), but with fluent programming style
MapReduceDriver<K1,V1,K2,V2,K3,V3>	withCounters(org.apache.hadoop.mapred.Counters ctrs) Sets the counters to use and returns self for fluent style
MapReduceDriver<K1,V1,K2,V2,K3,V3>	withMapper(org.apache.hadoop.mapred.Mapper<K1,V1,K2,V2> m) Sets the Mapper instance to use and returns self for fluent style
MapReduceDriver<K1,V1,K2,V2,K3,V3>	withOutputFormat(Class<? extends org.apache.hadoop.mapred.OutputFormat> outputFormatClass, Class<? extends org.apache.hadoop.mapred.InputFormat> inputFormatClass) Configure Reducer to output with a real OutputFormat.
MapReduceDriver<K1,V1,K2,V2,K3,V3>	withReducer(org.apache.hadoop.mapred.Reducer<K2,V2,K3,V3> r) Identical to setReducer(), but with fluent programming style

Methods inherited from class org.apache.hadoop.mrunit.MapReduceDriverBase

addAll, addInput, addInput, addInputFromString, getMapInputPath, preRunChecks, setKeyGroupingComparator, setKeyOrderComparator, setMapInputPath, withAll, withInput, withInput, withInputFromString, withKeyGroupingComparator, withKeyOrderComparator, withMapInputPath

Methods inherited from class org.apache.hadoop.mrunit.TestDriver

addAllOutput, addCacheArchive, addCacheArchive, addCacheFile, addCacheFile, addMultiOutput, addMultiOutput, addOutput, addOutput, addOutputFromString, cleanupDistributedCache, copy, copyPair, driverReused, formatPairList, formatValueList, getConfiguration, getExpectedEnumCounters, getExpectedOutputs, getExpectedStringCounters, getKeyComparator, getOutputSerializationConfiguration, getValueComparator, initDistributedCache, parseCommaDelimitedList, parseTabbedPair, printPreTestDebugLog, resetExpectedCounters, resetOutput, run, runTest, runTest, setCacheArchives, setCacheFiles, setConfiguration, setKeyComparator, setOutputSerializationConfiguration, setUsedOnceStatus, setValueComparator, thisAsTestDriver, validate, validate, validate, validateOutputList, withAllOutput, withCacheArchive, withCacheArchive, withCacheFile, withCacheFile, withConfiguration, withCounter, withCounter, withMultiOutput, withMultiOutput, withOutput, withOutput, withOutputFromString, withOutputSerializationConfiguration, withPathOutput, withPathOutput, withStrictCounterChecking

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

LOG

public static final org.apache.commons.logging.Log LOG

Constructor Detail

MapReduceDriver

public MapReduceDriver(org.apache.hadoop.mapred.Mapper<K1,V1,K2,V2> m,
               org.apache.hadoop.mapred.Reducer<K2,V2,K3,V3> r)

MapReduceDriver

public MapReduceDriver(org.apache.hadoop.mapred.Mapper<K1,V1,K2,V2> m,
               org.apache.hadoop.mapred.Reducer<K2,V2,K3,V3> r,
               org.apache.hadoop.mapred.Reducer<K2,V2,K2,V2> c)

MapReduceDriver

public MapReduceDriver()

Method Detail

getCounters

public org.apache.hadoop.mapred.Counters getCounters()

Returns:

the counters used in this test

setCounters

public void setCounters(org.apache.hadoop.mapred.Counters ctrs)

Sets the counters object to use for this test.

Parameters:

ctrs - The counters object to use.

withCounters

public MapReduceDriver<K1,V1,K2,V2,K3,V3> withCounters(org.apache.hadoop.mapred.Counters ctrs)

Sets the counters to use and returns self for fluent style

setMapper

public void setMapper(org.apache.hadoop.mapred.Mapper<K1,V1,K2,V2> m)

Set the Mapper instance to use with this test driver

Parameters:

m - the Mapper instance to use

withMapper

public MapReduceDriver<K1,V1,K2,V2,K3,V3> withMapper(org.apache.hadoop.mapred.Mapper<K1,V1,K2,V2> m)

Sets the Mapper instance to use and returns self for fluent style

getMapper

public org.apache.hadoop.mapred.Mapper<K1,V1,K2,V2> getMapper()

Returns:

the Mapper object being used by this test

setReducer

public void setReducer(org.apache.hadoop.mapred.Reducer<K2,V2,K3,V3> r)

Sets the reducer object to use for this test

Parameters:

r - The reducer object to use

withReducer

public MapReduceDriver<K1,V1,K2,V2,K3,V3> withReducer(org.apache.hadoop.mapred.Reducer<K2,V2,K3,V3> r)

Identical to setReducer(), but with fluent programming style

Parameters:

r - The Reducer to use

Returns:

this

getReducer

public org.apache.hadoop.mapred.Reducer<K2,V2,K3,V3> getReducer()

Returns:

the Reducer object being used for this test

setCombiner

public void setCombiner(org.apache.hadoop.mapred.Reducer<K2,V2,K2,V2> c)

Sets the reducer object to use as a combiner for this test

Parameters:

c - The combiner object to use

withCombiner

public MapReduceDriver<K1,V1,K2,V2,K3,V3> withCombiner(org.apache.hadoop.mapred.Reducer<K2,V2,K2,V2> c)

Identical to setCombiner(), but with fluent programming style

Parameters:

c - The Combiner to use

Returns:

this

getCombiner

public org.apache.hadoop.mapred.Reducer<K2,V2,K2,V2> getCombiner()

Returns:

the Combiner object being used for this test

withOutputFormat

public MapReduceDriver<K1,V1,K2,V2,K3,V3> withOutputFormat(Class<? extends org.apache.hadoop.mapred.OutputFormat> outputFormatClass,
                                                  Class<? extends org.apache.hadoop.mapred.InputFormat> inputFormatClass)

Configure Reducer to output with a real OutputFormat. Set InputFormat to read output back in for use with run* methods

Parameters:

outputFormatClass -

inputFormatClass -

Returns:

this for fluent style

run

public List<Pair<K3,V3>> run()
                      throws IOException

Description copied from class: TestDriver

Runs the test but returns the result set instead of validating it (ignores any addOutput(), etc calls made before this)

Specified by:

run in class MapReduceDriverBase<K1,V1,K2,V2,K3,V3,MapReduceDriver<K1,V1,K2,V2,K3,V3>>

Returns:

the list of (k, v) pairs returned as output from the test

Throws:

IOException

toString

public String toString()

Overrides:

toString in class Object

newMapReduceDriver

public static <K1,V1,K2,V2,K3,V3> MapReduceDriver<K1,V1,K2,V2,K3,V3> newMapReduceDriver()

Returns a new MapReduceDriver without having to specify the generic types on the right hand side of the object create statement.

Returns:

new MapReduceDriver

newMapReduceDriver

public static <K1,V1,K2,V2,K3,V3> MapReduceDriver<K1,V1,K2,V2,K3,V3> newMapReduceDriver(org.apache.hadoop.mapred.Mapper<K1,V1,K2,V2> mapper,
                                                                        org.apache.hadoop.mapred.Reducer<K2,V2,K3,V3> reducer)

Returns a new MapReduceDriver without having to specify the generic types on the right hand side of the object create statement.

Parameters:

mapper - passed to MapReduceDriver constructor

reducer - passed to MapReduceDriver constructor

Returns:

new MapReduceDriver

newMapReduceDriver

public static <K1,V1,K2,V2,K3,V3> MapReduceDriver<K1,V1,K2,V2,K3,V3> newMapReduceDriver(org.apache.hadoop.mapred.Mapper<K1,V1,K2,V2> mapper,
                                                                        org.apache.hadoop.mapred.Reducer<K2,V2,K3,V3> reducer,
                                                                        org.apache.hadoop.mapred.Reducer<K2,V2,K2,V2> combiner)

Returns a new MapReduceDriver without having to specify the generic types on the right hand side of the object create statement.

Parameters:

mapper - passed to MapReduceDriver constructor

reducer - passed to MapReduceDriver constructor

combiner - passed to MapReduceDriver constructor

Returns:

new MapReduceDriver