org.apache.hadoop.mrunit
Class MapReduceDriver<K1,V1,K2,V2,K3,V3>

java.lang.Object
  extended by org.apache.hadoop.mrunit.TestDriver<K1,V1,K3,V3>
      extended by org.apache.hadoop.mrunit.MapReduceDriverBase<K1,V1,K2,V2,K3,V3>
          extended by org.apache.hadoop.mrunit.MapReduceDriver<K1,V1,K2,V2,K3,V3>

public class MapReduceDriver<K1,V1,K2,V2,K3,V3>
extends MapReduceDriverBase<K1,V1,K2,V2,K3,V3>

Harness that allows you to test a Mapper and a Reducer instance together (along with an optional combiner). You provide the input key and value that should be sent to the Mapper, and outputs you expect to be sent by the Reducer to the collector for those inputs. By calling runTest(), the harness will deliver the input to the Mapper, feed the intermediate results to the Reducer (without checking them), and will check the Reducer's outputs against the expected results. This is designed to handle the (k, v)* -> (k, v)* case from the Mapper/Reducer pair, representing a single unit test. If a combiner is specified, then it will be run exactly once after the Mapper and before the Reducer.


Field Summary
static org.apache.commons.logging.Log LOG
           
 
Fields inherited from class org.apache.hadoop.mrunit.MapReduceDriverBase
inputList, keyGroupComparator, keyValueOrderComparator
 
Fields inherited from class org.apache.hadoop.mrunit.TestDriver
configuration, counterWrapper, expectedEnumCounters, expectedOutputs, expectedStringCounters
 
Constructor Summary
MapReduceDriver()
           
MapReduceDriver(org.apache.hadoop.mapred.Mapper<K1,V1,K2,V2> m, org.apache.hadoop.mapred.Reducer<K2,V2,K3,V3> r)
           
MapReduceDriver(org.apache.hadoop.mapred.Mapper<K1,V1,K2,V2> m, org.apache.hadoop.mapred.Reducer<K2,V2,K3,V3> r, org.apache.hadoop.mapred.Reducer<K2,V2,K2,V2> c)
           
 
Method Summary
 org.apache.hadoop.mapred.Reducer<K2,V2,K2,V2> getCombiner()
           
 org.apache.hadoop.mapred.Counters getCounters()
           
 org.apache.hadoop.mapred.Mapper<K1,V1,K2,V2> getMapper()
           
 org.apache.hadoop.mapred.Reducer<K2,V2,K3,V3> getReducer()
           
static
<K1,V1,K2,V2,K3,V3>
MapReduceDriver<K1,V1,K2,V2,K3,V3>
newMapReduceDriver()
          Returns a new MapReduceDriver without having to specify the generic types on the right hand side of the object create statement.
static
<K1,V1,K2,V2,K3,V3>
MapReduceDriver<K1,V1,K2,V2,K3,V3>
newMapReduceDriver(org.apache.hadoop.mapred.Mapper<K1,V1,K2,V2> mapper, org.apache.hadoop.mapred.Reducer<K2,V2,K3,V3> reducer)
          Returns a new MapReduceDriver without having to specify the generic types on the right hand side of the object create statement.
static
<K1,V1,K2,V2,K3,V3>
MapReduceDriver<K1,V1,K2,V2,K3,V3>
newMapReduceDriver(org.apache.hadoop.mapred.Mapper<K1,V1,K2,V2> mapper, org.apache.hadoop.mapred.Reducer<K2,V2,K3,V3> reducer, org.apache.hadoop.mapred.Reducer<K2,V2,K2,V2> combiner)
          Returns a new MapReduceDriver without having to specify the generic types on the right hand side of the object create statement.
 List<Pair<K3,V3>> run()
          Runs the test but returns the result set instead of validating it (ignores any addOutput(), etc calls made before this)
 void setCombiner(org.apache.hadoop.mapred.Reducer<K2,V2,K2,V2> c)
          Sets the reducer object to use as a combiner for this test
 void setCounters(org.apache.hadoop.mapred.Counters ctrs)
          Sets the counters object to use for this test.
 void setMapper(org.apache.hadoop.mapred.Mapper<K1,V1,K2,V2> m)
          Set the Mapper instance to use with this test driver
 void setReducer(org.apache.hadoop.mapred.Reducer<K2,V2,K3,V3> r)
          Sets the reducer object to use for this test
 String toString()
           
 MapReduceDriver<K1,V1,K2,V2,K3,V3> withCombiner(org.apache.hadoop.mapred.Reducer<K2,V2,K2,V2> c)
          Identical to setCombiner(), but with fluent programming style
 MapReduceDriver<K1,V1,K2,V2,K3,V3> withConfiguration(org.apache.hadoop.conf.Configuration configuration)
           
 MapReduceDriver<K1,V1,K2,V2,K3,V3> withCounter(Enum e, long expectedValue)
          Register expected enumeration based counter value
 MapReduceDriver<K1,V1,K2,V2,K3,V3> withCounter(String g, String n, long e)
          Register expected name based counter value
 MapReduceDriver<K1,V1,K2,V2,K3,V3> withCounters(org.apache.hadoop.mapred.Counters ctrs)
          Sets the counters to use and returns self for fluent style
 MapReduceDriver<K1,V1,K2,V2,K3,V3> withInput(K1 key, V1 val)
          Identical to addInput() but returns self for fluent programming style
 MapReduceDriver<K1,V1,K2,V2,K3,V3> withInput(Pair<K1,V1> input)
          Identical to addInput() but returns self for fluent programming style
 MapReduceDriver<K1,V1,K2,V2,K3,V3> withInputFromString(String input)
          Deprecated. No replacement due to lack of type safety and incompatibility with non Text Writables
 MapReduceDriver<K1,V1,K2,V2,K3,V3> withKeyGroupingComparator(org.apache.hadoop.io.RawComparator<K2> groupingComparator)
          Identical to MapReduceDriverBase.setKeyGroupingComparator(RawComparator), but with a fluent programming style
 MapReduceDriver<K1,V1,K2,V2,K3,V3> withKeyOrderComparator(org.apache.hadoop.io.RawComparator<K2> orderComparator)
          Identical to MapReduceDriverBase.setKeyOrderComparator(RawComparator), but with a fluent programming style
 MapReduceDriver<K1,V1,K2,V2,K3,V3> withMapper(org.apache.hadoop.mapred.Mapper<K1,V1,K2,V2> m)
          Sets the Mapper instance to use and returns self for fluent style
 MapReduceDriver<K1,V1,K2,V2,K3,V3> withOutput(K3 key, V3 val)
          Functions like addOutput() but returns self for fluent programming style
 MapReduceDriver<K1,V1,K2,V2,K3,V3> withOutput(Pair<K3,V3> outputRecord)
          Works like addOutput(), but returns self for fluent style
 MapReduceDriver<K1,V1,K2,V2,K3,V3> withOutputFromString(String output)
          Deprecated. No replacement due to lack of type safety and incompatibility with non Text Writables
 MapReduceDriver<K1,V1,K2,V2,K3,V3> withReducer(org.apache.hadoop.mapred.Reducer<K2,V2,K3,V3> r)
          Identical to setReducer(), but with fluent programming style
 
Methods inherited from class org.apache.hadoop.mrunit.MapReduceDriverBase
addInput, addInput, addInputFromString, addOutput, addOutput, addOutputFromString, runTest, setKeyGroupingComparator, setKeyOrderComparator, shuffle
 
Methods inherited from class org.apache.hadoop.mrunit.TestDriver
formatValueList, getConfiguration, getExpectedEnumCounters, getExpectedOutputs, getExpectedStringCounters, parseCommaDelimitedList, parseTabbedPair, resetExpectedCounters, resetOutput, runTest, setConfiguration, validate, validate
 
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait
 

Field Detail

LOG

public static final org.apache.commons.logging.Log LOG
Constructor Detail

MapReduceDriver

public MapReduceDriver(org.apache.hadoop.mapred.Mapper<K1,V1,K2,V2> m,
                       org.apache.hadoop.mapred.Reducer<K2,V2,K3,V3> r)

MapReduceDriver

public MapReduceDriver(org.apache.hadoop.mapred.Mapper<K1,V1,K2,V2> m,
                       org.apache.hadoop.mapred.Reducer<K2,V2,K3,V3> r,
                       org.apache.hadoop.mapred.Reducer<K2,V2,K2,V2> c)

MapReduceDriver

public MapReduceDriver()
Method Detail

getCounters

public org.apache.hadoop.mapred.Counters getCounters()
Returns:
the counters used in this test

setCounters

public void setCounters(org.apache.hadoop.mapred.Counters ctrs)
Sets the counters object to use for this test.

Parameters:
ctrs - The counters object to use.

withCounters

public MapReduceDriver<K1,V1,K2,V2,K3,V3> withCounters(org.apache.hadoop.mapred.Counters ctrs)
Sets the counters to use and returns self for fluent style


setMapper

public void setMapper(org.apache.hadoop.mapred.Mapper<K1,V1,K2,V2> m)
Set the Mapper instance to use with this test driver

Parameters:
m - the Mapper instance to use

withMapper

public MapReduceDriver<K1,V1,K2,V2,K3,V3> withMapper(org.apache.hadoop.mapred.Mapper<K1,V1,K2,V2> m)
Sets the Mapper instance to use and returns self for fluent style


getMapper

public org.apache.hadoop.mapred.Mapper<K1,V1,K2,V2> getMapper()
Returns:
the Mapper object being used by this test

setReducer

public void setReducer(org.apache.hadoop.mapred.Reducer<K2,V2,K3,V3> r)
Sets the reducer object to use for this test

Parameters:
r - The reducer object to use

withReducer

public MapReduceDriver<K1,V1,K2,V2,K3,V3> withReducer(org.apache.hadoop.mapred.Reducer<K2,V2,K3,V3> r)
Identical to setReducer(), but with fluent programming style

Parameters:
r - The Reducer to use
Returns:
this

getReducer

public org.apache.hadoop.mapred.Reducer<K2,V2,K3,V3> getReducer()
Returns:
the Reducer object being used for this test

setCombiner

public void setCombiner(org.apache.hadoop.mapred.Reducer<K2,V2,K2,V2> c)
Sets the reducer object to use as a combiner for this test

Parameters:
c - The combiner object to use

withCombiner

public MapReduceDriver<K1,V1,K2,V2,K3,V3> withCombiner(org.apache.hadoop.mapred.Reducer<K2,V2,K2,V2> c)
Identical to setCombiner(), but with fluent programming style

Parameters:
c - The Combiner to use
Returns:
this

getCombiner

public org.apache.hadoop.mapred.Reducer<K2,V2,K2,V2> getCombiner()
Returns:
the Combiner object being used for this test

withInput

public MapReduceDriver<K1,V1,K2,V2,K3,V3> withInput(K1 key,
                                                    V1 val)
Identical to addInput() but returns self for fluent programming style

Parameters:
key -
val -
Returns:
this

withInput

public MapReduceDriver<K1,V1,K2,V2,K3,V3> withInput(Pair<K1,V1> input)
Identical to addInput() but returns self for fluent programming style

Parameters:
input - The (k, v) pair to add
Returns:
this

withOutput

public MapReduceDriver<K1,V1,K2,V2,K3,V3> withOutput(Pair<K3,V3> outputRecord)
Works like addOutput(), but returns self for fluent style

Parameters:
outputRecord -
Returns:
this

withOutput

public MapReduceDriver<K1,V1,K2,V2,K3,V3> withOutput(K3 key,
                                                     V3 val)
Functions like addOutput() but returns self for fluent programming style

Parameters:
key -
val -
Returns:
this

withInputFromString

@Deprecated
public MapReduceDriver<K1,V1,K2,V2,K3,V3> withInputFromString(String input)
Deprecated. No replacement due to lack of type safety and incompatibility with non Text Writables

Identical to addInputFromString, but with a fluent programming style

Parameters:
input - A string of the form "key \t val". Trims any whitespace.
Returns:
this

withOutputFromString

@Deprecated
public MapReduceDriver<K1,V1,K2,V2,K3,V3> withOutputFromString(String output)
Deprecated. No replacement due to lack of type safety and incompatibility with non Text Writables

Identical to addOutputFromString, but with a fluent programming style

Parameters:
output - A string of the form "key \t val". Trims any whitespace.
Returns:
this

run

public List<Pair<K3,V3>> run()
                      throws IOException
Description copied from class: TestDriver
Runs the test but returns the result set instead of validating it (ignores any addOutput(), etc calls made before this)

Specified by:
run in class MapReduceDriverBase<K1,V1,K2,V2,K3,V3>
Returns:
the list of (k, v) pairs returned as output from the test
Throws:
IOException

toString

public String toString()
Overrides:
toString in class Object

withConfiguration

public MapReduceDriver<K1,V1,K2,V2,K3,V3> withConfiguration(org.apache.hadoop.conf.Configuration configuration)
Parameters:
configuration - The configuration object that will given to the mapper and reducer associated with the driver
Returns:
this driver object for fluent coding

withKeyGroupingComparator

public MapReduceDriver<K1,V1,K2,V2,K3,V3> withKeyGroupingComparator(org.apache.hadoop.io.RawComparator<K2> groupingComparator)
Identical to MapReduceDriverBase.setKeyGroupingComparator(RawComparator), but with a fluent programming style

Parameters:
groupingComparator - Comparator to use in the shuffle stage for key grouping
Returns:
this

withKeyOrderComparator

public MapReduceDriver<K1,V1,K2,V2,K3,V3> withKeyOrderComparator(org.apache.hadoop.io.RawComparator<K2> orderComparator)
Identical to MapReduceDriverBase.setKeyOrderComparator(RawComparator), but with a fluent programming style

Parameters:
orderComparator - Comparator to use in the shuffle stage for key value ordering
Returns:
this

withCounter

public MapReduceDriver<K1,V1,K2,V2,K3,V3> withCounter(Enum e,
                                                      long expectedValue)
Description copied from class: TestDriver
Register expected enumeration based counter value

Overrides:
withCounter in class TestDriver<K1,V1,K3,V3>
Parameters:
e - Enumeration based counter
expectedValue - Expected value
Returns:

withCounter

public MapReduceDriver<K1,V1,K2,V2,K3,V3> withCounter(String g,
                                                      String n,
                                                      long e)
Description copied from class: TestDriver
Register expected name based counter value

Overrides:
withCounter in class TestDriver<K1,V1,K3,V3>
Parameters:
g - Counter group
n - Counter name
e - Expected value
Returns:

newMapReduceDriver

public static <K1,V1,K2,V2,K3,V3> MapReduceDriver<K1,V1,K2,V2,K3,V3> newMapReduceDriver()
Returns a new MapReduceDriver without having to specify the generic types on the right hand side of the object create statement.

Returns:
new MapReduceDriver

newMapReduceDriver

public static <K1,V1,K2,V2,K3,V3> MapReduceDriver<K1,V1,K2,V2,K3,V3> newMapReduceDriver(org.apache.hadoop.mapred.Mapper<K1,V1,K2,V2> mapper,
                                                                                        org.apache.hadoop.mapred.Reducer<K2,V2,K3,V3> reducer)
Returns a new MapReduceDriver without having to specify the generic types on the right hand side of the object create statement.

Parameters:
mapper - passed to MapReduceDriver constructor
reducer - passed to MapReduceDriver constructor
Returns:
new MapReduceDriver

newMapReduceDriver

public static <K1,V1,K2,V2,K3,V3> MapReduceDriver<K1,V1,K2,V2,K3,V3> newMapReduceDriver(org.apache.hadoop.mapred.Mapper<K1,V1,K2,V2> mapper,
                                                                                        org.apache.hadoop.mapred.Reducer<K2,V2,K3,V3> reducer,
                                                                                        org.apache.hadoop.mapred.Reducer<K2,V2,K2,V2> combiner)
Returns a new MapReduceDriver without having to specify the generic types on the right hand side of the object create statement.

Parameters:
mapper - passed to MapReduceDriver constructor
reducer - passed to MapReduceDriver constructor
combiner - passed to MapReduceDriver constructor
Returns:
new MapReduceDriver


Copyright © 2012 The Apache Software Foundation. All Rights Reserved.