org.apache.hadoop.mrunit.mapreduce

Class MultipleInputsMapReduceDriver<K1,V1,K2,V2>

java.lang.Object

org.apache.hadoop.mrunit.TestDriver<K2,V2,T>

org.apache.hadoop.mrunit.internal.driver.MultipleInputsMapReduceDriverBase<org.apache.hadoop.mapreduce.Mapper,K1,V1,K2,V2,MultipleInputsMapReduceDriver<K1,V1,K2,V2>>

org.apache.hadoop.mrunit.mapreduce.MultipleInputsMapReduceDriver<K1,V1,K2,V2>

Type Parameters:

K1 - The common map output key type

V1 - The common map output value type

K2 - The reduce output key type

V2 - The reduce output value type

public class MultipleInputsMapReduceDriver<K1,V1,K2,V2>
extends org.apache.hadoop.mrunit.internal.driver.MultipleInputsMapReduceDriverBase<org.apache.hadoop.mapreduce.Mapper,K1,V1,K2,V2,MultipleInputsMapReduceDriver<K1,V1,K2,V2>>

Harness that allows you to test multiple Mappers and a Reducer instance together (along with an optional combiner). You provide the input keys and values that should be sent to each Mapper, and outputs you expect to be sent by the Reducer to the collector for those inputs. By calling runTest(), the harness will deliver the inputs to the respective Mappers, feed the intermediate results to the Reducer (without checking them), and will check the Reducer's outputs against the expected results. If a combiner is specified, it will run exactly once after all the Mappers and before the Reducer

Field Summary

Fields

Modifier and Type	Field and Description
static org.apache.commons.logging.Log	LOG

Fields inherited from class org.apache.hadoop.mrunit.internal.driver.MultipleInputsMapReduceDriverBase

inputs, keyGroupComparator, keyValueOrderComparator, mapInputPaths

Fields inherited from class org.apache.hadoop.mrunit.TestDriver

counterWrapper, expectedEnumCounters, expectedMultipleOutputs, expectedOutputs, expectedPathOutputs, expectedStringCounters, mos

Constructor Summary

Constructors

Constructor and Description
MultipleInputsMapReduceDriver() Construct a driver without specifying a Combiner nor a Reducer.
MultipleInputsMapReduceDriver(org.apache.hadoop.mapreduce.Reducer<K1,V1,K1,V1> combiner, org.apache.hadoop.mapreduce.Reducer<K1,V1,K2,V2> reducer) Construct a driver with the specified Combiner and Reducers
MultipleInputsMapReduceDriver(org.apache.hadoop.mapreduce.Reducer<K1,V1,K2,V2> reducer) Construct a driver with the specified Reducer.

Method Summary

Methods

Modifier and Type	Method and Description
<K,V> void	addAll(org.apache.hadoop.mapreduce.Mapper<K,V,K1,V1> mapper, List<Pair<K,V>> inputs) Add the specified input pairs to the specified mapper
<K,V> void	addInput(org.apache.hadoop.mapreduce.Mapper<K,V,K1,V1> mapper, K key, V val) Add the specified (key, val) pair to the specified mapper
<K,V> void	addInput(org.apache.hadoop.mapreduce.Mapper<K,V,K1,V1> mapper, Pair<K,V> input) Add the specified input pair to the specified mapper
<K,V> void	addMapper(org.apache.hadoop.mapreduce.Mapper<K,V,K1,V1> mapper) Add a mapper to use with this test driver
org.apache.hadoop.mapreduce.Reducer<K1,V1,K1,V1>	getCombiner()
org.apache.hadoop.mapreduce.Counters	getCounters()
Collection<org.apache.hadoop.mapreduce.Mapper>	getMappers()
org.apache.hadoop.mapreduce.Reducer<K1,V1,K2,V2>	getReducer()
static <K1,V1,K2,V2> MultipleInputsMapReduceDriver<K1,V1,K2,V2>	newMultipleInputMapReduceDriver() Static factory-style method to construct a driver instance without specifying a Combiner nor a Reducer.
static <K1,V1,K2,V2> MultipleInputsMapReduceDriver<K1,V1,K2,V2>	newMultipleInputMapReduceDriver(org.apache.hadoop.mapreduce.Reducer<K1,V1,K1,V1> combiner, org.apache.hadoop.mapreduce.Reducer<K1,V1,K2,V2> reducer) Static factory-style method to construct a driver instance with the specified Combiner and Reducer
static <K1,V1,K2,V2> MultipleInputsMapReduceDriver<K1,V1,K2,V2>	newMultipleInputMapReduceDriver(org.apache.hadoop.mapreduce.Reducer<K1,V1,K2,V2> reducer) Static factory-style method to construct a driver instance with the specified Reducer
protected void	preRunChecks(Set<org.apache.hadoop.mapreduce.Mapper> mappers, Object reducer)
List<Pair<K2,V2>>	run() Runs the test but returns the result set instead of validating it (ignores any addOutput(), etc calls made before this)
void	setCombiner(org.apache.hadoop.mapreduce.Reducer<K1,V1,K1,V1> combiner) Set the combiner to use with this test driver
void	setCounters(org.apache.hadoop.mapreduce.Counters counters) Sets the counters object to use for this test
void	setReducer(org.apache.hadoop.mapreduce.Reducer<K1,V1,K2,V2> reducer) Set the reducer to use with this test driver
protected List<KeyValueReuseList<K1,V1>>	sortAndGroup(List<Pair<K1,V1>> mapOutputs)
<K,V> MultipleInputsMapReduceDriver<K1,V1,K2,V2>	withAll(org.apache.hadoop.mapreduce.Mapper<K,V,K1,V1> mapper, List<Pair<K,V>> inputs) Identical to addInput but supports fluent programming style
MultipleInputsMapReduceDriver<K1,V1,K2,V2>	withCombiner(org.apache.hadoop.mapreduce.Reducer<K1,V1,K1,V1> combiner) Identical to setCombiner but supports a fluent programming style
MultipleInputsMapReduceDriver<K1,V1,K2,V2>	withCounter(org.apache.hadoop.mapreduce.Counters counters) Identical to setCounters but supports a fluent programming style
<K,V> MultipleInputsMapReduceDriver<K1,V1,K2,V2>	withInput(org.apache.hadoop.mapreduce.Mapper<K,V,K1,V1> mapper, K key, V val) Identical to addInput but supports fluent programming style
<K,V> MultipleInputsMapReduceDriver<K1,V1,K2,V2>	withInput(org.apache.hadoop.mapreduce.Mapper<K,V,K1,V1> mapper, Pair<K,V> input) Identical to addInput but supports fluent programming style
MultipleInputsMapReduceDriver<K1,V1,K2,V2>	withInputFormat(Class<? extends org.apache.hadoop.mapreduce.InputFormat> inputFormatClass) Set the InputFormat
<K,V> MultipleInputsMapReduceDriver<K1,V1,K2,V2>	withMapper(org.apache.hadoop.mapreduce.Mapper<K,V,K1,V1> mapper) Identical to addMapper but supports a fluent programming style
MultipleInputsMapReduceDriver<K1,V1,K2,V2>	withOutputFormat(Class<? extends org.apache.hadoop.mapreduce.OutputFormat> outputFormatClass) Configure Reducer to output with a real OutputFormat.
MultipleInputsMapReduceDriver<K1,V1,K2,V2>	withReducer(org.apache.hadoop.mapreduce.Reducer<K1,V1,K2,V2> reducer) Identical to setReducer but supports a fluent programming style

Methods inherited from class org.apache.hadoop.mrunit.internal.driver.MultipleInputsMapReduceDriverBase

addAll, addInput, addInput, getMapInputPath, setKeyGroupingComparator, setKeyOrderComparator, setMapInputPath, withAll, withInput, withInput, withKeyGroupingComparator, withKeyOrderComparator, withMapInputPath

Methods inherited from class org.apache.hadoop.mrunit.TestDriver

addAllOutput, addCacheArchive, addCacheArchive, addCacheFile, addCacheFile, addMultiOutput, addMultiOutput, addOutput, addOutput, addOutputFromString, cleanupDistributedCache, copy, copyPair, driverReused, formatPairList, formatValueList, getConfiguration, getExpectedEnumCounters, getExpectedOutputs, getExpectedStringCounters, getKeyComparator, getOutputSerializationConfiguration, getValueComparator, initDistributedCache, parseCommaDelimitedList, parseTabbedPair, printPreTestDebugLog, resetExpectedCounters, resetOutput, run, runTest, runTest, setCacheArchives, setCacheFiles, setConfiguration, setKeyComparator, setOutputSerializationConfiguration, setUsedOnceStatus, setValueComparator, thisAsTestDriver, validate, validate, validate, validateOutputList, withAllOutput, withCacheArchive, withCacheArchive, withCacheFile, withCacheFile, withConfiguration, withCounter, withCounter, withMultiOutput, withMultiOutput, withOutput, withOutput, withOutputFromString, withOutputSerializationConfiguration, withPathOutput, withPathOutput, withStrictCounterChecking

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

LOG

public static final org.apache.commons.logging.Log LOG

Constructor Detail

MultipleInputsMapReduceDriver

public MultipleInputsMapReduceDriver(org.apache.hadoop.mapreduce.Reducer<K1,V1,K2,V2> reducer)

Construct a driver with the specified Reducer. Note that a Combiner can be set separately.

Parameters:

reducer - The reducer to use

MultipleInputsMapReduceDriver

public MultipleInputsMapReduceDriver(org.apache.hadoop.mapreduce.Reducer<K1,V1,K1,V1> combiner,
                             org.apache.hadoop.mapreduce.Reducer<K1,V1,K2,V2> reducer)

Construct a driver with the specified Combiner and Reducers

Parameters:

combiner - The combiner to use

reducer - The reducer to use

MultipleInputsMapReduceDriver

public MultipleInputsMapReduceDriver()

Construct a driver without specifying a Combiner nor a Reducer. Note that these can be set with the appropriate set methods and that at least the Reducer must be set.

Method Detail

addMapper

public <K,V> void addMapper(org.apache.hadoop.mapreduce.Mapper<K,V,K1,V1> mapper)

Add a mapper to use with this test driver

Type Parameters:

K - The input key type to the mapper

V - The input value type to the mapper

Parameters:

mapper - The mapper instance to add

withMapper

public <K,V> MultipleInputsMapReduceDriver<K1,V1,K2,V2> withMapper(org.apache.hadoop.mapreduce.Mapper<K,V,K1,V1> mapper)

Identical to addMapper but supports a fluent programming style

Type Parameters:

K - The input key type to the mapper

V - The input value type to the mapper

Parameters:

mapper - The mapper instance to add

Returns:

this

getMappers

public Collection<org.apache.hadoop.mapreduce.Mapper> getMappers()

Returns:

The Mapper instances being used by this test

setCombiner

public void setCombiner(org.apache.hadoop.mapreduce.Reducer<K1,V1,K1,V1> combiner)

Set the combiner to use with this test driver

Parameters:

combiner - The combiner instance to use

withCombiner

public MultipleInputsMapReduceDriver<K1,V1,K2,V2> withCombiner(org.apache.hadoop.mapreduce.Reducer<K1,V1,K1,V1> combiner)

Identical to setCombiner but supports a fluent programming style

Parameters:

combiner - The combiner instance to use

Returns:

this

getCombiner

public org.apache.hadoop.mapreduce.Reducer<K1,V1,K1,V1> getCombiner()

Returns:

The combiner instance being used by this test

setReducer

public void setReducer(org.apache.hadoop.mapreduce.Reducer<K1,V1,K2,V2> reducer)

Set the reducer to use with this test driver

Parameters:

reducer - The reducer instance to use

withReducer

public MultipleInputsMapReduceDriver<K1,V1,K2,V2> withReducer(org.apache.hadoop.mapreduce.Reducer<K1,V1,K2,V2> reducer)

Identical to setReducer but supports a fluent programming style

Parameters:

reducer - The reducer instance to use

Returns:

this

getReducer

public org.apache.hadoop.mapreduce.Reducer<K1,V1,K2,V2> getReducer()

Returns:

Get the reducer instance being used by this test

getCounters

public org.apache.hadoop.mapreduce.Counters getCounters()

Returns:

The counters used in this test

setCounters

public void setCounters(org.apache.hadoop.mapreduce.Counters counters)

Sets the counters object to use for this test

Parameters:

counters - The counters object to use

withCounter

public MultipleInputsMapReduceDriver<K1,V1,K2,V2> withCounter(org.apache.hadoop.mapreduce.Counters counters)

Identical to setCounters but supports a fluent programming style

Parameters:

counters - The counters object to use

Returns:

this

withOutputFormat

public MultipleInputsMapReduceDriver<K1,V1,K2,V2> withOutputFormat(Class<? extends org.apache.hadoop.mapreduce.OutputFormat> outputFormatClass)

Configure Reducer to output with a real OutputFormat.

Parameters:

outputFormatClass - The OutputFormat class

Returns:

this

withInputFormat

public MultipleInputsMapReduceDriver<K1,V1,K2,V2> withInputFormat(Class<? extends org.apache.hadoop.mapreduce.InputFormat> inputFormatClass)

Set the InputFormat

Parameters:

inputFormatClass - The InputFormat class

Returns:

this

newMultipleInputMapReduceDriver

public static <K1,V1,K2,V2> MultipleInputsMapReduceDriver<K1,V1,K2,V2> newMultipleInputMapReduceDriver(org.apache.hadoop.mapreduce.Reducer<K1,V1,K1,V1> combiner,
                                                                                       org.apache.hadoop.mapreduce.Reducer<K1,V1,K2,V2> reducer)

Static factory-style method to construct a driver instance with the specified Combiner and Reducer

Type Parameters:

K1 - The common output key type of the mappers

V1 - The common output value type of the mappers

K2 - The output key type of the reducer

V2 - The output value type of the reducer

Parameters:

combiner - The combiner to use

reducer - The reducer to use

Returns:

this to support fluent programming style

newMultipleInputMapReduceDriver

public static <K1,V1,K2,V2> MultipleInputsMapReduceDriver<K1,V1,K2,V2> newMultipleInputMapReduceDriver(org.apache.hadoop.mapreduce.Reducer<K1,V1,K2,V2> reducer)

Static factory-style method to construct a driver instance with the specified Reducer

Type Parameters:

K1 - The common output key type of the mappers

V1 - The common output value type of the mappers

K2 - The output key type of the reducer

V2 - The output value type of the reducer

Parameters:

reducer - The reducer to use

Returns:

this to support fluent programming style

newMultipleInputMapReduceDriver

public static <K1,V1,K2,V2> MultipleInputsMapReduceDriver<K1,V1,K2,V2> newMultipleInputMapReduceDriver()

Static factory-style method to construct a driver instance without specifying a Combiner nor a Reducer. Note that these can be set separately by using the appropriate set (or with) methods and that at least a Reducer must be set

Type Parameters:

K1 - The common output key type of the mappers

V1 - The common output value type of the mappers

K2 - The output key type of the reducer

V2 - The output value type of the reducer

Returns:

this to support fluent programming style

addInput

public <K,V> void addInput(org.apache.hadoop.mapreduce.Mapper<K,V,K1,V1> mapper,
                  K key,
                  V val)

Add the specified (key, val) pair to the specified mapper

Type Parameters:

K - The type of the key

V - The type of the value

Parameters:

mapper - The mapper to add the input pair to

key - The key

val - The value

addInput

public <K,V> void addInput(org.apache.hadoop.mapreduce.Mapper<K,V,K1,V1> mapper,
                  Pair<K,V> input)

Add the specified input pair to the specified mapper

Type Parameters:

K - The type of the key

V - The type of the value

Parameters:

mapper - The mapper to add the input pair to

input - The (k,v) pair to add

addAll

public <K,V> void addAll(org.apache.hadoop.mapreduce.Mapper<K,V,K1,V1> mapper,
                List<Pair<K,V>> inputs)

Add the specified input pairs to the specified mapper

Type Parameters:

K - The type of the key

V - The type of the value

Parameters:

mapper - The mapper to add the input pairs to

inputs - The (k, v) pairs to add

withInput

public <K,V> MultipleInputsMapReduceDriver<K1,V1,K2,V2> withInput(org.apache.hadoop.mapreduce.Mapper<K,V,K1,V1> mapper,
                                                         K key,
                                                         V val)

Identical to addInput but supports fluent programming style

Type Parameters:

K - The type of the key

V - The type of the value

Parameters:

mapper - The mapper to add the input pair to

key - The key

val - The value

Returns:

this

withInput

public <K,V> MultipleInputsMapReduceDriver<K1,V1,K2,V2> withInput(org.apache.hadoop.mapreduce.Mapper<K,V,K1,V1> mapper,
                                                         Pair<K,V> input)

Identical to addInput but supports fluent programming style

Type Parameters:

K - The type of the key

V - The type of the value

Parameters:

mapper - The mapper to add the input pairs to

inputs - The (k, v) pairs to add

Returns:

this

withAll

public <K,V> MultipleInputsMapReduceDriver<K1,V1,K2,V2> withAll(org.apache.hadoop.mapreduce.Mapper<K,V,K1,V1> mapper,
                                                       List<Pair<K,V>> inputs)

Identical to addInput but supports fluent programming style

Type Parameters:

K - The type of the key

V - The type of the value

Parameters:

mapper - The mapper to add the input pairs to

inputs - The (k, v) pairs to add

Returns:

this

preRunChecks

protected void preRunChecks(Set<org.apache.hadoop.mapreduce.Mapper> mappers,
                Object reducer)

Overrides:

preRunChecks in class org.apache.hadoop.mrunit.internal.driver.MultipleInputsMapReduceDriverBase<org.apache.hadoop.mapreduce.Mapper,K1,V1,K2,V2,MultipleInputsMapReduceDriver<K1,V1,K2,V2>>

sortAndGroup

protected List<KeyValueReuseList<K1,V1>> sortAndGroup(List<Pair<K1,V1>> mapOutputs)

run

public List<Pair<K2,V2>> run()
                      throws IOException

Description copied from class: TestDriver

Runs the test but returns the result set instead of validating it (ignores any addOutput(), etc calls made before this)

Specified by:

run in class TestDriver<K2,V2,MultipleInputsMapReduceDriver<K1,V1,K2,V2>>

Returns:

the list of (k, v) pairs returned as output from the test

Throws:

IOException