public int getPartition(IntWritable key, Chromosome value, int numOfPartitions)
{
int partition = key.get();
if (partition < 0 || partition >= numOfPartitions)
{
partition = numOfPartitions-1;
}
System.out.println("partition "+partition );
return partition;
}
I wrote the custom partitioner above. But the problem is about the third parameter, numOfPartitions.
It is always "1" in pseudo-distributed mode. I have 4 mappers and 4 reducers, but only one of the reducers uses the real values. The others yield nothing, just empty files.
When I remove the if statement, hadoop complains about the partition number as "illegal partition for ...".
How can i set the number of partitions in pseudo-distributed mode?
Locked
