There is my solutions to tackle the disk spaces shortage problem I described in the previous post. The core principle of the solution is to reduce the number of output records at Mapper stage; the method I used is Filter, adding a filter, which I will explain later, to decrease the output records of Mapper, which in turn significantly decrease the Mapper’s Spill records, and fundamentally decrease the disk space usages. After applying the filter, with 30,661 records. some 200MB data set as inputs, the total Spill Records is 25,471,725, and it only takes about 509MB disk spaces!

Followed Filter

And now I’m going to reveal what’s kinda Filter it looks like, and how did I accomplish that filter. The true face of the FILTER is called Followed Filter, it filters users from computing co-followed combinations if their followed number does not satisfy a certain number, called Followed Threshold.

Followed Filter is used to reduce the co-followed combinations at Mapper stage. Say we set the followed threshold to 100, meaning users who doesn’t own 100 fans(be followed by 100 other users) will be ignored during co-followed combinations computing stage(to get the actual number of the threshold we need analyze statistics of user’s followed number of our data set).


Choosing followed filter is reasonable because how many user follows is a metric of user’s popularity/famousness.


In order to accomplish it, we need:

First, counting user’s followed number among our data set, which needs a new MapReduce Job;

Second, choosing a followed threshold after analyze the statistics perspective of followed number data set got in first step;

Third, using DistrbutedCache of Hadoop to cache users who satisfy the filter to all Mappers;

Forth, adding followed filter to Mapper class, only users satisfy filter condition will be passed into co-followed combination computing phrase;

Fifth, adding co-followed filter/threshold in Reducer side if necessary.


Here is the Hadoop Job Summary, after applying the followed filter with followed threshold of 1000, that means only users who are followed by 1000 users will have the opportunity to co-followed combinations, compared with the Job Summary in my previous post, most all metrics have significant improvements:

Counter Map Reduce Total
Bytes Written 0 1,798,185 1,798,185
Bytes Read 203,401,876 0 203,401,876
FILE_BYTES_READ 405,219,906 52,107,486 457,327,392
HDFS_BYTES_READ 203,402,751 0 203,402,751
FILE_BYTES_WRITTEN 457,707,759 52,161,704 509,869,463
HDFS_BYTES_WRITTEN 0 1,798,185 1,798,185
Reduce input groups 0 373,680 373,680
Map output materialized bytes 52,107,522 0 52,107,522
Combine output records 22,202,756 0 22,202,756
Map input records 30,661 0 30,661
Reduce shuffle bytes 0 52,107,522 52,107,522
Physical memory (bytes) snapshot 2,646,589,440 116,408,320 2,762,997,760
Reduce output records 0 373,680 373,680
Spilled Records 22,866,351 2,605,374 25,471,725
Map output bytes 2,115,139,050 0 2,115,139,050
Total committed heap usage (bytes) 2,813,853,696 84,738,048 2,898,591,744
CPU time spent (ms) 5,766,680 11,210 5,777,890
Virtual memory (bytes) snapshot 9,600,737,280 1,375,002,624 10,975,739,904
Map output records 117,507,725 0 117,507,725
Combine input records 137,105,107 0 137,105,107
Reduce input records 0 2,605,374 2,605,374


Frankly Speaking, chances are I am on the wrong way to Hadoop Programming, since I’m palying Pesudo Distribution Hadoop with my personal computer, which has 4 CUPs and 4G RAM, in real Hadoop Cluster disk spaces might never be a trouble, and all the tuning work I have done may turn into meaningless efforts. Before the Followed Filter, I also did some Hadoop tuning like customed Writable class, RawComparator, block size and io.sort.mb, etc.