There is my solutions to tackle the disk spaces shortage problem I described in the previous post. The core principle of the solution is to reduce the number of output records at Mapper stage; the method I used is Filter, adding a filter, which I will explain later, to decrease the output records of Mapper, which in turn significantly decrease the Mapper’s Spill records, and fundamentally decrease the disk space usages. After applying the filter, with 30,661 records. some 200MB data set as inputs, the total Spill Records is 25,471,725, and it only takes about 509MB disk spaces!

Followed Filter

And now I’m going to reveal what’s kinda Filter it looks like, and how did I accomplish that filter. The true face of the FILTER is called Followed Filter, it filters users from computing co-followed combinations if their followed number does not satisfy a certain number, called Followed Threshold.

Followed Filter is used to reduce the co-followed combinations at Mapper stage. Say we set the followed threshold to 100, meaning users who doesn’t own 100 fans(be followed by 100 other users) will be ignored during co-followed combinations computing stage(to get the actual number of the threshold we need analyze statistics of user’s followed number of our data set).

Reason

Choosing followed filter is reasonable because how many user follows is a metric of user’s popularity/famousness.

HOW

In order to accomplish it, we need:

First, counting user’s followed number among our data set, which needs a new MapReduce Job;

Second, choosing a followed threshold after analyze the statistics perspective of followed number data set got in first step;

Third, using DistrbutedCache of Hadoop to cache users who satisfy the filter to all Mappers;

Forth, adding followed filter to Mapper class, only users satisfy filter condition will be passed into co-followed combination computing phrase;

Fifth, adding co-followed filter/threshold in Reducer side if necessary.

Outcomes

Here is the Hadoop Job Summary, after applying the followed filter with followed threshold of 1000, that means only users who are followed by 1000 users will have the opportunity to co-followed combinations, compared with the Job Summary in my previous post, most all metrics have significant improvements:

 Counter Map Reduce Total Bytes Written 0 1,798,185 1,798,185 Bytes Read 203,401,876 0 203,401,876 FILE_BYTES_READ 405,219,906 52,107,486 457,327,392 HDFS_BYTES_READ 203,402,751 0 203,402,751 FILE_BYTES_WRITTEN 457,707,759 52,161,704 509,869,463 HDFS_BYTES_WRITTEN 0 1,798,185 1,798,185 Reduce input groups 0 373,680 373,680 Map output materialized bytes 52,107,522 0 52,107,522 Combine output records 22,202,756 0 22,202,756 Map input records 30,661 0 30,661 Reduce shuffle bytes 0 52,107,522 52,107,522 Physical memory (bytes) snapshot 2,646,589,440 116,408,320 2,762,997,760 Reduce output records 0 373,680 373,680 Spilled Records 22,866,351 2,605,374 25,471,725 Map output bytes 2,115,139,050 0 2,115,139,050 Total committed heap usage (bytes) 2,813,853,696 84,738,048 2,898,591,744 CPU time spent (ms) 5,766,680 11,210 5,777,890 Virtual memory (bytes) snapshot 9,600,737,280 1,375,002,624 10,975,739,904 SPLIT_RAW_BYTES 875 0 875 Map output records 117,507,725 0 117,507,725 Combine input records 137,105,107 0 137,105,107 Reduce input records 0 2,605,374 2,605,374

P.S.

Frankly Speaking, chances are I am on the wrong way to Hadoop Programming, since I’m palying Pesudo Distribution Hadoop with my personal computer, which has 4 CUPs and 4G RAM, in real Hadoop Cluster disk spaces might never be a trouble, and all the tuning work I have done may turn into meaningless efforts. Before the Followed Filter, I also did some Hadoop tuning like customed Writable class, RawComparator, block size and io.sort.mb, etc.

---EOF---