User_defined_counters
Enum name: group name / Enum filed: counter name
e.g.)
enum Temperature {// 카운터에서 Temperature는 그룹명, MISSING, MALFORMED는 카운터명으로 사용.
MISSING,
MALFORMED
}
public void map(LongWritable key, Text value,
OutputCollector<Text, IntWritable> output, Reporter reporter)
throws IOException {
parser.parse(value);
if (parser.isValidTemperature()) {
int airTemperature = parser.getAirTemperature();
output.collect(new Text(parser.getYear()),
new IntWritable(airTemperature));
} else if (parser.isMalformedTemperature()) {
System.err.println("Ignoring possibly corrupt input: " + value);
reporter.incrCounter(Temperature.MALFORMED, 1);
} else if (parser.isMissingTemperature()) {
reporter.incrCounter(Temperature.MISSING, 1);
}
// dynamic counter
reporter.incrCounter("TemperatureQuality", parser.getQuality(), 1);
}
Dynamic Counter <Instance Reporter> where it cannot define counters at the compile time.
public void incrCounter(Enum<?> arg0, long amount);
public void incrCounter(String group, String counter, long amount);
Properties를 통해 읽기 가능한 카운터 이름으로 나타낼 수 있다.
클래스명.properties or 클래스명_하위클래스명.properties
e.g MaxTemperatureWithCounters_Temperature.properties // properties 파일 위치는 enum과 같은 Dir
#properties 파일 내용
CounterGroupName = Air Temperature Records
MISSING.name = Missing
MALFORMED.name = Malformed
Counter 조회
RunningJob job = jobClient.getJob(JobID.forName(jobID));
Counters counters = job.getCounters();
Long missing = counters.getCounter(MaxTemperatureWithCounter.Temperature.MISSING);
Long total = counters.findCounter("org.apache.hadoop.mapred.Task$Counter","MAP_INPUT_RECORDS").getCounter();
http://diveintodata.org/2011/03/15/an-example-of-hadoop-mapreduce-counter/
http://www.programcreek.com/java-api-examples/index.php?api=org.apache.hadoop.mapreduce.Counter
http://stackoverflow.com/questions/13609028/is-there-a-list-of-all-standard-not-custom-hadoop-mapreduce-counters
http://develop.sunshiny.co.kr/899
'Distributed System Information' 카테고리의 다른 글
How to decide the number of reducers (0) | 2015.04.17 |
---|---|
Hadoop Compile (0) | 2015.02.10 |
Optimizing Top N on Wordcount example (0) | 2015.01.11 |
paper's latex file (0) | 2014.12.22 |
Inspirations from Master’s Defense (0) | 2014.12.11 |