Hadoop Counters Info

 

User_defined_counters

Enum name: group name / Enum filed: counter name

e.g.)

enum Temperature {// 카운터에서 Temperature는 그룹명, MISSING, MALFORMED는 카운터명으로 사용.

    MISSING,

    MALFORMED

  }

 

public void map(LongWritable key, Text value,

        OutputCollector<Text, IntWritable> output, Reporter reporter)

        throws IOException {

        

      parser.parse(value);

      if (parser.isValidTemperature()) {

        int airTemperature = parser.getAirTemperature();

        output.collect(new Text(parser.getYear()),

            new IntWritable(airTemperature));

      } else if (parser.isMalformedTemperature()) {

        System.err.println("Ignoring possibly corrupt input: " + value);

        reporter.incrCounter(Temperature.MALFORMED, 1);

      } else if (parser.isMissingTemperature()) {

        reporter.incrCounter(Temperature.MISSING, 1);

      }

        

      // dynamic counter

      reporter.incrCounter("TemperatureQuality", parser.getQuality(), 1);

        

 }

Dynamic Counter <Instance Reporter> where it cannot define counters at the compile time.

public void incrCounter(Enum<?> arg0, long amount);

public void incrCounter(String group, String counter, long amount);

 

Properties를 통해 읽기 가능한 카운터 이름으로 나타낼 수 있다.

클래스명.properties or 클래스명_하위클래스명.properties

e.g MaxTemperatureWithCounters_Temperature.properties // properties 파일 위치는 enum과 같은 Dir

#properties 파일 내용

CounterGroupName = Air Temperature Records

MISSING.name = Missing

MALFORMED.name = Malformed

 

Counter 조회

RunningJob job = jobClient.getJob(JobID.forName(jobID));

Counters counters = job.getCounters();

Long missing = counters.getCounter(MaxTemperatureWithCounter.Temperature.MISSING);

Long total = counters.findCounter("org.apache.hadoop.mapred.Task$Counter","MAP_INPUT_RECORDS").getCounter();

 

 

http://diveintodata.org/2011/03/15/an-example-of-hadoop-mapreduce-counter/    

http://www.programcreek.com/java-api-examples/index.php?api=org.apache.hadoop.mapreduce.Counter

http://stackoverflow.com/questions/13609028/is-there-a-list-of-all-standard-not-custom-hadoop-mapreduce-counters

http://develop.sunshiny.co.kr/899