Experimento com HDFS, YARN e MapReduce

Instalação do Hadoop

• Standalone
• Pseudo-Distributed
• Fully-Distributed

Principais passos:

• Para melhor desempenho e para não ter problemas com cota: Trabalhe no diretório temporário das máquinas do IC-03.
• Verifique a configuração do ssh

    $ ssh localhost
  

  Se uma password for solicitada:

    $ ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa
    $ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
    $ chmod 0600 ~/.ssh/authorized_keys
  

  Importante: utilize rsa e não dsa como sugerido no site do Hadoop.
• Faça o download da versão estável (atualmente 2.7.4)

    $ wget http://ftp.unicamp.br/pub/apache/hadoop/common/hadoop-2.7.4/hadoop-2.7.4.tar.gz
  

• Descompacte com

    $ tar xzf hadoop-2.7.4.tar.gz    
  

• Siga as orientações para a instalação do site do Hadoop.
• Verifique o funcionamento do HDFS em http://localhost:50070/
• Verifique o funcionamento do YARN em http://localhost:8088/
• Rode um dos exemplos prontos de mapreduce.

MapReduce

Para um exercício um pouco mais difícil implemente o índice invertido ou a recomendação de amigos.

Se você gosta de JAVA...

Exemplos do Hadoop Mapreduce

• wordcount: A map/reduce program that counts the words in the input files.
• wordmean: A map/reduce program that counts the average length of the words in the input files.
• wordmedian A map/reduce program that counts the median length of the words in the input files.
• wordstandarddeviation: A map/reduce program that counts the standard deviation of the length of the words in the input files.
• aggregatewordcount: An Aggregate based map/reduce program that counts the words in the input files.
• aggregatewordhist: An Aggregate based map/reduce program that computes the histogram of the words in the input files.
• grep: A map/reduce program that counts the matches of a regex in the input.
• randomwriter: A map/reduce program that writes 10GB of random data per node.
• randomtextwriter A map/reduce program that writes 10GB of random textual data per node.
• sort: A map/reduce program that sorts the data written by the random writer.
• pi: A map/reduce program that estimates the value of Pi using a quasi-Monte Carlo (qMC) method.
• bbp: A map/reduce program that uses Bailey-Borwein-Plouffe to compute exact digits of Pi.
• distbbp: A map/reduce program that uses a BBP-type method to compute exact binary digits of Pi. This program is designed for computing the n th bit of Pi, for large n, say n >= 10^8. For computing lower bits of Pi, consider using bbp.
• pentomino: A map/reduce tile laying program to find solutions to pentomino problems.
• secondarysort: An example defining a secondary sort to the reduce.
• sudoku: A sudoku solver.
• join: A job that effects a join over sorted, equally partitioned datasets.
• multifilewc: A job that counts words from several files.
• dbcount: An example job that count the pageview counts from a database.
• teragen: Generate data for the terasort.
• terasort: Run the terasort.
• teravalidate: Checking results of terasort

Se você prefere Python...

$ bin/hadoop jar share/hadoop/tools/lib/hadoop-streaming-2.7.4.jar -mapper mapper.py -reducer reducer.py -input /input -output /output