Objetivo e avaliação

Pontuação: 2.0 (2 pontos equivalem a 20% da média final)

• Ambientação (0.3)
• Escolha do artigo (0.3)
• Implementação (0.7)
• Apresentação (0.7)

Exemplos:

• Efficient Parallel Set-Similarity Joins Using MapReduce, Rares Vernica, Michael J. Carey, Chen Li, Proceedings of the 2010 ACM SIGMOD International Conference on Management of data. Veja os slides.
• Algoritmos em grafos
  • Introdução
  • Slides: MapReduce algorithm design e Mapreduce tutorial (a partir da página 47)
  • A MapReduce-based approach for shortest path problem in large-scale networks, Journal Engineering Applications of Artificial Intelligence, 2015
• Rapid parallel genome indexing with MapReduce, Proceedings of the second international workshop on MapReduce and its applications, 2011

Ambientação

Instalação do Hadoop

• Standalone
• Pseudo-Distributed
• Fully-Distributed

Principais passos:

• Verifique a configuração do ssh

    $ ssh localhost
  

  Se não conseguir logar sem digitar password:

    $ ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa
    $ cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
  

• Faça o download da versão 2.7.1
• Siga as orientações para a instalação. Instale no diretório temporário das máquinas do IC-03.
• Verifique o funcionamento do HDFS em http://localhost:50070/
• Verifique o funcionamento do YARN em http://localhost:8088/
• Rode um dos exemplos prontos de MapReduce.

MapReduce

WordCount

A partir deste passo, você poderá alterar este teste, para implementar o seu próprio código mapreduce. Por exemplo, você pode escrever uma variação que conta as palavras de mesmo tamanho ou as palavras que começam com uma dada letra (veja Developing First MapReduce Job).

Outros exemplos

• wordcount: A map/reduce program that counts the words in the input files.
• wordmean: A map/reduce program that counts the average length of the words in the input files.
• wordmedian A map/reduce program that counts the median length of the words in the input files.
• wordstandarddeviation: A map/reduce program that counts the standard deviation of the length of the words in the input files.
• aggregatewordcount: An Aggregate based map/reduce program that counts the words in the input files.
• aggregatewordhist: An Aggregate based map/reduce program that computes the histogram of the words in the input files.
• grep: A map/reduce program that counts the matches of a regex in the input.
• randomwriter: A map/reduce program that writes 10GB of random data per node.
• randomtextwriter A map/reduce program that writes 10GB of random textual data per node.
• sort: A map/reduce program that sorts the data written by the random writer.
• pi: A map/reduce program that estimates the value of Pi using a quasi-Monte Carlo (qMC) method.
• bbp: A map/reduce program that uses Bailey-Borwein-Plouffe to compute exact digits of Pi.
• distbbp: A map/reduce program that uses a BBP-type method to compute exact binary digits of Pi. This program is designed for computing the n th bit of Pi, for large n, say n >= 10^8. For computing lower bits of Pi, consider using bbp.
• pentomino: A map/reduce tile laying program to find solutions to pentomino problems.
• secondarysort: An example defining a secondary sort to the reduce.
• sudoku: A sudoku solver.
• join: A job that effects a join over sorted, equally partitioned datasets.
• multifilewc: A job that counts words from several files.
• dbcount: An example job that count the pageview counts from a database.
• teragen: Generate data for the terasort.
• terasort: Run the terasort.
• teravalidate: Checking results of terasort