Scientific Methodology

Monday 10:00-12:00 CC51

The course will cover issues on Scientific methodology both in Sciences in general and in Computer Science in particular. The statistical tests and the qualitative research parts will place more emphasis on the tests and qualitative techniques that are more frequently used on CS research.

But discussions on bibliographic search, on the quality of publications, on the limits of statistical tests, on the problems of reproducibility are within the context of Sciences in general.

Classes will start 26/2

The course will be in principle in English, but it all students agree, we will switch to Portuguese for the classes. Homeworks will remain in English.

The class for 5/2 (bibliographic search) will be in the amphitheater and it will be in Portuguese. It will be an open class also for all interested students.

Exercises

Exercise 1 - due 26/3 21:00 via email my solution

Exercise 2 - due 4/4 at 21:00 via email my solution

Exercise 3 - due 25/4 at 21:00 via email

Exercise 4 - due 21/5 at 21:00 via email

Class material

Publication ranks in CS and bibliographic search (in Portuguese)

Statistical analisys (in Portuguese)

Standard paired and non paired statistical significance tests

• Article showing that computer execution within Architecture are not normally distributed and to use standard parametric tests one should have over 160 data points (as opposed to the usual 30)
• The reference to an article that states that one should always use Wilcoxon tests instead of t-tests because the loss of power is not that large

Problems with p-values

Multiple groups/multiple comparisons

Equivalence tests and effect size

Some references on Bayesian estimation

Experiments, qualitative research and questionnaires

• A book chapter on all of these topics
• A book chapter on experiments
• A book chapter on action research
• A sequence of papers on questionnaires (in Software Engineering) 1 2 3 4 5

Topics

The course will cover the following topics:

• Hierarchy of publication quality with emphasis on Computer Science
• Bibliographic search
• Statistical tests - standard null hypothesis significance tests
• Modern techniques in statistical analysys - effect size and confidence intervals
• Bayesian analysis
• Fatorial designs
• Experimental designs
• Topics in experimental design and tests for some selected subareas of Computer Science such as machine learning, software engineering, human-machine interface, hardware design, and so on (according to the areas of research of the students in class)
• Quantitative experiments with people - ethical and experimental design issues
• Qualitative research (if there are students whose research require it).
• Reproducibility

The course has a practical emphasis. I will not write down any of the formulas for the different tests, just provide the name, the pressupositions of the tests, and the name of the R function that implement them.

Bibliographic references

Papers and notes on most of the topics will be made avaliable before the classes

Grading criteria

Weigthed average of classroom participation, exercises, and student presentation about his or her own research methods.

Exercises will be preferentially done in R