23set2026
09:00 Doctoral defense IC3 Auditorium
Topic on
Selective Learning: Investigating the Impact of Data Quality on Portuguese Continued Pretraining
Student
Thales Rogério Sales Almeida
Advisor / Teacher
Hélio Pedrini - Co-advisor: Rodrigo Frassetto Nogueira
Brief summary
Many studies document differences in the performance of large-scale language models depending on the language and regional context in which they are evaluated. In response, a growing body of work seeks to improve these models by continuing their pre-training on data that are underrepresented in the original corpus, ranging from texts in a specific language to particular domains. This thesis examines the role of data selection in this scenario and shows that, when adapting a model for a language with low exposure, the quality and composition of the adaptation data matter more than its volume, and this effect intensifies as the scale of the model increases. We established these results using Portuguese as a case study, although the questions, methods, and principles are intended to be generalized to other underrepresented languages. The work unfolds in three stages: measuring the language gap, building the necessary data and assessment infrastructure to study it, and conducting controlled experiments of continuous pre-training. To characterize the problem, we present the Timely Events Benchmark (TiEBe), with over 23.000 question-and-answer pairs covering 23 regions and 13 languages, which shows a strong correlation between factual retrieval and regional socioeconomic indicators. To enable controlled experiments, we built ClassiCC-PT, a large-scale Portuguese corpus extracted from Common Crawl through a documented and reproducible pipeline, whose core is a set of language-specific content classifiers that substantially outperform their counterparts trained only in English. We developed PoETa v2, the most comprehensive benchmark for evaluating language models in Portuguese to date, with 44 tasks, 12 of which were developed natively. It was used in the largest systematic evaluation of models in Portuguese ever conducted and offers the category-specific resolution needed to identify where interventions in the data have an effect. Based on these principles, we conducted our core experiments through continuous pre-training of checkpoints in English, producing the Curió family of models. The main result is that data quality can outweigh quantity, provided the model scales sufficiently: a model trained with a small set of tokens selected per classifier outperforms another trained with an order of magnitude more unfiltered tokens across all evaluation categories, consuming only a fraction of the computation. We also show that synthetic rewriting acts as a quality multiplier rather than an equalizer, amplifying high-quality data much more than low-quality data. The combined strategy produces the strongest models trained in this work. Two principles are repeated. Data quality consistently outweighs quantity, and the return on quality-driven curation grows with the scale of the model: interventions that reconfigure the results in 7B parameters leave the 1,1B models virtually unchanged. Ongoing pre-training proves to be an efficient path to language adaptation, as it transfers general English proficiency and requires only a modest budget of well-chosen tokens in the target language. All datasets, classifiers, models, and benchmarks are publicly available. Together, they offer a practical recipe and a set of empirical principles for adapting language models to underrepresented languages, with Portuguese as a demonstration case.
Examination Board
Headlines:
| Hélio Pedrini | IC / UNICAMP |
| Aline Marins Paes Carvalho | IC / UFF |
| Helena de Medeiros Caseli | CCET/UFSCar |
| Marcos Medeiros Raimundo | IC / UNICAMP |
| Marcelo da Silva Reis | IC / UNICAMP |
Substitutes:
| Anderson de Rezende Rocha | IC / UNICAMP |
| Thiago Alexandre Salgueiro Pardo | ICMC / USP |
| Ronaldo Cristiano Prati | CMCC / UFABC |