Browsing by Author "Carvallo, Andrés"

Now showing 1 - 2 of 2
  • Thumbnail Image
    Item
    Evaluation Benchmarks for Spanish Sentence Representations
    (European Language Resources Association (ELRA), 2022) Araujo Vasquez, Vladimir Giovanny; Carvallo, Andrés; Soto A.; Moens M.-F.; Kundu S.; Mercer R.E.; Canete J.; Bravo-Marquez F.; Mendoza M.
    © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0.Due to the success of pre-trained language models, versions of languages other than English have been released in recent years. This fact implies the need for resources to evaluate these models. In the case of Spanish, there are few ways to systematically assess the models' quality. In this paper, we narrow the gap by building two evaluation benchmarks. Inspired by previous work (Conneau and Kiela, 2018; Chen et al., 2019), we introduce Spanish SentEval and Spanish DiscoEval, aiming to assess the capabilities of stand-alone and discourse-aware sentence representations, respectively. Our benchmarks include considerable pre-existing and newly constructed datasets that address different tasks from various domains. In addition, we evaluate and analyze the most recent pre-trained Spanish language models to exhibit their capabilities and limitations. As an example, we discover that for the case of discourse evaluation tasks, mBERT, a language model trained on multiple languages, usually provides a richer latent representation than models trained only with documents in Spanish. We hope our contribution will motivate a fairer, more comparable, and less cumbersome way to evaluate future Spanish language models.
  • Thumbnail Image
    Item
    Simulating conversations on social media with generative agent-based models
    (2025) Jeon, Min Soo; Mendoza Rocha, Marcelo; Fernández Pizarro, Miguel; Providel, Eliana; Rodríguez Bórquez, Felipe; Espina Quilodrán, Nicolás Gonzalo; Carvallo, Andrés; Abeliuk, Andrés
    Large Language Models (LLMs) can generate realistic text resembling human-produced content. However, the ability of these models to simulate conversations on social media is still less explored. To investigate the potential and limitations of simulated text in this domain, we introduce network-simulator, a system to simulate conversations on social media. First, we simulate the macro structure of a conversation using Agent-Based Modeling (ABM). The generated structure defines who interacts with whom, the type of interaction, and the agent’s stance on the topic of the conversation. Subsequently, using the simulated interaction structure, our system generates prompts conditioned on the simulation variables, producing texts that are conditioned on the parameters of the predefined structure, guiding a micro simulation process. We compare human conversations with those simulated by our system. Based on stylistic and model-based metrics, we found that our system can simulate conversations on social media in various dimensions. However, we detected differences in metrics related to the predictability of text production. Furthermore, we examine the effect of true and false framings within simulated conversations, revealing that simulated discussions surrounding false information exhibit a more negative collective sentiment bias than those based on true content.