Safety of Large Language Models: training data attribution and efficient search

The performance of Large Language Models (LLMs) is highly determined by their training data. Despite the proliferation of open-weight LLMs, both the access to LLM training data has remained limited and the scale of the data makes it all but inscrutable. Accessing and understanding this data is therefore essential for LLM’s reliability and security. This talk presents two complementary directions: scalable search and indexing of training datasets, and training data attribution—methods that estimate which sources may have influenced a model’s answer. We will discuss what these methods can and cannot reveal about the origins of model-generated information and show recent results from our work on indexing of Apertus training data.