Croissant: Metadata for Machine Learning Systems

TY - GEN

T1 - Croissant: Metadata for Machine Learning Systems

AU - Tykhonov, Vyacheslav

AU - Miguelez, Joan Giner

PY - 2024/10/12

Y1 - 2024/10/12

N2 - Data is vital for machine learning (ML), yet managing it remains a challenge. We present Croissant, a metadata format that standardizes dataset representation across ML tools, frameworks, and platforms. Croissant enhances dataset discoverability, portability, and interoperability, already supporting hundreds of thousands of datasets in popular repositories. It allows seamless integration with widely-used ML frameworks regardless of data storage location. Human evaluations confirm Croissant's metadata as readable, concise, and complete. The vision is a shared Data Lake enabling federated search across platforms like Dataverse, Kaggle, and HuggingFace. A centralized approach focuses on standardization and repository-level harmonization, while a distributed approachadvocates agile, Linked Data-based solutions that empower diverse communities to integrate within a Distributed Data Network using Croissant ML and AI technologies.

AB - Data is vital for machine learning (ML), yet managing it remains a challenge. We present Croissant, a metadata format that standardizes dataset representation across ML tools, frameworks, and platforms. Croissant enhances dataset discoverability, portability, and interoperability, already supporting hundreds of thousands of datasets in popular repositories. It allows seamless integration with widely-used ML frameworks regardless of data storage location. Human evaluations confirm Croissant's metadata as readable, concise, and complete. The vision is a shared Data Lake enabling federated search across platforms like Dataverse, Kaggle, and HuggingFace. A centralized approach focuses on standardization and repository-level harmonization, while a distributed approachadvocates agile, Linked Data-based solutions that empower diverse communities to integrate within a Distributed Data Network using Croissant ML and AI technologies.

U2 - 10.5281/zenodo.14620784

DO - 10.5281/zenodo.14620784

M3 - Other contribution

PB - Schloss Dagstuhl - Leibniz-Zentrum für Informatik

ER -