Croissant: A Metadata Format for ML-Ready Datasets

Mubashara Akthar; Omar  Benjelloun; Costanza Conforti; Luca Foschini; Pieter Gijsbers; Joan Giner Miguelez; Sujata Goswami; Nitisha Jain; Michalis Karamousadakis; Satyapriya Krishna; Michael Kuchnik; Sylvain Lesage; Quentin Lhoest; Pierre Marcenac; Manil Maskey; Peter Mattson; Luis Oala; Hamidah Oderinwale; Pierre Ruyssen; Tim Santos; Rajat Shinde; Elena Simperl; Arjun Suresh; Geoffry Thomas; Vyacheslav Tykhonov; Joaquin Vanschoren; Susheel Varma; Jos van der Velde; Steffen Vogler; Carole-Jean Wu; Luyao Zhang

Croissant: A Metadata Format for ML-Ready Datasets

Mubashara Akthar, Omar Benjelloun, Costanza Conforti, Luca Foschini, Pieter Gijsbers, Joan Giner Miguelez, Sujata Goswami, Nitisha Jain, Michalis Karamousadakis, Satyapriya Krishna, Michael Kuchnik, Sylvain Lesage, Quentin Lhoest, Pierre Marcenac, Manil Maskey, Peter Mattson, Luis Oala, Hamidah Oderinwale, Pierre Ruyssen, Tim SantosRajat Shinde, Elena Simperl, Arjun Suresh, Geoffry Thomas, Vyacheslav Tykhonov, Joaquin Vanschoren, Susheel Varma, Jos van der Velde, Steffen Vogler, Carole-Jean Wu, Luyao Zhang

DANS-KNAW

Research output: Chapter in book/volume › Contribution to conference proceedings › Scientific › peer-review

Abstract

Data is a critical resource for machine learning (ML), yet working with data remains a key friction point. This paper introduces Croissant, a metadata format for datasets that creates a shared representation across ML tools, frameworks, and platforms. Croissant makes datasets more discoverable, portable, and interoperable, thereby addressing significant challenges in ML data management. Croissant is already supported by several popular dataset repositories, spanning hundreds of thousands of datasets, enabling easy loading into the most commonly-used ML frameworks, regardless of where the data is stored. Our initial evaluation by human raters shows that Croissant metadata is readable, understandable, complete, yet concise.

Original language	English
Title of host publication	38th Conference on Neural Information Processing Systems (NeurIPS 2024)
Subtitle of host publication	Track on Datasets and Benchmarks
Place of Publication	Vancouver, Canada
Publisher	NeurIPS
Number of pages	26
Publication status	Published - 12 Dec 2024

Keywords

machine learning
ml standard
Artificial Intelligence

Access to Document

https://ai.meta.com/research/publications/croissant-a-metadata-format-for-ml-ready-datasets/

Cite this

Akthar, M., Benjelloun, O., Conforti, C., Foschini, L., Gijsbers, P., Giner Miguelez, J., Goswami, S., Jain, N., Karamousadakis, M., Krishna, S., Kuchnik, M., Lesage, S., Lhoest, Q., Marcenac, P., Maskey, M., Mattson, P., Oala, L., Oderinwale, H., Ruyssen, P., ... Zhang, L. (2024). Croissant: A Metadata Format for ML-Ready Datasets. In 38th Conference on Neural Information Processing Systems (NeurIPS 2024): Track on Datasets and Benchmarks NeurIPS. https://ai.meta.com/research/publications/croissant-a-metadata-format-for-ml-ready-datasets/

@inbook{a335101f4b1e4aaa83116e56bd7ee5fe,

title = "Croissant: A Metadata Format for ML-Ready Datasets",

abstract = "Data is a critical resource for machine learning (ML), yet working with data remains a key friction point. This paper introduces Croissant, a metadata format for datasets that creates a shared representation across ML tools, frameworks, and platforms. Croissant makes datasets more discoverable, portable, and interoperable, thereby addressing significant challenges in ML data management. Croissant is already supported by several popular dataset repositories, spanning hundreds of thousands of datasets, enabling easy loading into the most commonly-used ML frameworks, regardless of where the data is stored. Our initial evaluation by human raters shows that Croissant metadata is readable, understandable, complete, yet concise.",

keywords = "machine learning, ml standard, Artificial Intelligence",

author = "Mubashara Akthar and Omar Benjelloun and Costanza Conforti and Luca Foschini and Pieter Gijsbers and {Giner Miguelez}, Joan and Sujata Goswami and Nitisha Jain and Michalis Karamousadakis and Satyapriya Krishna and Michael Kuchnik and Sylvain Lesage and Quentin Lhoest and Pierre Marcenac and Manil Maskey and Peter Mattson and Luis Oala and Hamidah Oderinwale and Pierre Ruyssen and Tim Santos and Rajat Shinde and Elena Simperl and Arjun Suresh and Geoffry Thomas and Vyacheslav Tykhonov and Joaquin Vanschoren and Susheel Varma and {van der Velde}, Jos and Steffen Vogler and Carole-Jean Wu and Luyao Zhang",

year = "2024",

month = dec,

day = "12",

language = "English",

booktitle = "38th Conference on Neural Information Processing Systems (NeurIPS 2024)",

publisher = "NeurIPS",

}

Akthar, M, Benjelloun, O, Conforti, C, Foschini, L, Gijsbers, P, Giner Miguelez, J, Goswami, S, Jain, N, Karamousadakis, M, Krishna, S, Kuchnik, M, Lesage, S, Lhoest, Q, Marcenac, P, Maskey, M, Mattson, P, Oala, L, Oderinwale, H, Ruyssen, P, Santos, T, Shinde, R, Simperl, E, Suresh, A, Thomas, G, Tykhonov, V, Vanschoren, J, Varma, S, van der Velde, J, Vogler, S, Wu, C-J & Zhang, L 2024, Croissant: A Metadata Format for ML-Ready Datasets. in 38th Conference on Neural Information Processing Systems (NeurIPS 2024): Track on Datasets and Benchmarks. NeurIPS, Vancouver, Canada. <https://ai.meta.com/research/publications/croissant-a-metadata-format-for-ml-ready-datasets/>

Croissant: A Metadata Format for ML-Ready Datasets. / Akthar, Mubashara; Benjelloun, Omar ; Conforti, Costanza et al.
38th Conference on Neural Information Processing Systems (NeurIPS 2024): Track on Datasets and Benchmarks. Vancouver, Canada: NeurIPS, 2024.

Research output: Chapter in book/volume › Contribution to conference proceedings › Scientific › peer-review

TY - CHAP

T1 - Croissant: A Metadata Format for ML-Ready Datasets

AU - Akthar, Mubashara

AU - Benjelloun, Omar

AU - Conforti, Costanza

AU - Foschini, Luca

AU - Gijsbers, Pieter

AU - Giner Miguelez, Joan

AU - Goswami, Sujata

AU - Jain, Nitisha

AU - Karamousadakis, Michalis

AU - Krishna, Satyapriya

AU - Kuchnik, Michael

AU - Lesage, Sylvain

AU - Lhoest, Quentin

AU - Marcenac, Pierre

AU - Maskey, Manil

AU - Mattson, Peter

AU - Oala, Luis

AU - Oderinwale, Hamidah

AU - Ruyssen, Pierre

AU - Santos, Tim

AU - Shinde, Rajat

AU - Simperl, Elena

AU - Suresh, Arjun

AU - Thomas, Geoffry

AU - Tykhonov, Vyacheslav

AU - Vanschoren, Joaquin

AU - Varma, Susheel

AU - van der Velde, Jos

AU - Vogler, Steffen

AU - Wu, Carole-Jean

AU - Zhang, Luyao

PY - 2024/12/12

Y1 - 2024/12/12

N2 - Data is a critical resource for machine learning (ML), yet working with data remains a key friction point. This paper introduces Croissant, a metadata format for datasets that creates a shared representation across ML tools, frameworks, and platforms. Croissant makes datasets more discoverable, portable, and interoperable, thereby addressing significant challenges in ML data management. Croissant is already supported by several popular dataset repositories, spanning hundreds of thousands of datasets, enabling easy loading into the most commonly-used ML frameworks, regardless of where the data is stored. Our initial evaluation by human raters shows that Croissant metadata is readable, understandable, complete, yet concise.

AB - Data is a critical resource for machine learning (ML), yet working with data remains a key friction point. This paper introduces Croissant, a metadata format for datasets that creates a shared representation across ML tools, frameworks, and platforms. Croissant makes datasets more discoverable, portable, and interoperable, thereby addressing significant challenges in ML data management. Croissant is already supported by several popular dataset repositories, spanning hundreds of thousands of datasets, enabling easy loading into the most commonly-used ML frameworks, regardless of where the data is stored. Our initial evaluation by human raters shows that Croissant metadata is readable, understandable, complete, yet concise.

KW - machine learning

KW - ml standard

KW - Artificial Intelligence

M3 - Contribution to conference proceedings

BT - 38th Conference on Neural Information Processing Systems (NeurIPS 2024)

PB - NeurIPS

CY - Vancouver, Canada

ER -