Unsupervised Improvement of Audio-Text Cross-Modal Representations
N/D
Autores
Wang, Zhepei
Subakan, Cem
Subramani, Krishna
Wu, Junkai
Smaragdis, Paris
Orientador
Co-orientadores
Citações na Scopus
Tipo de documento
Trabalho de Evento
Data
2023
Resumo
Recent advances in using language models to obtain cross-modal audio-text representations have overcome the limitations of conventional training approaches that use predefined labels. This has allowed the community to make progress in tasks like zero-shot classification, which would otherwise not be possible. However, learning such representations requires a large amount of human-annotated audio-text pairs. In this paper, we study unsupervised approaches to improve the learning framework of such representations with unpaired text and audio. We explore domain-unspecific and domain-specific curation methods to create audio-text pairs that we use to further improve the model. We also show that when domain-specific curation is used in conjunction with a soft-labeled contrastive loss, we are able to obtain significant improvement in terms of zero-shot classification performance on downstream sound event classification or acoustic scene classification tasks.
Palavras-chave
Audio-text representation learning; Data aug-mentation; Contrastive learning; Sound event classification; Acoustic scene classification
Titulo de periódico
URL da fonte
Título de Livro
URL na Scopus
Idioma
Inglês
Notas
Membros da banca
Área do Conhecimento CNPQ
CIENCIAS EXATAS E DA TERRA::CIENCIA DA COMPUTACAO
ENGENHARIAS::ENGENHARIA ELETRICA
ENGENHARIAS::ENGENHARIA ELETRICA