Unsupervised Improvement of Audio-Text Cross-Modal Representations
N/D
Autores
Wang, Zhepei
Subakan, Cem
Subramani, Krishna
Wu, Junkai
Smaragdis, Paris
Orientador
Co-orientadores
Citações na Scopus
Tipo de documento
Trabalho de Evento
Data
2023
Título da Revista
ISSN da Revista
Título do Volume
Resumo
Recent advances in using language models to obtain cross-modal audio-text representations have overcome the limitations of conventional training approaches that use predefined labels. This has allowed the community to make progress in tasks like zero-shot classification, which would otherwise not be possible. However, learning such representations requires a large amount of human-annotated audio-text pairs. In this paper, we study unsupervised approaches to improve the learning framework of such representations with unpaired text and audio. We explore domain-unspecific and domain-specific curation methods to create audio-text pairs that we use to further improve the model. We also show that when domain-specific curation is used in conjunction with a soft-labeled contrastive loss, we are able to obtain significant improvement in terms of zero-shot classification performance on downstream sound event classification or acoustic scene classification tasks.
Palavras-chave
Titulo de periódico
URL da fonte
Título de Livro
URL na Scopus
Idioma
Inglês
URL permanente
Notas
Membros da banca
Área do Conhecimento CNPQ
CIENCIAS EXATAS E DA TERRA::CIENCIA DA COMPUTACAO
ENGENHARIAS::ENGENHARIA ELETRICA
ENGENHARIAS::ENGENHARIA ELETRICA