Visually Grounded Interaction and Language (ViGIL)

Introduction

The dominant paradigm in modern natural language understanding is learning statistical language models from text-only corpora. This approach is founded on a distributional notion of semantics, i.e. that the "meaning" of a word is based only on its relationship to other words. While effective for many applications, methods in this family suffer from limited semantic understanding, as they miss learning from the multimodal and interactive environment in which communication often takes place - the symbols of language thus are not grounded in anything concrete. The symbol grounding problem first highlighted this limitation, that "meaningless symbols (i.e.) words cannot be grounded in anything but other meaningless symbols" [1].

On the other hand, humans acquire language by communicating about and interacting within a rich, perceptual environment. This behavior provides the necessary grounding for symbols, i.e. to concrete objects or concepts (i.e. physical or psychological). Thus, recent work has aimed to bridge vision, interactive learning, and natural language understanding through language learning tasks based on natural images (ReferIt [2], Visual Question Answering [3-6], Visual Dialog [7-10], Captioning [11, 32-35], Visual-Audio Correspondence [30]) or through embodied agents performing interactive tasks [12-29] in physically simulated environments (DeepMind Lab [12], Baidu XWorld [13], Habitat [14], StreetLearn [18], AI2-THOR [21], House3D [22], Matterport3D [23], GIBSON [27], MINOS [28]), often drawing on the recent successes of deep learning and reinforcement learning. We believe this line of research poses a promising, long-term solution to the grounding problem faced by current, popular language understanding models.

While machine learning research exploring visually-grounded language learning may be in its earlier stages, it may be possible to draw insights from the rich research literature on human language acquisition. In neuroscience, recent progress in fMRI technology has enabled better understanding of the interaction between language, vision and other modalities [17,18] suggesting that the brains share neural representations of concepts across vision and language. In concurrent work, developmental cognitive scientists have argued that word acquisition in children is closely linked to them learning the underlying physical concepts in the real world [15, 31], and that they generalize surprisingly well at this from sparse evidence [36].

This workshop thus aims to gather people from various backgrounds - machine learning, computer vision, natural language processing, neuroscience, cognitive science, psychology, and philosophy - to share and debate their perspectives on why grounding may (or may not) be important in building machines that truly understand natural language.

Important Dates

Call for Papers

We invite high-quality paper submissions on the following topics:

• language acquisition or learning through interactions
• image/video captioning, visual dialogues, visual question-answering, and other visually grounded language challenges
• reasoning in language and vision
• transfer learning in language and vision tasks
• audiovisual scene understanding and generation
• navigation and question answering in virtual worlds with natural-language instructions
• original multimodal works that can be extended to vision, language or interaction
• human-machine interaction with vision and language
• understanding and modeling the relationship between language and vision in humans semantic systems and modeling of natural language and visual stimuli representations in the human brain
• epistemology and research reflexions about language grounding, human embodiment and other related topics
• visual and linguistic cognition in infancy and/or adults

Submission

Please upload submissions at: cmt3.research.microsoft.com/VIGIL2019

Accepted papers will be presented during joint poster sessions, with exceptional submissions selected for spotlight oral presentations. Accepted papers will be made publicly available as non-archival reports, allowing future submissions to archival conferences or journals.

Submissions should be up to 4 pages excluding references, acknowledgements, and supplementary material, and should follow NeurIPS format. The CMT-based review process will be double-blind to avoid potential conflicts of interests.

We welcome published papers from *non-ML* conferences that are within the scope of the workshop (without re-formatting). These specific papers do not have to be anonymous. They are eligible for poster sessions and will only have a very light review process.

A limited pool of NeurIPS registrations might be available for accepted papers.

In case of any issues, feel free to email the workshop organizers at: vigilworkshop@gmail.com.

D-Day Workshop Information

Posters will be taped to the wall (we will provide tape). Please make sure they are printed on lightweight paper without lamination and no larger than 36 x 48 inches or 90 x 122 cm in portrait orientation.

Schedule

Invited Speakers

Accepted Papers

Organizers

Scientific Committee

Program Committee

• Aishwarya Agrawal
• Cătălina Cangea
• Volkan Cirik
• Meera Hahn
• Ethan Perez
• Rowan Zellers

• Ryan Benmalek
• Luca Celotti
• Daniel Fried
• Arjun Majumdar
• Hao Tan

Previous sessions

• ViGIL Workshop at NeurIPS 2017
• ViGIL Workshop at NeurIPS 2018

Sponsors

References