The brain is an extraordinarily dynamic, robust and adaptive processor of information that integrates vast amounts of evidence from a wide variety of disparate, noisy sensors carrying information from multiple simultaneous sources. From this immense quantity of information, the brain forms a unified and cohesive view of its universe. Even the very best systems in Artificial Intelligence (AI) and robotics have taken only tiny steps in this direction. Building a system that composes a global perspective from multiple distinct sources, types of data, and sensory modalities is a grand challenge of AI, yet it is specific enough that it can be studied quite rigorously and in such detail that the prospect for deep insights into these mechanisms, if pursued rigorously, is quite plausible in the near term. These insights would be extraordinarily beneficial not just to artificial intelligence and robotics, but also to neuroscience and cognition.
Throughout the world and on a grand scale, researchers in psychology and neuroscience are trying to tease out the underlying principles of the mind and brain. Simultaneously and generally separately, researchers in artificial intelligence are working to devise algorithms that tackle tasks previously requiring human intelligence. Although it would seem that opportunities should be abundant for these fields to complement each other, only in very rare cases does a discovery in one discipline increase understanding in the others. One example comes from reinforcement learning, which was originally developed for AI but is now the primary explanation of reward in the brain. Another example is the Self-Organizing Map, which was inspired by the brain but has since become one of the standard algorithms of AI. Yet as the fields evolve, it appears increasingly desirable for major areas of AI, psychology, and neuroscience to connect together, as each fills in different pieces of the puzzle—each studying and then understanding complementary functions and representations of the miracle of intelligence. We now have the opportunity to accelerate the confluence of these fields by focusing on an area of natural overlap: cross-modal learning.
The term crossmodal learning refers to the adaptive, synergistic synthesis of complex perceptions from multiple sensory modalities, such that the learning that occurs within any individual sensory modality can be enhanced with information from one or more other modalities. Crossmodal learning is crucial for human understanding of the world, and examples are ubiquitous, such as: learning to grasp and manipulate objects; learning to walk; learning to read and write; learning to understand language and its referents; etc. In all these examples, visual, auditory, somatosensory or other modalities have to be integrated, and learning must therefore be crossmodal. In fact, the broad range of acquired human skills are crossmodal, and many of the most advanced human capabilities, such as those involved in social cognition, require learning from the richest combinations of crossmodal information. Crossmodal learning is a broad, interdisciplinary topic that has not yet coalesced into a single, unified field. Instead, there are many separate fields, each tackling the concerns of crossmodal learning from its own perspective, with currently little overlap. We anticipate an accelerating trend towards integration of these areas and we intend to be prime movers of that integration. By focusing our research on crossmodal learning, we can bring together recent progress in artificial intelligence, psychology and neuroscience,along with some of the world’s leading experts in each field, to attack one of the most important open problems in all three fields.
CML Transregional Collaborative Research Centre
The purpose of the CML project is therefore to establish a new interdisciplinary Transregional Collaborative Research Centre (TRR) as an interdisciplinary cooperation between the existing fields of artificial intelligence, psychology and neuroscience, focused on establishing the topic of crossmodal learning as a new discipline. Our aim is therefore not only to establish crossmodal learning as a central nexus of these existing fields, but also to establish our collaborative centre as the primary research vehicle at the focal point of this new discipline. Based on an extensive groundwork of collaborative research between Germany and China, this centre is to be jointly funded by the DFG (Deutsche Forschungsgemeinschaft) and the NSFC (Natural Science Foundation of China) and is planned as a new international collaboration between the University of Hamburg, the Medical Center Hamburg Eppendorf and the three top universities in China (Tsinghua, Beijing Normal and Peking University) as well as the Institute of Psychology of the Chinese Academy of Sciences—all located in Beijing, China. The focus on crossmodal learning, and the necessarily interdisciplinary nature of its study, distinguishes our proposed research centre from other major research centres (e.g., the SFBs 636, 833, and 874).