Universität Hamburg - der Forschung, der Lehre, der Bildung, zur Homepage
leeres logo

Photo: UHH/Denstorf

Publications - Thematic Area A

Publications project A1 - Dynamic adaptation of multisensory processes by crossmodal recalibration

Publications of the current project

  • Rohlf, S., Bruns, P., & Röder, B. (2021). The effects of cue reliability on crossmodal recalibration in adults and children. Multisensory Research, 34, 743-761. https://doi.org/10.1163/22134808-bja10053
  • Bruns, P., Dinse, H. R., & Röder, B. (2020). Differential effects of the temporal and spatial distribution of audiovisual stimuli on cross-modal spatial recalibration. European Journal of Neuroscience, 52, 3763-3775. https://doi.org/10.1111/ejn.14779
  • Gudi-Mindermann, H., Rimmele, J. M., Bruns, P., Kloosterman, N. A., Donner, T. H., Engel, A. K., & Röder, B. (2020). Post-training load-related changes of auditory working memory – An EEG study. Frontiers in Human Neuroscience, 14, 72. https://doi.org/10.3389/fnhum.2020.00072
  • Kramer, A., Röder, B., & Bruns, P. (2020). Feedback modulates audio-visual spatial recalibration. Frontiers in Integrative Neuroscience, 13, 74. https://doi.org/10.3389/fnint.2019.00074
  • Li, L., Rehr, R., Bruns, P., Gerkmann, T., & Röder, B. (2020). A survey on probabilistic models in human perception and machines. Frontiers in Robotics and AI, 7, 85. https://doi.org/10.3389/frobt.2020.00085
  • Rohlf, S., Li, L., Bruns, P., & Röder, B. (2020). Multisensory integration develops prior to crossmodal recalibration. Current Biology, 30, 1726-1732. https://doi.org/10.1016/j.cub.2020.02.048
  • Tong, J., Li, L., Bruns, P., & Röder, B. (2020). Crossmodal associations modulate multisensory spatial integration. Attention, Perception, & Psychophysics, 82, 3490-3506. https://doi.org/10.3758/s13414-020-02083-2

Publications of the phase-1 project

  • Bruns, P. (2019). The ventriloquist illusion as a tool to study multisensory processing: An update. Frontiers in Integrative Neuroscience, 13, 51. https://doi.org/10.3389/fnint.2019.00051
  • Bruns, P., & Röder, B. (2019). Visual recalibration of auditory spatial perception decays at different time scales. Proceedings of the International Congress on Acoustics, 2019, 3915-3920. https://doi.org/10.18154/RWTH-CONV-239152
  • Gudi-Mindermann, H., Rimmele, J. M., Nolte, G., Bruns, P., Engel, A. K., & Röder, B. (2018). Working memory training in congenitally blind individuals results in an integration of occipital cortex in functional networks. Behavioural Brain Research, 348, 31-41. https://doi.org/10.1016/j.bbr.2018.04.002
  • Parisi, G. I., Tong, J., Barros, P., Röder, B., & Wermter, S. (2018). Towards Modeling the Interaction of Spatial-Associative Neural Network Representations for Multisensory Perception. Workshop on Computational Models for Crossmodal Learning (ICDL-EpiRob).
  • Zierul, B., Tong, J., Bruns, P., & Röder, B. (2019). Reduced multisensory integration of self-initiated stimuli, Cognition, 182, 349-359.
  • Bruns, P., Röder, B. (2019). Cross-modal learning in the auditory system. In A. K. C. Lee, M. Wallace, A. Coffin, A. N. Popper, & R. R. Fay (Eds.), Multisensory processes: The auditory perspective. Springer Handbook of Auditory Research.
  • Bruns, P., Röder, B. (2019). Repeated but not incremental training enhances cross-modal recalibration. Journal of Experimental Psychology: Human Perception and Performance, 45(4), 435.
  • Badde, S., Röder, B., & Bruns, P. (2018). Task-irrelevant sounds influence both temporal order and apparent motion judgments about tactile stimuli applied to crossed and uncrossed hands. Attention, Perception, & Psychophysics, 80(3), 773-783.
  • Bruns, P., Röder, B., (2017). Spatial and frequency specificity of the ventriloquism aftereffect revisited. Psychological Research, 1-16.
  • Katzakis, N., Tong, J., Nunez, O., Chen, L., Klinker, G., Röder, B., & Steinicke, F. (2017). Stylo and handifact: modulating haptic perception through visualisations for posture training in augmented reality. Proc. Spatial User Interaction, doi: 10.1145/3131277.3132181
  • Zierul, B., Röder, B., Tempelmann, C., Bruns, P., and Noesselt, T. (2017) The Role of Auditory Cortex in the Spatial Ventriloquism Aftereffect. NeuroImage, 162, 257-268.
  • Zhang, D., Hong, B., Gao, S., & Röder, B. (2017). Exploring the temporal dynamics of sustained and transient spatial attention using steady-state visual evoked potentials. Experimental Brain Research, 2017 May; 235(5):1575-1591. doi: 10.1007/s00221-017-4907-6.

Relevant previous work

  • Badde, S., Röder, B., Heed, T. (2014). Multiple spatial representations determine touch localization. Journal of Experimental Psychology: Human Perception and Performance, 40 (2), 784-801.
  • Bruns, P., Maiworm, M., & Röder, B. (2014) Reward expectation influences audiovisual spatial integration. Attention, Perception, & Psychophysics, 76, 1815-1827.
  • Bruns, P., Liebnau, R., & Röder, B. (2011). Crossmodal training induces changes in spatial representations early in the auditory processing pathway. Psychological Science, 22(9), 1120-1126.
  • Bruns, P. & Röder, B. (2015). Sensory recalibration integrates information from the immediate and the cumulative past. Scientific Reports, in press.
  • Bruns, P., Spence, C., & Röder, B. (2011). Tactile recalibration of auditory spatial representations. Experimental Brain Research, 209 (3), 333-344.
  • Bruns, P., & Röder, B. (2010). Tactile captu - Sensory capacity and information integration independently explain the cognitive status of healthy oder adults (Kooperation A3, B1, B4) - Crossmodal pattern discrimination in humans and robots: a visuo-tactile case study (Kooperation A3, A4, B5, Z3) - Reduced frontal white matter microstructure in healthy older adults with low tactile recognition performance (Deutsch-chinesische Ko-Autorenschaft) - Early parietofrontal network up regulation relates to future persistent deficits after severe stroke - a prospective cohort study re of auditory localization: An event-related potential study. European Journal of Neuroscience, 31, 1844-1857.
  • Heed, T., Buchholz, V., Engel, A., & Röder, B. (2015). Tactile remapping: from coordinate transformation to integration in sensorimotor processing. Trends in Cognitive Sciences, 19(5), 251-258.
  • Heed, T., Röder, B. (2014). Motor coordination uses external spatial coordinates independent of developmental vision. Cognition, 132, 1-15.
  • Maiworm, M., Bellantoni, M., Spence, C., & Röder, B. (2012). When emotional valence modulates audiovisual integration. Attention, Perception & Psychophysics, 74 (6), 1302-1311.
  • Zhang, D., Hong, B., Gao, X., Gao, S., Röder, B. (2011). Exploring steady-state visual evoked potentials as an index for intermodal and crossmodal spatial attention. Psychophysiology, 48, 665-675.

Publications project A2 - Using multiperturbation analysis to reveal the neural circuit basis of multi-sensory processing

Publications of the current project

  • Jiang Y, Fu X, Zhang Y, Wang SF, Zhu H, Wang WK, Zhang L, Wu P, Wong CCL, Li J, Ma J, Guan JS, Huang Y, Hui J.(2021). Rett syndrome linked to defects in forming the MeCP2/Rbfox/LASR complex in mouse models. Nat Commun. 12(1):5767. doi: 10.1038/s41467-021-26084-3
  • Guangyu Wang, Hong Xie, Yi Hu, Qinan Chen, Chenhui Liu, Kaiyuan Liu, Yuze Yan, Ji-Song Guan (2021). Egr1-EGFP transgenic mouse allows in vivo recording of Egr1 expression and neural activity. Journal of Neuroscience Methods, 363:109350. doi: 10.1016/j.jneumeth.2021.109350
  • Fei Wang, Weiqing Jiang, Li Gao, Chen Liu, Mingzhu Deng, Xiao Ren, Chenlu Zhu, Ji-Song Guan, Yonggang Wang (2021). Detecting Abnormal Neuronal Activity in a Chronic Migraine Model by Egr1-EGFP Transgenic Mice. Front Neurosci. 2021, Aug 12; 15:705938
  • Xue Zhang, Fengchao Wang, Yi Hu, Runze Chen, Dawei Meng, Liang Guo, Hailong Lv, Ji-Song Guan, Yichang Jia. (2020). In vivo stress granule misprocessing evidenced in a FUS knock-in ALS mouse model. Brain, 2020 May 1;143(5):1350-1367. doi: 10.1093/brain/awaa076
  • Shiguo Liu, Miaomiao Tian, Fan He, Jiani Li, Hong Xie, Wenmiao Liu, Yeting Zhang, Ru Zhang, Mingji Yi, Fengyuan Che, Xu Ma, Yi Zheng, Hao Deng, Guiju Wang, Lang Chen, Xue Sun, Yinglei Xu, Jingli Wang, Yucui Zang, Mengmeng Han, Xiuhai Wan, Hongzai Guan, Yinlin Ge, Chunmei Wu, Haiyan Wang, Hui Liang, Hui Li, Ni Ran, Zhaochuan Yang, Huanhuan Huang, Yanzhao Wei, Xueping Zheng, Xiangrong Sun, Xueying Feng, Lanlan Zheng, Tao Zhu, Wenhan Luo,Qinan Chen, Yuze Yan, Zuzhou Huang, Zhongcui Jing, Yixia Guo, Xuzhan Zhang, Christian P. Schaaf, Jinchuan Xing, Chuanyue Wang, FuliYu, Ji-Song Guan（2020). Mutations in ASH1L confer susceptibility to Tourette Syndrome. Molecular Psychiatry, Feb; 25(2):476-490
  • A Goulas, F Damicelli, C CHilgetag (2021). Bio-instantiated recurrent neural networks: Integrating neurobiology-based network topology in artificial networks. Neural Netw. 2021 Oct;142:608-618

Publications of the phase-1 project

  • Damicelli, F., Hilgetag, C. C., Hütt, M. T., & Messé, A. (2019). Topological reinforcement as a principle of modularity emergence in brain networks. Network Neuroscience, 3(2), 1-34.
  • Wang, G., Xie, H., Wang, L., Luo, W., Wang, Y., Jiang, J., ... & Guan, J. S. (2019). Switching From Fear to No Fear by Different Neural Ensembles in Mouse Retrosplenial Cortex. Cerebral Cortex pii: bhz050. doi: 10.1093/cercor/bhz050
  • Li, D., Zavaglia, M., Wang, G., Xie, H., Hu, Y., Werner, R., ... & Hilgetag, C. C. (2019). Discrimination of the hierarchical structure of cortical layers in 2-photon microscopy data by combined unsupervised and supervised machine learning. Scientific Reports, 9(1), 7424.
  • Goulas A, Zilles K, Hilgetag CC. (2018). Cortical gradients and laminar projections in mammals. Trends in Neuroscience, 41(11), 775-788.
  • Luo, W., & Guan. J. S. (2018). Do brain oscillations orchestrate memory? Brain Science Advances, 4(1), 16-33.
  • Jiang.J. Guanyu Wang, Ji-Song Guan. (2018). Mammillary body regulates state-dependent fear by alternating cortical oscillations Sci Rep, 8(1), 13471.
  • Messé A, Hütt MT, Hilgetag CC (2018). Toward a theory of coactivation patterns in excitable neural networks. PLoS Comput Biollgy, 14(4):e1006084.
  • Rollenhagen A, Ohana O, Sätzler K, Hilgetag C, Kuhl D, Lübke J (2018) Structural Properties of Synaptic Transmission and Temporal Dynamics at Excitatory Layer 5B Synapses in the Adult Rat Somatosensory Cortex. Front Synaptic Neurosci, 10:24.
  • Zou Y, Zhao Z, Yin D, Fan M, Small M, Liu Z, Hilgetag C, Kurths (2018) Brain anomaly networks uncover heterogeneous functional reorganization patterns after stroke Neuroimage-Clin. 20:523-530
  • Ruxing Fu, Wenhan Luo, Roy Nazempour, Daxin Tan, He Ding, Kaiyuan Zhang, Lan Yin, Ji-Song Guan*, Xin Sheng*. (2018). Implantable and Biodegradable Poly(L-lactic acid) Fibers for Optical Neural Interfaces, (2018), Adv Optical Materials. DOI: 10.1002/adom.201700941
  • Malherbe C, Umarova R, Zavaglia M, Kaller C, Beume L, Thomalla G, Weiller C, Hilgetag C (2018). Neural correlates of visuospatial bias in patients with left hemisphere stroke: a causal functional contribution analysis based on game theory, Neuropsychologia, 115: 142-153.
  • Fretter C, Lesne A, Hilgetag CC, Hütt MT. (2017). Topological determinants of self-sustained activity in a simple model of excitable dynamics on graphs. Sci Rep 7:42340.
  • Toba M, Zavaglia M, Rastelli F, Valabrégue R, Pradat-Diehl P, Valero-Cabré A, Hilgetag CC. (2017). Game theoretical mapping of causal interactions underlying visuo-spatial attention in the human brain based on stroke lesions. Hum Brain Mapp. doi: 10.1002/hbm.23601.
  • Damicelli F, Hilgetag CC, Hütt MT, Messé A. (2017). Modular topology emerges from plasticity in a minimalistic excitable network model. Chaos. 27(4):047406.
  • Chen Y, Wang S, Hilgetag CC, Zhou C. (2017). Features of spatial and functional segregation and integration of the primate connectome revealed by trade-off between wiring cost and efficiency. PLoS Comput Biol 13(9):e1005776.
  • Maier-Hein KH, Neher PF, ..., Hilgetag CC, Stieltjes B, Descoteaux M. (2017). The challenge of mapping the human connectome based on diffusion tractography. Nat Commun 8(1):1349.
  • Stellmann J, Hodecker S, Cheng B, Wanke N, Young K, Hilgetag C, Gerloff C, Heesen C, Thomalla G, Siemonsen S (2017) Reduced rich-club connectivity is related to disability in primary progressive MS Neurol Neuroimmunol Neuroinflamm. 4(5):e375
  • Goulas A, Uylings H, Hilgetag C (2017) Principles of ipsilateral and contralateral cortico-cortical connectivity in the mouse BRAIN STRUCT FUNCT. 222(3):1281-1295
  • Ding X, Liu S, Tian M, Zhang W, Zhu T, Li D, Wu J, Deng H, Jia Y, Xie W, Xie H, Guan JS (2017). Activity-induced histone modifications govern Neurexin-1 mRNA splicing and memory preservation. Nat Neurosci. 20(5):690-699.
  • Zavaglia M, Forkert N, Cheng B, Gerloff C, Thomalla G, Hilgetag C (2016). Technical considerations of a game-theoretical approach for lesion symptom mapping, BMC Neurosci 17:40
  • Hilgetag C, Amunts K (2016) Connectivity and cortical architecture e-Neuroforum. 7(3):56-63.
  • Hilgetag C, Medalla M, Beul S, Barbas H (2016) The primate connectome in context: Principles of connections of the cortical visual system. Neuroimage. 134:685-702.

Relevant previous work

  • Gräff J, Rei D., Guan J. S., Wang W. Y., Seo J., Hennig K. M., Nieland T. J., Fass D. M., Kao P. F., Kahn M., Su S. C., Samiei A., Joseph N., Haggarty S. J., Delalle I., & Tsai L. H. (2012). An epigenetic blockade of cognitive functions in the neurodegenerating brain, Nature, 483:222.
  • Guan J. S., Haggarty S. J., Giacometti E., Dannenberg J. H., Joseph N., Gao J., Nieland T. J., Zhou Y., Wang X., Mazitschek R., Bradner J. E., DePinho R. A., Jaenisch R., & Tsai L. H. (2009). HDAC2 negatively regulates memory formation and synaptic plasticity, Nature, 459: 55.
  • Hilgetag C. C., O'Neill M. A., & Young M. P. (1996). Indeterminate organization of the visual system, Science, 271: 776.
  • Hilgetag C. C., Théoret H., & Pascual-Leone A. (2001). Enhanced visual spatial attention ipsilateral to rTMS-induced ‘virtual lesions’ of human parietal cortex, Nature Neuroscience, 4, 953.
  • Keinan A., Sandbank B., & Hilgetag C. C. (2004). Meilijson I, Ruppin E: Fair attribution of functional contribution in artificial and biological networks, Neural Computation, 16: 1887.
  • Keinan A., Sandbank B., & Hilgetag C. C., Meilijson I, Ruppin E (2006). Axiomatic scalable neurocontroller analysis via the Shapley value, Artificial Life, 12: 333.
  • Sporns O., Chialvo D., Kaiser M., & Hilgetag C. C. (2004). Organization, development and function of complex brain networks, Trends in Cognitive Sciences, 8: 418-425.
  • Xie H., Liu Y., & Guan J. S. (2014). In vivo imaging of immediate early gene expression reveals layer-specific memory traces in the mammalian brain, PNAS, 111(7): 2788.
  • Young M. P., Hilgetag C. C., & Scannell J. W. (2000). On imputing function to structure from the behavioural effects of brain lesions, Phil Trans R Soc Lond B, 355: 147.
  • Zavaglia M., & Hilgetag C. C. (2015). Causal functional contributions and interactions in the attention network of the brain: An objective multi-perturbation analysis, Brain Struct Funct, DOI: 10.1007/s00429-015-1058-z.

Publications project A3 - Multisensory integration in the aging brain — mechanisms and facilitation

Publications of the current project

  • Focko L. Higgen, Hanna Braaß, Winifried Backhaus, Robert Schulz, Gui Xue, Christian Gerloff (2021). Reduced frontal white matter microstructure in healthy older adults with low tactile recognition performance, Scientific Reports volume 11, Article number: 111689, doi https://doi.org/10.1038/s41598-021-90995-w
  • Misselhorn, Göschl, Higgen* et al. (2020). Sensory capability and information integration independently explain the cognitive status of healthy older adults. Scientific Reports volume 10, Article number: 22437 (2020).
  • Higgen F.L., Heine C., Krawinkel L., Göschl F., Engel A.K., Hummel F.C., Xue Gui, Gerloff C. (2020). Crossmodal Congruency Enhances Performance of Healthy Older Adults in Visual-Tactile Pattern Matching. Frontiers in Aging Neuroscience 12, p. 74, 2020. doi: 10.3389/fnagi.2020.00074
  • Higgen F.L., Ruppel P., Görner M., Kerzel M., Hendrich N., Feldheim J., Wermter S., Zhang J., Gerloff C. (2020). Crossmodal pattern discrimination in humans and robots: a visuo-tactile case study. Frontiers in Robotics and AI, vol 7, 2020. doi: 10.3389/frobt.2020.540565
  • Winifried Backhaus, Hanna Braaß, Focko L. Higgen, Christian Gerloff and Robert Schulz (2021). Early parietofrontal network upregulation relates to future persistent deficits after severe stroke—a prospective cohort study, Brain Communications 2021, doi:10.1093/braincomms/fcab097

Publications of the phase-1 project

  • Higgen F.L., Braaß H., Schulz R., Xue G., Gerloff C. (2019). P35 A complex tactile recognition task as a marker for the aging brain. Clinical Neurophysiology 2019; 130(8): e162-e163. doi: 10.1016/j.clinph.2019.04.688
  • Huang, X., Zhang, H., Chen, C., Xue, G., He, Q. (2019). The neuroanatomical basis of the Gambler’s fallacy: A univariate and multivariate morphometric study. Hum Brain Mapp 40, 967–975. doi: https://doi.org/10.1002/hbm.24425
  • Zhu B., Chen C., Shao X., Liu W., Ye Z., Zhuang L., Zheng L., Loftus E.F., Xue G. (2019). Multiple interactive memory representations underlie the induction of false memory. Proceedings of the National Academy of Sciences U S A, 116, 3466-3475.
  • Cai, Y., Urgolites, Z., Wood, J., Chen, C., Li, S., Chen, A., Xue, G. (2018). Distinct neural substrates for visual short-term memory of actions. Hum Brain Mapp 39, 4119–4133. https://doi.org/10.1002/hbm.24236
  • Xue, G. (2018). The Neural Representations Underlying Human Episodic Memory. Trends Cogn. Sci. (Regul. Ed.) 22, 544–561. doi: https://doi.org/10.1016/j.tics.2018.03.004
  • Higgen F.L., Ruppel P., Görner M., Kerzel M., Magg S., Hendrich N. (2018). Crossmodal pattern discrimination in humans and robots: a visuo-tactile case study. Workshop on Crossmodal Learning, IROS.2018.
  • Qu, J., Qian, L., Chen, C., Xue, G., Li, H., Xie, P., Mei, L. (2017). Neural Pattern Similarity in the Left IFG and Fusiform Is Associated with Novel Word Learning. Front Hum Neurosci 11, 424. doi: https://doi.org/10.3389/fnhum.2017.00424

Relevant previous work

  • Cohen, L. G., Celnik, P., Pascual-Leone, A., Corwell, B., Falz, L., Dambrosia, J., Honda, M., Sadato, N., Gerloff, C., Catalá, M.D., & Hallett, M. (1997). Functional relevance of crossmodal plasticity in blind humans. Nature, 389(6647), 180-183.
  • Heise, K. F., Zimerman, M., Hoppe, J., Gerloff, C., Wegscheider, K., & Hummel, F. C. (2013). The aging motor system as a model for plastic changes of GABA-mediated intracortical inhibition and their behavioral relevance. J Neurosci, 33(21), 9039-9049.
  • Hummel, F., & Gerloff, C. (2005). Larger interregional synchrony is associated with greater behavioral success in a complex sensory integration task in humans. Cereb Cortex, 15(5), 670-678.
  • Hummel, F. C., Heise, K., Celnik, P., Floel, A., Gerloff, C., & Cohen, L. G. (2010). Facilitating skilled right hand motor function in older subjects by anodal polarization over the left primary motor cortex. Neurobiol Aging, 31(12), 2160-2168.
  • Plewnia, C., Rilk, A. J., Soekadar, S. R., Arfeller, C., Huber, H. S., Sauseng, P., Hummel, F.C., & Gerloff, C. (2008). Enhancement of long-range EEG coherence by synchronous bifocal transcranial magnetic stimulation. Eur J Neurosci, 27(6), 1577-1583.
  • Renzi, C., Bruns, P., Heise, K. F., Zimerman, M., Feldheim, J. F., Hummel, F. C., & Roder, B. (2013). Spatial remapping in the audio-tactile ventriloquism effect: a TMS investigation on the role of the ventral intraparietal area. J Cogn Neurosci, 25(5), 790-801.
  • Sauseng, P., Klimesch, W., Heise, K. F., Gruber, W. R., Holz, E., Karim, A. A., Glennon, M., Gerloff, C., Birbaumer, N., & Hummel, F. C. (2009). Brain oscillatory substrates of visual short-term memory capacity. Curr Biol, 19(21), 1846-1852.
  • Schulz, R., Zimerman, M., Timmermann, J. E., Wessel, M. J., Gerloff, C., & Hummel, F. C. (2014). White matter integrity of motor connections related to training gains in healthy aging. Neurobiol Aging, 35(6), 1404-1411.
  • Xue, G., Dong, Q., Chen, C., Lu, Z., Mumford, J. A., & Poldrack, R. A. (2010). Greater neural pattern similarity across repetitions is associated with better memory. Science, 330(6000), 97-101.
  • Zimerman, M., Nitsch, M., Giraux, P., Gerloff, C., Cohen, L. G., & Hummel, F. C. (2013). Neuroenhancement of the aging brain: restoring skill acquisition in old subjects. Ann Neurol, 73(1), 10-15.

Publications project A4 - Crossmodal joint sparse feature learning in robot tasks

Publications of the current project

  • Zhong Cao, Sen Cui, Changshui Zhang (2021). DCR: Disentangled component representation for sketch generation. Pattern Recognition Letters. vol. 145, pp. 16-22. 2021.
  • Zhong Cao, Jiang Lu, Sen Cui and Changshui Zhang (2020). Zero-Shot Handwritten Chinese Character Recognition with Hierarchical Decomposition Embedding. Pattern Recognition vol. 107, Nov. 2020.
  • Yiwen Guo, Long Chen, Yurong Chen, and Changshui Zhang (2020). On Connections between Regularizations for Improving DNN Robustness. IEEE Transactions on Pattern Analysis and Machine Intelligence (2020).
  • Yiwen Sun, Kun Fu, Zheng Wang, Changshui Zhang, and Jieping Ye (2021). Road Network Metric Learning for Estimated Time of Arrival. The 25th International Conference on Pattern Recognition (ICPR). Milan, Italy, 10-15 January, 2021.
  • Yiwen Sun, Yulu Wang, Kun Fu, Zheng Wang, Changshui Zhang, and Jieping Ye (2021). Constructing Geographic and Long-term Temporal Graph for Traffic Forecasting. The 25th International Conference on Pattern Recognition (ICPR). Milan, Italy, 10-15 January, 2021.
  • Yiwen Guo, Ming Lu, Wangmeng Zuo, Changshui Zhang, Yurong Chen (2021). Deep Likelihood Network for Image Restoration with Multiple Degradation Levels. IEEE Transactions on Image Processing (2021).
  • Yiwen Sun, Yulu Wang, Kun Fu, Zheng Wang, Ziang Yan, Changshui Zhang, Jieping Ye (2021). FMA-ETA: Estimating Travel Time Entirely Based on FFN With Attention. The 46th IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). June, 6-11, 2021.
  • Jiang Lu, Lei Li, and Changshui Zhang. Self-reinforcing Unsupervised Matching (2021). IEEE Transactions on Pattern Analysis and Machine Intelligence (2021).
  • Philipp Ruppel, Norman Hendrich, Jianwei Zhang (2021). Direct Policy Optimization with Differentiable Physical Consistency for Dexterous Manipulation. Proceedings IEEE ROBIO 2021.
  • Philipp Ruppel, Jianwei Zhang (2020). Learning Object Manipulation with Dexterous Hand-Arm Systems from Human Demonstration. IEEE/RSJ IROS 2020.
  • Niklas Fiedler, Philipp Ruppel, Yannick Jonetzko, Norman Hendrich, and Jianwei Zhang (2021). A Low-Cost Modular System of Customizable, Versatile, and Flexible Tactile Sensor Arrays. IEEE/RSJ IROS 2021.
  • Fangwei Zhong, Peng Sun, Wenhan Luo, Tingyun Yan, Yizhou Wang (2021). Towards Distraction-Robust Active Visual Tracking. ICML 2021.
  • Jing Xu, Fangwei Zhong, Yizhou Wang (2020). Learning Multi-Agent Coordination for Enhancing Target Coverage in Directional Sensor Networks. NeurIPS 2020.
  • Rongchang Xie, Chunyu Wang, Yizhou Wang (2020). MetaFuse: A Pre-trained Fusion Model for Human Pose Estimation. CVPR 2020.
  • Focko L. Higgen, Philipp Ruppel, Michael Görner, Matthias Kerzel, Norman Hendrich, Jan Feldheim, Stefan Wermter, Jianwei Zhang, Christian Gerloff (2020). Crossmodal Pattern Discrimination in Humans and Robots: AVisuo-Tactile Case Study. Frontiers Robotics and AI, 2020. DOI:10.3389/frobt.2020.540565
  • Lin Cong, Michael Görner, Philipp Ruppel, Hongzhuo Liang, Norman Hendrich, Jianwei Zhang (2020). Self-Adapting Recurrent Models for Object Pushing from Learning in Simulation. IROS 2020.
  • Yannick Jonetzko, Niklas Fiedler, Manfred Eppe, Jianwei Zhang (2020). Multimodal Object Analysis with Auditory and Tactile Sensing using Recurrent Neural Networks. ICCSIP 2020.

Publications of the phase-1 project

  • Liang, H., Li, S., Ma, X., Hendrich, N., Gerkmann T., Sun, F., & Zhang, J. (2019). Making sense of Audio Vibration for Liquid Height Estimation in Robotic Pouring. International Conference on Intelligent Robots and Systems (IROS), Macau, China (in press).
  • Yu, X., Guo, T., Fu, K, Li, L, Zhang, C. and Zhang J. (2019). Image Captioning with Partially Rewarded Imitation Learning. International Joint Conference on Neural Network (IJCNN), Budapest, Hungary.
  • Tang, S., Ji, Y., Lyu, J., Mi, J., Li, Q., & Zhang, J. (2019). Visual Domain Adaptation Exploiting Confidence-Samples. International Conference on Intelligent Robots and Systems (IROS), Macau, China (in press).
  • Chang, D., Lin, M., & Zhang, C. (2018). On the Generalization Ability of Online Gradient Descent Algorithm Under the Quadratic Growth Condition. IEEE Transactions on Neural Networks and Learning Systems, 29(10), 1-12.
  • Lu R., Duan Z. and Zhang C. (2018). Listen and Look: Audio–Visual Matching Assisted Speech Source Separation. IEEE Signal Processing Letters.
  • Lu, R., Duan, Z., & Zhang, C. (2018). Multi-Scale Recurrent Neural Network for Sound Event Detection. 2018 In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 131-135.
  • Lu, J., Li, J., Yan, Z., Mei, F., & Zhang, C. (2018). Attribute-Based Synthetic Network (ABS-Net): Learning more from pseudo feature representations. Pattern Recognition, 80, 129-142.
  • Lu, J., Cao, Z., Wu, K., Zhang, G., & Zhang, C. (2018). Boosting Few-Shot Image Recognition Via Domain Alignment Prototypical Networks. In 2018 IEEE 30th International Conference on Tools with Artificial Intelligence (ICTAI), 260-264.
  • Yan, C., Wu, K., & Zhang, C. (2018). A New Anchor-Labeling Method For Oriented Text Detection Using Dense Detection Framework. IEEE Signal Processing Letters, 25(9), 1295-1299.
  • Yan, C., Hu, J., & Zhang, C. (2018). Deep transformer: A framework for 2D text image rectification from planar transformations. Neurocomputing, 289, 32-43.
  • Chu, D., Lu, R., Li, J., Yu, X., Zhang, C., & Tao, Q. (2018). Optimizing Top-k Multiclass SVM via Semismooth Newton Algorithm. IEEE Transactions on Neural Networks and Learning Systems, 29(12), 6264-6275.
  • Fu, K., Li, J., Jin, J., & Zhang, C. (2018). Image-text surgery: Efficient concept learning in image captioning by generating pseudopairs. IEEE Transactions on Neural Networks and Learning Systems, (99), 1-12.
  • Yu, N., Hu, X., Song, B., Yang, J., & Zhang, J. (2018). Topic-Oriented Image Captioning Based on Order-Embedding. IEEE Transactions on Image Processing, 28(6), 2743-2754.
  • Ruppel, P., Jonetzko, Y., Görner, M., Hendrich, N., & Zhang, J.(2018). Simulation of the SynTouch BioTac Sensor. International Conference on Intelligent Autonomous Systems. DOI: 10.1007/978-3-030-01370-7_30
  • Ruppel, P., Hendrich, N., Starke, S., & Zhang, J. (2018). Cost functions to specify full-body motion and multi-goal manipulation tasks. In 2018 IEEE International Conference on Robotics and Automation (ICRA), 3152-3159.
  • Wu, K., & Zhang, C. (2018). Deep Generative Adversarial Networks for the Sparse Signal Denoising. In 2018 24th International Conference on Pattern Recognition (ICPR), 1127-1132.
  • Chu, D., Zhang, C., & Tao, Q. (2017). A faster cutting plane algorithm with accelerated line search for linear SVM. Pattern Recognition, 67, 127-138.
  • Liu, L., He, B., Zhuang, J., Zhang, L., & Lv, A. (2017). Force measurement system for invisalign based on thin film single force sensor. Measurement, 97, 1-7.
  • Hu, W., Zhuo, Q., Zhang, C., & Li, J. (2017). Fast branch convolutional neural network for traffic sign recognition. IEEE Intelligent Transportation Systems Magazine, 9(3), 114-126.
  • Lu, J., Hu, J., Zhao, G., Mei, F., & Zhang, C. (2017). An in-field automatic wheat disease diagnosis system. Computers and Electronics in Agriculture, 142, 369-379.
  • Lu, R., Wu, K., Duan, Z., & Zhang, C. (2017, March). Deep ranking: Triplet matchnet for music metric learning. In 2017 IEEE International Confer ence on Acoustics, Speech and Signal Processing (ICASSP), 121-125.
  • Cui, R., Liu, H., & Zhang, C. (2017). Recurrent convolutional neural networks for continuous sign language recognition by staged optimization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 7361-7369.
  • Tang, S., Chen, L., Mi, J., Ye, M., Li, Q., & Zhang, J. (2017). Adaptive pedestrian detection by modulating features in dynamical environment. IEEE International Conference on Robotics and Biomimetics (ROBIO), 62-67.
  • Liu, Z., Li, J., Shen, Z., Huang, G., Yan, S., & Zhang, C. (2017). Learning efficient convolutional networks through network slimming. In Proceedings of the IEEE International Conference on Computer Vision（ICCV）. 2736-2744.
  • Fu, K., Jin, J., Cui, R., Sha, F., & Zhang, C. (2016). Aligning where to see and what to tell: image captioning with region-based attention and scene-specific contexts. IEEE Transactions on Pattern Analysis and Machine Intelligence, 39(12), 2321-2334.
  • Guan, H., & Zhang, J. (2016). Multi-sensory based novel household object categorization system by using interactive behaviours. In 2016 IEEE International Conference on Robotics and Biomimetics (ROBIO), 1685-1690.
  • Zhang, L., He, B., Zhang, J., (2016). Multimodal Information Representation Based on Approximation Theory (in Chinese). Science Press, ISBN: 978-7-03-050253-7.

Relevant previous work

  • Gong, P., Zhang, C., Lu, Z., Huang, J.Z., & Ye, J. (2013). A general iterative shrinkage and thresholding algorithm for non-convex regularized optimization problems. In Machine learning: Proceedings of the International Conference on Machine Learning (pp. 37). NIH Public Access.
  • Gong, P., Ye, J., & Zhang, C.S. (2012). Multi-stage multi-task feature learning. In Advances in Neural Information Processing Systems (pp. 1988-1996).
  • Gong, P., & Zhang, C. (2011). A fast dual projected Newton Method for L1-regularized least squares. In The 22nd International Joint Conference on Artificial Intelligenc (IJCAI) (pp. 1275-1280). IEEE.
  • He, J., & Zhang, J. (2014). In-hand haptic perception in dexterous manipulations. Science China Information Sciences, 57(12), 1-11.
  • Hu, X., Zhang, J., Li, J., & Zhang, B. (2014). Sparsity-regularized HMAX for visual recognition. PLOS ONE, 9(1), 1–12.
  • Bai, T., Li, Y., & Zhou, X. (2014). Monocular human motion tracking with discriminative sparse representation. Advanced Robotics, 28(6), 403-414.
  • Pan, Z., & Zhang, C. (2015). Relaxed sparse eigenvalue conditions for sparse estimation via non-convex regularized regression. Pattern Recognition, 48(1), 231-243.
  • Pan, Z., Lin, M., Hou, G., Zhang, C. et al. (2015). Damping proximal coordinate descent algorithm for non-convex regularization. Neurocomputing, 152, 151-163.
  • Yu, J., Ding, R., Yang, Q., Tan, M., Wang, W., & Zhang, J. (2012). On a bio-inspired amphibious robot capable of multimodal motion. IEEE/ASME Transactions on Mechatronics, 17(5), 847-856.
  • Zhang, J., & Rössler, B. (2004). Self-valuing learning and generalization with application in visually guided grasping of complex objects. Journal of Robotics and Autonomous Systems, 47(2), 117-127.

Publications project A5 - Crossmodal learning in a neurobotic cortical and midbrain model

Publications of the current project

  • Cao, S., Fu, D., Yang, X., Barros, P., Wermter, S., Liu, X., & Wu, H. (2021). How Can AI Recognize Pain and Express Empathy. arXiv preprint arXiv:2110.04249. (under review)
  • Duczek, N., Kerzel, M., & Wermter, S. (2021). Continual Learning from Synthetic Data for a Humanoid Exercise Robot. arXiv preprint arXiv:2102.10034. (under review)
  • Fu, D., Abawi, F., Carneiro, H., Kerzel, M., Chen, Z., Strahl, E., Liu, X., & Wermter, S. (2021). A trained humanoid robot can perform human-like crossmodal social attention conflict resolution. https://arxiv.org/pdf/2111.01906.pdf (under review)
  • Barros, P., Barakova, E., & Wermter, S. (2020). Adapting the Interplay between Personalized and Generalized Affect Recognition based on an Unsupervised Neural Framework. IEEE Transactions on Affective Computing.
  • Wieser, I., Barros, P., Heinrich, S., & Wermter, S. (2020). Understanding Auditory Representations of Emotional Expressions with Neural Networks. Neural Computing and Applications, 32(4), 1007–1022.
  • Mohammadi, H. B., Xirakia, N., Abawi, F., … & Wermter, S. (2019). Designing a Personality-Driven Robot for a Human-Robot Interaction Scenario. Proceedings of The IEEE International Conference on Robotics and Automation ICRA-2019, 4317–4324.
  • Qu, L., Weber, C., & Wermter, S. (2019). LipSound: Neural Mel-spectrogram Reconstruction for Lip Reading. Proceedings of Interspeech2019.
  • Alpay, T., Abawi, F., & Wermter, S. (2019). Preserving Activations in Recurrent Neural Networks Based on Surprisal. Neurocomputing, 342, 75–82.
  • Carneiro, H., Weber, C., & Wermter, S. (2021). FaVoA: Face-Voice Association Favours Ambiguous Speaker Detection. Proceedings of The Thirtieth International Conference on Artificial Neural Networks ICANN-2021, 439-450.
  • Abawi, F., Weber, T., & Wermter, S. (2021). GASP: Gated Attention for Saliency Prediction. Proceedings of The Thirtieth International Joint Conference on Artificial Intelligence IJCAI-2021, 584-591.
  • Fu, D., Weber, C., Yang, G., Kerzel, M., Nan, W., Barros, P., ... & Wermter, S. (2020). What can computational models learn from human selective attention? A review from an audiovisual unimodal and crossmodal perspective.  Frontiers in Integrative Neuroscience, 14, 10.
  • Ahrens, K., Abawi, F., & Wermter, S. (2021). Dynamic Representations for Imbalanced Lifelong Learning. Proceedings of The Thirtieth International Conference on Artificial Neural Networks ICANN-2021, 409-420.
  • Fang, H., Carbajal, G., Wermter, S., & Gerkmann, T. (2021). Variational Autoencoder for Speech Enhancement with a Noise-Aware Encoder. 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 676-680.
  • Andriella, A., Siqueira, H., Fu, D., Magg, S., Barros, P., Wermter, S., ... & Alenya, G. (2020). Do I have a personality? Endowing care robots with context-dependent personality traits.  International Journal of Social Robotics, 1-22.
  • Higgen, F. L., Ruppel, P., Görner, M., Kerzel, M., Hendrich, N., Feldheim, J., ... & Gerloff, C. (2020). Crossmodal pattern discrimination in humans and robots: a visuo-tactile case study. Frontiers in Robotics and AI, 7.
  • Qu, L., Weber, C., & Wermter, S. (2020). Multimodal Target Speech Separation with Voice and Face References. Proceedings of Interspeech2020.
  • Weber, T., & Wermter, S. (2020). Integrating Intrinsic and Extrinsic Explainability: The Relevance of Understanding Neural Networks for Human-Robot Interaction. AI-HRI Symposium at AAAI-FSS.
  • Kerzel, M., Abawi, F., Eppe, M., & Wermter, S. (2020). Enhancing a Neurocognitive Shared Visuomotor Model for Object Identification, Localization, and Grasping With Learning From Auxiliary Tasks. IEEE Transactions on Cognitive and Developmental Systems, 1–13.

Publications of the phase-1 project

  • Marzouk, A., Barros, P., Eppe, M., & Wermter, S. (2019). The Conditional Boundary Equilibrium Generative Adversarial Network and its Application to Facial Attributes. Proceedings of The International Joint Conference on Neural Networks IJCNN-2019, 1–7.
  • Barros, P., Fliesswasser, E., Kerzel, M., & Wermter, S. (2019). Exploring Low-level and High-level Transfer Learning for Multi-task Facial Recognition with a Semi-supervised Neural Network. Proceedings of The IEEE/RSJ International Conference on Intelligent Robots and Systems IROS-2019, 1378–1384.
  • Mici, L., Parisi, G.I., & Wermter, S. (2019). Compositional Learning of Human Activities With a Self-Organizing Neural Architecture. Frontiers in Robotics and AI, 6.
  • Barros, P., Churamani, N., Lim, A., & Wermter, S. (2019). The OMG-Empathy Dataset: Evaluating the Impact of Affective Behavior in Storytelling. Proceedings of The Eighth IEEE International Conference on Affective Computing and Intelligent Interaction ACII-2019, 1–7.
  • Barros, P., Eppe, M., Parisi, G. I., Liu, X., & Wermter, S. (2019). Expectation Learning for Stimulus Prediction Across Modalities Improves Unisensory Classification. Frontiers in Robotics and AI, 6, 137.
  • Barros, P., Wermter, S., & Sciutti, A. (2019). Towards Learning How to Properly Play UNO with the iCub Robot. In Proceedings of the ICDL-EPIROB Workshop on Naturalistic Non-verbal and Affective Human-Robot Interactions. arXiv:1908.00744.
  • Barros, P., Parisi, G., & Wermter, S. (2019). A Personalized Affective Memory Model for Improving Emotion Recognition. In International Conference on Machine Learning (pp. 485-494).
  • Parisi, G., Barros, P., Fu, D., Magg, S., Wu, H., Liu, X., & Wermter, S. (2019). A Neurorobotic Experiment for Crossmodal Conflict Resolution. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops (pp. 33-36).
  • Lindt, A., Barros, P., Siqueira, H., & Wermter, S. (2019). Facial Expression Editing with Continuous Emotion Labels. In 2019 14th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2019) (pp. 1-8). IEEE.
  • Parisi, G. I., Kemker, R., Part, J. L., Kanan, C., & Wermter, S. (2019). Continual lifelong learning with neural networks: A review. Neural Networks.113，54-71.
  • Barros, P., Parisi, G. I., Fu, D., Liu, X., & Wermter, S. (2018). Expectation learning for adaptive crossmodal stimuli association. EUCognition Meeting, arXiv preprint arXiv:1801.07654.
  • Fu, D., Barros, P., Parisi, G. I., Wu, H., Magg, S., Liu, X., & Wermter, S. (2018). Assessing the Contribution of Semantic Congruency to Multisensory Integration and Conflict Resolution. arXiv preprint arXiv:1810.06748.
  • Barros, P., Parisi, G. I., Fu, D., Liu, X., & Wermter, S. (2018). Expectation Learning and Crossmodal Modulation with a Deep Adversarial Network. In 2018 International Joint Conference on Neural Networks (IJCNN), 1-8.
  • Mici, L., Parisi, G. I., & Wermter, S. (2018). An incremental self-organizing architecture for sensorimotor learning and prediction. IEEE Transactions on Cognitive and Developmental Systems, 10(4), 918-928
  • Cruz, F., Parisi, G. I., & Wermter, S. (2018). Multi-modal Feedback for Affordance-driven Interactive Reinforcement Learning. In 2018 International Joint Conference on Neural Networks (IJCNN), 1-8.
  • Churamani, N., Barros, P., Strahl, E., & Wermter, S. (2018). Learning empathy-driven emotion expressions using affective modulations. In 2018 International Joint Conference on Neural Networks (IJCNN) (pp. 1-8).
  • Parisi, G. I., Tong, J., Barros, P., Röder, B., & Wermter, S. (2018). Towards Modeling the Interaction of Spatial-Associative Neural Network Representations for Multisensory Perception. Workshop on Computational Models for Crossmodal Learning (ICDL-EpiRob).
  • Parisi, G. I., Barros, P., Fu, D., Magg, S., Wu, H., Liu, X., & Wermter, S. (2018). A neurorobotic experiment for crossmodal conflict resolution in complex environments. 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2330-2335.
  • Parisi, G.I., Barros, P., Kerzel, M., Wu, H., Yang, G., Li, Z., Liu, X., Wermter, S. (2017). A computational model of crossmodal processing for conflict resolution. Proc. International Conference on Development and Learning and on Epigenetic Robotics (ICDL-EPIROB), pages 33-38.
  • Parisi, G.I., Wermter, S. (2017). Lifelong learning of action representations with deep neural self-organization. AAAI Spring Symposium Series, pages 608-612.
  • Parisi, G.I., Barros, P., Wu, H., Yang, G., Li, Z., Liu, X., Wermter, S. (2017). A deep neural model for emotion-driven multimodal attention. AAAI Spring Symposium Series, pages 482-485.
  • Barros, P., Parisi, G.I., Wermter, S. (2017). Emotion-modulated attention improves expression recognition: A deep learning model. Neurocomputing, Volume 253, pages 104-114.
  • Barros, P., & Wermter, S. (2017).A self-organizing model for affective memory. International Joint Conference on Neural Networks (IJCNN), pp. 31-38.
  • Churamani, N., Kerzel, M., Strahl, E., Barros, P., & Wermter, S. (2017). Teaching emotion expressions to a human companion robot using deep neural architectures. International Joint Conference on Neural Networks (IJCNN), pp. 627-634.
  • Tsironi, E., Barros, P., Weber, C., & Wermter, S. (2017). An analysis of Convolutional Long Short-Term Memory Recurrent Neural Networks for gesture recognition. Neurocomputing.
  • Parisi, G.I., Tani, J., Weber, C., Wermter, S. (2017). Emergence of Multimodal Action Representations from Neural Network Self-Organization. Cognitive Systems Research
  • Cruz, F., Parisi, G. I., and Wermter, S. (2016). Learning Contextual Affordances with an Associative Neural Architecture. Proceedings of the European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning (ESANN), pp. 665-670, Bruges, Belgium, 2016.
  • Parisi, G.I., Wermter, S. A Neurocognitive Robot Assistant for Robust Event Detection. (2016). Trends in Ambient Intelligent Systems: Role of Computational Intelligence, Series "Studies in Computational Intelligence", pp. 1-28, Springer, 2016
  • Cruz, F., Parisi, G. I., & Wermter, S. (2016). Learning Contextual Affordances with an Associative Neural Architecture. Proceedings of the European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning (ESANN), 665-670, Bruges, Belgium.
  • Speck, D., Barros, P., Weber, C. and Wermter, S. (2016). Ball Localization for Robocup Soccer using Convolutional Neural Networks. RoboCup Symposium, Leipzig, Germany, 2016. - Best Paper Award
  • Barros, P. , Weber, C., Wermter, S. (2016). Learning Auditory Representations for Emotion Recognition. Proceedings of International Joint Conference on Neural Networks (IJCNN/WCCI), Vancouver, Canada, July 2016.
  • Mousavi, N., Siqueira, H., Barros, P., Fernandes, B., Wermter, S. (2016). Understanding How Deep Neural Networks Learn Face Expressions. Proceedings of International Joint Conference on Neural Networks (IJCNN/WCCI), Vancouver, Canada, July 2016.

Relevant previous work

  • Bauer, J., Dávila-Chacón, J., & Wermter, S. (2014). Modeling development of natural multi-sensory integration using neural self-organization and probabilistic population codes. Connection Science, 1-19.
  • Bauer, J., Magg, S., & Wermter, S. (2015). Attention modeled as information in learning multisensory integration. Neural Networks, 65, 44-52.
  • Dávila-Chacón, J., Magg, S., Liu, J., & Wermter, S. (2013). Neural and statistical processing of spatial cues for sound source localisation. In Proc. of Int. Joint Conf. on Neural Networks (pp. 1274-1281), IEEE.
  • Liu, X., Banich, M. T., Jacobson, B. L., & Tanabe, J. L. (2004). Common and distinct neural substrates of attentional control in an integrated Simon and spatial Stroop task as assessed by event-related fMRI. Neuroimage, 22(3), 1097-1106.
  • Liu, X., Banich, M. T., Jacobson, B. L., & Tanabe, J. L. (2006). Functional dissociation of attentional selection within PFC: response and non-response related aspects of attentional selection as ascertained by fMRI. Cereb Cortex, 16(6), 827-834.
  • Liu, X., Park, Y., Gu, X., & Fan, J. (2010). Dimensional overlap accounts for independence and integration of stimulus-response compatibility effects. Attention Perception & Psychophysics, 72(6), 1710-1720.
  • Murray, J., Erwin, H., & Wermter, S. (2009). Robotic sound-source localisation architecture using cross-correlation and recurrent neural networks. Neural Networks, 22(2). 173–189.
  • Parisi, GI., Weber, C., &Wermter, S. (2015). Self-organizing neural integration of pose-motion features for human action recognition. Frontiers in Neurorobotics. from: http://journal.frontiersin.org/article/10.3389/fnbot.2015.00003/
  • Wermter, S., & Elshaw, M. (2003). Learning robot actions based on self-organising language memory. Neural Networks, 16(5), 691–699.
  • Zhong, J., Weber, C., & Wermter, S. (2013). A predictive network architecture for a robust and smooth robot docking behavior. Journal of Behavioral Robotics, 3(4), 172-180.

Publications project A6 - Brain-inspired multimodal deep learning

Publications of the current project

  • Xiaolin Hu, Zhigang Zeng, “Bridging the functional and wiring properties of V1 neurons through sparse coding, Neural Computation. 1-34, 2021. doi 10.1162/neco_a_01453
  • Jianjin Xu, Zheyang Xiong, Xiaolin Hu (2021). Frame difference-based temporal loss for video stylization, preprint, arXiv:2102.05822, 2021.
  • Xiao Li, Jianmin Li, Ting Dai, Jie Shi, Jun Zhu, Xiaolin Hu (2021). Rethinking Natural Adversarial Examples for Classification Models, preprint arXiv:2102.1173.
  • Gang Zhang, Xin Lu, Jingru Tan, Jianmin Li, Zhaoxiang Zhang, Quanquan Li, Xiaolin Hu (2021). RefineMask: Towards high-quality instance segmentation with fine-grained features, Proc. of the 34th IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Virtual, June 19-25, 2021.
  • Chufeng Tang, Hang Chen, Xiao Li, Jianmin Li, Zhaoxiang Zhang, Xiaolin Hu (2021). Look closer to segment better: boundary patch refinement for instance segmentation, Proc. of the 34th IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Virtual, June 19-25, 2021.
  • Han Liu, Shifeng Zhang, Ke Lin, Jing Wen, Jianmin Li, Xiaolin Hu (2021). Vocabulary-wide credit assignment for training image captioning models, IEEE Transactions on Image Processing, vol. 30, pp. 2450-2460, 2021.
  • Xiaopei Zhu, Xiao Li, Jianmin Li, Zheyao Wang, Xiaolin Hu (2021). Fooling thermal infrared pedestrian detectors in real world using small bulbs, The Thirty-Fifth AAAI Conference on Artificial Intelligence (AAAI), Virtual, Feb 2-9, 2021.
  • Haoran Chen, Jianmin Li, Simone Frintrop, and Xiaolin Hu (2021). Annotation cleaning for the MSR-Video to Text dataset, arXiv:2102.06448, 2021 (under review).
  • Hu, Li, Zhang, Luo, Lemercier, Gerkmann (2021). Speech Separation Using an Asynchronous Fully Recurrent Convolutional Neural Network, NeurIPS, Dec. 2021.
  • Ge Gao, Mikko Lauri, Xiaolin Hu, Jianwei Zhang, Simone Frintrop (2021). CloudAAE: Learning 6D Object Pose Regression with On-line Data Synthesis on Point Clouds, Proceedings of the International Conference on Robotics and Automation (ICRA) 2021.
  • Haoran Chen, Ke Lin, Alexander Maye, Jianming Li, and Xiaolin Hu (2020). A Semantics-Assisted Video Captioning Model Trained with Scheduled Sampling, Frontiers in Robotics and AI, September 30, 2020
  • Ge Gao, Mikko Lauri, Yulong Wang, Xiaolin Hu, Jianwei Zhang and Simone Frintrop. (2020). 6D Object Pose Regression via Supervised Learning on Point Clouds. Proceedings of the IEEE International Conference on Robotics and Automation (ICRA) 2020, Paris, France. [Arxiv preprint 2001.08942], [Code].
  • Carbajal, Richter, and Gerkmann (2021). Guided Variational Autoencoder for Speech Enhancement with a Supervised Classifier, IEEE Int. Conf. Acoust., Speech, Signal Process. (ICASSP-2021), Toronto, Ontario, Canada, Jun. 2021.
  • Caus, Carbajal, Gerkmann, Frintrop (2021). See the silence: improving visual-only voice activity detection by optical flow and RGB fusion, Int. Conference on Computer Vision Systems (ICVS-2021), Sep. 2021.
  • Carbajal, Richter, and Gerkmann (2021). Disentanglement Learning for Variational Autoencoders Applied to Audio-Visual Speech Enhancement, IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA), New Paltz, NY, USA, Oct. 2021.
  • Fang, Carbajal, Wermter, Gerkmann (2021). Variational Autoencoder for Speech Enhancement With a Noise-Aware Encoder, IEEE Int. Conf. Acoust., Speech, Signal Process. (ICASSP-2021), Toronto, Ontario, Canada, Jun. 2021.
  • Fang, Carbajal, Wermter, Gerkmann (2021). Joint Reduction of Ego-noise and Environmental Noise with a Partially-adaptive Dictionary, ITG Conference on Speech Communication, Kiel, Germany, Sep. 2021.
  • Julius Richter, Guillaume Carbajal, Timo Gerkmann (2020). Speech Enhancement with Stochastic Temporal Convolutional Networks", ISCA Interspeech, Shanghai, China, Oct. 2020.
  • Nguyen, Richter, Lauri, Gerkmann, Frintrop (2020). Improving mix-and-separate training in audio-visual sound source separation with an object prior, ICPR 2020.
  • Li, Rehr, Bruns, Gerkmann, Röder (2020). A survey on probabilistic models in human perception and machines, Frontiers in Robotics and AI, Vol. 7, No. 85, pp. 1-9, Jul. 2020.
  • Liang, Zhou, Li, Ma, Hendrich, Gerkmann, Sun, Stoffel, Zhang (2020). Robust Robotic Pouring Using Audition and Haptics, IEEE/RSJ IROS 2020, Las Vegas, USA, Oct 2020.

Publications from phase 1

  • Feng, W., Liu, W., Li, T., Peng, J., Qian, C., & Hu, X. (2019). Turbo Learning Framework for Human-Object Interactions Recognition and Human Pose Estimation. The Thirty-Third AAAI Conference on Artificial Intelligence (AAAI), Honolulu, Hawaii, USA.
  • Yu, M., Weber, C., Hu, X. (2019). Learning Sparse Hidden States in Long Short-Term Memory. In Proc. of the International Conference on Artificial Neural Networks (ICANN2019)
  • Liao, F., Liang, M., Li, Z., Hu, X., & Song, S. (2019). Evaluate the Malignancy of Pulmonary Nodules Using the 3-D Deep Leaky Noisy-or Network. IEEE transactions on neural networks and learning systems.
  • Liu, W., Chen, J., Li, C., Qian, C., Chu, X., & Hu, X. (2018). A Cascaded Inception of Inception Network with Attention Modulated Feature Fusion for Human Pose Estimation. The Thirty-Second AAAI Conference on Artificial Intelligence.
  • Xiong, Z., Weber, C., & Hu, X. (2018). Frame Difference-Based Real-Time Video Stylization in Video Calls. In 2018 Ninth International Conference on Intelligent Control and Information Processing (ICICIP), 333-339.
  • Wang, Y., Su, H., Zhang, B., & Hu, X. (2018). Interpret neural networks by identifying critical data routing paths. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 8906-8914.
  • Liao, F., Liang, M., Dong, Y., Pang, T., Hu, X., & Zhu, J. (2018). Defense against adversarial attacks using high-level representation guided denoiser. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 1778-1787.
  • Springenberg, S., Lakomkin, E., Weber, C., & Wermter, S. (2018). Image-to-Text Transduction with Spatial Self-Attention. In 2018 proceedings, European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning (ESANN).
  • Dong, Y., Liao, F., Pang, T., Su, H., Zhu, J., Hu, X., & Li, J. (2018). Boosting adversarial attacks with momentum. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 9185-9193).
  • Sun, T., Wang, Y., Yang, J., Hu, X., (2017). Convolutional neural networks with two pathways for image style recognition, IEEE Transactions on Image Processing, vol. 26, no. 9, pp. 4102-4113, 2017.
  • Wang, J., & Hu, X. (2017). Gated recurrent convolution neural network for ocr. In International Conference on Advances in Neural Information Processing Systems(NeurIPS), 335-344.
  • Hu, Z., Wang. T., Hu, X., (2017). An STDP-based supervised learning algorithm for spiking neural networks, Proc. of 24th International Conference on Neural Information Processing (ICONIP), Guangzhou, China, Nov. 14-18, 2017.
  • Hu, X., Wang, T., (2017). Training the Hopfield Neural Network for Classification Using a STDP-Like Rule, Proc. of 24th International Conference on Neural Information Processing (ICONIP), Guangzhou, China, Nov. 14-18, 2017.
  • Yu, N., Qiu, S., Hu, X., Li, J., (2017). Accelerating Convolutional Neural Networks by Group-wise 2D-filter Pruning, Proc. of International Joint Conference on Neural Networks (IJCNN), Anchorage, Alaska, USA, May 14–19, 2017, pp. 2502-2509.
  • Wu, J., Ma, L., Hu, X., (2017). Delving deeper into convolutional neural networks for camera relocalization, Proc. of IEEE International Conference on Robotics and Automation (ICRA), Singapore, May 29- June 3, 2017, pp. 5644-5651.
  • Zhao, Y., Jin, X., Hu, X., (2017). Recurrent convolutional neural network for speech processing, Proc. of the 42nd IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), New Orleans, USA, March 5-9, 2017.
  • Zhou, X., Weber, C., Wermter, S. (2017). Robot Localization and Orientation Detection based on Place Cells and Head-direction Cells. Proceedings of the 26th International Conference on Artificial Neural Networks (ICANN 2017), Sep 2017.
  • Zhuang, C., Wang, Y., Yamins, D., & Hu, X. (2017). Deep learning predicts correlation between a functional signature of higher visual areas and sparse firing of neurons. Frontiers in Computational Neuroscience, 11, 100.
  • Chen, X., Hu, X., Zhou, H., & Xu, N. (2017, May). Fxpnet: Training a deep convolutional neural network in fixed-point representation. In 2017 International Joint Conference on Neural Networks (IJCNN), 2494-2501.
  • Wu, J., Ma, L., Hu, X., (2016). Predicting world coordinates of pixels in RGB images using convolutional neural network for camera relocalization, Proc. of the Seventh International Conference on Intelligent Control and Information Processing (ICICIP), Siem Reap, Cambodia, December 1-4, 2016.
  • Qin H., Yan J., Li X., Hu X., (2016). Joint Training of Cascaded CNN for Face Detection, Proc. of the 28th IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016, pp. 3456-3465.

Relevant previous work

  • Heinrich, S., Weber, C., & Wermter, S. (2013). Embodied language understanding with a multiple timescale recurrent neural network. In Artificial Neural Networks and Mahine Learning-ICANN 2013 (pp. 216-223). Springer Berlin Heidelberg.
  • Hu, X., Zhang, J., Qi, P., & Zhang, B. (2014). Modeling response properties of V2 neurons using a hierarchical K-means model. Neurocomputing, 134, 198-205.
  • Hu, X., Sun, C., & Zhang, B. (2010). Design of recurrent neural networks for solving constrained least absolute deviation problems. IEEE Transactions on Neural Networks, 21(7), 1073-1086.
  • Hu, X., & Zhang, B. (2009). An alternative recurrent neural network for solving variational inequalities and related optimization problems. IEEE Transactions on Systems, Man, and Cybernetics, Part B: Cybernetics, 39(6), 1640-1645.
  • Hu, X., & Zhang, B. (2010). A Gaussian attractor network for memory and recognition with experience-dependent Learning. Neural Computation, 22(5), 1333-1357.
  • Liang, M., & Hu, X. (2015). Predicting eye fixations with higher-level visual features. IEEE Transactions on Image Processing, 24(3), 1178-1189.
  • Liang, M., & Hu, X. (2015). Recurrent convolutional neural network for object recognition. In Proc. of the 28th IEEE Conference on Computer Vision and Pattern Recognition (pp. 3367-3375). IEEE.
  • Weber, C., & Triesch, J. (2008). A sparse generative model of V1 simple cells with intrinsic plasticity. Neural Computation, 20(5), 1261-84.
  • Wu, Y., Liu, Y., Li, J., Liu, H., & Hu, X. (2013). Traffic sign detection based on convolutional neural networks. In the 2013 International Joint Conference on Neural Network (IJCNN) (pp. 1-7). IEEE.
  • Zhong, J., Weber, C., & Wermter, S. (2013). A predictive network architecture for a robust and smooth robot docking behavior. Paladyn, Journal of Behavioral Robotics, 3(4), 172-180.