Publications - Thematic Area A

Publications project A1 - Dynamic adaptation of multisensory processes by crossmodal recalibration

Publications of the current project

Zierul, B., Tong, J., Bruns, P., & Röder, B. 2019. (in press). Reduced multisensory integration of self-initiated stimuli, Cognition, 182, 349-359.

Bruns, P., Röder, B. (2019). Cross-modal learning in the auditory system. In A. K. C. Lee, M. Wallace, A. Coffin, A. N. Popper, & R. R. Fay (Eds.), Multisensory processes: The auditory perspective. Springer Handbook of Auditory Research.

Bruns, P., Röder, B. (2019). Repeated but not incremental training enhances cross-modal recalibration. Journal of Experimental Psychology: Human Perception and Performance, 45(4), 435.

Badde, S., Röder, B., & Bruns, P. (2018). Task-irrelevant sounds influence both temporal order and apparent motion judgments about tactile stimuli applied to crossed and uncrossed hands. Attention, Perception, & Psychophysics, 80(3), 773-783.

Bruns, P., Röder, B., (2017). Spatial and frequency specificity of the ventriloquism aftereffect revisited. Psychological Research, 1-16.

Katzakis, N., Tong, J., Nunez, O., Chen, L., Klinker, G., Röder, B., & Steinicke, F. (2017). Stylo and handifact: modulating haptic perception through visualisations for posture training in augmented reality. Proc. Spatial User Interaction, doi: 10.1145/3131277.3132181

Zierul, B., Röder, B., Tempelmann, C., Bruns, P., and Noesselt, T. (2017) The Role of Auditory Cortex in the Spatial Ventriloquism Aftereffect. NeuroImage, 162, 257-268.

Zhang, D., Hong, B., Gao, S., & Röder, B. (2017). Exploring the temporal dynamics of sustained and transient spatial attention using steady-state visual evoked potentials. Experimental Brain Research, 2017 May; 235(5):1575-1591. doi: 10.1007/s00221-017-4907-6.

NEUROIMAGE

Relevant previous work

Badde, S., Röder, B., Heed, T. (2014). Multiple spatial representations determine touch localization. Journal of Experimental Psychology: Human Perception and Performance, 40 (2), 784-801.

Bruns, P., Maiworm, M., & Röder, B. (2014) Reward expectation influences audiovisual spatial integration. Attention, Perception, & Psychophysics, 76, 1815-1827.

Bruns, P., Liebnau, R., & Röder, B. (2011). Crossmodal training induces changes in spatial representations early in the auditory processing pathway. Psychological Science, 22(9), 1120-1126.

Bruns, P. & Röder, B. (2015). Sensory recalibration integrates information from the immediate and the cumulative past. Scientific Reports, in press.

Bruns, P., Spence, C., & Röder, B. (2011). Tactile recalibration of auditory spatial representations. Experimental Brain Research, 209 (3), 333-344.

Bruns, P., & Röder, B. (2010). Tactile capture of auditory localization: An event-related potential study. European Journal of Neuroscience, 31, 1844-1857.

Heed, T., Buchholz, V., Engel, A., & Röder, B. (2015). Tactile remapping: from coordinate transformation to integration in sensorimotor processing. Trends in Cognitive Sciences, 19(5), 251-258.

Heed, T., Röder, B. (2014). Motor coordination uses external spatial coordinates independent of developmental vision. Cognition, 132, 1-15.

Maiworm, M., Bellantoni, M., Spence, C., & Röder, B. (2012). When emotional valence modulates audiovisual integration. Attention, Perception & Psychophysics, 74 (6), 1302-1311.

Zhang, D., Hong, B., Gao, X., Gao, S., Röder, B. (2011). Exploring steady-state visual evoked potentials as an index for intermodal and crossmodal spatial attention. Psychophysiology, 48, 665-675.

Publications project A2 - Using multiperturbation analysis to reveal the neural circuit basis of multi-sensory processing

Publications of the current project

Damicelli, F., Hilgetag, C. C., Hütt, M. T., & Messé, A. (2019). Topological reinforcement as a principle of modularity emergence in brain networks. Network Neuroscience, 3(2), 1-34.

Wang, G., Xie, H., Wang, L., Luo, W., Wang, Y., Jiang, J., ... & Guan, J. S. (2019). Switching From Fear to No Fear by Different Neural Ensembles in Mouse Retrosplenial Cortex. Cerebral Cortex pii: bhz050. doi: 10.1093/cercor/bhz050

Li, D., Zavaglia, M., Wang, G., Xie, H., Hu, Y., Werner, R., ... & Hilgetag, C. C. (2019). Discrimination of the hierarchical structure of cortical layers in 2-photon microscopy data by combined unsupervised and supervised machine learning. Scientific Reports, 9(1), 7424.

Goulas A, Zilles K, Hilgetag CC. (2018). Cortical gradients and laminar projections in mammals. Trends in Neuroscience, 41(11), 775-788.

Luo, W., & Guan. J. S. (2018). Do brain oscillations orchestrate memory? Brain Science Advances, 4(1), 16-33.

Jiang.J. Guanyu Wang, Ji-Song Guan. (2018). Mammillary body regulates state-dependent fear by alternating cortical oscillations Sci Rep, 8(1), 13471.

Messé A, Hütt MT, Hilgetag CC (2018). Toward a theory of coactivation patterns in excitable neural networks. PLoS Comput Biollgy, 14(4):e1006084.

Rollenhagen A, Ohana O, Sätzler K, Hilgetag C, Kuhl D, Lübke J (2018) Structural Properties of Synaptic Transmission and Temporal Dynamics at Excitatory Layer 5B Synapses in the Adult Rat Somatosensory Cortex. Front Synaptic Neurosci, 10:24.

Zou Y, Zhao Z, Yin D, Fan M, Small M, Liu Z, Hilgetag C, Kurths (2018) Brain anomaly networks uncover heterogeneous functional reorganization patterns after stroke Neuroimage-Clin. 20:523-530

Ruxing Fu, Wenhan Luo, Roy Nazempour, Daxin Tan, He Ding, Kaiyuan Zhang, Lan Yin, Ji-Song Guan*, Xin Sheng*. (2018). Implantable and Biodegradable Poly(L-lactic acid) Fibers for Optical Neural Interfaces, (2018), Adv Optical Materials. DOI: 10.1002/adom.201700941

Malherbe C, Umarova R, Zavaglia M, Kaller C, Beume L, Thomalla G, Weiller C, Hilgetag C (2018). Neural correlates of visuospatial bias in patients with left hemisphere stroke: a causal functional contribution analysis based on game theory, Neuropsychologia, 115: 142-153.

Fretter C, Lesne A, Hilgetag CC, Hütt MT. (2017). Topological determinants of self-sustained activity in a simple model of excitable dynamics on graphs. Sci Rep 7:42340.

Toba M, Zavaglia M, Rastelli F, Valabrégue R, Pradat-Diehl P, Valero-Cabré A, Hilgetag CC. (2017). Game theoretical mapping of causal interactions underlying visuo-spatial attention in the human brain based on stroke lesions. Hum Brain Mapp. doi: 10.1002/hbm.23601.

Damicelli F, Hilgetag CC, Hütt MT, Messé A. (2017). Modular topology emerges from plasticity in a minimalistic excitable network model. Chaos. 27(4):047406.

Chen Y, Wang S, Hilgetag CC, Zhou C. (2017). Features of spatial and functional segregation and integration of the primate connectome revealed by trade-off between wiring cost and efficiency. PLoS Comput Biol 13(9):e1005776.

Maier-Hein KH, Neher PF, ..., Hilgetag CC, Stieltjes B, Descoteaux M. (2017). The challenge of mapping the human connectome based on diffusion tractography. Nat Commun 8(1):1349.

Stellmann J, Hodecker S, Cheng B, Wanke N, Young K, Hilgetag C, Gerloff C, Heesen C, Thomalla G, Siemonsen S (2017) Reduced rich-club connectivity is related to disability in primary progressive MS Neurol Neuroimmunol Neuroinflamm. 4(5):e375

Goulas A, Uylings H, Hilgetag C (2017) Principles of ipsilateral and contralateral cortico-cortical connectivity in the mouse BRAIN STRUCT FUNCT. 222(3):1281-1295

Ding X, Liu S, Tian M, Zhang W, Zhu T, Li D, Wu J, Deng H, Jia Y, Xie W, Xie H, Guan JS (2017). Activity-induced histone modifications govern Neurexin-1 mRNA splicing and memory preservation. Nat Neurosci. 20(5):690-699.

Zavaglia M, Forkert N, Cheng B, Gerloff C, Thomalla G, Hilgetag C (2016). Technical considerations of a game-theoretical approach for lesion symptom mapping, BMC Neurosci 17:40

Hilgetag C, Amunts K (2016) Connectivity and cortical architecture e-Neuroforum. 7(3):56-63.

Hilgetag C, Medalla M, Beul S, Barbas H (2016) The primate connectome in context: Principles of connections of the cortical visual system. Neuroimage. 134:685-702.

Relevant previous work

Gräff J, Rei D., Guan J. S., Wang W. Y., Seo J., Hennig K. M., Nieland T. J., Fass D. M., Kao P. F., Kahn M., Su S. C., Samiei A., Joseph N., Haggarty S. J., Delalle I., & Tsai L. H. (2012). An epigenetic blockade of cognitive functions in the neurodegenerating brain, Nature, 483:222.

Guan J. S., Haggarty S. J., Giacometti E., Dannenberg J. H., Joseph N., Gao J., Nieland T. J., Zhou Y., Wang X., Mazitschek R., Bradner J. E., DePinho R. A., Jaenisch R., & Tsai L. H. (2009). HDAC2 negatively regulates memory formation and synaptic plasticity, Nature, 459: 55.

Hilgetag C. C., O'Neill M. A., & Young M. P. (1996). Indeterminate organization of the visual system, Science, 271: 776.

Hilgetag C. C., Théoret H., & Pascual-Leone A. (2001). Enhanced visual spatial attention ipsilateral to rTMS-induced ‘virtual lesions’ of human parietal cortex, Nature Neuroscience, 4, 953.

Keinan A., Sandbank B., & Hilgetag C. C. (2004). Meilijson I, Ruppin E: Fair attribution of functional contribution in artificial and biological networks, Neural Computation, 16: 1887.

Keinan A., Sandbank B., & Hilgetag C. C., Meilijson I, Ruppin E (2006). Axiomatic scalable neurocontroller analysis via the Shapley value, Artificial Life, 12: 333.

Sporns O., Chialvo D., Kaiser M., & Hilgetag C. C. (2004). Organization, development and function of complex brain networks, Trends in Cognitive Sciences, 8: 418-425.

Xie H., Liu Y., & Guan J. S. (2014). In vivo imaging of immediate early gene expression reveals layer-specific memory traces in the mammalian brain, PNAS, 111(7): 2788.

Young M. P., Hilgetag C. C., & Scannell J. W. (2000). On imputing function to structure from the behavioural effects of brain lesions, Phil Trans R Soc Lond B, 355: 147.

Zavaglia M., & Hilgetag C. C. (2015). Causal functional contributions and interactions in the attention network of the brain: An objective multi-perturbation analysis, Brain Struct Funct, DOI: 10.1007/s00429-015-1058-z.

Publications project A3 - Multisensory integration in the aging brain — mechanisms and facilitation

Publications of the current project

Higgen F.L., Heine C., Krawinkel L., Göschl F., Engel A.K., Hummel F.C., Xue Gui, Gerloff C. (2020). Crossmodal Congruency Enhances Performance of Healthy Older Adults in Visual-Tactile Pattern Matching. Frontiers in Aging Neuroscience 12, p. 74, 2020. doi: 10.3389/fnagi.2020.00074

Higgen F.L., Braaß H., Schulz R., Xue G., Gerloff C. (2019). P35 A complex tactile recognition task as a marker for the aging brain. Clinical Neurophysiology 2019; 130(8): e162-e163. doi: 10.1016/j.clinph.2019.04.688

Huang, X., Zhang, H., Chen, C., Xue, G., He, Q. (2019). The neuroanatomical basis of the Gambler’s fallacy: A univariate and multivariate morphometric study. Hum Brain Mapp 40, 967–975. doi: https://doi.org/10.1002/hbm.24425

Zhu B., Chen C., Shao X., Liu W., Ye Z., Zhuang L., Zheng L., Loftus E.F., Xue G. (2019). Multiple interactive memory representations underlie the induction of false memory. Proceedings of the National Academy of Sciences U S A, 116, 3466-3475.

Cai, Y., Urgolites, Z., Wood, J., Chen, C., Li, S., Chen, A., Xue, G. (2018). Distinct neural substrates for visual short-term memory of actions. Hum Brain Mapp 39, 4119–4133. https://doi.org/10.1002/hbm.24236

Xue, G. (2018). The Neural Representations Underlying Human Episodic Memory. Trends Cogn. Sci. (Regul. Ed.) 22, 544–561. doi: https://doi.org/10.1016/j.tics.2018.03.004

Higgen F.L., Ruppel P., Görner M., Kerzel M., Magg S., Hendrich N. (2018). Crossmodal pattern discrimination in humans and robots: a visuo-tactile case study. Workshop on Crossmodal Learning, IROS.2018.

Qu, J., Qian, L., Chen, C., Xue, G., Li, H., Xie, P., Mei, L. (2017). Neural Pattern Similarity in the Left IFG and Fusiform Is Associated with Novel Word Learning. Front Hum Neurosci 11, 424. doi: https://doi.org/10.3389/fnhum.2017.00424

Relevant previous work

Cohen, L. G., Celnik, P., Pascual-Leone, A., Corwell, B., Falz, L., Dambrosia, J., Honda, M., Sadato, N., Gerloff, C., Catalá, M.D., & Hallett, M. (1997). Functional relevance of crossmodal plasticity in blind humans. Nature, 389(6647), 180-183.

Heise, K. F., Zimerman, M., Hoppe, J., Gerloff, C., Wegscheider, K., & Hummel, F. C. (2013). The aging motor system as a model for plastic changes of GABA-mediated intracortical inhibition and their behavioral relevance. J Neurosci, 33(21), 9039-9049.

Hummel, F., & Gerloff, C. (2005). Larger interregional synchrony is associated with greater behavioral success in a complex sensory integration task in humans. Cereb Cortex, 15(5), 670-678.

Hummel, F. C., Heise, K., Celnik, P., Floel, A., Gerloff, C., & Cohen, L. G. (2010). Facilitating skilled right hand motor function in older subjects by anodal polarization over the left primary motor cortex. Neurobiol Aging, 31(12), 2160-2168.

Plewnia, C., Rilk, A. J., Soekadar, S. R., Arfeller, C., Huber, H. S., Sauseng, P., Hummel, F.C., & Gerloff, C. (2008). Enhancement of long-range EEG coherence by synchronous bifocal transcranial magnetic stimulation. Eur J Neurosci, 27(6), 1577-1583.

Renzi, C., Bruns, P., Heise, K. F., Zimerman, M., Feldheim, J. F., Hummel, F. C., & Roder, B. (2013). Spatial remapping in the audio-tactile ventriloquism effect: a TMS investigation on the role of the ventral intraparietal area. J Cogn Neurosci, 25(5), 790-801.

Sauseng, P., Klimesch, W., Heise, K. F., Gruber, W. R., Holz, E., Karim, A. A., Glennon, M., Gerloff, C., Birbaumer, N., & Hummel, F. C. (2009). Brain oscillatory substrates of visual short-term memory capacity. Curr Biol, 19(21), 1846-1852.

Schulz, R., Zimerman, M., Timmermann, J. E., Wessel, M. J., Gerloff, C., & Hummel, F. C. (2014). White matter integrity of motor connections related to training gains in healthy aging. Neurobiol Aging, 35(6), 1404-1411.

Xue, G., Dong, Q., Chen, C., Lu, Z., Mumford, J. A., & Poldrack, R. A. (2010). Greater neural pattern similarity across repetitions is associated with better memory. Science, 330(6000), 97-101.

Zimerman, M., Nitsch, M., Giraux, P., Gerloff, C., Cohen, L. G., & Hummel, F. C. (2013). Neuroenhancement of the aging brain: restoring skill acquisition in old subjects. Ann Neurol, 73(1), 10-15.

Publications project A4 - Crossmodal joint sparse feature learning in robot tasks

Publications of the current project

Liang, H., Li, S., Ma, X., Hendrich, N., Gerkmann T., Sun, F., & Zhang, J. (2019). Making sense of Audio Vibration for Liquid Height Estimation in Robotic Pouring. International Conference on Intelligent Robots and Systems (IROS), Macau, China (in press).

Yu, X., Guo, T., Fu, K, Li, L, Zhang, C. and Zhang J. (2019). Image Captioning with Partially Rewarded Imitation Learning. International Joint Conference on Neural Network (IJCNN), Budapest, Hungary.

Tang, S., Ji, Y., Lyu, J., Mi, J., Li, Q., & Zhang, J. (2019). Visual Domain Adaptation Exploiting Confidence-Samples. International Conference on Intelligent Robots and Systems (IROS), Macau, China (in press).

Chang, D., Lin, M., & Zhang, C. (2018). On the Generalization Ability of Online Gradient Descent Algorithm Under the Quadratic Growth Condition. IEEE Transactions on Neural Networks and Learning Systems, 29(10), 1-12.

Lu R., Duan Z. and Zhang C. (2018). Listen and Look: Audio–Visual Matching Assisted Speech Source Separation. IEEE Signal Processing Letters.

Lu, R., Duan, Z., & Zhang, C. (2018). Multi-Scale Recurrent Neural Network for Sound Event Detection. 2018 In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 131-135.

Lu, J., Li, J., Yan, Z., Mei, F., & Zhang, C. (2018). Attribute-Based Synthetic Network (ABS-Net): Learning more from pseudo feature representations. Pattern Recognition, 80, 129-142.

Lu, J., Cao, Z., Wu, K., Zhang, G., & Zhang, C. (2018). Boosting Few-Shot Image Recognition Via Domain Alignment Prototypical Networks. In 2018 IEEE 30th International Conference on Tools with Artificial Intelligence (ICTAI), 260-264.

Yan, C., Wu, K., & Zhang, C. (2018). A New Anchor-Labeling Method For Oriented Text Detection Using Dense Detection Framework. IEEE Signal Processing Letters, 25(9), 1295-1299.

Yan, C., Hu, J., & Zhang, C. (2018). Deep transformer: A framework for 2D text image rectification from planar transformations. Neurocomputing, 289, 32-43.

Chu, D., Lu, R., Li, J., Yu, X., Zhang, C., & Tao, Q. (2018). Optimizing Top-k Multiclass SVM via Semismooth Newton Algorithm. IEEE Transactions on Neural Networks and Learning Systems, 29(12), 6264-6275.

Fu, K., Li, J., Jin, J., & Zhang, C. (2018). Image-text surgery: Efficient concept learning in image captioning by generating pseudopairs. IEEE Transactions on Neural Networks and Learning Systems, (99), 1-12.

Yu, N., Hu, X., Song, B., Yang, J., & Zhang, J. (2018). Topic-Oriented Image Captioning Based on Order-Embedding. IEEE Transactions on Image Processing, 28(6), 2743-2754.

Ruppel, P., Jonetzko, Y., Görner, M., Hendrich, N., & Zhang, J.(2018). Simulation of the SynTouch BioTac Sensor. International Conference on Intelligent Autonomous Systems. DOI: 10.1007/978-3-030-01370-7_30

Ruppel, P., Hendrich, N., Starke, S., & Zhang, J. (2018). Cost functions to specify full-body motion and multi-goal manipulation tasks. In 2018 IEEE International Conference on Robotics and Automation (ICRA), 3152-3159.

Wu, K., & Zhang, C. (2018). Deep Generative Adversarial Networks for the Sparse Signal Denoising. In 2018 24th International Conference on Pattern Recognition (ICPR), 1127-1132.

Chu, D., Zhang, C., & Tao, Q. (2017). A faster cutting plane algorithm with accelerated line search for linear SVM. Pattern Recognition, 67, 127-138.

Liu, L., He, B., Zhuang, J., Zhang, L., & Lv, A. (2017). Force measurement system for invisalign based on thin film single force sensor. Measurement, 97, 1-7.

Hu, W., Zhuo, Q., Zhang, C., & Li, J. (2017). Fast branch convolutional neural network for traffic sign recognition. IEEE Intelligent Transportation Systems Magazine, 9(3), 114-126.

Lu, J., Hu, J., Zhao, G., Mei, F., & Zhang, C. (2017). An in-field automatic wheat disease diagnosis system. Computers and Electronics in Agriculture, 142, 369-379.

Lu, R., Wu, K., Duan, Z., & Zhang, C. (2017, March). Deep ranking: Triplet matchnet for music metric learning. In 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 121-125.

Cui, R., Liu, H., & Zhang, C. (2017). Recurrent convolutional neural networks for continuous sign language recognition by staged optimization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 7361-7369.

Tang, S., Chen, L., Mi, J., Ye, M., Li, Q., & Zhang, J. (2017). Adaptive pedestrian detection by modulating features in dynamical environment. IEEE International Conference on Robotics and Biomimetics (ROBIO), 62-67.

Liu, Z., Li, J., Shen, Z., Huang, G., Yan, S., & Zhang, C. (2017). Learning efficient convolutional networks through network slimming. In Proceedings of the IEEE International Conference on Computer Vision（ICCV）. 2736-2744.

Fu, K., Jin, J., Cui, R., Sha, F., & Zhang, C. (2016). Aligning where to see and what to tell: image captioning with region-based attention and scene-specific contexts. IEEE Transactions on Pattern Analysis and Machine Intelligence, 39(12), 2321-2334.

Guan, H., & Zhang, J. (2016). Multi-sensory based novel household object categorization system by using interactive behaviours. In 2016 IEEE International Conference on Robotics and Biomimetics (ROBIO), 1685-1690.

Zhang, L., He, B., Zhang, J., (2016). Multimodal Information Representation Based on Approximation Theory (in Chinese). Science Press, ISBN: 978-7-03-050253-7.

Relevant previous work

Gong, P., Zhang, C., Lu, Z., Huang, J.Z., & Ye, J. (2013). A general iterative shrinkage and thresholding algorithm for non-convex regularized optimization problems. In Machine learning: Proceedings of the International Conference on Machine Learning (pp. 37). NIH Public Access.

Gong, P., Ye, J., & Zhang, C.S. (2012). Multi-stage multi-task feature learning. In Advances in Neural Information Processing Systems (pp. 1988-1996).

Gong, P., & Zhang, C. (2011). A fast dual projected Newton Method for L1-regularized least squares. In The 22nd International Joint Conference on Artificial Intelligenc (IJCAI) (pp. 1275-1280). IEEE.

He, J., & Zhang, J. (2014). In-hand haptic perception in dexterous manipulations. Science China Information Sciences, 57(12), 1-11.

Hu, X., Zhang, J., Li, J., & Zhang, B. (2014). Sparsity-regularized HMAX for visual recognition. PLOS ONE, 9(1), 1–12.

Bai, T., Li, Y., & Zhou, X. (2014). Monocular human motion tracking with discriminative sparse representation. Advanced Robotics, 28(6), 403-414.

Pan, Z., & Zhang, C. (2015). Relaxed sparse eigenvalue conditions for sparse estimation via non-convex regularized regression. Pattern Recognition, 48(1), 231-243.

Pan, Z., Lin, M., Hou, G., Zhang, C. et al. (2015). Damping proximal coordinate descent algorithm for non-convex regularization. Neurocomputing, 152, 151-163.

Yu, J., Ding, R., Yang, Q., Tan, M., Wang, W., & Zhang, J. (2012). On a bio-inspired amphibious robot capable of multimodal motion. IEEE/ASME Transactions on Mechatronics, 17(5), 847-856.

Zhang, J., & Rössler, B. (2004). Self-valuing learning and generalization with application in visually guided grasping of complex objects. Journal of Robotics and Autonomous Systems, 47(2), 117-127.

Publications project A5 - Crossmodal learning in a neurobotic cortical and midbrain model

Publications of the current project

Barros, P., Wermter, S., & Sciutti, A. (2019). Towards Learning How to Properly Play UNO with the iCub Robot. In Proceedings of the ICDL-EPIROB Workshop on Naturalistic Non-verbal and Affective Human-Robot Interactions. arXiv:1908.00744.

Barros, P., Parisi, G., & Wermter, S. (2019). A Personalized Affective Memory Model for Improving Emotion Recognition. In International Conference on Machine Learning (pp. 485-494).

Parisi, G., Barros, P., Fu, D., Magg, S., Wu, H., Liu, X., & Wermter, S. (2019). A Neurorobotic Experiment for Crossmodal Conflict Resolution. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops (pp. 33-36).

Lindt, A., Barros, P., Siqueira, H., & Wermter, S. (2019). Facial Expression Editing with Continuous Emotion Labels. In 2019 14th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2019) (pp. 1-8). IEEE.

Parisi, G. I., Kemker, R., Part, J. L., Kanan, C., & Wermter, S. (2019). Continual lifelong learning with neural networks: A review. Neural Networks.113，54-71.

Barros, P., Parisi, G. I., Fu, D., Liu, X., & Wermter, S. (2018). Expectation learning for adaptive crossmodal stimuli association. EUCognition Meeting, arXiv preprint arXiv:1801.07654.

Fu, D., Barros, P., Parisi, G. I., Wu, H., Magg, S., Liu, X., & Wermter, S. (2018). Assessing the Contribution of Semantic Congruency to Multisensory Integration and Conflict Resolution. arXiv preprint arXiv:1810.06748.

Barros, P., Parisi, G. I., Fu, D., Liu, X., & Wermter, S. (2018). Expectation Learning and Crossmodal Modulation with a Deep Adversarial Network. In 2018 International Joint Conference on Neural Networks (IJCNN), 1-8.

Mici, L., Parisi, G. I., & Wermter, S. (2018). An incremental self-organizing architecture for sensorimotor learning and prediction. IEEE Transactions on Cognitive and Developmental Systems, 10(4), 918-928

Cruz, F., Parisi, G. I., & Wermter, S. (2018). Multi-modal Feedback for Affordance-driven Interactive Reinforcement Learning. In 2018 International Joint Conference on Neural Networks (IJCNN), 1-8.

Churamani, N., Barros, P., Strahl, E., & Wermter, S. (2018). Learning empathy-driven emotion expressions using affective modulations. In 2018 International Joint Conference on Neural Networks (IJCNN) (pp. 1-8).

Parisi, G. I., Tong, J., Barros, P., Röder, B., & Wermter, S. (2018). Towards Modeling the Interaction of Spatial-Associative Neural Network Representations for Multisensory Perception. Workshop on Computational Models for Crossmodal Learning (ICDL-EpiRob).

Parisi, G. I., Barros, P., Fu, D., Magg, S., Wu, H., Liu, X., & Wermter, S. (2018). A neurorobotic experiment for crossmodal conflict resolution in complex environments. 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2330-2335.

Parisi, G.I., Barros, P., Kerzel, M., Wu, H., Yang, G., Li, Z., Liu, X., Wermter, S. (2017). A computational model of crossmodal processing for conflict resolution. Proc. International Conference on Development and Learning and on Epigenetic Robotics (ICDL-EPIROB), pages 33-38.

Parisi, G.I., Wermter, S. (2017). Lifelong learning of action representations with deep neural self-organization. AAAI Spring Symposium Series, pages 608-612.

Parisi, G.I., Barros, P., Wu, H., Yang, G., Li, Z., Liu, X., Wermter, S. (2017). A deep neural model for emotion-driven multimodal attention. AAAI Spring Symposium Series, pages 482-485.

Barros, P., Parisi, G.I., Wermter, S. (2017). Emotion-modulated attention improves expression recognition: A deep learning model. Neurocomputing, Volume 253, pages 104-114.

Barros, P., & Wermter, S. (2017).A self-organizing model for affective memory. International Joint Conference on Neural Networks (IJCNN), pp. 31-38.

Churamani, N., Kerzel, M., Strahl, E., Barros, P., & Wermter, S. (2017). Teaching emotion expressions to a human companion robot using deep neural architectures. International Joint Conference on Neural Networks (IJCNN), pp. 627-634.

Tsironi, E., Barros, P., Weber, C., & Wermter, S. (2017). An analysis of Convolutional Long Short-Term Memory Recurrent Neural Networks for gesture recognition. Neurocomputing.

Parisi, G.I., Tani, J., Weber, C., Wermter, S. (2017). Emergence of Multimodal Action Representations from Neural Network Self-Organization. Cognitive Systems Research

Cruz, F., Parisi, G. I., and Wermter, S. (2016). Learning Contextual Affordances with an Associative Neural Architecture. Proceedings of the European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning (ESANN), pp. 665-670, Bruges, Belgium, 2016.

Parisi, G.I., Wermter, S. A Neurocognitive Robot Assistant for Robust Event Detection. (2016). Trends in Ambient Intelligent Systems: Role of Computational Intelligence, Series "Studies in Computational Intelligence", pp. 1-28, Springer, 2016

Cruz, F., Parisi, G. I., & Wermter, S. (2016). Learning Contextual Affordances with an Associative Neural Architecture. Proceedings of the European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning (ESANN), 665-670, Bruges, Belgium.

Speck, D., Barros, P., Weber, C. and Wermter, S. (2016). Ball Localization for Robocup Soccer using Convolutional Neural Networks. RoboCup Symposium, Leipzig, Germany, 2016. - Best Paper Award

Barros, P. , Weber, C., Wermter, S. (2016). Learning Auditory Representations for Emotion Recognition. Proceedings of International Joint Conference on Neural Networks (IJCNN/WCCI), Vancouver, Canada, July 2016.

Mousavi, N., Siqueira, H., Barros, P., Fernandes, B., Wermter, S. (2016). Understanding How Deep Neural Networks Learn Face Expressions. Proceedings of International Joint Conference on Neural Networks (IJCNN/WCCI), Vancouver, Canada, July 2016.

Relevant previous work

Bauer, J., Dávila-Chacón, J., & Wermter, S. (2014). Modeling development of natural multi-sensory integration using neural self-organization and probabilistic population codes. Connection Science, 1-19.

Bauer, J., Magg, S., & Wermter, S. (2015). Attention modeled as information in learning multisensory integration. Neural Networks, 65, 44-52.

Dávila-Chacón, J., Magg, S., Liu, J., & Wermter, S. (2013). Neural and statistical processing of spatial cues for sound source localisation. In Proc. of Int. Joint Conf. on Neural Networks (pp. 1274-1281), IEEE.

Liu, X., Banich, M. T., Jacobson, B. L., & Tanabe, J. L. (2004). Common and distinct neural substrates of attentional control in an integrated Simon and spatial Stroop task as assessed by event-related fMRI. Neuroimage, 22(3), 1097-1106.

Liu, X., Banich, M. T., Jacobson, B. L., & Tanabe, J. L. (2006). Functional dissociation of attentional selection within PFC: response and non-response related aspects of attentional selection as ascertained by fMRI. Cereb Cortex, 16(6), 827-834.

Liu, X., Park, Y., Gu, X., & Fan, J. (2010). Dimensional overlap accounts for independence and integration of stimulus-response compatibility effects. Attention Perception & Psychophysics, 72(6), 1710-1720.

Murray, J., Erwin, H., & Wermter, S. (2009). Robotic sound-source localisation architecture using cross-correlation and recurrent neural networks. Neural Networks, 22(2). 173–189.

Parisi, GI., Weber, C., &Wermter, S. (2015). Self-organizing neural integration of pose-motion features for human action recognition. Frontiers in Neurorobotics. from: http://journal.frontiersin.org/article/10.3389/fnbot.2015.00003/

Wermter, S., & Elshaw, M. (2003). Learning robot actions based on self-organising language memory. Neural Networks, 16(5), 691–699.

Zhong, J., Weber, C., & Wermter, S. (2013). A predictive network architecture for a robust and smooth robot docking behavior. Journal of Behavioral Robotics, 3(4), 172-180.

Publications project A6 - Brain-inspired multimodal deep learning

Publications of the current project

Ge Gao, Mikko Lauri, Yulong Wang, Xiaolin Hu, Jianwei Zhang and Simone Frintrop. (2020). 6D Object Pose Regression via Supervised Learning on Point Clouds. Proceedings of the IEEE International Conference on Robotics and Automation (ICRA) 2020, Paris, France. [Arxiv preprint 2001.08942], [Code].

Publications from phase 1

Feng, W., Liu, W., Li, T., Peng, J., Qian, C., & Hu, X. (2019). Turbo Learning Framework for Human-Object Interactions Recognition and Human Pose Estimation. The Thirty-Third AAAI Conference on Artificial Intelligence (AAAI), Honolulu, Hawaii, USA.

Yu, M., Weber, C., Hu, X. (2019). Learning Sparse Hidden States in Long Short-Term Memory. In Proc. of the International Conference on Artificial Neural Networks (ICANN2019)

Liao, F., Liang, M., Li, Z., Hu, X., & Song, S. (2019). Evaluate the Malignancy of Pulmonary Nodules Using the 3-D Deep Leaky Noisy-or Network. IEEE transactions on neural networks and learning systems.

Liu, W., Chen, J., Li, C., Qian, C., Chu, X., & Hu, X. (2018). A Cascaded Inception of Inception Network with Attention Modulated Feature Fusion for Human Pose Estimation. The Thirty-Second AAAI Conference on Artificial Intelligence.

Xiong, Z., Weber, C., & Hu, X. (2018). Frame Difference-Based Real-Time Video Stylization in Video Calls. In 2018 Ninth International Conference on Intelligent Control and Information Processing (ICICIP), 333-339.

Wang, Y., Su, H., Zhang, B., & Hu, X. (2018). Interpret neural networks by identifying critical data routing paths. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 8906-8914.

Liao, F., Liang, M., Dong, Y., Pang, T., Hu, X., & Zhu, J. (2018). Defense against adversarial attacks using high-level representation guided denoiser. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 1778-1787.

Springenberg, S., Lakomkin, E., Weber, C., & Wermter, S. (2018). Image-to-Text Transduction with Spatial Self-Attention. In 2018 proceedings, European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning (ESANN).

Dong, Y., Liao, F., Pang, T., Su, H., Zhu, J., Hu, X., & Li, J. (2018). Boosting adversarial attacks with momentum. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 9185-9193).

Sun, T., Wang, Y., Yang, J., Hu, X., (2017). Convolutional neural networks with two pathways for image style recognition, IEEE Transactions on Image Processing, vol. 26, no. 9, pp. 4102-4113, 2017.

Wang, J., & Hu, X. (2017). Gated recurrent convolution neural network for ocr. In International Conference on Advances in Neural Information Processing Systems(NeurIPS), 335-344.

Hu, Z., Wang. T., Hu, X., (2017). An STDP-based supervised learning algorithm for spiking neural networks, Proc. of 24th International Conference on Neural Information Processing (ICONIP), Guangzhou, China, Nov. 14-18, 2017.

Hu, X., Wang, T., (2017). Training the Hopfield Neural Network for Classification Using a STDP-Like Rule, Proc. of 24th International Conference on Neural Information Processing (ICONIP), Guangzhou, China, Nov. 14-18, 2017.

Yu, N., Qiu, S., Hu, X., Li, J., (2017). Accelerating Convolutional Neural Networks by Group-wise 2D-filter Pruning, Proc. of International Joint Conference on Neural Networks (IJCNN), Anchorage, Alaska, USA, May 14–19, 2017, pp. 2502-2509.

Wu, J., Ma, L., Hu, X., (2017). Delving deeper into convolutional neural networks for camera relocalization, Proc. of IEEE International Conference on Robotics and Automation (ICRA), Singapore, May 29- June 3, 2017, pp. 5644-5651.

Zhao, Y., Jin, X., Hu, X., (2017). Recurrent convolutional neural network for speech processing, Proc. of the 42nd IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), New Orleans, USA, March 5-9, 2017.

Zhou, X., Weber, C., Wermter, S. (2017). Robot Localization and Orientation Detection based on Place Cells and Head-direction Cells. Proceedings of the 26th International Conference on Artificial Neural Networks (ICANN 2017), Sep 2017.

Zhuang, C., Wang, Y., Yamins, D., & Hu, X. (2017). Deep learning predicts correlation between a functional signature of higher visual areas and sparse firing of neurons. Frontiers in Computational Neuroscience, 11, 100.

Chen, X., Hu, X., Zhou, H., & Xu, N. (2017, May). Fxpnet: Training a deep convolutional neural network in fixed-point representation. In 2017 International Joint Conference on Neural Networks (IJCNN), 2494-2501.

Wu, J., Ma, L., Hu, X., (2016). Predicting world coordinates of pixels in RGB images using convolutional neural network for camera relocalization, Proc. of the Seventh International Conference on Intelligent Control and Information Processing (ICICIP), Siem Reap, Cambodia, December 1-4, 2016.

Qin H., Yan J., Li X., Hu X., (2016). Joint Training of Cascaded CNN for Face Detection, Proc. of the 28th IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016, pp. 3456-3465.

Relevant previous work

Heinrich, S., Weber, C., & Wermter, S. (2013). Embodied language understanding with a multiple timescale recurrent neural network. In Artificial Neural Networks and Mahine Learning-ICANN 2013 (pp. 216-223). Springer Berlin Heidelberg.

Hu, X., Zhang, J., Qi, P., & Zhang, B. (2014). Modeling response properties of V2 neurons using a hierarchical K-means model. Neurocomputing, 134, 198-205.

Hu, X., Sun, C., & Zhang, B. (2010). Design of recurrent neural networks for solving constrained least absolute deviation problems. IEEE Transactions on Neural Networks, 21(7), 1073-1086.

Hu, X., & Zhang, B. (2009). An alternative recurrent neural network for solving variational inequalities and related optimization problems. IEEE Transactions on Systems, Man, and Cybernetics, Part B: Cybernetics, 39(6), 1640-1645.

Hu, X., & Zhang, B. (2010). A Gaussian attractor network for memory and recognition with experience-dependent Learning. Neural Computation, 22(5), 1333-1357.

Liang, M., & Hu, X. (2015). Predicting eye fixations with higher-level visual features. IEEE Transactions on Image Processing, 24(3), 1178-1189.

Liang, M., & Hu, X. (2015). Recurrent convolutional neural network for object recognition. In Proc. of the 28th IEEE Conference on Computer Vision and Pattern Recognition (pp. 3367-3375). IEEE.

Weber, C., & Triesch, J. (2008). A sparse generative model of V1 simple cells with intrinsic plasticity. Neural Computation, 20(5), 1261-84.

Wu, Y., Liu, Y., Li, J., Liu, H., & Hu, X. (2013). Traffic sign detection based on convolutional neural networks. In the 2013 International Joint Conference on Neural Network (IJCNN) (pp. 1-7). IEEE.

Zhong, J., Weber, C., & Wermter, S. (2013). A predictive network architecture for a robust and smooth robot docking behavior. Paladyn, Journal of Behavioral Robotics, 3(4), 172-180.