Publications - Thematic Area B

Publications project B1 - Neural dynamics in crossmodal prediction

Publications of the current project

• Maÿe Alexander, Wang Tiezhi, Engel Andreas K. (2021). Neuronal Oscillatory Signatures of Joint Attention and Intersubjectivity in Arrhythmic Coaction, Frontiers in Human Neuroscience, vol 15, p. 655, 2021. DOI=10.3389/fnhum.2021.767208

Publications from phase 1

• Maye A, Wang P, Daume J, Hu X, Engel AK (2019). An oscillator ensemble model of sequence learning. Frontiers in Integrative Neuroscience 13: 43.
• Wang P, Göschl F, Friese U, König P, Engel AK (2019). Long-range functional coupling predicts performance: oscillatory EEG networks in multisensory processing. Neuroimage 196: 114-125.
• Rimmele JM, Gudi-Mindermann H, Nolte G, Röder B, Engel AK (2019). Working memory training integrates visual cortex into beta-band networks in congenitally blind individuals. Neuroimage 194: 259-271.
• Chen J, Li Z, Hong B, Maye A, Engel AK, Zhang D (2019). A single-stimulus, multi-target BCI based on retinotopic mapping of motion-onset VEPs. IEEE Transactions on Biomedical Engineering 66: 464-470.
• Nolte G, Aburidi M, Engel AK (2019). Robust calculation of slopes in detrended fluctuation analysis and its application to envelopes of human alpha rhythm. Scientific Reports 9: 6339.
• Shahbazi Avarvand F, Bartz S, Andreou C, Samek W, Leicht G, Mulert C, Engel AK, Nolte G (2018). Localizing bicoherence from EEG and MEG. Neuroimage 174: 352-363.
• Chen, J., Zhang, D., Engel, A. K., Gong, Q., and Maye, A. (2017). Application of a single-flicker online SSVEP BCI for spatial navigation, PloS one, 12(5), e0178385.
• Maye A, Zhang D, Engel AK (2017). Utilizing retinotopic mapping for a multi-target SSVEP BCI with a single flicker frequency. IEEE Transactions on Neural Systems and Rehabilitation Engineering 25: 1026–1036.

Relevant previous work

• Engel, A. K., Fries, P., & Singer, W. (2001). Dynamic predictions: oscillations and synchrony in top-down processing. Nature Reviews Neuroscience, 2, 704-716.
• Engel, A. K., Gerloff, C., Hilgetag, C. C., & Nolte, G. (2013). Intrinsic coupling modes: multiscale interactions in ongoing brain activity. Neuron, 80, 867-886.
• Hipp, J. F., Engel, A. K., & Siegel, M. (2011). Oscillatory synchronization in large-scale cortical networks predicts perception. Neuron, 69, 387-396.
• Hipp, J. F., Hawellek, D., Corbetta, M., Siegel, M., & Engel, A. K. (2012). Large-scale cortical correlation structure of spontaneous oscillatory activity. Nature Neuroscience, 15, 884-890.
• Höfle, M., Pomper, U., Hauck, M., Engel, A. K., & Senkowski, D. (2013). Spectral signatures of viewing a needle approaching one' own body when anticipating pain. European Journal of Neuroscience, 7, 3089-3098.
• Hu, X., Sun, C., & Zhang, B. (2010). Design of recurrent neural networks for solving constrained least absolute deviation problems. IEEE Transactions on Neural Networks, 21, 1073-1086.
• Liang, M., & Hu, X. (2015). Recurrent convolutional neural network for object recognition. Proceedings of the 28th IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 3367-3375.
• Schepers, I. M., Hipp, J. F., Schneider, T. R., Röder, B., & Engel, A. K. (2012). Functionally specific oscillatory activity correlates between visual and auditory cortex in the blind. Brain, 135, 922-934.
• Siegel, M., Donner, T. H., Oostenveld, R., Fries, P., & Engel, A. K. (2008). Neuronal synchronization along the dorsal visual pathway reflects the focus of spatial attention. Neuron, 60, 709-719.
• Womelsdorf, T., Schoffelen, J.-M., Oostenveld, R., Singer, W., Desimone, R., Engel, A. K., & Fries, P. (2007). Modulation of neuronal interactions through neuronal synchronization. Science, 316, 1609-1612.

Publications project B2 - Bayesian analysis of the interaction of learning, semantics and social influence
with crossmodal integration

Publications from phase 1

• Oganian, Y., Heekeren, H. R., & Korn, C. W. (2019). Low foreign language proficiency reduces optimism about the personal future. Quarterly Journal of Experimental Psychology, 72(1), 60-75.
• Korn, C. W., Heekeren, H. R., & Oganian, Y. (2019). The framing effect in a monetary gambling task is robust in minimally verbal language switching contexts. Quarterly Journal of Experimental Psychology, 72(1), 52-59.
• Liu, C., Zhuo, J., & Zhu, J. (2019). Understanding MCMC Dynamics as Flows on the Wasserstein Space. In International Conference on Machine Learning (ICML), Long Beach, CA, USA.
• Liu, C., Zhuo, J., Chen P., Zhang, R., & Zhu, J. (2019). 135 Understand and Accelerate Particle-based Variational Inference. In International Conference on Machine Learning (ICML), Long Beach, CA, USA.
• Shi, J., Khan, M. E., & Zhu, J. (2019). Scalable Training of Inference Networks for Gaussian-Process Models. In International Conference on Machine Learning (ICML), Long Beach, CA, USA.
• Pang, T., Xu, K., Du, C., Chen, N., & Zhu, J. (2019). Improving Adversarial Robustness via Promoting Ensemble Diversity. In International Conference on Machine Learning (ICML), Long Beach, CA, USA.
• Wang, Z., Ren, T., Zhu, J., & Zhang, B. (2019). Function space particle optimization for Bayesian neural networks. In International Conference on Learning Representations (ICLR), New Orleans, Louisiana, USA.
• Dong, Y., Pang, T., Su, H., & Zhu, J. (2019). Evading Defenses to Transferable Adversarial Examples by Translation-Invariant Attacks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 4312-4321.
• Dong, Y., Su, H., Wu, B., Li, Z., Liu, W., Zhang, T., & Zhu, J. (2019). Efficient Decision-based Black-box Adversarial Attacks on Face Recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 7714-7722.
• Wei, X., Zhu, J., & Su, H. (2019). Sparse adversarial perturbations for videos. In the 33rd AAAI Conference on Artificial Intelligence (AAAI), Honolulu, Hawaii, USA.
• Liu, C., & Zhu, J. (2018). Riemannian Stein variational gradient descent for Bayesian inference. Thirty-Second AAAI Conference on Artificial Intelligence.
• Chen, J., Zhu, J., Teh, Y. W., & Zhang, T. (2018). Stochastic Expectation Maximization with variance reduction. Advances in Neural Information Processing Systems(NeurIPS), 7967-7977.
• Shi, J., Sun, S., & Zhu, J. (2018). A spectral approach to gradient estimation for implicit distributions. Proc. of the 35th International Conference on Machine Learning (ICML), arXiv preprint arXiv:1806.02925.
• Tian, T., Zhou, Y., & Zhu, J. (2018). Selective Verification Strategy for Learning from Crowds. In Thirty-Second AAAI Conference on Artificial Intelligence.
• Du, C., Li, C., Zheng, Y., Zhu, J., & Zhang, B. (2018). Collaborative filtering with user-item co-autoregressive models. In Thirty-Second AAAI Conference on Artificial Intelligence.
• Bayer, J., Gläscher, J., Finsterbusch, J., Schulte, L. H., & Sommer, T. (2018). Linear and inverted U-shaped dose-response functions describe estrogen effects on hippocampal activity in young women. Nature Communications, 9(1), 1220.
• Korn, C. W., & Bach, D. R. (2018). Heuristic and optimal policy computations in the human brain during sequential decision-making. Nature Communications, 9(1), 325.
• Rosenblau, G., Korn, C. W., & Pelphrey, K. A. (2018). A computational account of optimizing social predictions reveals that adolescents are conservative learners in social contexts. Journal of Neuroscience, 38(4), 974-988.
• Pang, T., Du, C., Dong, Y., & Zhu, J. (2018). Towards robust detection of adversarial examples. In Proc. of Advances in Neural Information Processing Systems (NeurIPS), Montreal, Canada, 4579-4589. (Spotlight, NVIDIA Pioneering Research Award)
• Li, C., Welling, M., Zhu, J., & Zhang, B. (2018). Graphical Generative Adversarial Networks, In Proc. of Advances in Neural Information Processing Systems (NeurIPS), Montreal, Canada
• Luo, Y., Tian, T., Shi, J., Zhu, J., & Zhang, B. (2018). Semi-crowdsourced clustering with deep generative models. In Proc. of Advances in Neural Information Processing Systems (NeurIPS), Montreal, Canada, 3212-3222.
• Li, J., Su, H., Zhu, J., & Zhang, B. (2018). Essay-Anchor Attentive Multi-Modal Bilinear Pooling for Textbook Question Answering. In 2018 IEEE International Conference on Multimedia and Expo (ICME), 1-6.
• Wei, X., Zhu, J., Feng, S., & Su, H. (2018). Video-to-video translation with global temporal consistency. In 2018 ACM Multimedia Conference on Multimedia Conference, Seoul, Korea, 18-25.
• Shi, J., Sun, S., & Zhu, J. (2017). Kernel implicit variational inference. Proc. of the 6th International Conference on Learning Representations (ICLR), arXiv preprint arXiv:1705.10119.
• Zhuo, J., Liu, C., Shi, J., Zhu, J., Chen, N., & Zhang, B. (2017). Message passing stein variational gradient descent. Proc. of the 35th International Conference on Machine Learning (ICML), arXiv preprint arXiv:1711.04425.
• Li, C., Zhu, J., & Zhang, B. (2017). Max-margin deep generative models for (semi-) supervised learning. IEEE transactions on pattern analysis and machine intelligence, 40(11), 2762-2775.
• Ren, Y., Wang, Y., & Zhu, J. (2017). Spectral learning for supervised topic models. IEEE transactions on pattern analysis and machine intelligence, 40(3), 726-739.
• Li, C., Xu, K., Zhu, J., & Zhang, B. (2017). Triple generative adversarial nets. 31st Conference on Neural Information Processing Systems(NeurIPS), Long Beach, CA, USA, 4088-4098.
• Liu, M., Shi, J., Cao, K., Zhu, J., & Liu, S. (2017). Analyzing the training processes of deep generative models. IEEE transactions on visualization and computer graphics, 24(1), 77-87.
• Deng, Z., Zhang, H., Liang, X., Yang, L., Xu, S., Zhu, J., & Xing, E. P. (2017). Structured generative adversarial networks. Proc. of Advances in Neural Information Processing Systems(NeurIPS), 3899-3909.
• Zhou, Y., Li, J., & Zhu, J. (2017). Identify the Nash Equilibrium in static games with random payoffs. In Proceedings of the 34th International Conference on Machine Learning (ICML), 4160-4169.
• Liu, S., Xiao, J., Liu, J., Wang, X., Wu, J., & Zhu, J. (2017). Visual diagnosis of tree boosting methods. IEEE transactions on visualization and computer graphics, 24(1), 163-173.
• Dong, Y., Ni, R., Li, J., Chen, Y., Zhu, J., & Su, H. (2017). Learning accurate low-bit deep neural networks with stochastic quantization. The 28th British Machine Vision Conference (BMVC), arXiv preprint arXiv:1708.01001.
• Ren, Y., & Zhu, J. (2017). Distributed Accelerated Proximal Coordinate Gradient Methods. Proc. of International Joint Conference on Artificial Intelligence (IJCAI), 2655-2661.
• Hu, W., Zhu, J., Su, H., Zhuo, J., & Zhang, B. (2017). Semi-supervised Max-margin Topic Model with Manifold Posterior Regularization. Proc. of International Joint Conference on Artificial Intelligence (IJCAI), 1865-1871.
• Liu, M., Jiang, L., Liu, J., Wang, X., Zhu, J., & Liu, S. (2017). Improving Learning-from-Crowds through Expert Validation. Proc. of International Joint Conference on Artificial Intelligence (IJCAI), 2329-2336.
• Dong, Y., Su, H., Zhu, J., & Zhang, B. (2017). Improving interpretability of deep neural networks with semantic information. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 4306-4314.
• Tian, T., Chen, N., & Zhu, J. (2017). Learning attributes from the crowdsourced relative labels. Thirty-First AAAI Conference on Artificial Intelligence.
• Korn, C. W., Zaiser, J., Schalk, L., Oganian, Y., & Saalbach, H. (2017). Hard-to-read fonts do not influence the framing effect. Psychonom Bull & Rev 25:2:696-703.
• Chen, J., Zhu, J., & Song, L. (2017). Stochastic training of graph convolutional networks with variance reduction. In Proc. of the 35th International Conference on Machine Learning (ICML), Stockholm, Sweden.

Relevant previous work

• Gläscher, J., & Büchel, C. (2005). Formal learning theory dissociates brain regions with different temporal integration. Neuron, 47, 295-306.
• Gläscher, J., Hampton, A. N., & O’Doherty, J. (2009). Determining a role for ventromedial prefrontal cortex in encoding action-based value signals during reward-related decision making. Cereb Cortex, 19, 483-495.
• Gläscher, J., & O’Doherty, J. (2010). Model-based approaches to neuroimaging: combining reinforcement learning theory with fMRI data. In Sanfey AG, Nadel L (Eds.), Wiley Interdisciplinary Reviews: Cognitive Science, 1, 501-510.
• Gläscher, J., Daw, N., Dayan, P., & O’Doherty, J. P. (2010). States versus rewards: Dissociable neural prediction error signals underlying model-based and model-free reinforcement learning. Neuron, 66, 585-595.
• Gläscher, J., Adolphs, R., Damasio, H., Bechara, A., Rudrauf, D., Calamia, M., Paul, L. K., & Tranel, D. (2012). Lesion map-ping of cognitive control and value-based decision-making in the prefrontal cortex. Proc Natl Acad Sci USA, 109, 14681-14686.
• Chen, N., Zhu, J., Sun, F., & Xing, E. P. (2012). Large-margin Subspace Learning for Multi-view Data Analysis. IEEE Trans. on Pattern Analysis and Machine Intelligence, 34, 2365-2378.
• Zhu, J., Ahmed, A., & Xing, E. P. (2012). MedLDA: Maximum Margin Supervised Topic Models. Journal of Machine Learning Research, 13, 2237-2278.
• Zhu, J., & Xing, E. P. (2009). Maximum Entropy Discrimination Markov Networks. Journal of Machine Learning Research, 10, 2531-2569.
• Zhu, J., Chen, N., & Xing, E. P. (2014a) Bayesian Inference with Posterior Regularization and applications to Infinite Latent SVMs. Journal of Machine Learning Research, 15, 1799-1847.
• Zhu, J., Chen, N., Perkins, H., & Zhang, B. (2014b) Gibbs Max-margin Topic Models with Data Augmentation. Journal of Machine Learning Research, 15, 1073-1110.

Publications project B3 - Neurocognitive mechanisms for implicit learning of crossmodal predictions

Publications of the current project

• Taesler, P., Jablonowski, J., Fu, Q., & Rose, M. (2019). Modeling implicit learning in a cross-modal audio-visual serial reaction time task. Cognitive Systems Research, 54, 154-164.
• Fu, Q., Sun, H., Dienes, Z., & Fu, X. (2019). Dataset of implicit sequence learning of chunking and abstract structures. Data in brief, 22, 72-75.
• Zhou, X., Fu, Q., Rose, M. R., & Sun, Y. (2019). Which Matters More in Incidental Category Learning: Edge-based vs. Surface-based Features. Frontiers in Psychology, 10, 183.
• Sun, X, Sun, Y., & Fu, Q. (2019). Cross-modal learning and its cognitive and neural mechanisms. Progress in Biochemistry and Biophysics, 46(6), 565-577. (in Chinese)
• Wu, J., Fu*, Q., Zhou, X., & Sun, X. (2018). The effect of presenting mode of different features on the acquisition of rule-based and similarity-based knowledge in category learning. Journal of Psychological Science, 41(5), 1–6. (in Chinese).
• Jablonowski, J., Taesler, P., Fu, Q., & Rose, M. (2018). Implicit acoustic sequence learning recruits the hippocampus. PloS one, 13(12), e0209590.
• Fu, Q., Sun, H., Dienes, Z., & Fu, X. (2018). Implicit sequence learning of chunking and abstract structures. Consciousness and Cognition, 62, 42-56.
• Clos, M., Sommer, T., Schneider, S. L., & Rose, M. (2018). Enhanced transformation of incidentally learned knowledge into explicit memory by dopaminergic modulation. PloS one, 13(6), e0199013.
• Fu, Q., Liu, Y. J., Dienes, Z., Wu, J., Chen, W., & Fu, X. (2017). Neural correlates of subjective awareness for natural scene categorization of color photographs and line-drawings. Frontiers in psychology, 8, 210.
• Taesler, P., & Rose, M. (2017). Psychophysically-anchored, Robust Thresholding in Studying Pain-related Lateralization of Oscillatory Prestimulus Activity. Journal of Visualized Experiments, (119), e55228.

Relevant previous work

• Fu, Q., Bin G, Dienes, Z., Fu, X., & Gao, X. (2013a). Learning without consciously knowing: Evidence from event-related potentials in sequence learning. Conscious Cogn, 22, 22–34.
• Fu, Q., Dienes Z, & Fu, X. (2010). Can unconscious knowledge allow control in sequence learning? Conscious Cogn, 19, 462–474.
• Fu, Q., Dienes, Z, Shang, J., & Fu, X. (2013b). Who learns more? Cultural differences in implicit sequence learning. PLoS One, 8, e71625.
• Fu, Q., Fu, X., & Dienes, Z. (2008). Implicit sequence learning and conscious awareness. Conscious Cogn, 17, 185–202.
• Gao, X., Xu, D., Cheng, M., & Gao, S. (2003). A BCI-based environmental controller for the motion-disabled. IEEE Trans Neural Syst Rehabil Eng, 11, 137-140.
• Haider, H., Eberhardt, K., Kunde, A., & Rose, M. (2013). Implicit visual learning and the expression of learning. Conscious Cogn, 22, 82-98.
• Rose, M., Haider, H., & Büchel, C. (2010). The emergence of explicit memory during learning. Cerebral Cortex, 12, 2787-2797.
• Rose, M., Haider, H., Salari, N. & Büchel, C. (2011). Functional dissociation of hippocampal mechanism during implicit learning based on the domain of associations. The Journal of Neuroscience, 31, 13739-13745.
• Rose, M., Haider, H., Weiller, C., & Büchel, C. (2002). The role of medial temporal lobe structures in implicit learning: an event-related fMRI study. Neuron, 36, 1221-1231.
• Schmid, C., Büchel, C., & Rose, M. (2011). The neural basis of visual dominance in the context of audio-visual object processing. Neuroimage, 55, 304-311.

Publications project B4 - Brain dynamics of crossmodal learning and conflict processing

Publications of the current project

• Nolte, G., Galindo-Leon, E., Li, Z., Liu, X., and Engel, A K., (2020). Mathematical relations between measures of brain connectivity estimated from electrophysiological recordings for gaussian distributed data. Frontiers in Neuroscience, 14(1127).
• Li, Z., Göschl, F., Yang, G. (2020). Dissociated neural mechanisms of target and distractor processing facilitated by expectations. Journal of Neuroscience (Journal Club), 40 (10), 1997-1999.
• Li, Z., Yang, G., Wu, H., Li, Q., Xu, H., Göschl, F., ... & Liu, X. (2021). Modality-specific neural mechanisms of cognitive control in a Stroop-like task.  Brain and Cognition, 147, 105662.
• Misselhorn, J., Göschl, F., Higgen, F. L., Hummel, F. C., Gerloff, C., & Engel, A. K. (2020). Sensory capability and information integration independently explain the cognitive status of healthy older adults. Scientific Reports, 10(1), 22437.
• J Li, B Hong, G Nolte, AK Engel, D Zhang (2021). Preparatiory delta phase response is correlared with naturalistic speech comprehension performance. Cognitive Neurodynamics, 1-16, 2021.
• Li Z, Li J, Hong B, Nolte G, Engel AK, Zhang D (2021). Speaker-Listener Neural Coupling Reveals an Adaptive Mechanism for Speech Comprehension in a Noisy Environment. Cereb Cortex. 2021 Aug 26;31(10):4719-4729.

Publications from phase 1

• Avarvand, F. S., Bartz, S., Andreou, C., Samek, W., Leicht, G., Mulert, C., ... & Nolte, G. (2018). Localizing bicoherence from EEG and MEG. Neuroimage, 174, 352-363.
• Daume, J., Graetz, S., Gruber, T., Engel, A. K., & Friese, U. (2017). Cognitive control during audiovisual working memory engages frontotemporal theta-band interactions. Scientific Reports, 7(1), 12585.
• Li, Q.*, Yang, G.*, Li, Z., Qi, Y., Cole, M. W., & Liu, X. (2017). Conflict detection and resolution rely on a combination of common and distinct cognitive control networks. Neuroscience & Biobehavioral Reviews, 83, 123-131 (* contributed equally).
• Li, Q., Wang, K., Nan, W., Zheng, Y., Wu, H., Wang, H., & Liu, X. (2015). Electrophysiological dynamics reveal distinct processing of stimulus-stimulus and stimulus-response conflicts. Psychophysiology, 52(4), 562-571.
• Li, Z, Yang, G., Nan, W., Li, Q., & Liu, X. (2018). Attentional regulation mechanisms of cognitive control in conflict resolution. Advances in Psychological Science, 26(6), 966-974.
• Nolte, G., Aburidi, M., & Engel, A. K. (2019) Robust calculation of slopes in detrended fluctuation analysis and its application to envelopes of human alpha rhythms. Scientific Reports, 9 (1), 6339.
• Wang, P.*, Göschl, F.*, Friese, U., König, P., & Engel, A.K. (2019). Long-range functional coupling predicts performance: Oscillatory EEG networks in multisensory processing. NeuroImage, 196, 114-125 (* contributed equally).
• Yang, G., Li, Z., Wu, H., & Liu, X. (2019). Generality and specificity of cognitive control: research logics and debates. Acta Physiologica Sinica, 71(01), 140-148.
• Yang, G., Nan, W., Zheng, Y., Wu, H., Li, Q., & Liu, X. (2017). Distinct cognitive control mechanisms as revealed by modality-specific conflict adaptation effects. Journal of experimental psychology: human perception and performance, 43(4), 807-818.
• Parisi, G., Barros, P., Fu, D., Magg, S., Wu, H., Liu, X., & Wermter, S. (2019). A Neurorobotic Experiment for Crossmodal Conflict Resolution. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, 33-36.
• Zamani, M. A., Magg, S., Weber, C., Wermter, S., & Fu, D. (2018). Deep reinforcement learning using compositional representations for performing instructions. Paladyn Journal of Behavioral Robotics, 9(1), 358-373.
• Barros, P., Parisi, G.I., Fu, D., Liu, X., Wermter, S. (2017) Adaptive crossmodal stimuli association using expectation learning. European Society for Cognitive Systems, Zurich, Switzerland.
• 杨国春, 李政汉, 伍海燕, & 刘勋. (2019). 认知控制的一般性/特异性机制: 研究逻辑和争论. 生理学报, (1), 14.
• 李政汉, 杨国春, 南威治, 李琦, & 刘勋. (2018). 冲突解决过程中认知控制的注意调节机制. 心理科学进展, 26(6), 966-974.

Relevant previous work

• Ewald, A., Marzetti, L., Zappasodi, F., Meinecke, F. C., & Nolte, G. (2012). Estimating true brain connectivity from EEG/MEG data invariant to linear and static transformations in sensor space. NeuroImage, 60(1), 476-488.
• Li, Q., Wang, K., Nan, W., Zheng, Y., Wu, H., Wang, H., & Liu, X. (2015). Electrophysiological dynamics reveal distinct processing of stimulus-stimulus and stimulus-response conflicts. Psychophysiology, 52(4), 562-571.
• Liu, X., Banich, M. T., Jacobson, B. L., & Tanabe, J. L. (2004). Common and distinct neural substrates of attentional control in an integrated Simon and spatial Stroop task as assessed by event-related fMRI. Neuroimage, 22(3), 1097-1106.
• Liu, X., Banich, M. T., Jacobson, B. L., & Tanabe, J. L. (2006). Functional dissociation of attentional selection within PFC: response and non-response related aspects of attentional selection as ascertained by fMRI. Cerebral Cortex, 16(6), 827-834.
• Liu, X., Park, Y., Gu, X., & Fan, J. (2010). Dimensional overlap accounts for independence and integration of stimulus—response compatibility effects.Attention, Perception, & Psychophysics, 72(6), 1710-1720.
• Luft, C. D. B., Nolte, G., & Bhattacharya, J. (2013). High-learners present larger mid-frontal theta power and connectivity in response to incorrect performance feedback. The Journal of Neuroscience, 33(5), 2029-2038.
• Nolte, G. (2003). The magnetic lead field theorem in the quasi-static approximation and its use for magnetoencephalography forward calculation in realistic volume conductors. Physics in medicine and biology, 48(22), 3637.
• Nolte, G., Bai, O., Wheaton, L., Mari, Z., Vorbach, S., & Hallett, M. (2004). Identifying true brain interaction from EEG data using the imaginary part of coherency. Clinical neurophysiology, 115(10), 2292-2307.
• Nolte, G., Ziehe, A., Nikulin, V. V., Schlögl, A., Krämer, N., Brismar, T., & Müller, K. R. (2008). Robustly estimating the flow direction of information in complex physical systems. Physical review letters, 100(23), 234101.
• Wang, K., Li, Q., Zheng, Y., Wang, H., & Liu, X. (2014). Temporal and spectral profiles of stimulus–stimulus and stimulus–response conflict processing. Neuroimage, 89, 280-288.

Publications project B5 - Crossmodal fusion for dexterous manipulation in proactive human-robot collaboration

Publications of the current project

• Yikai Wang, Wenbing Huang, Bin Fang, Fuchun Sun (2021). Elastic Tactile Simulation Towards Tactile-Visual Perception. ACM Multimedia (oral) 2021.
• Fan Yang, Chao Yang, Fuchun Sun, et al. (2021). Fault-Aware Robust Control via Adversarial Reinforcement Learning. ICRA 2021.
• Mingxuan Jing, Wenbing Huang, Fuchun Sun, Xiaojian Ma, Tao Kong, Chuang Gan, Lei Li. (2021). Adversarial Option-Aware Hierarchical Imitation Learning. ICML 2021
• Philipp Ruppel, Norman Hendrich, and Jianwei Zhang (2021). Direct Policy Optimization with Differentiable Physical Consistency for Dexterous Manipulation, 2021 IEEE Intl. Conference on Systems Robotics and Biomimetics (ROBIO-2021), Sanya, China, to appear.
• C. Zeng, S. Li, Y. Jiang, Q. Li,Z. Chen, C. Yang, and J. Zhang (2021). Learning compliant grasping and manipulation by teleoperation with adaptive force control, International Conference on Intelligent Robots and Systems (IROS) 2021, Prague, Czech Republic.
• Niklas Fiedler, Philipp Ruppel, Yannick Jonetzko, Norman Hendrich, and Jianwei Zhang (2021). A Low-Cost Modular System of Customizable, Versatile, and Flexible Tactile Sensor Arrays. International Conference on Intelligent Robots and Systems (IROS) 2021, Prague, Czech Republic.
• Máximo A. Roa, Mehmet Dogar, Jordi Pages, Carlos Vivas, Antonio Morales, Nikolaus Correll, Michael Görner, Jan Rosell, Sergi Foix, Raphael Memmesheimer, and Francesco Ferro (2021). Mobile Manipulation Hackathon: Moving Into Real-World Applications, IEEE Robotics & Automation Magazine, March 2021, p. 2-14. DOI: 10.1109/MRA.2021.3061951
• Philipp Ruppel, Jianwei Zhang (2020). Learning Object Manipulation with Dexterous Hand-Arm Systems from Human Demonstration, International Conference on Intelligent Robots and Systems (IROS) 2020, Las Vegas, USA.
• Hongzhuo Liang, Chuangchuang Zhou, Shuang Li, Xiaojian Ma, Norman Hendrich, Timo Gerkmann, Fuchun Sun and Jianwei Zhang (2020). Robust Robotic Pouring using Audition and Haptics, International Conference on Intelligent Robots and Systems (IROS) 2020, Las Vegas, USA.
• Shuang Li, Jiaxi Jiang, Philipp Ruppel, Hongzhuo Liang, Xiaojian Ma, Norman Hendrich, Fuchun Sun, Jianwei Zhang (2020). A Mobile Robot Hand-Arm Teleoperation System by Vision and IMU, International Conference on Intelligent Robots and Systems (IROS) 2020, Las Vegas, USA.
• Ronja Güldenring, Michael Görner, Norman Hendrich, Niels Jul Jacobsen, Jianwei Zhang (2020). Learning Local Planners for Human-aware Navigation in Indoor Environment, International Conference on Intelligent Robots and Systems (IROS) 2020, Las, Vegas, USA.
• Lin Cong, Michael Görner, Philipp Ruppel, Hongzhuo Liang, Norman Hendrich, Jianwei Zhang (2020). Self-Adapting Recurrent Models for Object Pushing from Learning in Simulation, International Conference on Intelligent Robots and Systems (IROS) 2020, Las Vegas, USA. /
• Jinpeng Mi, Jianzhi Lyu, Song Tang, Qingdu Li and Jianwei Zhang (2020). Interactive Natural Language Grounding via Referring Expression Comprehension and Scene Graph Parsing, Frontiers in Neurorobotics.
• Jinpeng Mi, Hongzhuo Liang, Nikolaos Katzakis, Changshui Zhang and Jianwei Zhang (2020). Intention-Related Natural Language Grounding via Object Affordance Detection and Intention Semantic Extraction, Frontiers in Neurorobotics.
• Philipp Ruppel, Norman Hendrich, Jianwei Zhang (2019). Low-cost multi-view pose tracking using active markers, IEEE ICPS/MFI 2019, pp. 261-268, Taipei, Taiwan. DOI: 10.1109/ICPHYS.2019.8780297

Publications from phase 1

• Michael Görner, Robert Haschke, Helge, Ritter, Jianwei Zhang (2019. Moveit! Task Constructor for Task-Level Motion Planning, IEEE International Conference on Robotics and Automation, ICRA 2019, Montreal, Canada.
• Hongzhuo Liang, Xiaojian Ma, Shuang Li, Michael Görner, Song Tang, Bin Fang, Fuchun Sun, and Jianwei Zhang (2019). PointNetGPD: Detecting Grasp Configurations from Point Sets, IEEE International Conference on Robotics and Automation, ICRA 2019, Montreal, Canada. (pdf)
• Shuang Li, Xiaojian Ma, Hongzhuo Liang, Michael Görner, Philipp Ruppel, Bin Fang, Fuchun Sun, and Jianwei Zhang (2019). Vision-based Teleoperation of Shadow Dexterous Hand using End-to-End Deep Neural Network, IEEE International Conference on Robotics and Automation, ICRA 2019, Montreal, Canada. (pdf)
• Liu, C., Fang, B., Sun, F., Li, X., & Huang, W. (2019). Learning to Grasp Familiar Objects Based on Experience and Objects' Shape Affordance. IEEE Transactions on Systems, Man, and Cybernetics: Systems.
• Wang, T., Yang, C., Kirchner, F., Du, P., Sun, F., & Fang, B. (2019). Multimodal grasp data set: A novel visual–tactile data set for robotic manipulation. International Journal of Advanced Robotic Systems, 16(1), 1729881418821571.
• Fuchun, Sun, and Liu Huaping (2019). A Novel Multi-modal Tactile Sensor Design using Thermochromic Material. SCIENCE CHINA Information Sciences.
• Fang, B., Wei, X., Sun, F., Huang, H., Yu, Y., & Liu, H. (2019). Skill learning for human-robot interaction using wearable device. Tsinghua Science and Technology, 24(6), 654-662.
• Deng, Z., Guan, H., Huang, R., Liang, H., Zhang, L., & Zhang, J. (2019). Combining Model-Based Q-Learning With Structural Knowledge Transfer for Robot Skill Learning. IEEE Transactions on Cognitive and Developmental Systems, 11(1), 26-35.
• Starke, S., Hendrich, N., & Zhang, J. (2019). Memetic Evolution for Generic Full-Body Inverse Kinematics in Robotics and Animation. IEEE Transactions on Evolutionary Computation, 23(3), 406-420.
• Deng, Z., Gao, G., Frintrop, S., Zhang, J. (2019). Attention based visual analysis for fast grasp planning with multi-fingered robotic hand. Frontiers in Neurorobotics. doi: 10.3389/fnbot.2019.00060.
• Jing, M., Ma, X., Huang, W., Sun, F., & Liu, H. (2018). ask Transfer by Preference-Based Cost Learning. arXiv preprint arXiv:1805.04686.
• Deng, Z., Zheng, X., Zhang, L., & Zhang, J. (2018). A learning framework for semantic reach-to-grasp tasks integrating machine learning and optimization. Robotics and Autonomous Systems, 108, 140-152.
• Starke, S., Hendrich, N., Zhang, J. (2018). A Forward Kinematics Data Structure for Efficient Evolutionary Inverse Kinematics, in S. Zeghloul et al. (eds.), Computational Kinematics, Mechanisms and Machine Science 50, Springer 2018. DOI: 10.1007/978-3-319-60867-9_64
• Fang, B., Sun, F., Yang, C., Xue, H., Chen, W., Zhang, C., Guo, D., Liu, H. (2018). A Dual-Modal Vision-based Tactile Sensor for Robotic Hand Grasping, IEEE International Conference on Robotics and Automation, (ICRA-2018), Brisbane, Australia.
• Gao, G., Lauri, M., Zhang, J., and Frintrop, S. (2018). Occlusion Resistant Object Rotation Regression from Point Cloud Segments, Proceedings of the ECCV workshop on Recovering 6D Object Pose, 2018. https://arxiv.org/abs/1808.05498.
• Tao Kong, Fuchun Sun, et al. (2018). Deep Feature Pyramid Reconfiguration for Object Dectection. ECCV, 2018.
• Liang, H., Li, S., Görner, M., and Zhang, J. (2018). Generating Robust Grasps for Unknown Objects in Clutter Using Point Cloud Data. Shanghai International Symposium on Human-Centered Robotics (HCR), 299-303, 2018.
• Liang, H., and Zhao, Q. (2018). Multi-View CNNs for 3D Hand Pose Estimation, Shanghai International Symposium on Human-Centered Robotics (HCR), 255-259, 2018.
• Sebastian Starke, Norman Hendrich, and Jianwei Zhang (2018). Memetic Evolution for Generic Full-Body Inverse Kinematics in Robotics and Animation, IEEE Transactions on Evolutionary Computation, 2018. doi: 10.1109/TEVC.2018.2867601
• Jing M, Ma X, Sun F, et al. (2018). Learning and Inference Movement with Deep Generative Model. arXiv preprint arXiv:1805.07252, 2018.
• Ma X, Jing M, Sun F, et al. (2018). Adversarial Task Transfer from Preference. arXiv preprint arXiv:1805.04686, 2018
• Fuchun Sun, Wenchang Zhang, et al. (2018). Fused Fuzzy Petri Nets: A Shared Control Method for Brain Computer Interface Systems, IEEE Trans. On Cognitive and Developmental Systems, 2018.
• Ruppel, P., Hendrich, N., Starke, S., Zhang, J. (2018). Cost Functions to Specify Full-Body Motion and Multi-Goal Manipulation Tasks, IEEE International Conference on Robotics and Automation, (ICRA-2018), Brisbane, Australia.
• Han, D., Nie, H., Chen, J., Chen, M., Deng, Z., Zhang, J. (2018). Multi-modal haptic image recognition based on deep learning. Sensor Review.
• Ruppel, P., Jonetzko, J., Görner, M., Hendrich, N., Zhang, J. (2018). Simulation of the SynTouch BioTac Sensor, The 15th International Conference on Intelligent Autonomous Systems, (IAS-15 2018), Baden Baden, Germany.
• Tao Kong, Fuchun Sun, (2017). RON: Reverse Connection with Objectness Prior Networks for Object Detection, CVPR 2017.
• Huang, W., Harandi, M., Zhang, T., Fan, L., Sun, F., & Huang, J. (2017). Efficient optimization for linear dynamical systems with applications to clustering and sparse coding. Advances in Neural Information Processing Systems (NeurIPS), 3444-3454.
• Liu, C., Sun, F., Wang, C., Wang, F., & Yuille, A. (2017). MAT: A multimodal attentive translator for image captioning. Proc. International Joint Conference on Artificial Intelligence (IJCAI), arXiv preprint arXiv:1702.05658.
• Huang, Z., Sun, F., Min, H., Fang, B., Zhang, W., & Hu, X. (2017). A novel wearable tactile sensor array designed for fingertip motion recognition. 2017 IEEE International Conference on Robotics and Biomimetics (ROBIO) (pp. 165-170)
• Li, L., Sun, F., Fang, B., Huang, Z. Yang, C., & Jing, M. (2017). Learning to detect slip for stable grasping. In 2017 IEEE International Conference on Robotics and Biomimetics (ROBIO).
• Wasserfall, F., Hendrich, N., & Zhang, J. (2017). Adaptive slicing for the FDM process revisited. 2017 13th IEEE Conference on Automation Science and Engineering (CASE) (pp. 49-54)
• Starke, S., Hendrich, N., & Zhang, J. (2017). A memetic evolutionary algorithm for real-time articulated kinematic motion. 2017 IEEE Congress on Evolutionary Computation (CEC) (pp. 2473-2479).
• Starke, S., Hendrich, N., Krupke, D., & Zhang, J. (2017). Evolutionary multi-objective inverse kinematics on highly articulated and humanoid robots. 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (pp. 6959-6966).
• Bestmann, M., Wasserfall, F., Hendrich, N., & Zhang, J. (2017). Replacing cables on robotic arms by using serial via Bluetooth. 2017 IEEE International Conference on Robotics and Biomimetics (ROBIO) (pp. 189-195).
• Huaping Liu, Fuchun Sun, Di Guo, Bin Fang, (2017). Structured output-associated dictionary learning for haptic understanding, IEEE Transactions on Systems, Man and Cybernetics: Systems, vol.47, no.7, 2017, pp.1564-1574
• Fiedler, N., Bestmann, M., Hendrich, N. (2018). ImageTagger: An Open Source Online Platform for Collaborative Image Labeling, Robocup Symposium 2018, Montreal, Canada.
• Deng, Z., Guan, H., Huang, R., Liang, H., Zhang, L., Zhang, J. (2017). Combining Model-based Q-learning with Structural Knowledge Transfer for Robot Skill Learning. IEEE Transactions on Cognitive and Developmental Systems.
• Huang, T., Sun, F., Fang, B., (2017). A novel wearable tactile sensor for fingertip motion recognition. IEEE Intl. Conference on Robotics and Biomimetics, (ROBIO-2017), Macau, China.
• Li, X., Sun, F., Fang, B.,(2017). Learning to detect slip for stable grasping. IEEE Intl. Conference on Robotics and Biomimetics, (ROBIO-2017), Macau, China.
• Wei, X., Sun, F., Fang, B., (2017). Robotic skills learning using dynamical movement primitives from a wearable device. IEEE Intl. Conference on Robotics and Biomimetics, (ROBIO-2017), Macau, China.
• Bestmann, M., Wasserfall, F., Hendrich, N., Zhang, J. (2017). Replacing Cables on Robotic Arms by Using Serial via Bluetooth, IEEE Intl. Conference on Robotics and Biomimetics, (ROBIO-2017), Macau, China. DOI: 10.1109/ROBIO.2017.8324416
• Starke, S., Hendrich, N., Zhang, J. (2017). Multi-Objective Evolutionary Optimisation for Inverse Kinematics on Highly Articulated and Humanoid Robots, IEEE Intl. Conference on Intelligent Robots and Systems, (IROS-2017), Vancouver, Canada. DOI: 10.1109/IROS.2017.8206620
• Wasserfall, F., Hendrich, N., Fiedler, F., Zhang, J. (2017). 3D-Printed Low-Cost Modular Force Sensors, 20th Intl. Conference on Climbing and Walking Robots (CLAWAR-2017). In "Human-Centric Robotics", ISBN: 978-981-3231-03-0 (hardcover). ISBN: 978-981-3231-05-4 (ebook).
• Wasserfall, F., Hendrich, N., Zhang, J. (2017). Adaptive Slicing for the FDM Process Revisited, 13th IEEE Conference on Automation Science and Engineering (CASE-2017). DOI: 10.1109/COASE.2017.8256074
• Starke, S., Hendrich, N., Zhang, J. (2017). A Memetic Evolutionary Algorithm for Real-Time Articulated Kinematic Motion, IEEE Congress on Evolutionary Computation (CEC 2017), p 2473-2479, Sán Sebastian, Spain, 2017. DOI: 10.1109/CEC.2017.7969605
• Starke, S., Hendrich, N., Magg, S., Zhang, J. (2016). An Efficient Hybridization of Genetic Algorithms and Particle Swarm Optimization for Inverse Kinematics, IEEE Intl. Conference on Robotics and Biomimetics (ROBIO 2016), 3-7 Dec. 2016, Qingdao, China. DOI: 10.1109/ROBIO.2016.7866587

Relevant previous work

• Bernardino, A., Henriques, M., Hendrich, N., & Zhang, J. (2013, December). Precision grasp synergies for dexterous robotic hands. In Proc. IEEE Conference on Robotics and Biomimetics, ROBIO 2013 (pp. 62-67). IEEE. (Best Conference Paper Award).
• Chen, N., Zhu, J., Sun, F., & Xing, E. P. (2012). Large-margin predictive latent subspace learning for multiview data analysis. Pattern Analysis and Machine Intelligence, IEEE Transactions on, 34(12), 2365-2378.
• Cheng, G., Hendrich, N., & Zhang, J. (2012, October). Action gist based automatic segmentation for periodic in-hand manipulation movement learning. In Proc. IEEE Conference on Intelligent Robots and Systems, IROS 2012 pp. 4768-4775). IEEE.
• Guo, D., Sun, F., Zhang, J., & Liu, H. (2013, December). A grasp synthesis and grasp synergy analysis for anthropomorphic hand. Proc IEEE Conference on Robotics and Biomimetics, ROBIO 2013 (pp. 1617-1622). IEEE.
• Hendrich, N., Klimentjew, D., & Zhang, J. (2010, September). Multi-sensor based segmentation of human manipulation tasks. Proc. IEEE Conference on Multisensor Fusion and Integration for Intelligent Systems, MFI 2010. (pp. 223-229). IEEE.
• Hueser, M., Baier, T., & Zhang, J. (2006, May). Learning of demonstrated grasping skills by stereoscopic tracking of human head configuration. Proc. IEEE Conference on Robotics and Automation, ICRA 2006 (pp. 2795-2800). IEEE.
• Liu, H., & Sun, F. (2012). Fusion tracking in color and infrared images using joint sparse representation. In Science China Information Sciences, 55(3), 590-599.
• Pei, D., Liu, H., Liu, Y., & Sun, F. (2013, August). Unsupervised multimodal feature learning for semantic image segmentation. In Proc IEEE Conference on Neural Networks, IJCNN 2013 (pp. 1-6). IEEE.
• Wang, H. Q., Sun, F. C., Cai, Y. N., Ding, L. G., & Chen, N. (2010). An unbiased LSSVM model for classification and regression. Soft Computing, 14(2), 171-180.
• Zhang, J., & Rössler, B. (2004). Self-valuing learning and generalization with application in visually guided grasping of complex objects. Robotics and Autonomous systems, 47(2), 117-127.