Universität Hamburg - der Forschung, der Lehre, der Bildung, zur Homepage
leeres logo

Photo: UHH/Denstorf

Publications - Thematic Area C

Publications project C1 - Crossmodal speech representations and their implications in speech learning

Publications of the current project

  • Guo N, Si X, Zhang Y, Ding Y, Zhou W, Zhang D, Hong B. (2021). Speech frequency-following response in human auditory cortex is more than a simple tracking. Neuroimage, 2021 Feb 1;226:117545. doi: 10.1016/j.neuroimage.2020.117545
  • Yan Y, Qian T, Xu X, Han H, Ling Z, Zhou W, Liu H, Hong B (2019). Human cortical networking by probabilistic and frequency-specific coupling. Neuroimage 2020 Feb 15;207:116363. doi: 10.1016/j.neuroimage.2019.116363
  • Chen, Jingjing, Alexander Maye, Andreas K. Engel, Yijun Wang, Xiaorong Gao, and Dan Zhang. (2019). Simultaneous Decoding of Eccentricity and Direction Information for a Single-Flicker SSVEP BCI, Electronics 8, no. 12: 1554. https://doi.org/10.3390/electronics8121554

Publications of the phase-1 project

  • Zhang Q, Hu X, Hong B, Zhang B. (2019). A hierarchical sparse coding model predicts acoustic feature encoding in both auditory midbrain and cortex. PLoS Computational Biology.
  • Chen, J., Li, Z., Hong, B., Maye, A., Engel, A. K., & Zhang, D. (2019). A Single-Stimulus, Multitarget BCI Based on Retinotopic Mapping of Motion-Onset VEPs. IEEE Transactions on Biomedical Engineering, 66(2), 464-470.
  • Si X, Zhou W, Hong B. (2017). Cooperative cortical network for categorical processing of Chinese lexical tone. Proceedings of the National Academy of Sciences. 114(46):12303-12308.

Relevant previous work

  • Ewald, A., Aristei, S., Nolte, G., & Rahman, R. A. (2012). Brain oscillations and functional connectivity during overt language production. Frontiers in psychology, 3.
  • Guo, J., Gao, S., & Hong, B. (2010). An auditory brain–computer interface using active mental response. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 18(3), 230-235.
  • Hong, B., Guo, F., Liu, T., Gao, X., & Gao, S. (2009). N200-speller using motion-onset visual response. Clinical neurophysiology, 120(9), 1658-1666.
  • Lou, B., Hong, B., Gao, X., & Gao, S. (2008). Bipolar electrode selection for a motor imagery based brain–computer interface. Journal of Neural Engineering, 5(3), 342.
  • Marzetti, L., Del Gratta, C., & Nolte, G. (2008). Understanding brain connectivity from EEG data by identifying systems composed of interacting sources. Neuroimage, 42(1), 87-98.
  • Nolte, G., Bai, O., Wheaton, L., Mari, Z., Vorbach, S., & Hallett, M. (2004). Identifying true brain interaction from EEG data using the imaginary part of coherency. Clinical neurophysiology, 115(10), 2292-2307.
  • Nolte, G., Ziehe, A., Nikulin, V. V., Schlögl, A., Krämer, N., Brismar, T., & Müller, K. R. (2008). Robustly estimating the flow direction of information in complex physical systems. Physical review letters, 100(23), 234101.
  • Qian, T., Zhou, W., Ling, Z., Gao, S., Liu, H., & Hong, B. (2013). Fast presurgical functional mapping using task-related intracranial high gamma activity: Laboratory investigation. Journal of neurosurgery, 119(1), 26-36.
  • Zhang, D., Gong, E., Wu, W., Lin, J., Zhou, W., & Hong, B. (2012, August). Spoken sentences decoding based on intracranial high gamma response using dynamic time warping. In Engineering in Medicine and Biology Society (EMBC), 2012 Annual International Conference of the IEEE (pp. 3292-3295). IEEE.
  • Zhang, D., Song, H., Xu, R., Zhou, W., Ling, Z., & Hong, B. (2013). Toward a minimally invasive brain–computer interface using a single subdural channel: a visual speller study. Neuroimage, 71, 30-41.

Publications project C2 - Crossmodal learning of language – mechanisms and facilitation

Publications of the phase 1 project

  • Huang, X., Zhang, H., Chen, C., Xue, G., & He, Q. (2019). The neuroanatomical basis of the Gambler's fallacy: A univariate and multivariate morphometric study. Human brain mapping, 40(3), 967-975.
  • Feng, K., Zhao, X., Liu, J., Cai, Y., Ye, Z., Chen, C., & Xue, G. (2019). Spaced learning enhances episodic memory by increasing neural pattern similarity across repetitions. Journal of Neuroscience, 39(27), 5351-5360.
  • Zhu, B., Chen, C., Shao, X., Liu, W., Ye, Z., Zhuang, L., ... & Xue, G. (2019). Multiple interactive memory representations underlie the induction of false memory. Proceedings of the National Academy of Sciences, 116(9), 3466-3475.
  • Xue, G. (2018). The neural representations underlying human episodic memory. Trends in cognitive sciences, 22(6), 544-561.
  • Qu, J., Qian, L., Chen, C., Xue, G., Li, H., Xie, P., & Mei, L. (2017). Neural pattern similarity in the left IFG and fusiform is associated with novel word learning. Frontiers in human neuroscience, 11, 424.
  • Xue, H., Zhao, L., Wang, Y., Dong, Q., Chen, C., & Xue, G. (2017). Anodal transcranial direct current stimulation over the left temporoparietal cortex facilitates assembled phonology. Trends in neuroscience and education, 8, 10-17.
  • Zhao, L., Chen, C., Shao, L., Wang, Y., Xiao, X., Chen, C., Yang, J., Zevin, Jason., & Xue, G. (2016). Orthographic and phonological representations in the fusiform cortex. Cerebral Cortex, 27(11), 5197-5210.

Relevant previous work

  • He, Q., Xue, G., Chen, C., Chen, C., Lu, Z. L., & Dong, Q. (2013). Decoding the neuroanatomical basis of reading ability: A multimoxel morphometric study. Journal of Neuroscience, 33, 12835-12843.
  • Helfrich, R. F., Knepper, H., Nolte, G., Strüber, D., Rach, S., Herrmann, C. S., Schneider, T. R., & Engel, A. K. (2014). Selective modulation of interhemispheric functional connectivity by HD-tACS shapes perception. PLoS Biology, 12, e1002031.
  • Hipp, J. F., Engel, A. K., & Siegel, M. (2011). Oscillatory synchronization in large-scale cortical networks predicts perception. Neuron, 69, 387-396.
  • Xue, G., Chen, C., Zhen, J., & Dong, Q. (2006). Language experience shapes fusiform activation in processing a logographic artificial language: an fMRI Training Study. Neuroimage, 31, 1315-1326.
  • Xue, G., Dong, Q., Chen, C., Lu, Z. L., Mumford, J. A., & Poldrack, R. A. (2013). Complementary role of frontoparietal activity and cortical pattern similarity in successful episodic memory encoding. Cerebral Cortex, 23, 1562-1571.
  • Xue, G., Dong, Q., Chen, C., Lu, Z. L., & Poldrack, R. A. (2010). Greater neural pattern similarity across repetitions is associated with better later memory. Science, 330, 97-101.
  • Xue, G., Mei, L., Chen, C., Lu, Z. L., Poldrack, R. A., & Dong, Q. (2010). Facilitating memory for novel characters by reducing the neural repetition suppression in the left fusiform cortex. PLoS One, 5, e13204.
  • Xue, G., Juan, C. H., Chang, C. F., Lu, Z. L., & Dong, Q. (2012). Lateral prefrontal cortex contributes to maladaptive decisions. Proceedings of the National Academy of Sciences U. S. A., 109, 4401-4406.
  • Lu, Y., Wang, C., Chen, C., & Xue, G. (2015). Spatiotemporal neural pattern similarity supports episodic memory. Current Biology, 25, 780-785.
  • Womelsdorf, T., Schoffelen, J.-M., Oostenveld, R., Singer, W., Desimone, R., Engel, A. K., & Fries, P. (2007). Modulation of neuronal interactions through neuronal synchronization. Science, 316, 1609-1612.

Publications project C4 - Vision- and action-embodied language learning

Publications of the current project

  • Yuan Yao, Ao Zhang, Xu Han, Mengdi Li, Cornelius Weber, Zhiyuan Liu, Stefan Wermter, Maosong Sun (2021). Visual Distant Supervision for Scene Graph Generation. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021, pp. 15816-15826.
  • Mengdi Li, Cornelius Weber, Matthias Kerzel, Jae Hee Lee, Zheni Zeng, Zhiyuan Liu, Stefan Wermter (2021). Robotic Occlusion Reasoning for Efficient Object Existence Prediction. IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Oct 2021.
  • Huajian Fang, Guillaume Carbajal, Stefan Wermter, Timo Gerkmann (2021). Variational Autoencoder for Speech Enhancement with a Noise-Aware Encoder. IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2021, pp. 676-680, doi: 10.1109/ICASSP39728.2021.9414060.
  • Ozan Özdemir, Matthias Kerzel, Stefan Wermter (2021). Embodied Language Learning with Paired Variational Autoencoders. IEEE International Conference on Development and Learning (ICDL), pages 1-6, doi:10.1109/ICDL49984.2021.9515668, Aug 2021.
  • Hugo Carneiro, Cornelius Weber, Stefan Wermter (2021). FaVoA: Face-Voice Association Favours Ambiguous Speaker Detection. Proceedings of the 30th International Conference on Artificial Neural Networks (ICANN 2021), Volume LNCS 12891, pages 439-450, doi:10.1007/978-3-030-86362-3_36, Sep 2021.
  • Julien Scholz, Cornelius Weber, Burhan Hafez, Stefan Wermter (2021). Improving Model-Based Reinforcement Learning with Internal State Representations through Self-Supervision. Proceedings of the International Joint Conference on Neural Networks (IJCNN 2021), Jul 2021.
  • Aaron Eisermann, Jae Hee Lee, Cornelius Weber, Stefan Wermter (2021). Generalization in Multimodal Language Learning from Simulation. Proceedings of the International Joint Conference on Neural Networks (IJCNN 2021), Jul 2021.
  • Frank Röder, Ozan Özdemir, Phuong D. H. Nguyen, Stefan Wermter and Manfred Eppe (2021). The Embodied Crossmodal Self Forms Language and Interaction: A Computational Cognitive Review. Frontiers in Psychology, Vol. 12, doi: 10.3389/fpsyg.2021.716671
  • Matthias Möller, Johannes Twiefel, Cornelius Weber, Stefan Wermter (2021). Controlling the Noise Robustness of End-to-End Automatic Speech Recognition Systems. Proceedings of the International Joint Conference on Neural Networks (IJCNN 2021), Jul 2021.
  • Victor Uc-Cetina, Nicolás Navarro-Guerrero, Anabel Martin-Gonzalez, Cornelius Weber, Stefan Wermter (2021). Survey on reinforcement learning for language processing. arXiv: 2104.05565, Apr 2021.
  • Vadym Gryshchuk, Cornelius Weber, Chu Kiong Loo, Stefan Wermter (2021). Lifelong Learning from Event-based Data. European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning (ESANN '21), Oct 2021.
  • Christoper Wulur, Cornelius Weber, Stefan Wermter (2021). Planning-integrated Policy for Efficient Reinforcement Learning in Sparse-reward Environments. Proceedings of the International Joint Conference on Neural Networks (IJCNN 2021), Jul 2021.
  • Jie Zhou, Shengding Hu, Xin Lv, Cheng Yang, Zhiyuan Liu, Wei Xu, Jie Jiang, Juanzi Li, Maosong Sun (2021). KACC: A Multi-task Benchmark for Knowledge Abstraction, Concretization and Completion. Findings of the Association for Computational Linguistics (ACL-IJCNLP 2021), August 2021.

Publications of the phase 1 project

  • Springenberg S., Lakomkin E., Weber C., Wermter S.(2019) Predictive Auxiliary Variational Autoencoder for Representation Learning of Global Speech Characteristics. Proc. International Conference on Spoken Language Processing (INTERSPEECH), 2019.
  • Yu, N., Weber, C., & Hu, X. (2019) Learning sparse hidden states in Long Short-Term Memory. Proc. 28th International Conference on Artificial Neural Networks and Machine Learning, ICANN 2019.
  • Xie, R., Heinrich, S., Liu, Z., Weber, C., Yao, Y., Wermter, S., & Sun, M. (2019). Integrating Image-based and Knowledge-based Representation Learning. IEEE Transactions on Cognitive and Developmental Systems, early access.
  • Heinrich, S., Springstübe, P., Knöppler, T., Kerzel, M., & Wermter, S. (2019). Continuous convolutional object tracking in developmental robot scenarios. Neurocomputing. 342, 137-144.
  • Weber, C. (2019). Lernverfahren fur neuronale Netze und Anwendungen in der Robotik. Mitt. Math. Ges. Hamburg, 39, 27-58
  • Hinz, T., Heinrich, S., & Wermter, S. (2019). Generating multiple objects at spatially distinct locations. International Conference on Learning Representations (ICLR), arxiv.org/abs/1901.00686
  • Zhang Z., Han X., Liu Z., Jiang X., Sun M., Liu Q. (2019). ERNIE: Enhanced Language Representation with Informative Entities. Proc. of the 57th Annual Meeting of the Association for Computational Linguistics (ACL).
  • Wang, X., Han, X., Lin, Y., Liu, Z., & Sun, M. (2018). Adversarial multi-lingual neural relation extraction. Proceedings of the 27th International Conference on Computational Linguistics, 1156-1166.
  • Han, X., Zhu, H., Yu, P., Wang, Z., Yao, Y., Liu, Z., & Sun, M. (2018). Fewrel: A large-scale supervised few-shot relation classification dataset with state-of-the-art evaluation. arXiv preprint arXiv:1810.10147.
  • Han, X., Yu, P., Liu, Z., Sun, M., & Li, P. (2018). Hierarchical Relation Extraction with Coarse-to-Fine Grained Attention. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. 2236-2245.
  • Shen, S. Q., Chen, Y., Yang, C., Liu, Z. Y., & Sun, M. S. (2018). Zero-Shot Cross-Lingual Neural Headline Generation. IEEE/ACM Transactions on Audio, Speech and Language Processing (TASLP), 26(12), 2319-2327.
  • Heinrich, S., & Wermter, S. (2018). Interactive natural language acquisition in a multi-modal recurrent neural architecture. Connection Science, 30(1), 99-133.
  • Heinrich, S., Alpay, T., Wermter, S. (2018). Adaptive and Variational Continuous Time Recurrent Neural Networks. Proc. IEEE ICDL-EpiRob 2018.
  • Heinrich, S., Kerzel, M., Strahl, E., Wermter, S. (2018). Embodied Multi-modal Interaction in Language learning: the EMIL data collection. Proc. ICDL-EpiRob Workshop on Active Vision, Attention, and Learning 2018.
  • Springstübe, P., Heinrich, S., Wermter, S. (2018). Continuous convolutional object tracking. Proc. ESANN, 2018.
  • Lin Y., Liu Z., Sun M. (2017). Neural Relation Extraction with Multi-lingual Attention. The 55th Annual Meeting of the Association for Computational Linguistics (ACL), 34-43.
  • Niu Y., Xie R., Liu Z., Sun M. (2017). Improved Word Representation Learning with Sememes. The 55th Annual Meeting of the Association for Computational Linguistics (ACL), 2049-2058.
  • Ng, H.G. et al., (2017). Hey Robot, Why Don't You Talk to Me? Proc. 26th IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN 2017).
  • Xie, R., Liu Z., Luan H., Sun M. (2017). Image-embodied Knowledge Representation Learning. Proceedings of International Joint Conference on Artificial Intelligence.
  • Heinrich, S., Weber, C., Wermter, S., Xie, R., Lin, Y., Liu, Z. (2016). Crossmodal language grounding, learning, and teaching. NIPS2016 Workshop on Cognitive Computation, pp. 62-68, Barcelona, Spain.

Relevant previous work

  • Chen, X., Liu, Z., & Sun, M. (2014). A Unified Model for Word Sense Representation and Disambiguation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP'14) (pp.1025-1035). Association for Computational Linguistics.
  • Heinrich, S., Magg, S., & Wermter, S. (2015). Analysing the Multiple Timescale Recurrent Neural Network for Embodied Language Understanding. In Artificial Neural Networks - Methods and Applications (pp. 149-174). Springer International Publishing.
  • Heinrich, S., & Wermter, S. (2014). Interactive Language Understanding with Multiple Timescale Recurrent Neural Networks. In Proceedings of the 24th International Conference on Artificial Neural Networks (ICANN 2014) (pp. 193-200), Springer International Publishing.
  • Kleesiek, J., Engel, A. K., Weber, C., & Wermter, S. (2011, August). Reward-driven learning of sensorimotor laws and visual features. In Proceedings of the International Conference on Development and Learning and Epigenetic Robotics (ICDL-EpiRob) (pp. 1-6). IEEE Xpress.
  • Lin, Y., Liu, Z., Sun, M., Liu, Y., & Zhu, X. (2015, February). Learning Entity and Relation Embeddings for Knowledge Graph Completion. In Proceedings of the29th AAAI Conference on Artificial Intelligence (AAAI'15) (pp. 2181-2187). Association for the Advancement of Artificial Intelligence.
  • Saeb, S., Weber, C., & Triesch, J. (2009). Goal-directed learning of features and forward models. Neural Networks, 22(5-6), 586-592, Elsevier
  • Weber, C., Wermter, S., & Elshaw, M. (2006). A hybrid generative and predictive model of the motor cortex. Neural Networks, 19(4), 339-353.
  • Wermter, S., Weber, C., Elshaw, M., Gallese, V., & Pulvermüller, F. (2005). Grounding neural robot language in action. In Wermter S, Palm G, & Elshaw M. (Eds.), Biomimetic Neural Learning for Intelligent Robots (pp.162-181). Springer Berlin Heidelberg.
  • Zhao, Y., Liu, Z., & Sun, M. (2015, February). Phrase Type Sensitive Tensor Indexing Model for Semantic Composition. In Proceedings of the 29th Conference on Artificial Intelligence (AAAI 2015) (pp. 2195-2201).
  • Zhong, J., Weber, C., & Wermter, S. (2012). A predictive network architecture for a robust and smooth robot docking behavior. Paladyn. Journal of Behavioral Robotics, 3(4), 172-180.

Publications project C5 - Combining active vision with multimodal speech comprehension

Publications of the phase 1 project

  • Alaçam, Ö. (2019). Enhancing natural language understanding through cross-modal interaction: Meaning recovery from acoustically noisy speech. In Proceedings of the 22nd Nordic Conference on Computational Linguistics (NoDaLiDa), Association for Computational Linguistics, Turku, Finland.
  • Alaçam, Ö., Menzel, W., and Staron, T. (2019-accepted). How does visual complexity influence predictive language processing in a situated context? In Proceedings of the 15th KONVENS – Bridging the gap between NLP and human understanding, Friedrich-Alexander-Universität Erlangen-Nürnberg, Germany.
  • Alaçam, Ö., Menzel, W. and Staron, T. (2019). Syntactic and Semantic Disambiguation in a Situated Language Setting. EuroCogsci. 2019. Bochum.
  • Qu, Q., & Damian, M. F. (2019). Orthographic effects in Mandarin spoken language production. Memory & cognition, 47(2), 326-334.
  • Lou, Y., Cai, H., Liu, X., & Li, X. (2019). Effects of self-enhancement on eye movements during reading. Frontiers in psychology, 10.
  • Qu, Q., & Damian, M. F. (2019). The role of orthography in second-language spoken word production: Evidence from Tibetan Chinese bilinguals. Quarterly Journal of Experimental Psychology.
  • Xu, P., Qu, Q., Shen, W., & Li, X. (2019). Co-activation of taxonomic and thematic relations in spoken word comprehension: Evidence from eye movements. Frontiers in psychology, 10, 964.
  • 韩海宾, 许萍萍, 屈青青, 程茜, & 李兴珊. (2019). 语言加工过程中的视听跨通道整合. 心理科学进展, 27(3), 475-489.
  • Chen, L., Zhou, X., Müller, H. J., & Shi, Z. (2018). What you see depends on what you hear: Temporal averaging and crossmodal integration. Journal of Experimental Psychology: General, 147(12), 1851.
  • Salama, A. R., Alaçam, Ö and Menzel, W. (2018) Text Completion using Context-Integrated Dependency Parsing. In the Proceedings of 3rd Workshop on Representation Learning for NLP, ACL 2018.
  • Qu, Q., Cui, Z. L., and Damian, M. F(2018). Orthographic effects in second-language spoken word recognition. Journal of Experimental Psychology: Learning, Memory, and Cognition, 44(8), 1325 1332.
  • Alaçam, Ö., Staron, T., and Menzel, W. (2018). A multi-modal data-set for systematic analyses of linguistic ambiguities in situated contexts. In Annual International Symposium on Information Management and Big Data (pp. 106-122). Springer.
  • Staron, T., Alaçam, Ö., and Menzel, W. (2018). Incorporating Contextual Information for Language-Independent, Dynamic Disambiguation Tasks. In Proceedings of the 11th Language Resources and Evaluation Conference (LREC), Miyazaki, Japan.
  • 屈青青*，刘维琳，李兴珊 (2018). 汉语言语产生的语音加工单元—基于音位的研究. 心理科学进展, 26(9), 1535-1544.
  • Alaçam, Ö., Staron, T. and Menzel, W. (2017). Towards a Systematic Analysis of Linguistic and Visual Complexity in Disambiguation and Structural Prediction. In the Proceedings of the 5th International Conference on Information Management and Big Data 4-6 September 2017. Lima, Peru.
  • Alaçam, Ö. and Habel, C. (2017). Negation in Gesture Accompanying Speech: A Cross-Modal Study. Journal of Multimodal Communication Studies, Vol. 4, issue 1-2.
  • Shen, W., Qu, Q., Ni, A., Zhou, J., & Li, X. (2017). The time course of morphological processing during spoken word recognition in Chinese. Psychonomic Bulletin & Review, 1-7. doi:10.3758/s13423-017-1274-z.
  • Qu, Q. Q., & Damian, M. F. (2017). Orthographic effects in spoken word recognition: Evidence from Chinese, Psychonomic Bulletin & Review, 1-6.

Relevant previous work

  • Baumgärtner, C., Beuck, N., & Menzel, W. (2012). An Architecture for Incremental Information Fusion of Cross-Modal Representations. In Proc. IEEE Conference on Multisensor Fusion and Integration for Intelligent Systems, MFI-2012 (pp. 498-503). IEEE
  • Beuck, N., & Menzel, W. (2013). Structural prediction in Incremental Dependency Parsing, In Proc. 14th Int. Conf. in Intelligent Text Processing and Computational Linguistics, CICLing-2013 (pp. 245-257). Springer
  • Foth, K. A., & Menzel, W. (2006). Hybrid Parsing: Using Probabilistic Models as Predictors for a Symbolic Parser, In Proc. 21st Int. Conf. on Computational Linguistics and 44th Annual Meeting of the Association for Computational Linguistics, Coling-ACL-2006 (pp. 321-328). IEEE.
  • Köhn, A., & Menzel, W. (2014). Incremental Predictive Parsing with TurboParser, In Proc. 52nd Annual Meeting of the Association for Computational Linguistics (pp. 803-808).
  • Li, X., Bicknell, K., Liu, P., Wei, W., & Rayner, K. (2014). Reading is fundamentally similar across disparate writing systems: A systematic characterization of how words and characters influence eye movements in Chinese
  • reading. Journal of Experimental Psychology: General, 143, 895-913.
  • Li, X., Gu, J., Liu, P., & Rayner, K. (2013). The advantage of word-based processing in Chinese reading: Evidence from eye movements. Journal of Experimental Psychology: Learning, Memory and Cognition, 39, 879-889.
  • Li, X., & Logan, G. D. (2008). Object-based attention in Chinese readers of Chinese words: Beyond Gestalt principles. Psychonomic Bulletin & Review, 15, 945-949.
  • McCrae, P., Menzel, W., & Sun, M. (2011). A computational model of concept generalization in cross-modal reference. Tsinghua Science & Technology, 16, 113-120.
  • Menzel, W. (2009). Towards radically incremental parsing of natural language. In Nicolov, N., & Angelova, G. (Eds.), Proc. 5th Int. Conf. on Recent Advances in Natural Language Processing (pp. 41-56). Amsterdam &
  • Philadelphia: John Benjamin's Publishers.
  • Rayner, K., Li, X., & Pollatsek, A. (2007). Extending the E-Z Reader model of eye movement control to Chinese reading. Cognitive Science, 31, 1021-1033.

Publications project C6 - Multisensory perception, learning and action for crossmodal tele-operations

Publications of the phase 1 project

  • Zeng, H., & Chen, L. (2019). Robust temporal averaging of time intervals between action and sensation. Frontiers in psychology, 10, 511.
  • Chen, L. (2019). Discrimination of empty and filled intervals marked by auditory signals with different durations and directions of intensity change. PsyCh journal, 8(2), 187-202.
  • Shen, L., Han, B., Chen, L., & Chen, Q. (2019). Perceptual inference employs intrinsic alpha frequency to resolve perceptual ambiguity. PLoS biology, 17(3), e3000025.
  • Zheng, W., & Chen, L. (2019). Illusory perception of auditory filled duration is task‐and context‐dependent. British Journal of Psychology.
  • Li, B., Chen, L., & Fang, F. (2019). Somatotopic representation of tactile duration: evidence from tactile duration aftereffect. Behavioural brain research, 371, 111954.
  • He, D., Wang, Y., & Fang, F. (2019). The Critical Role of V2 Population Receptive Fields in Visual Orientation Crowding. Current Biology.
  • Katzakis, N., Chen,L., Ariza, O., Teather,R., and Steinicke, F.(2019).Evaluation of 3D Pointing Accuracy in the Fovea and Periphery in Immersive Head-Mounted Display Environments, IEEE Transaction on visualization and computer graphics
  • Wang, Y., Zhu, Z., Chen, B., & Fang, F. (2019). Perceptual learning and recognition confusion reveal the underlying relationships among the six basic emotions. Cognition and Emotion, 33(4), 754-767.
  • Chen, L.*, Zhou, X., Müller, H.J., Shi, Z. (in press) What you see depends on what you hear: temporal averaging and crossmodal integration, Journal of Experimental Psychology: General.
  • Zheng,W.* & Chen, L.* (2018). The Roles of Attentional Shifts and Attentional Reengagement in Resolving The Spatial Compatibility Effect in Tactile Simon-like Tasks. Scientific Reports, 8:8760. doi:10.1038/s41598-018-27114-9.
  • Wan, Y. & Chen, L.* (2018). Temporal Reference, Attentional Modulation, and Crossmodal Assimilation. Front. Comput. Neurosci. 12:39. doi: 10.3389/fncom.2018.00039.
  • Na R., Bi T., Tjan B.S., Liu Z. and Fang F. (2018). Effect of task difficulty on blood-oxygen-level-dependent signal: a functional magnetic resonance imaging study in a motion discrimination task. PLoS ONE 13(6): e0199440.
  • Wang Y., Zhu Z., Chen B. and Fang F. (2018) Perceptual learning and recognition confusion reveal the underlying relationships among the six basic emotions. Cognition and Emotion. https://doi.org/10.1080/02699931.2018.1491831
  • Yiltiz, H. & Chen, L.*（2018）. Emotional cues and social anxiety resolve ambiguous perception of biological motion. Exp Brain Res. 236：1409-1420
  • Tian,Y. & Chen, L.* (2018). Cross-modal attention modulates tactile subitizing but not tactile numerosity estimation. Attention, Perception, & Psychophysics, https://doi.org/10.3758/s13414-018-1507-x
  • Mo C.*, He D.* and Fang F. (2018). Attention priority map of face images in human early visual cortex. Journal of Neuroscience. 38(1), 149-157
  • Zhang, J., Langbehn, E., Krupke, D., Katzakis, N., & Steinicke, F. (2018). Detection Thresholds for Rotation and Translation Gains in 360° Video-Based Telepresence Systems. IEEE transactions on visualization and computer graphics, 24(4), 1671-1680.
  • Zhang, J., Langbehn, E., Krupke, D., Katzakis, N., & Steinicke, F. (2018). A 360 video-based robot platform for telepresent redirected walking.
  • Freiwald, J. P., Katzakis, N., & Steinicke, F. (2018). Camera time warp: compensating latency in video see-through head-mounted-displays for reduced cybersickness effects. In Proceedings of the 24th ACM Symposium on Virtual Reality Software and Technology. 9.
  • Ariza, O., Bruder, G., Katzakis, N., & Steinicke, F. (2018). Analysis of Proximity-Based Multimodal Feedback for 3D Selection in Immersive Virtual Environments. In 2018 IEEE Conference on Virtual Reality and 3D User Interfaces (VR). 327-334.
  • Katzakis, N., & Steinicke, F. (2018). Excuse me! perception of abrupt direction changes using body cues and paths on mixed reality avatars. arXiv preprint arXiv:1801.05085.
  • Chen N., Lu J., Shao H., Weng X. and Fang F. (2017). Neural mechanisms of motion perceptual learning in noise. Human Brain Mapping. 38(12), 6029-6042
  • Chen, L., Guo, L., Bao,M. (2017). "Sleep-dependent consolidation benefits fast transfer of time interval training". Experimental Brain Research, 235(3),661-672.
  • Guo,L., Bao, M., Guan, L., Chen, L.. (2017). Cognitive Styles Differentiate Crossmodal Correspondences Between Pitch Glide and Visual Apparent Motion, Multisensory Research.30,363-385.
  • He D., Mo C. and Fang F. (2017). Predictive feature remapping before saccadic eye movements. Journal of Vision 17(5):14, 1-12.

Relevant previous work

  • Bruder, G., Steinicke, F., & Sturzlinger, W. (2013, July). To touch or not to touch?: comparing 2D touch and 3D mid-air interaction on stereoscopic tabletop surfaces. In Proceedings of the 1st symposium on Spatial user interaction (pp. 9-16). ACM.
  • Bruder, G., Lubos, P., & Steinicke, F. (2015). Cognitive Resource Demands of Redirected Walking. IEEE Trans. on Visualization and Computer Graphics (TVCG), 4, 539-544.
  • Chen, L., Wang, Q., & Bao, M. (2014). Spatial references and audio-tactile interaction in cross-modal dynamic capture. Multisensory Research, 27, 55-70.
  • Chen, L., & Vroomen, J (2013). Intersensory Binding across Space and Time: A Tutorial Review. Attention, Perception, & Psychophysics, 75, 790-811.
  • Chen, L., Shi, Z., & Müller, H. J. (2011). Interaction of Perceptual Grouping and Crossmodal Temporal Capture in Tactile Apparent-Motion. PLoS ONE, 6(2), e17130.
  • Chen, J., He, Y., Zhu, Z., Zhou, T., Peng, Y., Zhang, X., & Fang, F. (2014). Attention-dependent early Cortical Suppression contributes to Crowding. Journal of Neuroscience. 34(32), 10465-10474.
  • Daiber, F., Valkov, D., Steinicke, F., Hinrich, K., & Krüger, A. (2012). Towards Object Prediction based on Hand Postures for Reach to Grasp Interaction. In Proceedings of the ACM, the 3rd Dimension of CHI (3DCHI) (pp.99-106). ACM.
  • Fang, F., & He, S. (2005). Cortical Responses to Invisible Objects in Human Dorsal and Ventral Pathways. Nature Neuroscience. 8(10), 1380-1385.
  • Steinicke, F., Bruder, G., Jerald, J., Frenz, H., & Lappe, M. (2010). Estimation of Detection Thresholds for Redirected Walking Techniques. IEEE Trans. on Visualization and Computer Graphics (TVCG), 16(1), 17-27
  • Zhang, X., Zhaoping, L., Zhou, T., & Fang, F. (2012). Neural Activities in V1 create a Bottom-Up Saliency Map. Neuron 73(1), 183-192.

Publications project C7 - Cross-modal Learning for Improving Human Reading

Publications of the current project

  • Yao, P., Slattery, T. J., & Li, X. (in press). Sentence context modulates the neighborhood frequency effect: Evidence from eye movements while reading Chinese.  Journal of Experimental Psychology: Learning, Memory, and Cognition.
  • Yao, P., Staub, A., & Li, X. (2021). Predictability Eliminates Neighborhood Effects During Chinese Sentence.  Psychonomic Bulletin & Review. https://doi.org/10.3758/s13423-021-01966-1
  • Huang, L., Staub, A., & Li, X. (2021). Prior context influences lexical competition when segmenting Chinese overlapping ambiguous strings.  Journal of Memory and Language, 118
  • Florian Schneider, Özge Alaçam, Xintong Wang, Chris Biemann (2021). Towards Multi-Modal Text-Image Retrieval to improve Human Reading. NAACL SRW 2021.
  • Özge Alaçam, Ganeshan Malhotra, Eugen Ruppert, Chris Biemann (2021). Gaze-based Multimodal Meaning Recovery for Noisy / Complex Environments. ICMI 2021: 673-681
  • Li, X., & Pollatsek, A. (2020). An integrated model of word processing and eye-movement control during Chinese reading.  Psychological Review, 127(6), 1139-1162.
  • Han, H. & Li, X.(2020). Degree of conceptual overlap affects eye movements in visual world paradigm. Language, Cognition and Neuroscience, 35(10), 1456-1464. 
  • Huang, L., & Li, X. (2020). Early, but not overwhelming: The effect of prior context on segmenting overlapping ambiguous strings when reading Chinese.  Quarterly Journal of Experimental Psychology. 73(9), 1382-1395. https://doi.org/10.1177/1747021820926012

Publications project C8 - Crossmodal bindings and plasticity during visual-haptic interactions for novel forms of therapy

Publications of the current project

  • Hartfill, Gabel, Kruse, Schmidt, Riebandt, Kühn & Steinicke (2021). Analysis of Detection Thresholds for Hand Redirection during Mid-Air Interactions in Virtual Reality. VRST ’21, December 8–10, 2021, Osaka, Japan.
  • Katzakis, Chen, Ariza, Teacher, & Steinicke (2021). Evaulation of 3D Pointing Accuracy in the Fovea and Periphery in Immersive Head-Mounted Display Environments. IEEE Transaction on Visualization and Computer Graphics, 27:3,1929-1936.
  • Gong, Gu, Wang & Chen (2021). Interoception visualization relieves acute pain. Biological Psychology, in minor revision.
  • Katzakis, Chen & Steinicke (2020). Visual-Haptic Size Estimation in Peripersonal Space. Frontiers in Neurorobotics, 14:18.

Publications project C9 - The role of mental models and sense of agency in learning crossmodal communicative acts

Publications of the current project