Publications - Thematic Area C
Publications project C1 - Crossmodal speech representations and their implications in speech learning
Publications of the current project
-
Guo N, Si X, Zhang Y, Ding Y, Zhou W, Zhang D, Hong B. (2021).
Speech frequency-following response in human auditory cortex is more than a simple tracking.
Neuroimage, 2021 Feb 1;226:117545. doi: 10.1016/j.neuroimage.2020.117545
-
Yan Y, Qian T, Xu X, Han H, Ling Z, Zhou W, Liu H, Hong B (2019).
Human cortical networking by probabilistic and frequency-specific coupling.
Neuroimage 2020 Feb 15;207:116363. doi: 10.1016/j.neuroimage.2019.116363
-
Chen, Jingjing, Alexander Maye, Andreas K. Engel, Yijun Wang, Xiaorong Gao, and Dan Zhang. (2019).
Simultaneous Decoding of Eccentricity and Direction Information for a Single-Flicker SSVEP BCI,
Electronics 8, no. 12: 1554. https://doi.org/10.3390/electronics8121554
Publications of the phase-1 project
- Zhang Q, Hu X, Hong B, Zhang B. (2019). A hierarchical sparse coding model predicts acoustic feature encoding in both auditory midbrain and cortex. PLoS Computational Biology.
- Chen, J., Li, Z., Hong, B., Maye, A., Engel, A. K., & Zhang, D. (2019). A Single-Stimulus, Multitarget BCI Based on Retinotopic Mapping of Motion-Onset VEPs. IEEE Transactions on Biomedical Engineering, 66(2), 464-470.
- Si X, Zhou W, Hong B. (2017). Cooperative cortical network for categorical processing of Chinese lexical tone. Proceedings of the National Academy of Sciences. 114(46):12303-12308.
Relevant previous work
- Ewald, A., Aristei, S., Nolte, G., & Rahman, R. A. (2012). Brain oscillations and functional connectivity during overt language production. Frontiers in psychology, 3.
- Guo, J., Gao, S., & Hong, B. (2010). An auditory brain–computer interface using active mental response. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 18(3), 230-235.
- Hong, B., Guo, F., Liu, T., Gao, X., & Gao, S. (2009). N200-speller using motion-onset visual response. Clinical neurophysiology, 120(9), 1658-1666.
- Lou, B., Hong, B., Gao, X., & Gao, S. (2008). Bipolar electrode selection for a motor imagery based brain–computer interface. Journal of Neural Engineering, 5(3), 342.
- Marzetti, L., Del Gratta, C., & Nolte, G. (2008). Understanding brain connectivity from EEG data by identifying systems composed of interacting sources. Neuroimage, 42(1), 87-98.
- Nolte, G., Bai, O., Wheaton, L., Mari, Z., Vorbach, S., & Hallett, M. (2004). Identifying true brain interaction from EEG data using the imaginary part of coherency. Clinical neurophysiology, 115(10), 2292-2307.
- Nolte, G., Ziehe, A., Nikulin, V. V., Schlögl, A., Krämer, N., Brismar, T., & Müller, K. R. (2008). Robustly estimating the flow direction of information in complex physical systems. Physical review letters, 100(23), 234101.
- Qian, T., Zhou, W., Ling, Z., Gao, S., Liu, H., & Hong, B. (2013). Fast presurgical functional mapping using task-related intracranial high gamma activity: Laboratory investigation. Journal of neurosurgery, 119(1), 26-36.
- Zhang, D., Gong, E., Wu, W., Lin, J., Zhou, W., & Hong, B. (2012, August). Spoken sentences decoding based on intracranial high gamma response using dynamic time warping. In Engineering in Medicine and Biology Society (EMBC), 2012 Annual International Conference of the IEEE (pp. 3292-3295). IEEE.
- Zhang, D., Song, H., Xu, R., Zhou, W., Ling, Z., & Hong, B. (2013). Toward a minimally invasive brain–computer interface using a single subdural channel: a visual speller study. Neuroimage, 71, 30-41.
Publications project C2 - Crossmodal learning of language – mechanisms and facilitation
Publications of the phase 1 project
- Huang, X., Zhang, H., Chen, C., Xue, G., & He, Q. (2019). The neuroanatomical basis of the Gambler's fallacy: A univariate and multivariate morphometric study. Human brain mapping, 40(3), 967-975.
- Feng, K., Zhao, X., Liu, J., Cai, Y., Ye, Z., Chen, C., & Xue, G. (2019). Spaced learning enhances episodic memory by increasing neural pattern similarity across repetitions. Journal of Neuroscience, 39(27), 5351-5360.
- Zhu, B., Chen, C., Shao, X., Liu, W., Ye, Z., Zhuang, L., ... & Xue, G. (2019). Multiple interactive memory representations underlie the induction of false memory. Proceedings of the National Academy of Sciences, 116(9), 3466-3475.
- Xue, G. (2018). The neural representations underlying human episodic memory. Trends in cognitive sciences, 22(6), 544-561.
- Qu, J., Qian, L., Chen, C., Xue, G., Li, H., Xie, P., & Mei, L. (2017). Neural pattern similarity in the left IFG and fusiform is associated with novel word learning. Frontiers in human neuroscience, 11, 424.
- Xue, H., Zhao, L., Wang, Y., Dong, Q., Chen, C., & Xue, G. (2017). Anodal transcranial direct current stimulation over the left temporoparietal cortex facilitates assembled phonology. Trends in neuroscience and education, 8, 10-17.
- Zhao, L., Chen, C., Shao, L., Wang, Y., Xiao, X., Chen, C., Yang, J., Zevin, Jason., & Xue, G. (2016). Orthographic and phonological representations in the fusiform cortex. Cerebral Cortex, 27(11), 5197-5210.
Relevant previous work
- He, Q., Xue, G., Chen, C., Chen, C., Lu, Z. L., & Dong, Q. (2013). Decoding the neuroanatomical basis of reading ability: A multimoxel morphometric study. Journal of Neuroscience, 33, 12835-12843.
- Helfrich, R. F., Knepper, H., Nolte, G., Strüber, D., Rach, S., Herrmann, C. S., Schneider, T. R., & Engel, A. K. (2014). Selective modulation of interhemispheric functional connectivity by HD-tACS shapes perception. PLoS Biology, 12, e1002031.
- Hipp, J. F., Engel, A. K., & Siegel, M. (2011). Oscillatory synchronization in large-scale cortical networks predicts perception. Neuron, 69, 387-396.
- Xue, G., Chen, C., Zhen, J., & Dong, Q. (2006). Language experience shapes fusiform activation in processing a logographic artificial language: an fMRI Training Study. Neuroimage, 31, 1315-1326.
- Xue, G., Dong, Q., Chen, C., Lu, Z. L., Mumford, J. A., & Poldrack, R. A. (2013). Complementary role of frontoparietal activity and cortical pattern similarity in successful episodic memory encoding. Cerebral Cortex, 23, 1562-1571.
- Xue, G., Dong, Q., Chen, C., Lu, Z. L., & Poldrack, R. A. (2010). Greater neural pattern similarity across repetitions is associated with better later memory. Science, 330, 97-101.
- Xue, G., Mei, L., Chen, C., Lu, Z. L., Poldrack, R. A., & Dong, Q. (2010). Facilitating memory for novel characters by reducing the neural repetition suppression in the left fusiform cortex. PLoS One, 5, e13204.
- Xue, G., Juan, C. H., Chang, C. F., Lu, Z. L., & Dong, Q. (2012). Lateral prefrontal cortex contributes to maladaptive decisions. Proceedings of the National Academy of Sciences U. S. A., 109, 4401-4406.
- Lu, Y., Wang, C., Chen, C., & Xue, G. (2015). Spatiotemporal neural pattern similarity supports episodic memory. Current Biology, 25, 780-785.
- Womelsdorf, T., Schoffelen, J.-M., Oostenveld, R., Singer, W., Desimone, R., Engel, A. K., & Fries, P. (2007). Modulation of neuronal interactions through neuronal synchronization. Science, 316, 1609-1612.
Publications project C4 - Vision- and action-embodied language learning
Publications of the current project
- Yuan Yao, Ao Zhang, Xu Han, Mengdi Li, Cornelius Weber, Zhiyuan Liu, Stefan Wermter, Maosong Sun (2021).
Visual Distant Supervision for Scene Graph Generation.
Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021, pp. 15816-15826.
- Mengdi Li, Cornelius Weber, Matthias Kerzel, Jae Hee Lee, Zheni Zeng, Zhiyuan Liu, Stefan Wermter (2021).
Robotic Occlusion Reasoning for Efficient Object Existence Prediction.
IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Oct 2021.
- Huajian Fang, Guillaume Carbajal, Stefan Wermter, Timo Gerkmann (2021).
Variational Autoencoder for Speech Enhancement with a Noise-Aware Encoder.
IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2021, pp. 676-680,
doi: 10.1109/ICASSP39728.2021.9414060.
- Ozan Özdemir, Matthias Kerzel, Stefan Wermter (2021).
Embodied Language Learning with Paired Variational Autoencoders.
IEEE International Conference on Development and Learning (ICDL), pages 1-6,
doi:10.1109/ICDL49984.2021.9515668, Aug 2021.
- Hugo Carneiro, Cornelius Weber, Stefan Wermter (2021).
FaVoA: Face-Voice Association Favours Ambiguous Speaker Detection.
Proceedings of the 30th International Conference on Artificial Neural Networks (ICANN 2021), Volume LNCS 12891, pages 439-450, doi:10.1007/978-3-030-86362-3_36, Sep 2021.
- Julien Scholz, Cornelius Weber, Burhan Hafez, Stefan Wermter (2021).
Improving Model-Based Reinforcement Learning with Internal State Representations through Self-Supervision.
Proceedings of the International Joint Conference on Neural Networks (IJCNN 2021), Jul 2021.
- Aaron Eisermann, Jae Hee Lee, Cornelius Weber, Stefan Wermter (2021).
Generalization in Multimodal Language Learning from Simulation.
Proceedings of the International Joint Conference on Neural Networks (IJCNN 2021), Jul 2021.
- Frank Röder, Ozan Özdemir, Phuong D. H. Nguyen, Stefan Wermter and Manfred Eppe (2021).
The Embodied Crossmodal Self Forms Language and Interaction: A Computational Cognitive Review.
Frontiers in Psychology, Vol. 12, doi: 10.3389/fpsyg.2021.716671
- Matthias Möller, Johannes Twiefel, Cornelius Weber, Stefan Wermter (2021).
Controlling the Noise Robustness of End-to-End Automatic Speech Recognition Systems.
Proceedings of the International Joint Conference on Neural Networks (IJCNN 2021), Jul 2021.
- Victor Uc-Cetina, Nicolás Navarro-Guerrero, Anabel Martin-Gonzalez, Cornelius Weber, Stefan Wermter (2021).
Survey on reinforcement learning for language processing.
arXiv: 2104.05565, Apr 2021.
- Vadym Gryshchuk, Cornelius Weber, Chu Kiong Loo, Stefan Wermter (2021).
Lifelong Learning from Event-based Data.
European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning (ESANN '21), Oct 2021.
- Christoper Wulur, Cornelius Weber, Stefan Wermter (2021).
Planning-integrated Policy for Efficient Reinforcement Learning in Sparse-reward Environments.
Proceedings of the International Joint Conference on Neural Networks (IJCNN 2021), Jul 2021.
- Jie Zhou, Shengding Hu, Xin Lv, Cheng Yang, Zhiyuan Liu, Wei Xu, Jie Jiang, Juanzi Li, Maosong Sun (2021).
KACC: A Multi-task Benchmark for Knowledge Abstraction, Concretization and Completion.
Findings of the Association for Computational Linguistics (ACL-IJCNLP 2021), August 2021.
Publications of the phase 1 project
- Springenberg S., Lakomkin E., Weber C., Wermter S.(2019) Predictive Auxiliary Variational Autoencoder for Representation Learning of Global Speech Characteristics.
Proc. International Conference on Spoken Language Processing (INTERSPEECH), 2019.
- Yu, N., Weber, C., & Hu, X. (2019) Learning sparse hidden states in Long Short-Term Memory. Proc. 28th International Conference on Artificial Neural Networks and Machine Learning, ICANN 2019.
- Xie, R., Heinrich, S., Liu, Z., Weber, C., Yao, Y., Wermter, S., & Sun, M. (2019). Integrating Image-based and Knowledge-based Representation Learning. IEEE Transactions on Cognitive and Developmental Systems,
early access.
- Heinrich, S., Springstübe, P., Knöppler, T., Kerzel, M., & Wermter, S. (2019). Continuous convolutional object tracking in developmental robot scenarios. Neurocomputing. 342, 137-144.
- Weber, C. (2019). Lernverfahren fur neuronale Netze und Anwendungen in der Robotik. Mitt. Math. Ges. Hamburg, 39, 27-58
- Hinz, T., Heinrich, S., & Wermter, S. (2019). Generating multiple objects at spatially distinct locations. International Conference on Learning Representations (ICLR), arxiv.org/abs/1901.00686
- Zhang Z., Han X., Liu Z., Jiang X., Sun M., Liu Q. (2019). ERNIE: Enhanced Language Representation with Informative Entities. Proc. of the 57th Annual Meeting of the Association for Computational Linguistics (ACL).
- Wang, X., Han, X., Lin, Y., Liu, Z., & Sun, M. (2018). Adversarial multi-lingual neural relation extraction. Proceedings of the 27th International Conference on Computational Linguistics, 1156-1166.
- Han, X., Zhu, H., Yu, P., Wang, Z., Yao, Y., Liu, Z., & Sun, M. (2018). Fewrel: A large-scale supervised few-shot relation classification dataset with state-of-the-art evaluation. arXiv preprint arXiv:1810.10147.
- Han, X., Yu, P., Liu, Z., Sun, M., & Li, P. (2018). Hierarchical Relation Extraction with Coarse-to-Fine Grained Attention. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. 2236-2245.
- Shen, S. Q., Chen, Y., Yang, C., Liu, Z. Y., & Sun, M. S. (2018). Zero-Shot Cross-Lingual Neural Headline Generation. IEEE/ACM Transactions on Audio, Speech and Language Processing (TASLP), 26(12), 2319-2327.
- Heinrich, S., & Wermter, S. (2018). Interactive natural language acquisition in a multi-modal recurrent neural architecture. Connection Science, 30(1), 99-133.
- Heinrich, S., Alpay, T., Wermter, S. (2018). Adaptive and Variational Continuous Time Recurrent Neural Networks. Proc. IEEE ICDL-EpiRob 2018.
- Heinrich, S., Kerzel, M., Strahl, E., Wermter, S. (2018). Embodied Multi-modal Interaction in Language learning: the EMIL data collection. Proc. ICDL-EpiRob Workshop on Active Vision, Attention, and Learning 2018.
- Springstübe, P., Heinrich, S., Wermter, S. (2018). Continuous convolutional object tracking. Proc. ESANN, 2018.
- Lin Y., Liu Z., Sun M. (2017). Neural Relation Extraction with Multi-lingual Attention. The 55th Annual Meeting of the Association for Computational Linguistics (ACL), 34-43.
- Niu Y., Xie R., Liu Z., Sun M. (2017). Improved Word Representation Learning with Sememes. The 55th Annual Meeting of the Association for Computational Linguistics (ACL), 2049-2058.
- Ng, H.G. et al., (2017). Hey Robot, Why Don't You Talk to Me? Proc. 26th IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN 2017).
- Xie, R., Liu Z., Luan H., Sun M. (2017). Image-embodied Knowledge Representation Learning. Proceedings of International Joint Conference on Artificial Intelligence.
- Heinrich, S., Weber, C., Wermter, S., Xie, R., Lin, Y., Liu, Z. (2016). Crossmodal language grounding, learning, and teaching. NIPS2016 Workshop on Cognitive Computation, pp. 62-68, Barcelona, Spain.
Relevant previous work
- Chen, X., Liu, Z., & Sun, M. (2014). A Unified Model for Word Sense Representation and Disambiguation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP'14) (pp.1025-1035). Association for Computational Linguistics.
- Heinrich, S., Magg, S., & Wermter, S. (2015). Analysing the Multiple Timescale Recurrent Neural Network for Embodied Language Understanding. In Artificial Neural Networks - Methods and Applications (pp. 149-174). Springer International Publishing.
- Heinrich, S., & Wermter, S. (2014). Interactive Language Understanding with Multiple Timescale Recurrent Neural Networks. In Proceedings of the 24th International Conference on Artificial Neural Networks (ICANN 2014) (pp. 193-200), Springer International Publishing.
- Kleesiek, J., Engel, A. K., Weber, C., & Wermter, S. (2011, August). Reward-driven learning of sensorimotor laws and visual features. In Proceedings of the International Conference on Development and Learning and Epigenetic Robotics (ICDL-EpiRob) (pp. 1-6). IEEE Xpress.
- Lin, Y., Liu, Z., Sun, M., Liu, Y., & Zhu, X. (2015, February). Learning Entity and Relation Embeddings for Knowledge Graph Completion. In Proceedings of the29th AAAI Conference on Artificial Intelligence (AAAI'15) (pp. 2181-2187). Association for the Advancement of Artificial Intelligence.
- Saeb, S., Weber, C., & Triesch, J. (2009). Goal-directed learning of features and forward models. Neural Networks, 22(5-6), 586-592, Elsevier
- Weber, C., Wermter, S., & Elshaw, M. (2006). A hybrid generative and predictive model of the motor cortex. Neural Networks, 19(4), 339-353.
- Wermter, S., Weber, C., Elshaw, M., Gallese, V., & Pulvermüller, F. (2005). Grounding neural robot language in action. In Wermter S, Palm G, & Elshaw M. (Eds.), Biomimetic Neural Learning for Intelligent Robots (pp.162-181). Springer Berlin Heidelberg.
- Zhao, Y., Liu, Z., & Sun, M. (2015, February). Phrase Type Sensitive Tensor Indexing Model for Semantic Composition. In Proceedings of the 29th Conference on Artificial Intelligence (AAAI 2015) (pp. 2195-2201).
- Zhong, J., Weber, C., & Wermter, S. (2012). A predictive network architecture for a robust and smooth robot docking behavior. Paladyn. Journal of Behavioral Robotics, 3(4), 172-180.
Publications project C5 - Combining active vision with multimodal speech comprehension
Publications of the phase 1 project
- Alaçam, Ö. (2019). Enhancing natural language understanding through cross-modal interaction: Meaning recovery from acoustically noisy speech. In Proceedings of the 22nd Nordic Conference on Computational Linguistics (NoDaLiDa),
Association for Computational Linguistics, Turku, Finland.
- Alaçam, Ö., Menzel, W., and Staron, T. (2019-accepted).
How does visual complexity influence predictive language processing in a situated context?
In Proceedings of the 15th KONVENS – Bridging the gap between NLP and human understanding,
Friedrich-Alexander-Universität Erlangen-Nürnberg, Germany.
- Alaçam, Ö., Menzel, W. and Staron, T. (2019). Syntactic and Semantic Disambiguation in a Situated Language Setting.
EuroCogsci. 2019. Bochum.
- Qu, Q., & Damian, M. F. (2019). Orthographic effects in Mandarin spoken language production.
Memory & cognition, 47(2), 326-334.
- Lou, Y., Cai, H., Liu, X., & Li, X. (2019). Effects of self-enhancement on eye movements during reading.
Frontiers in psychology, 10.
- Qu, Q., & Damian, M. F. (2019). The role of orthography in second-language spoken word production: Evidence from Tibetan Chinese bilinguals. Quarterly Journal of Experimental Psychology.
- Xu, P., Qu, Q., Shen, W., & Li, X. (2019). Co-activation of taxonomic and thematic relations in spoken word comprehension: Evidence from eye movements. Frontiers in psychology, 10, 964.
- 韩海宾, 许萍萍, 屈青青, 程茜, & 李兴珊. (2019). 语言加工过程中的视听跨通道整合. 心理科学进展, 27(3), 475-489.
- Chen, L., Zhou, X., Müller, H. J., & Shi, Z. (2018). What you see depends on what you hear: Temporal averaging and crossmodal integration. Journal of Experimental Psychology: General, 147(12), 1851.
- Salama, A. R., Alaçam, Ö and Menzel, W. (2018) Text Completion using Context-Integrated Dependency Parsing. In the Proceedings of 3rd Workshop on Representation Learning for NLP, ACL 2018.
- Qu, Q., Cui, Z. L., and Damian, M. F(2018). Orthographic effects in second-language spoken word recognition. Journal of Experimental Psychology: Learning, Memory, and Cognition, 44(8), 1325 1332.
- Alaçam, Ö., Staron, T., and Menzel, W. (2018). A multi-modal data-set for systematic analyses of linguistic ambiguities in situated contexts. In Annual International Symposium on Information Management and Big Data (pp. 106-122). Springer.
- Staron, T., Alaçam, Ö., and Menzel, W. (2018). Incorporating Contextual Information for Language-Independent, Dynamic Disambiguation Tasks. In Proceedings of the 11th Language Resources and Evaluation Conference (LREC), Miyazaki, Japan.
- 屈青青*,刘维琳,李兴珊 (2018). 汉语言语产生的语音加工单元—基于音位的研究. 心理科学进展, 26(9), 1535-1544.
- Alaçam, Ö., Staron, T. and Menzel, W. (2017). Towards a Systematic Analysis of Linguistic and Visual Complexity in Disambiguation and Structural Prediction. In the Proceedings of the 5th International Conference on Information Management and Big Data 4-6 September 2017. Lima, Peru.
- Alaçam, Ö. and Habel, C. (2017). Negation in Gesture Accompanying Speech: A Cross-Modal Study. Journal of Multimodal Communication Studies, Vol. 4, issue 1-2.
- Shen, W., Qu, Q., Ni, A., Zhou, J., & Li, X. (2017). The time course of morphological processing during spoken word recognition in Chinese. Psychonomic Bulletin & Review, 1-7. doi:10.3758/s13423-017-1274-z.
- Qu, Q. Q., & Damian, M. F. (2017). Orthographic effects in spoken word recognition: Evidence from Chinese, Psychonomic Bulletin & Review, 1-6.
Relevant previous work
- Baumgärtner, C., Beuck, N., & Menzel, W. (2012). An Architecture for Incremental Information Fusion of Cross-Modal Representations. In Proc. IEEE Conference on Multisensor Fusion and Integration for Intelligent Systems, MFI-2012 (pp. 498-503). IEEE
- Beuck, N., & Menzel, W. (2013). Structural prediction in Incremental Dependency Parsing, In Proc. 14th Int. Conf. in Intelligent Text Processing and Computational Linguistics, CICLing-2013 (pp. 245-257). Springer
- Foth, K. A., & Menzel, W. (2006). Hybrid Parsing: Using Probabilistic Models as Predictors for a Symbolic Parser, In Proc. 21st Int. Conf. on Computational Linguistics and 44th Annual Meeting of the Association for Computational Linguistics, Coling-ACL-2006 (pp. 321-328). IEEE.
- Köhn, A., & Menzel, W. (2014). Incremental Predictive Parsing with TurboParser, In Proc. 52nd Annual Meeting of the Association for Computational Linguistics (pp. 803-808).
- Li, X., Bicknell, K., Liu, P., Wei, W., & Rayner, K. (2014). Reading is fundamentally similar across disparate writing systems: A systematic characterization of how words and characters influence eye movements in Chinese
- reading. Journal of Experimental Psychology: General, 143, 895-913.
- Li, X., Gu, J., Liu, P., & Rayner, K. (2013). The advantage of word-based processing in Chinese reading: Evidence from eye movements. Journal of Experimental Psychology: Learning, Memory and Cognition, 39, 879-889.
- Li, X., & Logan, G. D. (2008). Object-based attention in Chinese readers of Chinese words: Beyond Gestalt principles. Psychonomic Bulletin & Review, 15, 945-949.
- McCrae, P., Menzel, W., & Sun, M. (2011). A computational model of concept generalization in cross-modal reference. Tsinghua Science & Technology, 16, 113-120.
- Menzel, W. (2009). Towards radically incremental parsing of natural language. In Nicolov, N., & Angelova, G. (Eds.), Proc. 5th Int. Conf. on Recent Advances in Natural Language Processing (pp. 41-56). Amsterdam &
- Philadelphia: John Benjamin's Publishers.
- Rayner, K., Li, X., & Pollatsek, A. (2007). Extending the E-Z Reader model of eye movement control to Chinese reading. Cognitive Science, 31, 1021-1033.
Publications project C6 - Multisensory perception, learning and action for crossmodal tele-operations
Publications of the phase 1 project
- Zeng, H., & Chen, L. (2019). Robust temporal averaging of time intervals between action and sensation. Frontiers in psychology, 10, 511.
- Chen, L. (2019). Discrimination of empty and filled intervals marked by auditory signals with different durations and directions of intensity change. PsyCh journal, 8(2), 187-202.
- Shen, L., Han, B., Chen, L., & Chen, Q. (2019). Perceptual inference employs intrinsic alpha frequency to resolve perceptual ambiguity. PLoS biology, 17(3), e3000025.
- Zheng, W., & Chen, L. (2019). Illusory perception of auditory filled duration is task‐and context‐dependent. British Journal of Psychology.
- Li, B., Chen, L., & Fang, F. (2019). Somatotopic representation of tactile duration: evidence from tactile duration aftereffect. Behavioural brain research, 371, 111954.
- He, D., Wang, Y., & Fang, F. (2019). The Critical Role of V2 Population Receptive Fields in Visual Orientation Crowding. Current Biology.
- Katzakis, N., Chen,L., Ariza, O., Teather,R., and Steinicke, F.(2019).Evaluation of 3D Pointing Accuracy in the Fovea and Periphery in Immersive Head-Mounted Display Environments, IEEE Transaction on visualization and computer graphics
- Wang, Y., Zhu, Z., Chen, B., & Fang, F. (2019). Perceptual learning and recognition confusion reveal the underlying relationships among the six basic emotions. Cognition and Emotion, 33(4), 754-767.
- Chen, L.*, Zhou, X., Müller, H.J., Shi, Z. (in press) What you see depends on what you hear: temporal averaging and crossmodal integration, Journal of Experimental Psychology: General.
- Zheng,W.* & Chen, L.* (2018). The Roles of Attentional Shifts and Attentional Reengagement in Resolving The Spatial Compatibility Effect in Tactile Simon-like Tasks. Scientific Reports, 8:8760. doi:10.1038/s41598-018-27114-9.
- Wan, Y. & Chen, L.* (2018). Temporal Reference, Attentional Modulation, and Crossmodal Assimilation. Front. Comput. Neurosci. 12:39. doi: 10.3389/fncom.2018.00039.
- Na R., Bi T., Tjan B.S., Liu Z. and Fang F. (2018). Effect of task difficulty on blood-oxygen-level-dependent signal: a functional magnetic resonance imaging study in a motion discrimination task. PLoS ONE 13(6): e0199440.
- Wang Y., Zhu Z., Chen B. and Fang F. (2018) Perceptual learning and recognition confusion reveal the underlying relationships among the six basic emotions. Cognition and Emotion. https://doi.org/10.1080/02699931.2018.1491831
- Yiltiz, H. & Chen, L.*(2018). Emotional cues and social anxiety resolve ambiguous perception of biological motion. Exp Brain Res. 236:1409-1420
- Tian,Y. & Chen, L.* (2018). Cross-modal attention modulates tactile subitizing but not tactile numerosity estimation. Attention, Perception, & Psychophysics, https://doi.org/10.3758/s13414-018-1507-x
- Mo C.*, He D.* and Fang F. (2018). Attention priority map of face images in human early visual cortex. Journal of Neuroscience. 38(1), 149-157
- Zhang, J., Langbehn, E., Krupke, D., Katzakis, N., & Steinicke, F. (2018). Detection Thresholds for Rotation and Translation Gains in 360° Video-Based Telepresence Systems. IEEE transactions on visualization and computer graphics, 24(4), 1671-1680.
- Zhang, J., Langbehn, E., Krupke, D., Katzakis, N., & Steinicke, F. (2018). A 360 video-based robot platform for telepresent redirected walking.
- Freiwald, J. P., Katzakis, N., & Steinicke, F. (2018). Camera time warp: compensating latency in video see-through head-mounted-displays for reduced cybersickness effects. In Proceedings of the 24th ACM Symposium on Virtual Reality Software and Technology. 9.
- Ariza, O., Bruder, G., Katzakis, N., & Steinicke, F. (2018). Analysis of Proximity-Based Multimodal Feedback for 3D Selection in Immersive Virtual Environments. In 2018 IEEE Conference on Virtual Reality and 3D User Interfaces (VR). 327-334.
- Katzakis, N., & Steinicke, F. (2018). Excuse me! perception of abrupt direction changes using body cues and paths on mixed reality avatars. arXiv preprint arXiv:1801.05085.
- Chen N., Lu J., Shao H., Weng X. and Fang F. (2017). Neural mechanisms of motion perceptual learning in noise. Human Brain Mapping. 38(12), 6029-6042
- Chen, L., Guo, L., Bao,M. (2017). "Sleep-dependent consolidation benefits fast transfer of time interval training". Experimental Brain Research, 235(3),661-672.
- Guo,L., Bao, M., Guan, L., Chen, L.. (2017). Cognitive Styles Differentiate Crossmodal Correspondences Between Pitch Glide and Visual Apparent Motion, Multisensory Research.30,363-385.
- He D., Mo C. and Fang F. (2017). Predictive feature remapping before saccadic eye movements. Journal of Vision 17(5):14, 1-12.
Relevant previous work
- Bruder, G., Steinicke, F., & Sturzlinger, W. (2013, July). To touch or not to touch?: comparing 2D touch and 3D mid-air interaction on stereoscopic tabletop surfaces. In Proceedings of the 1st symposium on Spatial user interaction (pp. 9-16). ACM.
- Bruder, G., Lubos, P., & Steinicke, F. (2015). Cognitive Resource Demands of Redirected Walking. IEEE Trans. on Visualization and Computer Graphics (TVCG), 4, 539-544.
- Chen, L., Wang, Q., & Bao, M. (2014). Spatial references and audio-tactile interaction in cross-modal dynamic capture. Multisensory Research, 27, 55-70.
- Chen, L., & Vroomen, J (2013). Intersensory Binding across Space and Time: A Tutorial Review. Attention, Perception, & Psychophysics, 75, 790-811.
- Chen, L., Shi, Z., & Müller, H. J. (2011). Interaction of Perceptual Grouping and Crossmodal Temporal Capture in Tactile Apparent-Motion. PLoS ONE, 6(2), e17130.
- Chen, J., He, Y., Zhu, Z., Zhou, T., Peng, Y., Zhang, X., & Fang, F. (2014). Attention-dependent early Cortical Suppression contributes to Crowding. Journal of Neuroscience. 34(32), 10465-10474.
- Daiber, F., Valkov, D., Steinicke, F., Hinrich, K., & Krüger, A. (2012). Towards Object Prediction based on Hand Postures for Reach to Grasp Interaction. In Proceedings of the ACM, the 3rd Dimension of CHI (3DCHI) (pp.99-106). ACM.
- Fang, F., & He, S. (2005). Cortical Responses to Invisible Objects in Human Dorsal and Ventral Pathways. Nature Neuroscience. 8(10), 1380-1385.
- Steinicke, F., Bruder, G., Jerald, J., Frenz, H., & Lappe, M. (2010). Estimation of Detection Thresholds for Redirected Walking Techniques. IEEE Trans. on Visualization and Computer Graphics (TVCG), 16(1), 17-27
- Zhang, X., Zhaoping, L., Zhou, T., & Fang, F. (2012). Neural Activities in V1 create a Bottom-Up Saliency Map. Neuron 73(1), 183-192.
Publications project C7 - Cross-modal Learning for Improving Human Reading
Publications of the current project
- Yao, P., Slattery, T. J., & Li, X. (in press).
Sentence context modulates the neighborhood frequency effect: Evidence from eye movements while reading Chinese.
Journal of Experimental Psychology: Learning, Memory, and Cognition.
- Yao, P., Staub, A., & Li, X. (2021).
Predictability Eliminates Neighborhood Effects During Chinese Sentence.
Psychonomic Bulletin & Review. https://doi.org/10.3758/s13423-021-01966-1
- Huang, L., Staub, A., & Li, X. (2021).
Prior context influences lexical competition when segmenting Chinese overlapping ambiguous strings.
Journal of Memory and Language, 118
- Florian Schneider, Özge Alaçam, Xintong Wang, Chris Biemann (2021).
Towards Multi-Modal Text-Image Retrieval to improve Human Reading.
NAACL SRW 2021.
- Özge Alaçam, Ganeshan Malhotra, Eugen Ruppert, Chris Biemann (2021).
Gaze-based Multimodal Meaning Recovery for Noisy / Complex Environments.
ICMI 2021: 673-681
- Li, X., & Pollatsek, A. (2020).
An integrated model of word processing and eye-movement control during Chinese reading.
Psychological Review, 127(6), 1139-1162.
- Han, H. & Li, X.(2020).
Degree of conceptual overlap affects eye movements in visual world paradigm.
Language, Cognition and Neuroscience, 35(10), 1456-1464.
- Huang, L., & Li, X. (2020).
Early, but not overwhelming: The effect of prior context on segmenting overlapping ambiguous strings when reading Chinese.
Quarterly Journal of Experimental Psychology. 73(9), 1382-1395. https://doi.org/10.1177/1747021820926012
Publications project C8 - Crossmodal bindings and plasticity during visual-haptic interactions for novel forms of therapy
Publications of the current project
-
Hartfill, Gabel, Kruse, Schmidt, Riebandt, Kühn & Steinicke (2021).
Analysis of Detection Thresholds for Hand Redirection during Mid-Air Interactions in Virtual Reality.
VRST ’21, December 8–10, 2021, Osaka, Japan.
- Katzakis, Chen, Ariza, Teacher, & Steinicke (2021).
Evaulation of 3D Pointing Accuracy in the Fovea and Periphery in Immersive Head-Mounted Display Environments.
IEEE Transaction on Visualization and Computer Graphics, 27:3,1929-1936.
- Gong, Gu, Wang & Chen (2021).
Interoception visualization relieves acute pain.
Biological Psychology, in minor revision.
- Katzakis, Chen & Steinicke (2020).
Visual-Haptic Size Estimation in Peripersonal Space.
Frontiers in Neurorobotics, 14:18.
Publications project C9 - The role of mental models and sense of agency in learning crossmodal communicative acts
Publications of the current project