Publications | Kalin Stefanov

2023

MARLIN: Masked Autoencoder for Facial Video Representation Learning

Zhixi Cai, Shreya Ghosh, Kalin Stefanov, Abhinav Dhall, Jianfei Cai, Hamid Rezatofighi, Reza Haffari, and Munawar Hayat

In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023

Abs Bib PDF Code

This paper proposes a self-supervised approach to learn universal facial representations from videos, that can transfer across a variety of facial analysis tasks such as Facial Attribute Recognition (FAR), Facial Expression Recognition (FER), DeepFake Detection (DFD), and Lip Synchronization (LS). Our proposed framework, named MARLIN, is a facial video masked autoencoder, that learns highly robust and generic facial embeddings from abundantly available non-annotated web crawled facial videos. As a challenging auxiliary task, MARLIN reconstructs the spatio-temporal details of the face from the densely masked facial regions which mainly include eyes, nose, mouth, lips, and skin to capture local and global aspects that in turn help in encoding generic and transferable features. Through a variety of experiments on diverse downstream tasks, we demonstrate MARLIN to be an excellent facial video encoder as well as feature extractor, that performs consistently well across a variety of downstream tasks including FAR (1.13% gain over supervised benchmark), FER (2.64% gain over unsupervised benchmark), DFD (1.86% gain over unsupervised benchmark), LS (29.36% gain for Frechet Inception Distance), and even in low data regime. Our code and models are available at https://github.com/ControlNet/MARLIN.
@inproceedings{10203837, author = {Cai, Zhixi and Ghosh, Shreya and Stefanov, Kalin and Dhall, Abhinav and Cai, Jianfei and Rezatofighi, Hamid and Haffari, Reza and Hayat, Munawar}, title = {MARLIN: Masked Autoencoder for Facial Video Representation Learning}, booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition}, year = {2023}, pages = {1493-1504}, doi = {10.1109/CVPR52729.2023.00150}, pdf = {10203837.pdf}, code = {https://github.com/ControlNet/MARLIN}, bibtex_show = {true}, preview = {paper.png} }
Glitch in the Matrix: A Large Scale Benchmark for Content Driven Audio-Visual Forgery Detection and Localization

Zhixi Cai, Shreya Ghosh, Abhinav Dhall, Tom Gedeon, Kalin Stefanov, and Munawar Hayat

Computer Vision and Image Understanding, 2023

Abs Bib PDF Code

Most deepfake detection methods focus on detecting spatial and/or spatio-temporal changes in facial attributes and are centered around the binary classification task of detecting whether a video is real or fake. This is because available benchmark datasets contain mostly visual-only modifications present in the entirety of the video. However, a sophisticated deepfake may include small segments of audio or audio–visual manipulations that can completely change the meaning of the video content. To addresses this gap, we propose and benchmark a new dataset, Localized Audio Visual DeepFake (LAV-DF), consisting of strategic content-driven audio, visual and audio–visual manipulations. The proposed baseline method, Boundary Aware Temporal Forgery Detection (BA-TFD), is a 3D Convolutional Neural Network-based architecture which effectively captures multimodal manipulations. We further improve (i.e. BA-TFD+) the baseline method by replacing the backbone with a Multiscale Vision Transformer and guide the training process with contrastive, frame classification, boundary matching and multimodal boundary matching loss functions. The quantitative analysis demonstrates the superiority of BA-TFD+ on temporal forgery localization and deepfake detection tasks using several benchmark datasets including our newly proposed dataset. The dataset, models and code are available at https://github.com/ControlNet/LAV-DF.
@article{S1077314223001984, title = {Glitch in the Matrix: A Large Scale Benchmark for Content Driven Audio-Visual Forgery Detection and Localization}, author = {Cai, Zhixi and Ghosh, Shreya and Dhall, Abhinav and Gedeon, Tom and Stefanov, Kalin and Hayat, Munawar}, year = {2023}, journal = {Computer Vision and Image Understanding}, volume = {236}, doi = {https://doi.org/10.1016/j.cviu.2023.103818}, url = {https://www.sciencedirect.com/science/article/pii/S1077314223001984}, keywords = {Datasets, Deepfake, Localization, Detection}, pdf = {S1077314223001984.pdf}, code = {https://github.com/ControlNet/LAV-DF}, bibtex_show = {true}, preview = {paper.png} }

2022

Graph-Based Group Modelling for Backchannel Detection

Garima Sharma, Kalin Stefanov, Abhinav Dhall, and Jianfei Cai

In Proceedings of the ACM International Conference on Multimedia, 2022

Abs Bib PDF

The brief responses given by listeners in group conversations are known as backchannels rendering the task of backchannel detection an essential facet of group interaction analysis. Most of the current backchannel detection studies explore various audio-visual cues for individuals. However, analysing all group members is of utmost importance for backchannel detection, like any group interaction. This study uses a graph neural network to model group interaction through all members’ implicit and explicit behaviours. The proposed method achieves the best and second best performance on agreement estimation and backchannel detection tasks, respectively, of the 2022 MultiMediate: Multi-modal Group Behaviour Analysis for Artificial Mediation challenge.
@inproceedings{10.1145/3503161.3551605, author = {Sharma, Garima and Stefanov, Kalin and Dhall, Abhinav and Cai, Jianfei}, title = {Graph-Based Group Modelling for Backchannel Detection}, year = {2022}, isbn = {9781450392037}, publisher = {Association for Computing Machinery}, address = {New York, NY, USA}, url = {https://doi.org/10.1145/3503161.3551605}, doi = {10.1145/3503161.3551605}, booktitle = {Proceedings of the ACM International Conference on Multimedia}, pages = {7190--7194}, numpages = {5}, keywords = {backchannel detection, agreement estimation, graph neural networks}, location = {Lisboa, Portugal}, series = {MM '22}, pdf = {3503161.3551605.pdf}, bibtex_show = {true}, preview = {paper.png} }
Do You Really Mean That? Content Driven Audio-Visual Deepfake Dataset and Multimodal Method for Temporal Forgery Localization

Zhixi Cai, Kalin Stefanov, Abhinav Dhall, and Munawar Hayat

In Proceedings of the International Conference on Digital Image Computing: Techniques and Applications, 2022

Abs Bib PDF Code

Due to its high societal impact, deepfake detection is getting active attention in the computer vision community. Most deepfake detection methods rely on identity, facial attributes, and adversarial perturbation-based spatio-temporal modifications at the whole video or random locations while keeping the meaning of the content intact. However, a sophisticated deepfake may contain only a small segment of video/audio manipulation, through which the meaning of the content can be, for example, completely inverted from a sentiment perspective. We introduce a content-driven audio-visual deepfake dataset, termed Localized Audio Visual DeepFake (LAV-DF), explicitly designed for the task of learning temporal forgery localization. Specifically, the content-driven audio-visual manipulations are performed strategically to change the sentiment polarity of the whole video. Our baseline method for benchmarking the proposed dataset is a 3DCNN model, termed as Boundary Aware Temporal Forgery Detection (BA-TFD), which is guided via contrastive, boundary matching, and frame classification loss functions. Our extensive quantitative and qualitative analysis demonstrates the proposed method’s strong performance for temporal forgery localization and deepfake detection tasks.
@inproceedings{10034605, author = {Cai, Zhixi and Stefanov, Kalin and Dhall, Abhinav and Hayat, Munawar}, booktitle = {Proceedings of the International Conference on Digital Image Computing: Techniques and Applications}, title = {Do You Really Mean That? Content Driven Audio-Visual Deepfake Dataset and Multimodal Method for Temporal Forgery Localization}, year = {2022}, pages = {1-10}, doi = {10.1109/DICTA56598.2022.10034605}, pdf = {10034605.pdf}, code = {https://github.com/ControlNet/LAV-DF}, bibtex_show = {true}, preview = {paper.png} }
Hierarchical Residual Learning Based Vector Quantized Variational Autoencoder for Image Reconstruction and Generation

Mohammad Adiban, Kalin Stefanov, Sabato M. Siniscalchi, and Giampiero Salvi

In Proceedings of the British Machine Vision Conference, 2022

Abs Bib PDF Code

We propose a multi-layer variational autoencoder method, we call HR-VQVAE, that learns hierarchical discrete representations of the data. By utilizing a novel objective function, each layer in HR-VQVAE learns a discrete representation of the residual from previous layers through a vector quantized encoder. Furthermore, the representations at each layer are hierarchically linked to those at previous layers. We evaluate our method on the tasks of image reconstruction and generation. Experimental results demonstrate that the discrete representations learned by HR-VQVAE enable the decoder to reconstruct high-quality images with less distortion than the baseline methods, namely VQVAE and VQVAE-2. HR-VQVAE can also generate high-quality and diverse images that outperform state-of-the-art generative models, providing further verification of the efficiency of the learned representations. The hierarchical nature of HR-VQVAE i) reduces the decoding search time, making the method particularly suitable for high-load tasks and ii) allows to increase the codebook size without incurring the codebook collapse problem.
@inproceedings{0636, author = {Adiban, Mohammad and Stefanov, Kalin and Siniscalchi, Sabato M. and Salvi, Giampiero}, title = {Hierarchical Residual Learning Based Vector Quantized Variational Autoencoder for Image Reconstruction and Generation}, booktitle = {Proceedings of the British Machine Vision Conference}, publisher = {{BMVA} Press}, year = {2022}, url = {https://bmvc2022.mpi-inf.mpg.de/0636.pdf}, pdf = {0636.pdf}, code = {https://github.com/mohammad-adiban/Video-Prediction}, bibtex_show = {true}, preview = {paper.png} }
Visual Representations of Physiological Signals for Fake Video Detection

Kalin Stefanov, Bhawna Paliwal, and Abhinav Dhall

2022

Abs Bib PDF

Realistic fake videos are a potential tool for spreading harmful misinformation given our increasing online presence and information intake. This paper presents a multimodal learning-based method for detection of real and fake videos. The method combines information from three modalities - audio, video, and physiology. We investigate two strategies for combining the video and physiology modalities, either by augmenting the video with information from the physiology or by novelly learning the fusion of those two modalities with a proposed Graph Convolutional Network architecture. Both strategies for combining the two modalities rely on a novel method for generation of visual representations of physiological signals. The detection of real and fake videos is then based on the dissimilarity between the audio and modified video modalities. The proposed method is evaluated on two benchmark datasets and the results show significant increase in detection performance compared to previous methods.
@misc{2207.08380, title = {Visual Representations of Physiological Signals for Fake Video Detection}, author = {Stefanov, Kalin and Paliwal, Bhawna and Dhall, Abhinav}, year = {2022}, eprint = {2207.08380}, archiveprefix = {arXiv}, primaryclass = {cs.CV}, pdf = {2207.08380.pdf}, bibtex_show = {true}, preview = {paper.png} }

2021

Spatial Bias in Vision-Based Voice Activity Detection

Kalin Stefanov, Mohammad Adiban, and Giampiero Salvi

In Proceedings of the International Conference on Pattern Recognition, 2021

Abs Bib PDF

We develop and evaluate models for automatic vision-based voice activity detection (VAD) in multiparty human-human interactions that are aimed at complementing acoustic VAD methods. We provide evidence that this type of vision-based VAD models are susceptible to spatial bias in the dataset used for their development; the physical settings of the interaction, usually constant throughout data acquisition, determines the distribution of head poses of the participants. Our results show that when the head pose distributions are significantly different in the train and test sets, the performance of the vision-based VAD models drops significantly. This suggests that previously reported results on datasets with a fixed physical configuration may overestimate the generalization capabilities of this type of models. We also propose a number of possible remedies to the spatial bias, including data augmentation, input masking and dynamic features, and provide an in-depth analysis of the visual cues used by the developed vision-based VAD models.
@inproceedings{10.1109/ICPR48806.2021.9413345, author = {Stefanov, Kalin and Adiban, Mohammad and Salvi, Giampiero}, booktitle = {Proceedings of the International Conference on Pattern Recognition}, title = {Spatial Bias in Vision-Based Voice Activity Detection}, year = {2021}, pages = {10433--10440}, doi = {10.1109/ICPR48806.2021.9413345}, issn = {1051-4651}, pdf = {ICPR48806.2021.9413345.pdf}, bibtex_show = {true}, preview = {paper.png} }
Analysis of Behavior Classification in Motivational Interviewing

Leili Tavabi, Trang Tran, Kalin Stefanov, Brian Borsari, Joshua Woolley, Stefan Scherer, and Mohammad Soleymani

In Proceedings of the Seventh Workshop on Computational Linguistics and Clinical Psychology: Improving Access, 2021

Abs Bib PDF

Analysis of client and therapist behavior in counseling sessions can provide helpful insights for assessing the quality of the session and consequently, the client’s behavioral outcome. In this paper, we study the automatic classification of standardized behavior codes (annotations) used for assessment of psychotherapy sessions in Motivational Interviewing (MI). We develop models and examine the classification of client behaviors throughout MI sessions, comparing the performance by models trained on large pretrained embeddings (RoBERTa) versus interpretable and expert-selected features (LIWC). Our best performing model using the pretrained RoBERTa embeddings beats the baseline model, achieving an F1 score of 0.66 in the subject-independent 3-class classification. Through statistical analysis on the classification results, we identify prominent LIWC features that may not have been captured by the model using pretrained embeddings. Although classification using LIWC features underperforms RoBERTa, our findings motivate the future direction of incorporating auxiliary tasks in the classification of MI codes.
@inproceedings{10.18653/v1/2021.clpsych-1.13, title = {Analysis of Behavior Classification in Motivational Interviewing}, author = {Tavabi, Leili and Tran, Trang and Stefanov, Kalin and Borsari, Brian and Woolley, Joshua and Scherer, Stefan and Soleymani, Mohammad}, booktitle = {Proceedings of the Seventh Workshop on Computational Linguistics and Clinical Psychology: Improving Access}, year = {2021}, address = {Online}, publisher = {Association for Computational Linguistics}, url = {https://aclanthology.org/2021.clpsych-1.13}, doi = {10.18653/v1/2021.clpsych-1.13}, pages = {110--115}, pdf = {2021.clpsych-1.13.pdf}, bibtex_show = {true}, preview = {paper.png} }
Group-Level Focus of Visual Attention for Improved Next Speaker Prediction

Chris Birmingham, Kalin Stefanov, and Maja Mataric

In Proceedings of the ACM International Conference on Multimedia, 2021

Abs Bib PDF

In this work we address the Next Speaker Prediction sub challenge of the ACM ’21 MultiMediate Grand Challenge. This challenge poses the problem of turn taking prediction in physically situated multiparty interaction. Solving this problem is essential for enabling fluent real-time multiparty human-machine interaction. This problem is made more difficult by the need for a robust solution that can perform effectively across a wide variety of settings and contexts. Prior work has shown that current state-of-the-art methods rely on machine learning approaches that do not generalize well to new settings and feature distributions. To address this problem, we propose the use of group-level focus of visual attention as additional information. We show that a simple combination of group-level focus of visual attention features and publicly available audio-video synchronizer models is competitive with state-of-the-art methods fine-tuned for the challenge dataset.
@inproceedings{10.1145/3474085.3479213, author = {Birmingham, Chris and Stefanov, Kalin and Mataric, Maja}, title = {Group-Level Focus of Visual Attention for Improved Next Speaker Prediction}, year = {2021}, isbn = {9781450386517}, publisher = {Association for Computing Machinery}, address = {New York, NY, USA}, url = {https://doi.org/10.1145/3474085.3479213}, doi = {10.1145/3474085.3479213}, booktitle = {Proceedings of the ACM International Conference on Multimedia}, pages = {4838--4842}, numpages = {5}, keywords = {focus of visual attention, neural networks, turn taking prediction}, location = {Virtual Event, China}, series = {MM '21}, pdf = {3474085.3479213.pdf}, bibtex_show = {true}, preview = {paper.png} }
Group-Level Focus of Visual Attention for Improved Active Speaker Detection

Christopher Birmingham, Maja Mataric, and Kalin Stefanov

In Companion Publication of the ACM International Conference on Multimodal Interaction, 2021

Abs Bib PDF

This work addresses the problem of active speaker detection in physically situated multiparty interactions. This challenge requires a robust solution that can perform effectively across a wide range of speakers and physical contexts. Current state-of-the-art active speaker detection approaches rely on machine learning methods that do not generalize well to new physical settings. We find that these methods do not transfer well even between similar datasets. We propose the use of group-level focus of visual attention in combination with a general audio-video synchronizer method for improved active speaker detection across speakers and physical contexts. Our dataset-independent experiments demonstrate that the proposed approach outperforms state-of-the-art methods trained specifically for the task of active speaker detection.
@inproceedings{10.1145/3461615.3485430, author = {Birmingham, Christopher and Mataric, Maja and Stefanov, Kalin}, title = {Group-Level Focus of Visual Attention for Improved Active Speaker Detection}, year = {2021}, isbn = {9781450384711}, publisher = {Association for Computing Machinery}, address = {New York, NY, USA}, url = {https://doi.org/10.1145/3461615.3485430}, doi = {10.1145/3461615.3485430}, booktitle = {Companion Publication of the ACM International Conference on Multimodal Interaction}, pages = {37--42}, numpages = {6}, keywords = {neural networks, focus of visual attention, active speaker detection}, location = {Montreal, QC, Canada}, series = {ICMI '21 Companion}, pdf = {3461615.3485430.pdf}, bibtex_show = {true}, preview = {paper.png} }

2020

Self-Supervised Vision-Based Detection of the Active Speaker as Support for Socially Aware Language Acquisition

Kalin Stefanov, Jonas Beskow, and Giampiero Salvi

IEEE Transactions on Cognitive and Developmental Systems, 2020

Abs Bib PDF

This paper presents a self-supervised method for visual detection of the active speaker in a multiperson spoken interaction scenario. Active speaker detection is a fundamental prerequisite for any artificial cognitive system attempting to acquire language in social settings. The proposed method is intended to complement the acoustic detection of the active speaker, thus improving the system robustness in noisy conditions. The method can detect an arbitrary number of possibly overlapping active speakers based exclusively on visual information about their face. Furthermore, the method does not rely on external annotations, thus complying with cognitive development. Instead, the method uses information from the auditory modality to support learning in the visual domain. This paper reports an extensive evaluation of the proposed method using a large multiperson face-to-face interaction data set. The results show good performance in a speaker dependent setting. However, in a speaker independent setting the proposed method yields a significantly lower performance. We believe that the proposed method represents an essential component of any artificial cognitive system or robotic platform engaging in social interactions.
@article{10.1109/TCDS.2019.2927941, author = {Stefanov, Kalin and Beskow, Jonas and Salvi, Giampiero}, journal = {IEEE Transactions on Cognitive and Developmental Systems}, title = {Self-Supervised Vision-Based Detection of the Active Speaker as Support for Socially Aware Language Acquisition}, year = {2020}, volume = {12}, number = {2}, pages = {250--259}, doi = {10.1109/TCDS.2019.2927941}, issn = {2379-8939}, pdf = {TCDS.2019.2927941}, bibtex_show = {true}, preview = {paper.png} }
Emotion or Expressivity? An Automated Analysis of Nonverbal Perception in a Social Dilemma

Su Lei, Kalin Stefanov, and Jonathan Gratch

In Proceedings of the IEEE International Conference on Automatic Face and Gesture Recognition, 2020

Abs Bib PDF

An extensive body of research has examined how specific emotional expressions shape social perceptions and social decisions, yet recent scholarship in emotion research has raised questions about the validity of emotion as a construct. In this article, we contrast the value of measuring emotional expressions with the more general construct of expressivity (in the sense of conveying a thought or emotion through any nonverbal behavior) and develop models that can automatically extract perceived expressivity from videos. Although less extensive, a solid body of research has shown expressivity to be an important element when studying interpersonal perception, particularly in psychiatric contexts. Here we examine the role expressivity plays in predicting social perceptions and decisions in the context of a social dilemma. We show that perceivers use more than facial expressions when making judgments of expressivity and see these expressions as conveying thoughts as well as emotions (although facial expressions and emotional attributions explain most of the variance in these judgments). We next show that expressivity can be predicted with high accuracy using Lasso and random forests. Our analysis shows that features related to motion dynamics are particularly important for modeling these judgments. We also show that learned models of expressivity have value in recognizing important aspects of a social situation. First, we revisit a previously published finding which showed that smile intensity was associated with the unexpectedness of outcomes in social dilemmas; instead, we show that expressivity is a better predictor (and explanation) of this finding. Second, we provide preliminary evidence that expressivity is useful for identifying “moments of interest” in a video sequence.
@inproceedings{10.1109/FG47880.2020.00123, author = {Lei, Su and Stefanov, Kalin and Gratch, Jonathan}, booktitle = {Proceedings of the IEEE International Conference on Automatic Face and Gesture Recognition}, title = {Emotion or Expressivity? An Automated Analysis of Nonverbal Perception in a Social Dilemma}, year = {2020}, pages = {544--551}, doi = {10.1109/FG47880.2020.00123}, pdf = {FG47880.2020.00123.pdf}, bibtex_show = {true}, preview = {paper.png} }
Multimodal Automatic Coding of Client Behavior in Motivational Interviewing

Leili Tavabi, Kalin Stefanov, Larry Zhang, Brian Borsari, Joshua D. Woolley, Stefan Scherer, and Mohammad Soleymani

In Proceedings of the ACM International Conference on Multimodal Interaction, 2020

Abs Bib PDF

Motivational Interviewing (MI) is defined as a collaborative conversation style that evokes the client’s own intrinsic reasons for behavioral change. In MI research, the clients’ attitude (willingness or resistance) toward change as expressed through language, has been identified as an important indicator of their subsequent behavior change. Automated coding of these indicators provides systematic and efficient means for the analysis and assessment of MI therapy sessions. In this paper, we study and analyze behavioral cues in client language and speech that bear indications of the client’s behavior toward change during a therapy session, using a database of dyadic motivational interviews between therapists and clients with alcohol-related problems. Deep language and voice encoders, ie BERT and VGGish, trained on large amounts of data are used to extract features from each utterance. We develop a neural network to automatically detect the MI codes using both the clients’ and therapists’ language and clients’ voice, and demonstrate the importance of semantic context in such detection. Additionally, we develop machine learning models for predicting alcohol-use behavioral outcomes of clients through language and voice analysis. Our analysis demonstrates that we are able to estimate MI codes using clients’ textual utterances along with preceding textual context from both the therapist and client, reaching an F1-score of 0.72 for a speaker-independent three-class classification. We also report initial results for using the clients’ data for predicting behavioral outcomes, which outlines the direction for future work.
@inproceedings{10.1145/3382507.3418853, author = {Tavabi, Leili and Stefanov, Kalin and Zhang, Larry and Borsari, Brian and Woolley, Joshua D. and Scherer, Stefan and Soleymani, Mohammad}, title = {Multimodal Automatic Coding of Client Behavior in Motivational Interviewing}, year = {2020}, isbn = {9781450375818}, publisher = {Association for Computing Machinery}, address = {New York, NY, USA}, url = {https://doi.org/10.1145/3382507.3418853}, doi = {10.1145/3382507.3418853}, booktitle = {Proceedings of the ACM International Conference on Multimodal Interaction}, pages = {406--413}, numpages = {8}, keywords = {mental health, machine learning, motivational interviewing, human behavior}, location = {Virtual Event, Netherlands}, series = {ICMI '20}, pdf = {3382507.3418853.pdf}, bibtex_show = {true}, preview = {paper.png} }
OpenSense: A Platform for Multimodal Data Acquisition and Behavior Perception

Kalin Stefanov, Baiyu Huang, Zongjian Li, and Mohammad Soleymani

In Proceedings of the ACM International Conference on Multimodal Interaction, 2020

Abs Bib PDF Code

Automatic multimodal acquisition and understanding of social signals is an essential building block for natural and effective human-machine collaboration and communication. This paper introduces OpenSense, a platform for real-time multimodal acquisition and recognition of social signals. OpenSense enables precisely synchronized and coordinated acquisition and processing of human behavioral signals. Powered by the Microsoft’s Platform for Situated Intelligence, OpenSense supports a range of sensor devices and machine learning tools and encourages developers to add new components to the system through straightforward mechanisms for component integration. This platform also offers an intuitive graphical user interface to build application pipelines from existing components. OpenSense is freely available for academic research.
@inproceedings{10.1145/3382507.3418832, author = {Stefanov, Kalin and Huang, Baiyu and Li, Zongjian and Soleymani, Mohammad}, title = {OpenSense: A Platform for Multimodal Data Acquisition and Behavior Perception}, year = {2020}, isbn = {9781450375818}, publisher = {Association for Computing Machinery}, address = {New York, NY, USA}, url = {https://doi.org/10.1145/3382507.3418832}, doi = {10.1145/3382507.3418832}, booktitle = {Proceedings of the ACM International Conference on Multimodal Interaction}, pages = {660--664}, numpages = {5}, keywords = {platform, perception, multimodal, behavior, open source}, location = {Virtual Event, Netherlands}, series = {ICMI '20}, pdf = {3382507.3418832.pdf}, code = {https://github.com/intelligent-human-perception-laboratory/OpenSense}, bibtex_show = {true}, preview = {paper.png} }

2019

Modeling of Human Visual Attention in Multiparty Open-World Dialogues

Kalin Stefanov, Giampiero Salvi, Dimosthenis Kontogiorgos, Hedvig Kjellström, and Jonas Beskow

ACM Transactions on Human-Robot Interaction, 2019

Abs Bib PDF

This study proposes, develops, and evaluates methods for modeling the eye-gaze direction and head orientation of a person in multiparty open-world dialogues, as a function of low-level communicative signals generated by his/hers interlocutors. These signals include speech activity, eye-gaze direction, and head orientation, all of which can be estimated in real time during the interaction. By utilizing these signals and novel data representations suitable for the task and context, the developed methods can generate plausible candidate gaze targets in real time. The methods are based on Feedforward Neural Networks and Long Short-Term Memory Networks. The proposed methods are developed using several hours of unrestricted interaction data and their performance is compared with a heuristic baseline method. The study offers an extensive evaluation of the proposed methods that investigates the contribution of different predictors to the accurate generation of candidate gaze targets. The results show that the methods can accurately generate candidate gaze targets when the person being modeled is in a listening state. However, when the person being modeled is in a speaking state, the proposed methods yield significantly lower performance.
@article{10.1145/3323231, author = {Stefanov, Kalin and Salvi, Giampiero and Kontogiorgos, Dimosthenis and Kjellstr\"{o}m, Hedvig and Beskow, Jonas}, title = {Modeling of Human Visual Attention in Multiparty Open-World Dialogues}, year = {2019}, issue_date = {June 2019}, publisher = {Association for Computing Machinery}, address = {New York, NY, USA}, volume = {8}, number = {2}, url = {https://doi.org/10.1145/3323231}, doi = {10.1145/3323231}, journal = {ACM Transactions on Human-Robot Interaction}, articleno = {8}, numpages = {21}, keywords = {multiparty, eye-gaze direction, open-world dialogue, head orientation, human-human interaction}, pdf = {3323231.pdf}, bibtex_show = {true}, preview = {paper.png} }

Towards Digitally-Mediated Sign Language Communication

Kalin Stefanov, and Mayumi Bono

In Proceedings of the International Conference on Human-Agent Interaction, 2019

Abs Bib PDF

This paper presents our efforts towards building an architecture for digitally-mediated sign language communication. The architecture is based on a client-server model and enables a near real-time recognition of sign language signs on a mobile device. The paper describes the two main components of the architecture, a recognition engine (server-side) and a mobile application (client-side), and outlines directions for future work.

@inproceedings{10.1145/3349537.3352794,
  author = {Stefanov, Kalin and Bono, Mayumi},
  title = {Towards Digitally-Mediated Sign Language Communication},
  year = {2019},
  isbn = {9781450369220},
  publisher = {Association for Computing Machinery},
  address = {New York, NY, USA},
  url = {https://doi.org/10.1145/3349537.3352794},
  doi = {10.1145/3349537.3352794},
  booktitle = {Proceedings of the International Conference on Human-Agent Interaction},
  pages = {286--288},
  numpages = {3},
  keywords = {sign language, key word signing, recognition},
  location = {Kyoto, Japan},
  series = {HAI '19},
  pdf = {3349537.3352794.pdf},
  bibtex_show = {true},
  preview = {paper.png}
}

Multimodal Learning for Identifying Opportunities for Empathetic Responses

Leili Tavabi, Kalin Stefanov, Setareh Nasihati Gilani, David Traum, and Mohammad Soleymani

In Proceedings of the ACM International Conference on Multimodal Interaction, 2019

Abs Bib PDF

Embodied interactive agents possessing emotional intelligence and empathy can create natural and engaging social interactions. Providing appropriate responses by interactive virtual agents requires the ability to perceive users’ emotional states. In this paper, we study and analyze behavioral cues that indicate an opportunity to provide an empathetic response. Emotional tone in language in addition to facial expressions are strong indicators of dramatic sentiment in conversation that warrant an empathetic response. To automatically recognize such instances, we develop a multimodal deep neural network for identifying opportunities when the agent should express positive or negative empathetic responses. We train and evaluate our model using audio, video and language from human-agent interactions in a wizard-of-Oz setting, using the wizard’s empathetic responses and annotations collected on Amazon Mechanical Turk as ground-truth labels. Our model outperforms a text-based baseline achieving F1-score of 0.71 on a three-class classification. We further investigate the results and evaluate the capability of such a model to be deployed for real-world human-agent interactions.
@inproceedings{10.1145/3340555.3353750, author = {Tavabi, Leili and Stefanov, Kalin and Nasihati Gilani, Setareh and Traum, David and Soleymani, Mohammad}, title = {Multimodal Learning for Identifying Opportunities for Empathetic Responses}, year = {2019}, isbn = {9781450368605}, publisher = {Association for Computing Machinery}, address = {New York, NY, USA}, url = {https://doi.org/10.1145/3340555.3353750}, doi = {10.1145/3340555.3353750}, booktitle = {Proceedings of the ACM International Conference on Multimodal Interaction}, pages = {95--104}, numpages = {10}, keywords = {virtual human, machine learning, multimodal sentiment, empathy, human behavior}, location = {Suzhou, China}, series = {ICMI '19}, pdf = {3340555.3353750.pdf}, bibtex_show = {true}, preview = {paper.png} }
Multimodal Analysis and Estimation of Intimate Self-Disclosure

Mohammad Soleymani, Kalin Stefanov, Sin-Hwa Kang, Jan Ondras, and Jonathan Gratch

In Proceedings of the ACM International Conference on Multimodal Interaction, 2019

Abs Bib PDF

Self-disclosure to others has a proven benefit for one’s mental health. It is shown that disclosure to computers can be similarly beneficial for emotional and psychological well-being. In this paper, we analyzed verbal and nonverbal behavior associated with self-disclosure in two datasets containing structured human-human and human-agent interviews from more than 200 participants. Correlation analysis of verbal and nonverbal behavior revealed that linguistic features such as affective and cognitive content in verbal behavior, and nonverbal behavior such as head gestures are associated with intimate self-disclosure. A multimodal deep neural network was developed to automatically estimate the level of intimate self-disclosure from verbal and nonverbal behavior. Between modalities, verbal behavior was the best modality for estimating self-disclosure within-corpora achieving r = 0.66. However, the cross-corpus evaluation demonstrated that nonverbal behavior can outperform language modality in cross-corpus evaluation. Such automatic models can be deployed in interactive virtual agents or social robots to evaluate rapport and guide their conversational strategy.
@inproceedings{10.1145/3340555.3353737, author = {Soleymani, Mohammad and Stefanov, Kalin and Kang, Sin-Hwa and Ondras, Jan and Gratch, Jonathan}, title = {Multimodal Analysis and Estimation of Intimate Self-Disclosure}, year = {2019}, isbn = {9781450368605}, publisher = {Association for Computing Machinery}, address = {New York, NY, USA}, url = {https://doi.org/10.1145/3340555.3353737}, doi = {10.1145/3340555.3353737}, booktitle = {Proceedings of the ACM International Conference on Multimodal Interaction}, pages = {59--68}, numpages = {10}, keywords = {neural networks, nonverbal behavior, self-disclosure, natural language understanding}, location = {Suzhou, China}, series = {ICMI '19}, pdf = {3340555.3353737.pdf}, bibtex_show = {true}, preview = {paper.png} }

2018

Recognition and Generation of Communicative Signals: Modeling of Hand Gestures, Speech Activity and Eye-Gaze in Human-Machine Interaction

Kalin Stefanov

KTH Royal Institute of Technology, 2018

Abs Bib PDF

Nonverbal communication is essential for natural and effective face-to-face human-human interaction. It is the process of communicating through sending and receiving wordless (mostly visual, but also auditory) signals between people. Consequently, a natural and effective face-to-face human-machine interaction requires machines (e.g., robots) to understand and produce such human-like signals. There are many types of nonverbal signals used in this form of communication including, body postures, hand gestures, facial expressions, eye movements, touches and uses of space. This thesis investigates two of these nonverbal signals: hand gestures and eye-gaze. The main goal of the thesis is to propose computational methods for real-time recognition and generation of these two signals in order to facilitate natural and effective human-machine interaction. The first topic addressed in the thesis is the real-time recognition of hand gestures and its application to recognition of isolated sign language signs. Hand gestures can also provide important cues during human-robot interaction, for example, emblems are type of hand gestures with specific meaning used to substitute spoken words. The thesis has two main contributions with respect to the recognition of hand gestures: 1) a newly collected dataset of isolated Swedish Sign Language signs, and 2) a real-time hand gestures recognition method. The second topic addressed in the thesis is the general problem of real-time speech activity detection in noisy and dynamic environments and its application to socially-aware language acquisition. Speech activity can also provide important information during human-robot interaction, for example, the current active speaker’s hand gestures and eye-gaze direction or head orientation can play an important role in understanding the state of the interaction. The thesis has one main contribution with respect to speech activity detection: a real-time vision-based speech activity detection method. The third topic addressed in the thesis is the real-time generation of eye-gaze direction or head orientation and its application to human-robot interaction. Eye-gaze direction or head orientation can provide important cues during human-robot interaction, for example, it can regulate who is allowed to speak when and coordinate the changes in the roles on the conversational floor (e.g., speaker, addressee, and bystander). The thesis has two main contributions with respect to the generation of eye-gaze direction or head orientation: 1) a newly collected dataset of face-to-face interactions, and 2) a real-time eye-gaze direction or head orientation generation method.
@phdthesis{stefanov2018recognition, title = {Recognition and Generation of Communicative Signals: Modeling of Hand Gestures, Speech Activity and Eye-Gaze in Human-Machine Interaction}, author = {Stefanov, Kalin}, year = {2018}, school = {KTH Royal Institute of Technology}, pdf = {phd.thesis.pdf}, bibtex_show = {true}, preview = {paper.png} }

2017

Vision-based Active Speaker Detection in Multiparty Interaction

Kalin Stefanov, Jonas Beskow, and Giampiero Salvi

In Proceedings of the International Workshop on Grounding Language Understanding, 2017

Abs Bib PDF

This paper presents a supervised learning method for automatic visual detection of the active speaker in multiparty interactions. The presented detectors are built using a multimodal multiparty interaction dataset previously recorded with the purpose to explore patterns in the focus of visual attention of humans. Three different conditions are included: two humans involved in taskbased interaction with a robot; the same two humans involved in task-based interaction where the robot is replaced by a third human, and a free three-party human interaction. The paper also presents an evaluation of the active speaker detection method in a speaker dependent experiment showing that the method achieves good accuracy rates in a fairly unconstrained scenario using only image data as input. The main goal of the presented method is to provide real-time detection of the active speaker within a broader framework implemented on a robot and used to generate natural focus of visual attention behavior during multiparty human-robot interactions.
@inproceedings{10.21437/GLU.2017-10, author = {Stefanov, Kalin and Beskow, Jonas and Salvi, Giampiero}, title = {Vision-based Active Speaker Detection in Multiparty Interaction}, year = {2017}, booktitle = {Proceedings of the International Workshop on Grounding Language Understanding}, pages = {47--51}, doi = {10.21437/GLU.2017-10}, pdf = {GLU.2017-10.pdf}, bibtex_show = {true}, preview = {paper.png} }
A Real-time Gesture Recognition System for Isolated Swedish Sign Language Signs

Kalin Stefanov, and Jonas Beskow

In Proceedings of the European and Nordic Symposium on Multimodal Communication, 2017

Abs Bib PDF

This paper describes a method for automatic recognition of isolated Swedish Sign Language signs for the purpose of educational signing-based games. Two datasets consisting of 51 signs have been recorded from a total of 7 (experienced) and 10 (inexperienced) adult signers. The signers performed all of the signs 5 times and were captured with a RGB-D (Kinect) sensor, via a purpose-built recording application. A recognizer based on manual components of sign language is presented and tested on the collected datasets. Signer-dependent recognition rate is 95.3% for the most consistent signer. Signer-independent recognition rate is on average 57.9% for the experienced signers and 68.9% for the inexperienced.
@inproceedings{ecp17141004, title = {A Real-time Gesture Recognition System for Isolated Swedish Sign Language Signs}, author = {Stefanov, Kalin and Beskow, Jonas}, year = {2017}, series = {Link{\"o}ping Electronic Conference Proceedings}, publisher = {Link{\"o}ping University Electronic Press}, pages = {18--27}, booktitle = {Proceedings of the European and Nordic Symposium on Multimodal Communication}, pdf = {ecp17141004.pdf}, bibtex_show = {true}, preview = {paper.png} }

2016

A Multi-party Multi-modal Dataset for Focus of Visual Attention in Human-human and Human-robot Interaction

Kalin Stefanov, and Jonas Beskow

In Proceedings of the International Conference on Language Resources and Evaluation, 2016

Abs Bib PDF

This papers describes a data collection setup and a newly recorded dataset. The main purpose of this dataset is to explore patterns in the focus of visual attention of humans under three different conditions - two humans involved in task-based interaction with a robot; same two humans involved in task-based interaction where the robot is replaced by a third human, and a free three-party human interaction. The dataset contains two parts - 6 sessions with duration of approximately 3 hours and 9 sessions with duration of approximately 4.5 hours. Both parts of the dataset are rich in modalities and recorded data streams - they include the streams of three Kinect v2 devices (color, depth, infrared, body and face data), three high quality audio streams, three high resolution GoPro video streams, touch data for the task-based interactions and the system state of the robot. In addition, the second part of the dataset introduces the data streams from three Tobii Pro Glasses 2 eye trackers. The language of all interactions is English and all data streams are spatially and temporally aligned.
@inproceedings{L16-1703, title = {A Multi-party Multi-modal Dataset for Focus of Visual Attention in Human-human and Human-robot Interaction}, author = {Stefanov, Kalin and Beskow, Jonas}, booktitle = {Proceedings of the International Conference on Language Resources and Evaluation}, year = {2016}, address = {Portoro{\v{z}}, Slovenia}, publisher = {European Language Resources Association}, url = {https://aclanthology.org/L16-1703}, pages = {4440--4444}, pdf = {L16-1703.pdf}, bibtex_show = {true}, preview = {paper.png} }
Look Who’s Talking: Visual Identification of the Active Speaker in Multi-Party Human-Robot Interaction

Kalin Stefanov, Akihiro Sugimoto, and Jonas Beskow

In Proceedings of the International Workshop on Advancements in Social Signal Processing for Multimodal Interaction, 2016

Abs Bib PDF

This paper presents analysis of a previously recorded multi-modal interaction dataset. The primary purpose of that dataset is to explore patterns in the focus of visual attention of humans under three different conditions - two humans involved in task-based interaction with a robot; the same two humans involved in task-based interaction where the robot is replaced by a third human, and a free three-party human interaction. The paper presents a data-driven methodology for automatic visual identification of the active speaker based on facial action units (AUs). The paper also presents an evaluation of the proposed methodology on 12 different interactions with an approximate length of 4 hours. The methodology will be implemented on a robot and used to generate natural focus of visual attention behavior during multi-party human-robot interactions.
@inproceedings{10.1145/3005467.3005470, author = {Stefanov, Kalin and Sugimoto, Akihiro and Beskow, Jonas}, title = {Look Who's Talking: Visual Identification of the Active Speaker in Multi-Party Human-Robot Interaction}, year = {2016}, isbn = {9781450345576}, publisher = {Association for Computing Machinery}, address = {New York, NY, USA}, url = {https://doi.org/10.1145/3005467.3005470}, doi = {10.1145/3005467.3005470}, booktitle = {Proceedings of the International Workshop on Advancements in Social Signal Processing for Multimodal Interaction}, pages = {22--27}, numpages = {6}, keywords = {human-robot interaction, active speaker identification, multi-modal interaction}, location = {Tokyo, Japan}, series = {ASSP4MI '16}, pdf = {3005467.3005470.pdf}, bibtex_show = {true}, preview = {paper.png} }
Gesture Recognition System for Isolated Sign Language Signs

Kalin Stefanov, and Jonas Beskow

In Proceedings of the European and Nordic Symposium on Multimodal Communication, 2016

Abs Bib PDF

This paper describes a method for automatic recognition of isolated Swedish Sign Language signs for the purpose of educational signing-based games. Two datasets consisting of 51 signs have been recorded from a total of 7 (experienced) and 10 (inexperienced) adult signers. The signers performed all of the signs 5 times and were captured with a RGB-D (Kinect) sensor, via a purpose-built recording application. A recognizer based on manual components of sign language is presented and tested on the collected datasets. Signer-dependent recognition rate is 95.3% for the most consistent signer. Signer-independent recognition rate is on average 57.9% for the experienced signers and 68.9% for the inexperienced.
@inproceedings{stefanov2016gesture, title = {Gesture Recognition System for Isolated Sign Language Signs}, author = {Stefanov, Kalin and Beskow, Jonas}, booktitle = {Proceedings of the European and Nordic Symposium on Multimodal Communication}, pages = {57--59}, year = {2016}, pdf = {stefanov2016gesture}, bibtex_show = {true}, preview = {paper.png} }

2015

Public Speaking Training with a Multimodal Interactive Virtual Audience Framework

Mathieu Chollet, Kalin Stefanov, Helmut Prendinger, and Stefan Scherer

In Proceedings of the ACM International Conference on Multimodal Interaction, 2015

Abs Bib PDF

We have developed an interactive virtual audience platform for public speaking training. Users’ public speaking behavior is automatically analyzed using multimodal sensors, and ultimodal feedback is produced by virtual characters and generic visual widgets depending on the user’s behavior. The flexibility of our system allows to compare different interaction mediums (e.g. virtual reality vs normal interaction), social situations (e.g. one-on-one meetings vs large audiences) and trained behaviors (e.g. general public speaking performance vs specific behaviors).

@inproceedings{10.1145/2818346.2823294,
  author = {Chollet, Mathieu and Stefanov, Kalin and Prendinger, Helmut and Scherer, Stefan},
  title = {Public Speaking Training with a Multimodal Interactive Virtual Audience Framework},
  year = {2015},
  isbn = {9781450339124},
  publisher = {Association for Computing Machinery},
  address = {New York, NY, USA},
  url = {https://doi.org/10.1145/2818346.2823294},
  doi = {10.1145/2818346.2823294},
  booktitle = {Proceedings of the ACM International Conference on Multimodal Interaction},
  pages = {367--368},
  numpages = {2},
  keywords = {automatic behavior recognition, public speaking training, virtual audience},
  location = {Seattle, Washington, USA},
  series = {ICMI '15},
  pdf = {2818346.2823294.pdf},
  bibtex_show = {true},
  preview = {paper.png}
}

2014

Tutoring Robots

Samer Al Moubayed, Jonas Beskow, Bajibabu Bollepalli, Ahmed Hussen-Abdelaziz, Martin Johansson, Maria Koutsombogera, José David Lopes, Jekaterina Novikova, Catharine Oertel, Gabriel Skantze, Kalin Stefanov, and Gül Varol

In Innovative and Creative Developments in Multimodal Interaction Systems, 2014

Abs Bib PDF

This project explores a novel experimental setup towards building spoken, multi-modally rich, and human-like multiparty tutoring agent. A setup is developed and a corpus is collected that targets the development of a dialogue system platform to explore verbal and nonverbal tutoring strategies in multiparty spoken interactions with embodied agents. The dialogue task is centered on two participants involved in a dialogue aiming to solve a card-ordering game. With the participants sits a tutor that helps the participants perform the task and organizes and balances their interaction. Different multimodal signals captured and auto-synchronized by different audio-visual capture technologies were coupled with manual annotations to build a situated model of the interaction based on the participants personalities, their temporally-changing state of attention, their conversational engagement and verbal dominance, and the way these are correlated with the verbal and visual feedback, turn-management, and conversation regulatory actions generated by the tutor. At the end of this chapter we discuss the potential areas of research and developments this work opens and some of the challenges that lie in the road ahead.
@inproceedings{10.1007/978-3-642-55143-7_4, author = {Al Moubayed, Samer and Beskow, Jonas and Bollepalli, Bajibabu and Hussen-Abdelaziz, Ahmed and Johansson, Martin and Koutsombogera, Maria and Lopes, Jos{\'e} David and Novikova, Jekaterina and Oertel, Catharine and Skantze, Gabriel and Stefanov, Kalin and Varol, G{\"u}l}, editor = {Rybarczyk, Yves and Cardoso, Tiago and Rosas, Jo{\~a}o and Camarinha-Matos, Luis M.}, title = {Tutoring Robots}, booktitle = {Innovative and Creative Developments in Multimodal Interaction Systems}, year = {2014}, publisher = {Springer Berlin Heidelberg}, address = {Berlin, Heidelberg}, pages = {80--113}, isbn = {978-3-642-55143-7}, pdf = {978-3-642-55143-7_4.pdf}, bibtex_show = {true}, preview = {paper.png} }
Human-Robot Collaborative Tutoring using Multiparty Multimodal Spoken Dialogue

Samer Al Moubayed, Jonas Beskow, Bajibabu Bollepalli, Joakim Gustafson, Ahmed Hussen-Abdelaziz, Martin Johansson, Maria Koutsombogera, José David Lopes, Jekaterina Novikova, Catharine Oertel, Gabriel Skantze, Kalin Stefanov, and Gül Varol

In Proceedings of the ACM/IEEE International Conference on Human-Robot Interaction, 2014

Abs Bib PDF

In this paper, we describe a project that explores a novel experimental setup towards building a spoken, multi-modally rich, and human-like multiparty tutoring robot. A human-robot interaction setup is designed, and a human-human dialogue corpus is collected. The corpus targets the development of a dialogue system platform to study verbal and nonverbal tutoring strategies in multiparty spoken interactions with robots which are capable of spoken dialogue. The dialogue task is centered on two participants involved in a dialogue aiming to solve a card-ordering game. Along with the participants sits a tutor (robot) that helps the participants perform the task, and organizes and balances their interaction. Different multimodal signals captured and autosynchronized by different audio-visual capture technologies, such as a microphone array, Kinects, and video cameras, were coupled with manual annotations. These are used build a situated model of the interaction based on the participants personalities, their state of attention, their conversational engagement and verbal dominance, and how that is correlated with the verbal and visual feedback, turn-management, and conversation regulatory actions generated by the tutor. Driven by the analysis of the corpus, we will show also the detailed design methodologies for an affective, and multimodally rich dialogue system that allows the robot to measure incrementally the attention states, and the dominance for each participant, allowing the robot head Furhat to maintain a wellcoordinated, balanced, and engaging conversation, that attempts to maximize the agreement and the contribution to solve the task. This project sets the first steps to explore the potential of using multimodal dialogue systems to build interactive robots that can serve in educational, team building, and collaborative task solving applications.
@inproceedings{8542606, author = {Moubayed, Samer Al and Beskow, Jonas and Bollepalli, Bajibabu and Gustafson, Joakim and Hussen-Abdelaziz, Ahmed and Johansson, Martin and Koutsombogera, Maria and Lopes, José David and Novikova, Jekaterina and Oertel, Catharine and Skantze, Gabriel and Stefanov, Kalin and Varol, Gül}, booktitle = {Proceedings of the ACM/IEEE International Conference on Human-Robot Interaction}, title = {Human-Robot Collaborative Tutoring using Multiparty Multimodal Spoken Dialogue}, year = {2014}, pages = {112--113}, issn = {2167-2121}, pdf = {8542606.pdf}, bibtex_show = {true}, preview = {paper.png} }
The Tutorbot Corpus — A Corpus for Studying Tutoring Behaviour in Multiparty Face-to-Face Spoken Dialogue

Maria Koutsombogera, Samer Al Moubayed, Bajibabu Bollepalli, Ahmed Hussen Abdelaziz, Martin Johansson, José David Aguas Lopes, Jekaterina Novikova, Catharine Oertel, Kalin Stefanov, and Gül Varol

In Proceedings of the International Conference on Language Resources and Evaluation, 2014

Abs Bib PDF

This paper describes a novel experimental setup exploiting state-of-the-art capture equipment to collect a multimodally rich game-solving collaborative multiparty dialogue corpus. The corpus is targeted and designed towards the development of a dialogue system platform to explore verbal and nonverbal tutoring strategies in multiparty spoken interactions. The dialogue task is centered on two participants involved in a dialogue aiming to solve a card-ordering game. The participants were paired into teams based on their degree of extraversion as resulted from a personality test. With the participants sits a tutor that helps them perform the task, organizes and balances their interaction and whose behavior was assessed by the participants after each interaction. Different multimodal signals captured and auto-synchronized by different audio-visual capture technologies, together with manual annotations of the tutors behavior constitute the Tutorbot corpus. This corpus is exploited to build a situated model of the interaction based on the participants temporally-changing state of attention, their conversational engagement and verbal dominance, and their correlation with the verbal and visual feedback and conversation regulatory actions generated by the tutor.
@inproceedings{L14-1641, title = {The Tutorbot Corpus {---} A Corpus for Studying Tutoring Behaviour in Multiparty Face-to-Face Spoken Dialogue}, author = {Koutsombogera, Maria and Moubayed, Samer Al and Bollepalli, Bajibabu and Abdelaziz, Ahmed Hussen and Johansson, Martin and Lopes, Jos{\'e} David Aguas and Novikova, Jekaterina and Oertel, Catharine and Stefanov, Kalin and Varol, G{\"u}l}, booktitle = {Proceedings of the International Conference on Language Resources and Evaluation}, year = {2014}, address = {Reykjavik, Iceland}, publisher = {European Language Resources Association}, url = {http://www.lrec-conf.org/proceedings/lrec2014/pdf/832_Paper.pdf}, pages = {4196--4201}, pdf = {L14-1641.pdf}, bibtex_show = {true}, preview = {paper.png} }
A Data-Driven Approach to Detection of Interruptions in Human-Human Conversations

Raveesh Meena, Saeed Dabbaghchian, and Kalin Stefanov

In Proceedings of the FONETIK, 2014

Abs Bib PDF

We report the results of our initial efforts towards automatic detection of user’s interruptions in a spoken human–machine dialogue. In a first step, we explored the use of automatically extractable acoustic features, frequency and intensity, in discriminating listener’s interruptions in human–human conversations. A preliminary analysis of interaction snippets from the HCRC Map Task corpus suggests that for the task at hand, intensity is a stronger feature than frequency, and using intensity in combination with feature loudness offers the best results for a k-means clustering algorithm.
@inproceedings{774962, title = {A Data-Driven Approach to Detection of Interruptions in Human-Human Conversations}, author = {Meena, Raveesh and Dabbaghchian, Saeed and Stefanov, Kalin}, booktitle = {Proceedings of the FONETIK}, pages = {29--32}, year = {2014}, pdf = {774962.pdf}, bibtex_show = {true}, preview = {paper.png} }

Tivoli - Learning Signs Through Games and Interaction for Children with Communicative Disorders

Jonas Beskow, Simon Alexanderson, Kalin Stefanov, Britt Claesson, Sandra Derbring, Morgan Fredriksson, J. Starck, and E. Axelsson

In Proceedings of the Biennial Conference of the International Society for Augmentative and Alternative Communication, 2014

Bib

@inproceedings{beskow2014tivoli,
  title = {Tivoli - Learning Signs Through Games and Interaction for Children with Communicative Disorders},
  author = {Beskow, Jonas and Alexanderson, Simon and Stefanov, Kalin and Claesson, Britt and Derbring, Sandra and Fredriksson, Morgan and Starck, J. and Axelsson, E.},
  booktitle = {Proceedings of the Biennial Conference of the International Society for Augmentative and Alternative Communication},
  year = {2014},
  bibtex_show = {true},
  preview = {paper.png}
}

2013

A Kinect Corpus of Swedish Sign Language Signs

Kalin Stefanov, and Jonas Beskow

In Proceedings of the Workshop on Multimodal Corpora: Beyond Audio and Video, 2013

Abs Bib PDF

We describe a corpus of Swedish sign language signs, recorded for the purpose of an educational signing game. The primary target group of the game is children with communicative disabilities, and the goal is to offer a playful and interactive way of learning and practicing sign language signs to these children, as well as to their friends and family. As a first step, a dataset consisting of 51 signs has been recorded for a total of 10 adult signers. The signers performed all of the signs five times and were captured with an RGB-D (Microsoft Kinect) sensor, via a purpose-built recording application.
@inproceedings{e2e46e0abff94d4d944bec6ab0553087, title = {A Kinect Corpus of Swedish Sign Language Signs}, author = {Stefanov, Kalin and Beskow, Jonas}, year = {2013}, booktitle = {Proceedings of the Workshop on Multimodal Corpora: Beyond Audio and Video}, pdf = {e2e46e0abff94d4d944bec6ab0553087.pdf}, bibtex_show = {true}, preview = {paper.png} }

The Tivoli System–A Sign-driven Game for Children with Communicative Disorders

Jonas Beskow, Simon Alexanderson, Kalin Stefanov, Britt Claesson, Sandra Derbring, and Morgan Fredriksson

In Proceedings of the Symposium on Multimodal Communication, 2013

Bib

@inproceedings{beskow2013tivoli,
  title = {The Tivoli System--A Sign-driven Game for Children with Communicative Disorders},
  author = {Beskow, Jonas and Alexanderson, Simon and Stefanov, Kalin and Claesson, Britt and Derbring, Sandra and Fredriksson, Morgan},
  booktitle = {Proceedings of the Symposium on Multimodal Communication},
  year = {2013},
  bibtex_show = {true},
  preview = {paper.png}
}

Web-Enabled 3D Talking Avatars Based on WebGL and HTML5

Jonas Beskow, and Kalin Stefanov

In Proceedings of the International Conference on Intelligent Virtual Agents, 2013

Abs Bib PDF

We describe a system for plugin-free deployment of 3D talking characters on the web. The system employs the WebGL capabilites of modern web browsers in order to produce real-time animation of speech movements, in synchrony with text-to-speech synthesis, played back using HTML5 audio functionalty. The implementation is divided into a client and a server part, where the server delivers the audio waveform and the animation tracks for lip synchronisation, and the client takes care of audio playback and rendering of the avatar in the browser.
@inproceedings{3903, title = {Web-Enabled 3D Talking Avatars Based on WebGL and HTML5}, author = {Beskow, Jonas and Stefanov, Kalin}, booktitle = {Proceedings of the International Conference on Intelligent Virtual Agents}, volume = {8108}, pages = {486}, year = {2013}, organization = {Springer}, pdf = {3903.pdf}, bibtex_show = {true}, preview = {paper.png} }

2012

Multimodal Multiparty Social Interaction with the Furhat Head

Samer Al Moubayed, Gabriel Skantze, Jonas Beskow, Kalin Stefanov, and Joakim Gustafson

In Proceedings of the ACM International Conference on Multimodal Interaction, 2012

Abs Bib PDF

We will show in this demonstrator an advanced multimodal and multiparty spoken conversational system using Furhat, a robot head based on projected facial animation. Furhat is a human-like interface that utilizes facial animation for physical robot heads using back-projection. In the system, multimodality is enabled using speech and rich visual input signals such as multi-person real-time face tracking and microphone tracking. The demonstrator will showcase a system that is able to carry out social dialogue with multiple interlocutors simultaneously with rich output signals such as eye and head coordination, lips synchronized speech synthesis, and non-verbal facial gestures used to regulate fluent and expressive multiparty conversations.
@inproceedings{10.1145/2388676.2388736, author = {Al Moubayed, Samer and Skantze, Gabriel and Beskow, Jonas and Stefanov, Kalin and Gustafson, Joakim}, title = {Multimodal Multiparty Social Interaction with the Furhat Head}, year = {2012}, isbn = {9781450314671}, publisher = {Association for Computing Machinery}, address = {New York, NY, USA}, url = {https://doi.org/10.1145/2388676.2388736}, doi = {10.1145/2388676.2388736}, booktitle = {Proceedings of the ACM International Conference on Multimodal Interaction}, pages = {293--294}, numpages = {2}, keywords = {speech, spoken dialog, gaze, robot head, microphone tracking, gesture, facial animation, furhat, multiparty interaction, multimodal systems}, location = {Santa Monica, California, USA}, series = {ICMI '12}, pdf = {2388676.2388736.pdf}, bibtex_show = {true}, preview = {paper.png} }
Socially Aware Many-to-Machine Communication

F. Eyben, E. Gilmartin, C. Joder, E. Marchi, C. Munier, Kalin Stefanov, F. Weninger, and B. Schuller

In Proceedings of the International Summer Workshop on Multimodal Interfaces, 2012

Abs Bib

This reports describes the output of the project P5: Socially Aware Many-to-Machine Communication (M2M) at the eNTERFACE’12 workshop. In this project, we designed and implemented a new front-end for handling multi-user interaction in a dialog system. We exploit the Microsoft Kinect device for capturing multimodal input and extract some features describing user and face positions. These data are then analyzed in real-time to robustly detect speech and determine both who is speaking and whether the speech is directed to the system or not. This new front-end is integrated to the SEMAINE (Sustained Emotionally colored Machine-human Interaction using Nonverbal Expression) system. Furthermore, a multimodal corpus has been created, capturing all of the system inputs in two different scenarios involving human-human and human-computer interaction.
@inproceedings{eyben2012socially, title = {Socially Aware Many-to-Machine Communication}, author = {Eyben, F. and Gilmartin, E. and Joder, C. and Marchi, E. and Munier, C. and Stefanov, Kalin and Weninger, F. and Schuller, B.}, booktitle = {Proceedings of the International Summer Workshop on Multimodal Interfaces}, year = {2012}, bibtex_show = {true}, preview = {paper.png} }

2011

D2. 4.3 Spreading Activation Components v3 - LarKC Project Deliverable

M. Grinberg, H. Stefanov, Kalin Stefanov, and Ivan Peikov

2011

Bib

@article{grinberg2011d2,
  title = {D2. 4.3 Spreading Activation Components v3 - LarKC Project Deliverable},
  author = {Grinberg, M. and Stefanov, H. and Stefanov, Kalin and Peikov, Ivan},
  year = {2011},
  bibtex_show = {true},
  preview = {paper.png}
}

2010

Webcam-based Eye Gaze Tracking under Natural Head Movement

Kalin Stefanov

University of Amsterdam, 2010

Abs Bib PDF

This manuscript investigates and proposes a visual gaze tracker that tackles the problem using only an ordinary web camera and no prior knowledge in any sense (scene set-up, camera intrinsic and/or extrinsic parameters). The tracker we propose is based on the observation that our desire to grant the freedom of natural head movement to the user requires 3D modeling of the scene set-up. Although, using a single low resolution web camera bounds us in dimensions (no depth can be recovered), we propose ways to cope with this drawback and model the scene in front of the user. We tackle this three-dimensional problem by realizing that it can be viewed as series of two-dimensional special cases. Then, we propose a procedure that treats each movement of the user’s head as a special two-dimensional case, hence reducing the complexity of the problem back to two dimensions. Furthermore, the proposed tracker is calibration free and discards this tedious part of all previously mentioned trackers. Experimental results show that the proposed tracker achieves good results, given the restrictions on it. We can report that the tracker commits a mean error of (56.95, 70.82) pixels in x and y direction, respectively, when the user’s head is as static as possible (no chin-rests are used). Furthermore, we can report that the proposed tracker commits a mean error of (87.18, 103.86) pixels in x and y direction, respectively, under natural head movement.
@mastersthesis{1803.11088, author = {Stefanov, Kalin}, title = {Webcam-based Eye Gaze Tracking under Natural Head Movement}, school = {University of Amsterdam}, year = {2010}, eprint = {1803.11088}, archiveprefix = {arXiv}, primaryclass = {cs.CV}, pdf = {1803.11088.pdf}, bibtex_show = {true}, preview = {paper.png} }