Automatic Identification of Mice Social Behavior Through Multi-Modal Latent Space Clustering

Lab mice are widely used in the medical and biological fields for various purposes. For example, the social behavior of mice can provide important information about the effect of drugs or mutations in mice. However, studying behavior of mice is difficult because often experiments can generate long video data that a researcher must manually annotate and view. This study proposes a behavioral clustering model that takes simple features from video data and forms separate clusters for different mice interactions in latent space. In contrast to past works that detect a predefined set of mice interactions, our proposed model can handle any number of different interactions. Moreover, we also detect ultrasonic vocalizations of mice to study both vocal and non-vocal interactions in the same framework. Although our model was developed on two interacting mice data, it can be easily extended to other animals because of the simple features we use for clustering. Contributions-We present a multi-modal framework that takes mice video and sound data and automatically cluster social behavior in latent space that can be easily visualized in a single figure.