Automatic Infant Touch Detection¶

Author: Lukas Navara
Course: NI-MLP
Date: 30. 12. 2025

Abstract¶

This report studies the problem of automatic detection of infant touch events in video recordings. A custom dataset of infant videos was used to train a baseline deep learning model that predicts touch events at the frame level. The model was evaluated using quantitative performance metrics and qualitative inspection of temporal predictions.

The results show that automatic touch detection is feasible using the proposed approach. The model is able to generalize to infants that were not seen during training. Possible directions for future work are discussed, such as pose-based post-processing for touch zone estimation, and using the model as an assistive tool for faster data annotation.

Table of Contents¶

  1. Introduction
  2. Data Understanding
  3. Data Cleaning and Label Construction
  4. Modeling
  5. Evaluation
  6. Conclusions and Future Work
  7. Appendix

1 Introduction¶

No description has been provided for this image Figure 1: TinyTouch.

In the laboratory of Matěj Hoffmann at FEL CTU, we study infant development. A key topic is how infants explore their own bodies through touch. Touch events are annotated by human coders who review video recordings frame by frame. In the past, this process was done using Datavyu. The workflow was slow and error-prone because coders had to manually assign a body zone for each touch event.

To improve this process, I developed TinyTouch, a custom annotation tool. It allows coders to label touch events by clicking directly on a body diagram. This makes annotation faster and more consistent than the previous workflow. However, even with this improvement, the dataset is large. Fully manual annotation still requires a substantial amount of time and effort. This motivates the use of automation.

The goal of this project is to use existing labeled data to train a model that can automatically detect touch events. In the short term, the model can assist human coders by highlighting frames that are likely to contain touch events. With sufficient performance, it could later reduce the amount of manual work or partially replace manual annotation.

Coders do not record only binary touch versus no-touch labels. They also annotate the exact body location of the touch (Figure 1). In principle, a model could be trained to predict these touch zones as well. In this work, however, I focus on a simpler task: detecting whether a touch event occurs. This follows a step-by-step approach from a basic problem to more complex ones. The right hand is used as the target limb. If the approach is successful, the same pipeline can be applied to other limbs, such as the left hand or the legs.

Research hypothesis
A supervised machine learning model trained on existing annotations can detect right-hand touch events from video frames with performance meaningfully above a simple baseline.

Approach overview
The project follows a standard data analysis workflow. First, the annotated dataset is explored and cleaned. Next, input features are prepared and a supervised classification model is trained. Finally, model performance is evaluated using appropriate metrics.

Objective
Detect right-hand touch events from video frames.

2 Data Understanding¶

2.1 Data Source and Scope¶

The main data source is the file type_exp.csv. It contains a list of video recordings and their metadata, such as experiment type, recording conditions, and the annotation software used. From this dataset, recordings of four infants are included in this study: AA, CA, TH, and RT.

2.2 Subset Selection¶

To obtain a dataset that is consistent and easy to interpret, only a subset of recordings was selected. The selection was based on the following criteria:

  • the experiment involved spontaneous touch behavior,
  • the infant was recorded in the supine position,
  • touch events were annotated using the Datavyu software.

Buzzer-based experiments were excluded because touch events in these sessions are labeled only within predefined time intervals. Recordings where infants were seated in a car seat were also excluded. These setups introduce additional variability in posture and camera viewpoint. Overall, the selection strategy focuses on the simplest and most homogeneous experimental conditions.

2.3 Annotation Format¶

Datavyu annotations provide temporal labels defined by touch onset and offset timestamps. In this work, only labels generated with Datavyu are used. This avoids dealing with multiple export formats from TinyTouch and ensures that all annotations follow the same labeling scheme.

2.4 Final Dataset¶

The final dataset was split into training and validation sets at the video level. This prevents information leakage between the two sets.

Infant Training videos Validation videos
AA AA_st-2_s_s07_22w-0d_supine
AA_st-3_s_s06_19w-4d_supine
AA_st-3_s_s15_37w-1d_supine
AA_st-4_s_s09_25w-5d_supine
AA_st-6_s_s09_25w-5d_supine
AA_st-4_s_s08_23w-4d_supine
AA_st-2_s_s08_23w-4d_supine
AA_st-3_s_s08_23w-4d_supine
AA_st-5_s_s08_23w-4d_supine
AA_st-3_s_s03_13w-4d_supine		 AA_st-3_s_s02_11w-3d_supine
AA_st-4_s_s01_8w-4d_supine
CA CA_st-1_s_s06_20w-0d_supine
CA_st-1_s_s05_18w-0d_supine
CA_st-2_s_s07_22w-2d_supine
CA_st-3_s_s05_18w-3d_supine
CA_st-2_s_s05_18w-2d_supine
CA_st-6_s_s05_18w-4d_supine
CA_st-1_s_s03_14w-0d_supine		 CA_st-3_s_s04_16w-0d_supine
CA_st-2_r-3_s_s09_26w-3d_supine
TH TH_st_s_28_27w-6d_supine
TH_st-3_s_27_25w-4d_supine
TH_st-2_s_24_23w-5d_supine
TH_st-2_s_s22_21w-6d_supine
TH_st_s_s14_15w-3d_supine
TH_st_s_s23_23w-4d_supine
TH_st_s_s19_19w-5d_supine
TH_st-1_s_s04_7w-0d_supine
TH_st-2_s_s17_17w-4_supine
TH_st-3_s_s05_8w-3d_supine
TH_st-2_s_s21_21w-5d_supine
TH_st_s_s06_8w-4d_supine
TH_st-2_s_s07_8w-6d_supine
TH_st-1_s_s10_12w-3d_supine
TH_st_s_s15_15w-6d_supine
TH_st-1_s_s11_12w-4d_supine
TH_st-1_s_s12_12w-5d_supine
TH_st_s_s13_13w-6d_supine		 TH_st_s_s03_6w-6d_supine
TH_st-1_s_s08_10w-5d_supine
RT — RT_st_s_s02_17w-0d_supine
RT_st-2_s_s04_20w-1d_supine
RT_st-1_s_s04_20w-1d_supine
RT_st-3_s_s04_20w-1d_supine

Table 1: Training and validation video splits for each infant.

The training set contains 470,428 frame-level samples with a positive rate of approximately 32.0%. The validation set contains 149,364 samples with a positive rate of 31.9%. Figure 4 shows the distribution of touch and no-touch frames in the selected dataset.

2.5 Known Data Limitations¶

A known limitation of the dataset is that very short touch events are not annotated. Coders are instructed not to label contacts that last fewer than approximately seven video frames. As a result, some frames may visually appear to contain a touch even though no annotation is present. These cases follow the annotation guidelines and are therefore not considered labeling errors, but a limitation of the annotation protocol.

In addition, all labels are created manually by human coders. Some level of noise in the annotations is therefore expected. Touch events may be missed due to occlusions, unclear limb positions, or subjective judgment in ambiguous cases. Such imperfections should be seen as a natural property of human-annotated data rather than as systematic errors.

2.6 Video Properties¶

For each selected video, the frame rate was obtained from the video metadata. Individual frames were then extracted using the script extract_frames_for_dataset.py. These extracted frames form the basic input for all subsequent analysis and modeling steps.

3 Data Cleaning and Label Construction¶

This section describes how the raw Datavyu annotations and video recordings were converted into a frame-level dataset suitable for supervised learning.

3.1 Raw Annotations¶

The original annotations were exported from Datavyu. Each annotation represents a touch event as a time interval defined by touch onset and offset timestamps. The annotations also include additional metadata, such as a help flag that indicates whether a caregiver was involved.

3.2 Frame Extraction and Timestamp Assignment¶

Each video was processed using a script that extracts individual frames. Based on the video frame rate, a timestamp was assigned to every extracted frame. This produces a time-ordered sequence of frames for each recording.

The touch onset and offset timestamps were then mapped onto this frame sequence. Frames with timestamps that fall inside a touch interval were labeled as touch. All remaining frames were labeled as no-touch. This results in a binary, frame-level labeling that is suitable for training a classification model. The labels are stored in the file labels_v9.csv.

3.3 Filtering of Annotations¶

All annotation segments marked with the help flag were removed. These segments correspond to caregiver-assisted interactions and do not represent spontaneous infant touch, which is the focus of this work.

No description has been provided for this image

Figure 2: Infant detection and pose estimation. The detected right wrist keypoint defines the wrist-centered crop.

3.4 Pose Extraction and Wrist Localization¶

Body pose information was extracted for each frame to localize the right wrist. A two-step pipeline was used. First, YOLOv8 detected the infant and selected the largest bounding box. Second, ViTPose was applied within this box to estimate COCO-17 body keypoints. As shown in Figure 2, pose estimation is used to localize the right wrist.

The resulting keypoint coordinates and confidence scores were stored for each frame in the final dataset (labels_v9.csv). These keypoints are later used to crop the image around the right wrist. This provides a consistent region of interest for touch detection and reduces irrelevant visual information.

3.5 Resulting Dataset¶

The final dataset contains one row per video frame and includes:

  • a video identifier,
  • a path to the extracted frame,
  • a binary label indicating the presence of a right-hand touch,
  • a timestamp derived from the video frame rate,
  • pose keypoint coordinates and confidence scores,
  • a binary label indicating the train or validation split.
No description has been provided for this image

Figure 3: Frames per video in the training and validation sets

No description has been provided for this image

Figure 4: Overall frame distribution

4 Modeling¶

4.1 Task Formulation¶

The task is formulated as a binary classification problem at the frame level. For each video frame, the model predicts whether a right-hand touch event is present. The model outputs a probability score, which is later thresholded to produce a binary decision.

4.2 Input Representation¶

Instead of using the full video frame, the model operates on a wrist-centered image crop. For each frame, the right wrist position is obtained from pose estimation and used to extract a fixed-size region around the wrist. This reduces background noise and focuses the model on the most relevant visual information for touch detection.

To capture short-term motion, the input can optionally include a temporal window of neighboring frames. In this case, several consecutive wrist crops centered on the same frame are stacked and processed together. This allows the model to learn simple temporal patterns, such as the hand moving toward or away from the body.

4.3 Data Augmentation and Normalization¶

During training, standard image augmentations are applied to improve generalization. These include geometric and color transformations. When temporal windows are used, the same augmentation parameters are applied to all frames in the window. This preserves temporal consistency and avoids introducing artificial motion artifacts.

All images are normalized using ImageNet mean and standard deviation values. Figure 5 shows an example of a temporal window with consistent augmentation applied across frames.

No description has been provided for this image

Figure 5: Temporal window centered on a single frame. The same augmentation parameters are applied to all frames to preserve temporal consistency.

4.4 Model Architecture¶

The model uses a convolutional–recurrent architecture with three main components:

  1. Visual feature extractor
    Each input frame is processed by an EfficientNet-based convolutional neural network. This network extracts a compact feature representation from the wrist crop.

  2. Temporal aggregation
    When multiple frames are provided, the per-frame features are passed through a gated recurrent unit (GRU). A bidirectional GRU is used so that the representation of the central frame can leverage both past and future context within the temporal window.

  3. Classification head
    The aggregated feature representation is passed to a small multilayer perceptron. This network outputs a single logit that represents the likelihood of a right-hand touch.

4.5 Training Strategy¶

The dataset shows a strong imbalance between touch and no-touch frames and large differences in video length. Without additional controls, training would be dominated by long recordings and by the majority no-touch class. This could lead to a biased model with poor sensitivity to touch events.

To address these issues, a custom data loading pipeline was implemented. Frame access, wrist-centered cropping, optional temporal window construction, and metadata handling are encapsulated in a dedicated dataset class (WristCropDataset). This ensures consistent handling of spatial localization and temporal context during training and validation.

Data augmentation for temporal inputs is implemented using a custom transform (WristCropConsistentTransform). This transform applies the same randomly sampled augmentation parameters to all frames in a temporal window. This prevents artificial motion artifacts while still providing regularization.

Class imbalance and video-length bias are handled during batch construction using a custom sampler (PerVideoBalancedBatchSampler). Batches are formed to contain an approximately balanced ratio of touch and no-touch frames within each video whenever possible. In addition, a per-video sampling cap is applied in each epoch. This prevents long recordings from contributing a disproportionate number of samples. Across epochs, different subsets of frames are sampled, allowing the model to see a wide range of examples without over-representing any single video.

The model is trained using binary cross-entropy loss with logits and optimized with the AdamW optimizer. Separate learning rates can be used for the convolutional backbone and the classification head to improve training stability.

4.6 Model Selection¶

Model selection is based on validation performance. Several configurations were explored during development, but only the best-performing configuration is reported in the Evaluation section.

5 Evaluation¶

This section evaluates the performance of the best-performing model, WCTE-BiGRU (Wrist-Centered Temporal EfficientNet with BiGRU). Evaluation is based on quantitative metrics and qualitative inspection of model predictions.

5.1 Post-processing of Predictions¶

As described in Section 2.5, the ground-truth annotations follow a labeling policy in which very short touch events (approximately fewer than seven frames) are intentionally not labeled. This affects how predictions should be compared to the available labels.

To account for this during evaluation, a simple post-processing step was applied. Predicted touch segments shorter than the minimum labeled duration were removed before computing evaluation metrics. This avoids penalizing the model for detecting short events that cannot be matched to any ground-truth annotation by design.

5.2 Quantitative Evaluation Metrics¶

Model performance is evaluated using confusion-matrix-based metrics. Balanced accuracy is reported to measure performance across both classes, as it gives equal importance to touch and no-touch frames and is robust to class imbalance.

The primary metric is the F1 score, which captures the balance between precision and recall for the touch class. This is the most relevant measure for the task of touch detection.

Table 2 summarizes validation results for raw frame-level predictions and for predictions after post-processing.

Type TP FP FN TN Precision Recall Specificity Balanced Accuracy F1
Raw 38 982 19 342 8 640 82 400 0.6684 0.8186 0.8099 0.8142 0.7359
Filtered 37 867 16 980 9 755 84 762 0.6904 0.7952 0.8331 0.8141 0.7391

Table 2: Validation performance for raw predictions and predictions after post-processing.

Post-processing slightly reduces recall but improves precision and specificity. This leads to a small improvement in F1 score, indicating better alignment with the annotation policy. Overall performance remains stable, as reflected by similar balanced accuracy values.

5.3 Qualitative Evaluation and Error Analysis¶

While quantitative metrics provide an overall summary, they do not fully describe model behavior. For this reason, predictions were also inspected visually in the time domain.

For each validation video, ground-truth labels were plotted together with temporally filtered model predictions. In many cases, predicted touch intervals align well with annotated labels. In other sequences, the alignment is weaker but still reasonable.

A common pattern among false positives was identified. The model often predicts touch when the infant’s hand is in contact with the crib or surrounding structures. These events are not labeled as touch in the dataset and are therefore counted as false positives. An example of this behavior can be seen in the RT_st-2_s_s04_20w-1d_supine video (Figure 6).

This behavior suggests that the model captures the general concept of physical contact, even when the contact does not match the definition of a body touch. These errors appear to be systematic rather than random. They could be addressed in future work through post-processing. For example, pose information could be used to check whether the wrist is close to another body part. If the wrist is instead contacting the environment, the prediction could be filtered out.

Temporal plots of this type were generated for all infants in the validation set, allowing consistent qualitative inspection across subjects.

Temporal plot for infant AA

Temporal plot for infant CA

Temporal plot for infant TH

Temporal plot for infant RT

Figure 6: Temporal comparison of ground-truth labels, filtered predictions, and predicted touch probability for all validation infants (AA, CA, TH, RT). The dashed line shows the decision threshold.

5.4 Model Interpretation and Sanity Check (Grad-CAM)¶

To further validate model behavior, Gradient-weighted Class Activation Mapping (Grad-CAM) was used as an interpretation tool. The goal was not to improve performance, but to verify whether the model focuses on meaningful visual regions when predicting touch events.

The activation maps show that, for positive touch predictions, the model consistently attends to the wrist area. This matches domain expectations and provides an additional sanity check. It suggests that model decisions are driven by relevant visual cues rather than background artifacts.

Figure 7 shows an example Grad-CAM visualization overlaid on an input frame window.

Grad-CAM visualization highlighting wrist region
Figure 7: Grad-CAM visualization for a positive touch prediction, highlighting the wrist region.

6 Conclusion and Future Work¶

This work presented a baseline model for automatic infant touch detection. Quantitative results, together with qualitative inspection and Grad-CAM analysis, show that the proposed approach can successfully detect right-hand touch events. The task is therefore feasible using the available data and methods. The model should be seen as a proof of feasibility and a strong baseline, rather than a final production-ready system.

Several directions for future improvement were identified. A key issue is the presence of false positive predictions caused by contact with the crib or surrounding structures, as discussed in the qualitative evaluation. These errors could be reduced through dedicated post-processing, for example by using pose information to verify whether the wrist is close to another part of the infant’s body. In addition, fine-tuning the model on newly labeled or higher-quality data could further improve performance and robustness.

Binary touch detection is only the first step in the full analysis pipeline. For downstream behavioral studies, it is also necessary to identify which body region is being touched. This work therefore focuses on a simplified version of the problem.

There are two main ways to extend the model toward touch zone identification. One option is to train a multi-class model that directly predicts the touched body region. This approach would require more detailed annotations and would likely be a more challenging learning task. An alternative approach is to combine touch detection with pose estimation and infer the touched region from spatial relationships between the wrist and other body parts. Given the availability of reliable pose estimation models, this second approach appears more practical and better aligned with the current data.

From a data and evaluation perspective, an important next step is to conduct an inter-rater reliability study. Model performance could then be evaluated relative to human agreement, providing a more realistic upper bound on achievable accuracy and better context for model errors.

Based on the current results, fully automatic labeling of all recordings is likely unrealistic. However, the model already shows strong potential as an assistive annotation tool, where predicted touch events can be reviewed and corrected by a human annotator.

At the same time, the results suggest that for videos with high visual quality—such as good resolution, stable lighting, and clear visibility of the infant’s limbs—the model may be reliable enough for fully automatic labeling. For such recordings, an inter-rater reliability study comparing model outputs with human annotations would be particularly valuable to assess whether model performance approaches human agreement levels.

Notably, the best qualitative and quantitative performance was observed on videos with these favorable visual properties, even when the corresponding infant was not included in the training set. This suggests that video quality plays a major role in model performance and that the model generalizes well across subjects when the visual conditions are suitable.

Finally, once right-hand touch detection reaches sufficient reliability, the same approach can be extended to detect left-hand and leg touches using the same modeling pipeline.

7 Appendix A: Source Code¶

The source code used in this project is stored in a private Git repository. The repository is kept private as a precaution to prevent accidental disclosure of sensitive infant-related data or derived artifacts that may appear during development and reporting.

The code can be shared upon request after prior agreement. Contact: navarlu2@fel.cvut.cz.

At the time of submission, the most up-to-date implementation corresponds to the gpu-train-3 branch.