A research team investigating assistive computer vision frameworks has engineered a hybrid deep learning model combining the YOLOv8 object detection architecture with InceptionV3 feature extraction layers, demonstrating high precision and rapid inference speeds for real-time sign language recognition. Reported by Mehedi Hasan, Bikash Kumar Paul, Nazrul Islam, and Rafid Mostafiz in BMC Artificial Intelligence, the study addresses persistent computational bottlenecks that have historically challenged automated gesture translation systems on standard video interfaces. Digital platforms such as Zoom and Google Meet increasingly serve as core interaction spaces for deaf and hard-of-hearing communities, yet conventional computer vision tools have frequently struggled to balance the spatial accuracy required to differentiate intricate hand geometries with the execution speed needed to support live video feeds without perceptual latency.

To overcome these challenges, the investigators investigated whether restructuring the foundational feature extraction layers of a modern real-time detector could improve localization precision without imposing excessive computational overhead. Traditional gesture recognition systems often depended on handcrafted spatial features or cumbersome wearable physical sensors that proved impractical and expensive for broad public deployment. While subsequent approaches turned to deep convolutional neural networks, recurrent neural networks, and long short-term memory architectures to capture spatial hierarchies and temporal movement patterns, earlier region-based methods like Faster R-CNN maintained latency levels unsuitable for immediate, frame-by-frame video parsing. Earlier iterations of the You Only Look Once family, including YOLOv3 and YOLOv4, improved throughput but encountered difficulties when resolving subtle finger articulations against shifting background textures.

The experimental methodology centered on constructing a specialized, fine-grained sign language dataset before executing systematic structural modifications to the detection network. The research team generated a self-collected corpus comprising 520 high-resolution smartphone images across 34 distinct gesture classes, featuring both left- and right-hand executions recorded from multiple viewing angles under auto-exposure conditions. The captured vocabulary included everyday conversational and informational signs such as Love, Okay, Power, No, Washroom, and I Love You. Because raw image sets gathered in natural settings present varying frequencies per category, with sample counts ranging from 10 to 48 images per sign, the authors implemented targeted data balancing strategies, including focal loss adjustments, oversampling, and systematic augmentation routines to prevent the network from developing biased predictive preferences toward dominant classes.

Data pre-processing and precise spatial bounding box annotation were executed via the Roboflow platform to ensure rigorous ground-truth definitions across all training instances. The 520 raw images were partitioned into an experimental distribution comprising 70 percent for training (accounting for 364 images in baseline splits, with expanded configurations utilizing 463 images), 10 percent for validation (52 images), and 20 percent for testing (104 images). The input imagery, originally captured at a median resolution of 3024 by 4032 pixels, underwent standardized auto-orientation and dimensional transformation to 640 by 640 pixels to conform to the detector pipeline. To fortify the detector against real-world optical noise without generating synthetic labeling errors, the authors applied systematic rotation, dimensional scaling, horizontal flipping, brightness adjustments, and artificial noise injection, thereby presenting the training loop with comprehensive variations in hand orientation, subject-to-camera distance, and ambient illumination.

At the architectural level, the proposed model fundamentally alters the standard CSPDarknet53 backbone of YOLOv8 by substituting its initial convolutional layer with an InceptionV3 convolutional block. Within this modified backbone, input samples are handled in parallel with an initial resizing to 299 by 299 pixels to ensure compatibility with pre-trained Inception convolutional weights before the extracted feature maps pass into successive C2F modules and convolutional stages. The backbone concludes with a Spatial Pyramid Pooling-Fast module that generates fixed-length spatial representations across varying target dimensions without incurring informational loss or requiring additional spatial distortion.

Bridging the modified backbone and the network neck, the authors introduced an Adaptive Feature Fusion Module designed to dynamically process and refine multi-scale feature hierarchies before routing them to the prediction stages. Within this module, a Multi-Scale Dynamic Fusion mechanism computes weighted sums of low-, medium-, and high-resolution feature maps using learned parameter coefficients, allowing the model to adaptively prioritize the most relevant spatial representations. Concurrently, a Hierarchical Attention Mechanism applies sequential spatial and channel-wise attention masks to selectively amplify salient gesture contours while suppressing irrelevant background textures. The resulting fused representations are directed into an enhanced neck structure incorporating a Scale Selective Up-sampling layer and a Cross-Stage Partial Path Aggregation Network, feeding an Enhanced Detection Head specialized in predicting small, medium, and large gesture targets through multi-scale feature pyramids.

Model training was conducted within a Google Colab GPU environment utilizing an NVIDIA Tesla T4 graphics processing unit running CUDA 12.2 and driver version 535.104.05, completing a 50-epoch training schedule on the pre-trained YOLOv8s baseline in approximately 0.176 hours. During iterative backpropagation, the overall loss objective balanced three distinct mathematical components: a localization loss quantifying coordinate offsets between predicted and ground-truth bounding box centers and dimensions, a confidence loss penalizing positive and negative objectness predictions using weighted beta factors, and a multi-class classification loss evaluating softmax probability distributions against true gesture labels.

Upon formal evaluation, the customized YOLOv8-InceptionV3 system demonstrated remarkable detection metrics on the custom sign language dataset. The model attained a mean Average Precision at a 50 percent Intersection over Union threshold of 99.5 percent, a strict multi-threshold mAP50-95 of 86.2 percent, an overall precision of 93.3 percent, a recall of 97.6 percent, and an F1-score of 95.40 percent. Crucially for real-world deployment viability, the average inference latency was recorded at 4.6 milliseconds per frame, comfortably outperforming standard real-time video thresholds.

Comparative benchmarking conducted against both standalone classification networks and alternative object detection frameworks highlighted the distinct advantages of the hybrid architecture. When evaluated as pure image classifiers on the custom data, InceptionV3 led legacy baselines with 92.50 percent accuracy and a 94.10 percent F1-score, surpassing VGG-19, ResNet-50, and ResNet-101. When evaluated against full object detection systems, the customized model surpassed standard YOLOv5, YOLOv7, and unmodified YOLOv8 in localization precision across stringent overlap thresholds. While standard YOLOv8 registered marginally higher precision at 93.7 percent and an F1-score of 95.74 percent, the hybrid system achieved an 86.2 percent mAP50-95 compared to YOLOv8’s 84.6 percent, confirming that the parallel Inception filters and multi-scale attention mechanisms provide superior spatial delineation when localizing complex or partially occluding hand signs.

To evaluate broader generalization capabilities beyond the controlled 34-class dataset, the researchers assessed the framework on external benchmark corpora representing diverse sign languages, including American Sign Language, Indian Sign Language, and Bengali Sign Language. Across these diverse benchmark sets, which encompass distinct signing styles, differing signer hand physiologies, and varied environmental backgrounds, the customized architecture consistently maintained a reported recognition accuracy of 99.5 percent, outperforming comparative evaluations involving Faster R-CNN and VGG16. The authors observed that the model effectively generalized across the broad arm trajectories typical of American Sign Language, the nuanced finger configurations central to Indian Sign Language, and the culturally unique regional hand shapes found in Bengali Sign Language datasets.

Despite these robust performance indicators, the authors documented specific operational boundaries and error profiles that define targets for subsequent engineering efforts. Inspection of the confusion matrix revealed localized misclassifications, particularly between structurally similar signs such as Okay and Need Help, which share overlapping finger configurations and hand elevations. Furthermore, analysis across stringent IoU thresholds demonstrated reduced detection precision for intricate phrases such as Give Me Time, Thirsty, and And Say. The researchers also noted that while the curated dataset provided rigorous annotations, its overall volume remains relatively compact at 520 base images, creating potential overfitting vulnerabilities that necessitate further cross-validation on expansive unconstrained video sets.

Subject of Research: Real-time sign language gesture detection using a customized YOLOv8 and InceptionV3 neural network

Article Title: Advancing real-time sign language detection for deaf and hearing-impaired communities: a customized YOLOv8 approach with tailored annotations in computer vision

Article References: Hasan, M., Paul, B. K., Islam, N., & Mostafiz, R. (2025). Advancing real-time sign language detection for deaf and hearing-impaired communities: a customized YOLOv8 approach with tailored annotations in computer vision. BMC Artificial Intelligence, 1(1), Article 11. https://doi.org/10.1186/s44398-025-00010-9

Image Credits: AI Generated

DOI: 10.1186/s44398-025-00010-9

Keywords: Computer Vision, Sign Language Detection, YOLOv8, InceptionV3, Deep Learning, Assistive Technology, Object Detection, Roboflow, Advancing, real-time, sign, language