Sensor Fusion Using Machine Learning for Robust Object Detection in Adverse Weather Conditions for Self-Driving Cars

Abstract

Autonomous vehicles must navigate safely through a spectrum of environmental conditions—from clear skies to dense fog, torrential rain, and blinding snow. This thesis investigates a holistic sensor‑fusion framework that leverages complementary strengths of camera, LiDAR, and radar modalities to achieve robust 3D object detection under adverse weather. We begin by characterizing the failure modes of each sensor: vision systems suffer from contrast loss and occlusion in precipitation; LiDAR range returns scatter when mist or raindrops intrude; and radar, while inherently resilient to particulates, provides coarser spatial resolution. Building on both classical probabilistic fusion and cutting‑edge deep‑learning paradigms, we propose a multi‑level fusion network that integrates raw data, mid‑level features, and high‑level detection outputs. Our architecture employs modality‑specific backbones—ResNet for images, VoxelNet for point clouds, and range‑Doppler CNNs for radar scans—merged via attention‑driven feature fusion. To counteract training bias toward clear weather, we curate and augment a diverse training corpus drawn from nuScenes, Waymo Open Dataset, Oxford RobotCar (including its radar extension), A*3D, and synthetically fogged/raindrop‑enhanced KITTI sequences. Extensive experiments demonstrate that our fusion model retains over 80% of clear‑weather detection performance in heavy fog and rain—yielding a mean Average Precision (mAP) increase of 25–40% compared to camera‑only or LiDAR‑only baselines. Ablation studies quantify the incremental gains of each sensor combination, revealing that LiDAR+radar fusion counters extreme particulate interference, while camera+LiDAR excels at fine-grained classification. Against state‑of‑the‑art fusion methods, our approach achieves new benchmarks on adverse‑weather subsets, reducing missed detections by up to 30%. By systematically dissecting sensor behaviors, innovating data augmentation for inclement conditions, and designing a flexible fusion backbone, this work delivers a practical path toward truly all‑weather autonomous perception. The proposed framework not only elevates safety margins in real‑world deployment but also establishes a modular template for future multi‑modal extensions. Finally, we discuss the scalability and modularity of our fusion framework, emphasizing its extensibility to incorporate emerging sensor modalities such as thermal imaging and 4D radar. By providing a flexible backbone architecture, our approach facilitates rapid integration of future sensor technologies and fusion algorithms, paving the way toward truly all-weather autonomous perception systems. This work thus not only advances state-of-the-art sensor fusion methodologies but also lays practical groundwork for deploying safer, more reliable self-driving vehicles capable of navigating the full spectrum of environmental challenges.