Road Brain

2018-10

1. Background

This summer I joined Research Team for Smart Transport Emerging computing and Perception for an engineering project internship, instructed by Professor Yuchuan Du. The main purpose of this project is to build a “digital twin” for the Inovation Port of Shanghai Automobile City, and provide a monitor platform and postioning service for the driveless vehicle tested in the park.

Road Brain Demo

2. Pipeline

The whole pipeline of this project is camera-based. I started with building the 3D reconstruction map of the park and using computer vision method to detect and track vehicles.

I splitted the park into several intersections, each intersection contains 1-2 monocular survilience cameras. We collected large amount of geo-tagged images captured by a LadyBug camera for each intersection, and then used Structure-from-Motion method to build 3D reconstruction map.

Based on video streams collected by surveillance cameras, I implemented CNN based detection and feature-based tracking algorithm to detect and track car’s 3d pose. Afterward, a simple calibration algorithm is implemented for locating the cars in the 3D map.

3D reconstrcution map of an intersection. Shanghai International Automobile City

I tested some detection models, such as YOLO V3 and SSD, and finally used a keypoints-based GNN (Graph neural network) model to detect the car keypoints, then projected the vehicle keypoints into the reconstrcution map by camera matrix calibrated before.

Vehicle Keypoints Detection & Localization

The whole pipeline can run in real-time. We can analyze the entire trajectory of a car when it driving across multiple cameras. Since the whole system is camera-based, it can also work well in the underground parking lot, where the GPS service may not be available. The tracking pipeline is the same, but the re-identification model should be enhanced since the occlusion of the car. We use plate number combined with feature-based re-identification model (Deep SORT) to make sure the vehicle could be tracked across different cameras.

Underground Calibration

Underground Tracking and ReID Track

3. Future Work

Multi-classes Tracking: Now we only focus on vehicle tracking and localization, this pipeline can also be extented to any targets on road (bicycle, pedestrian) and reconstruct them in a digital world.
Data Fusion: We are also testing millimeter-wave radar to detect speed and direction of target, the video camera data and radar data can be merged to imporve the accuracy and speed.