Introduction to nuScenes: Dataset for Autonomous Driving

Published 2020-05-25 14:42
Tagged on nuscenes autonomous-driving

### 1. Introduction ------ According to **nuScenes** by [Aptiv](https://www.nuscenes.org) - > nuScenes is a public large-scale dataset for autonomous driving. It enables researchers to study challenging urban driving situations using the full sensor suite of a real self-driving car.  *Figure: LIDAR point cloud of an example scene in nuScenes dataset*  *Figure: Bounding box annotation of an example scene in nuScenes dataset. Source: [nuScenes](https://www.nuscenes.org/explore/scene-0011/0)* In this post, I am going to - - show how to install the nuScenes devkit, - download the mini dataset from nuScenes website, - define basic functionalities of the nuScenes devkit and data format, - explain the terms *scene*, *instance*, *sample*, *sample annotation* and *keyframe*, - look into different scenes and select a random instance to plot the *lidar* and *camera* measurements for a specific frame along with the targets annotated bounding box. ### 3. Devkit Installation and Dataset Download ------ I experimented on my *Ubuntu 18.04* machine with python 3.6.8 virtual environment. Other configurations might work also. - Install nuScenes devkit using pip command. ``` > pip install nuscenes-devkit ``` - Download the mini dataset from [here](https://www.nuscenes.org/download). You might open an account to be able to download the datasets. We are ready to explore nuScenes dataset now. But before doing that, we are going to define few terminologies of nuScenes dataset. ### 3. Terminologies ------ I will quote the definitions of the terms from nuScenes' official [tutorial](https://www.nuscenes.org/tutorial). One can also follow that tutorial for basic understanding. However, I am going to show things in my way. The dataset comprises of elemental building blocks that are the following: 1. `scene` - 20 second snippet of a car's journey. 2. `sample` - An annotated snapshot of a scene at a particular timestamp. 3. `sample_data` - Data collected from a particular sensor. 4. `sample_annotation` - An annotated instance of an object within our interest. 5. `instance` - Enumeration of all object instance we observed. 6. `category` - Taxonomy of object categories (e.g. vehicle, human). 7. `attribute` - Property of an instance that can change while the category remains the same. 8. `visibility` - Fraction of pixels visible in all the images collected from 6 different cameras.. 9. `sensor` - A specific sensor type. 10. `calibrated sensor` - Definition of a particular sensor as calibrated on a particular vehicle. 11. `ego_pose` - Ego vehicle poses at a particular timestamp. 12. `log` - Log information from which the data was extracted. 13. `map` - Map data that is stored as binary semantic masks from a top-down view. Here is an overview of how the dataset is arranged.  ### 4. Explore ------ Let's import nuScenes library and load the the mini dataset we have just downloaded. ``` %matplotlib inline from nuscenes.nuscenes import NuScenes nusc = NuScenes(version='v1.0-mini', dataroot='data/nuscenes/', verbose=True) ``` Output: <pre> ====== Loading NuScenes tables for version v1.0-mini... 23 category, 8 attribute, 4 visibility, 911 instance, 12 sensor, 120 calibrated_sensor, 31206 ego_pose, 8 log, 10 scene, 404 sample, 31206 sample_data, 18538 sample_annotation, 4 map, Done loading in 0.5 seconds. =.===== Reverse indexing ... Done reverse indexing in 0.1 seconds. =.===== </pre> We can see that this mini dataset consists of 10 scenes. Let's take the 4th scene and see what data is associated with a scene. ``` scene = nusc.scene[3] scene ``` Output: <pre> { 'token': 'bebf5f5b2a674631ab5c88fd1aa9e87a', 'log_token': '6f7fe59adf984e55a82571ab4f17e4e2', 'nbr_samples': 41, 'first_sample_token': '5991fad3280c4f84b331536c32001a04', 'last_sample_token': '35833ae5808e4ef186d1fdebac3d9cf6', 'name': 'scene-0655', 'description': 'Parking lot, parked cars, jaywalker, bendy bus, gardening vehicles' } </pre> Here `token` is one sort of unique identifier for retrieving details for a scene. Not just scene, a unique identifier is used for every kind of element in this dataset. As we already know that each scene has 20 seconds of a car's journey and in every second there are approximately two samples (it's like snapshot or video frame). That means there are approximately 40 samples in one single scene. `first_sample_token` is the token or identifier of the first sample. Let's take the first sample token and see what is inside a sample data. ``` sample = nusc.get('sample', scene['first_sample_token']) sample ``` Output: <pre> { 'token': '5991fad3280c4f84b331536c32001a04', 'timestamp': 1535385092150099, 'prev': '', 'next': '2578329fc3ae484bb23ef766808f4be5', 'scene_token': 'bebf5f5b2a674631ab5c88fd1aa9e87a', 'data': {'RADAR_FRONT': '9c0e927f845e412c93cb2846bb72f27f', 'RADAR_FRONT_LEFT': 'fbb8f507b73c435ab16bab9f509f2218', 'RADAR_FRONT_RIGHT': '2484789d50b645bbabacc8e735d8af13', 'RADAR_BACK_LEFT': '0fc32f1d2160443f9eedaa8edcfd05bf', 'RADAR_BACK_RIGHT': 'da0903784e9646c39125737bea931db6', 'LIDAR_TOP': '603a96c23db34acf8b5953d958df293d', 'CAM_FRONT': 'be95ce80ec624f81ba6737eb67241215', 'CAM_FRONT_RIGHT': 'de7f64fb4882443e99b21e51997756d7', 'CAM_BACK_RIGHT': '0cdb6b298b814b38bf9d7544f7f502a5', 'CAM_BACK_LEFT': '0d0a7c385ac84d64a4e8bd0545aff787', 'CAM_BACK': 'f256db4ccc1d41d9920e168184d2411f', 'CAM_FRONT_LEFT': '514d62f8cd534483a9d9509b26e87041'}, 'anns': ['8b979f63281c4955afe07d57a21aa0e1', 'be3536192b93448394597729df1cf52f', 'b6ded29415ae4ad2b76f0bf73fb674ce', '15bc2adc7b6b48bf86a1713fb124d488', 'f113b14fb3b94511b9c2534c2c39b127', '3816b7dfcb7d4dae85978b667a1cf4e8', 'e5fcecd0bd494406b62a2f8dd04c4fe3', '7dcab22e134c4b7e8752214d1753ae27', 'e6c3323d5ee34552b772a3225c408619', '25f5977dc8804754ae0760b98ca61871', '0fd3543039bb40c887e834ef6d21ef64', 'd24d107bd33540bda5934a87cdbb58aa', '1df5cfd5c87c44f690621e261e9755c7', '2455a5cb496647d2987f59febc73ec6f', 'aaf9c3cf65f2457299bdbf28d1f452c6', 'b7d029fc6387426c9848bac24a1223d4', '0ea15cdd8c6f43aca76381fe03628e00', '68a415bab9ca4fcab9c41d01b13343fd', 'f12e80b7479a4633bc00e5bda4bd6cba', 'bc73622a87a9424bbbea119251494369', 'fe596e0f70d74340a684e7b2a43a652f', '8d5d7cf4f0654fc7b63f28d2910e021d', '9e932ac2046b4babbf1f108255fcc4de', '6354c8160c8e4811a2ec878f50ce966e', 'a2191c6a0d074546bf6e423f5da1aae0', 'dbe2979694df4cd0a8d44dcc4d434930', '4a577db32137439581e414c65f278880', '5128acbbe4f54fa28ba184510fc624c8', 'b54f177b18de4d70a28e77fdbb89d9d8', 'f4fe6ff41049433f8d0fe5d1fe7325f4', 'ae72217b75ef4c24976aae2fd250e90c', 'beb5987591c64d699d0aaf9979be2bff', '7eeb173063184b208e87a61e4e515bb8', '2c24e9aae33b4ce0b3704ec21b709297'] } </pre> Since this sample is the first sample of the 4th scene of this dataset, it has no `prev ` (sample) token. It has `next` sample token to iterate through all the samples followed by it.. Naturally, the last sample of this scene should not contain `next` token. And all other samples have `prev` and `next` tokens. Other than `prev` and `next` samples' tokens, a sample contains tokens of sensor data i.e. radar, lidar and camera measurements. There is also a list of tokens of annotated data i.e. bounding box of different kinds of vehicles, persons etc. Let's take a look at 21th annotation of the sample data. ``` annotation = nusc.get('sample_annotation', sample['anns'][20]) annotation ``` Output: <pre> { 'token': 'fe596e0f70d74340a684e7b2a43a652f', 'sample_token': '5991fad3280c4f84b331536c32001a04', 'instance_token': 'd5aae9e790aa468aa42ad8b1baafe051', 'visibility_token': '4', 'attribute_tokens': ['cb5118da1ab342aa947717dc53544259'], 'translation': [1716.132, 870.368, 1.137], 'size': [1.993, 6.049, 2.0], 'rotation': [-0.31551654578626376, 0.0, 0.0, 0.9489200753146202], 'prev': '', 'next': 'a2ca591240484a449f17e191cb227046', 'num_lidar_pts': 2, 'num_radar_pts': 13, 'category_name': 'vehicle.truck' } </pre> The `category ` of this annotated object is `vehicle.truck`. I will talk about `instance` now. In nuScenes dataset, those objects which are being tracked in several consecutive frames are called `instance`. Let's take the instance token and see what's inside that. ``` instance = nusc.get('instance', annotation['instance_token']) instance ``` Output: <pre> { 'token': 'd5aae9e790aa468aa42ad8b1baafe051', 'category_token': '6021b5187b924d64be64a702e5570edf', 'nbr_annotations': 11, 'first_annotation_token': 'fe596e0f70d74340a684e7b2a43a652f', 'last_annotation_token': '57ad7889cfe24d28964f888a977ba061' } </pre> We have got `first_annotation_token` and `last_annotation_token` of an instance. We can loop through all the tokens from first to last annotation token of this instance. We will see that in other post. We are going to explore some of the sensor data today. Let's find out the very first sample data of this instance. ``` first_annotation = nusc.get('sample_annotation', instance['first_annotation_token']) first_sample = nusc.get('sample', first_annotation['sample_token']) ``` Let's get the lidar data and back camera data and render using nuScenes' in-built method. ``` first_lidar_data = first_sample['data']['LIDAR_TOP'] first_cam_back_data = first_sample['data']['CAM_BACK'] nusc.render_sample_data(first_lidar_data) nusc.render_sample_data(first_cam_back_data) ```   This will give you all the annotation in the sample. But if we want the data associated with the instance, then we can do the following- ``` nusc.render_instance(instance['token']) ```  ### 5. Conclusion ------ So far, we have learned what nuScenes dataset is, how to install devkit, download mini dataset and basic terminologies used in the dataset. We have also learned how the dataset is arranged, looked into what's inside a sample, annotation, sample_data etc. In the next post, we will generate an animation of an instance's journey.

---