As part of the bigger interdisciplinary project called Human-in-the-loop Sensing and Control for Commercial Building Energy Efficiency and Occupant Comfort (DOE#: DE-EE0007682), we develop a prototype called FORK (Fine grained Occupancy estimatoR using Kinect on ARM Embedded Platform). The goal is to explore the potential of using depth sensors to detect, identify, estimate, and track occupants in buildings on a cheaper and low power ARM processor in real-time. Unlike RGB approaches, using depth images makes the system much less privacy invasive.This project involves professors and students from different departments at Carnegie Mellon University, Stony Brook University, and Bosch Research and Technology Center (RTC) as a private industry collaborator.
I was in charge of deploying the sensing platform in different locations on CMU campus and at Bosch RTC’s offices. The sensing platform consists in an Odroid-XU4 running the FORK algorithm developed by Sirajum Munir, Bosch RTC, and a Microsoft Kinect V2 as the depth sensor. We also collected ground truth data in the form of RGB-D images in 4 different locations including classrooms, conference rooms, and student lounges. The data collection lasted 1 week in each location adding up to 16 TB of Depth and RGB images. We designed a two-step data annotation process in order to facilitate the cumbersome process. The data was stored in ~1,200 folders of 10,000 images each. We first manually detected the set of folders that contained people entering/exiting the room by displaying all folders at 5 frames per second. This resulted in a smaller set of folders that contained human activity. The second step involved annotating the range of frames where a person is visible, as well as whether they were entering or exiting the room. Although the process was tedious and time-consuming, we avoided using automation tools for people identification, since they would’ve compromised the authenticity of a ground truth dataset. Aftwards, we compared the performace of the current version of the algorithm on the annotated data and were able to achieve a TPR ratio above 95%. We presented the results as a Demo paper at BuildSys 2017 in Delft, Netherlands.
As of now, we have 10 working deployments operating on CMU campus and we are also testing other depth cameras such as Intel’s RealSense products, as well as deep learning approaches for human detection.