Blog post 4: The First Fruits of Labour

With demo day fast approaching on the 13th of June, the team has once again been hard at work this week. This hard work certainly paid off when we observed encouraging results through a working model with still room for improvement.

This week, the relevant Machine Learning tools and Framework i.e TensorFlow, Keras, OpenCV, etc were set up on the Jetson Nano. One of the main hardware limitations we faced was the lack of RAM on the Jetson Nano. Therefore, we reduced the original C3D+LSTM Model to obtain a miniature C3D+LSTM model with 10 times less parameters which improved inference speed from 0.8 Inferences per second to 9.0 Inferences per second. This miniature model can now also fit within the 2.5GB (about 1.5GB of 4GB RAM available is used by the kernel of the CPU) of GPU RAM available on the Jetson Nano.

We also started on the end to end system and deployment of the machine learning model on the Jetson Nano. The system takes in a sequence of 30 frames from a camera input sampled at a rate of 15 FPS. Each sequence is then passed into our miniature C3D+LSTM Model where a prediction is made at rate of 9 Inferences per second.  For the C3D+LSTM Model to make an accurate prediction on what hand sign was captured, the system requires at least 15 frames of the 30 frame long LSTM to be filled by frames of the action to dominate the LSTM units. Hence, a standard latency of the system is about 1 second given that the camera is sampling frames at 15 FPS. A working model that demonstrates the capabilities of the system was also implemented. This can be seen in the short video clip below, with the model being able to correctly identify the 5 Makaton signs. This working model gives us confidence that we are certainly heading in the right direction. The aim is now to further fine tune the model to make it as robust and as accurate as possible.

This week, we also recorded more training videos from helpful volunteers around college- a total of 126 people to be exact. The training data has become more diverse, reaching a diversity of around 50% by this week. When training the model, however, we did not use all the data, to ensure that the model does not overfit on the 3 group members which consist of more than 50% of the dataset. Furthermore, to make the validation accuracy more representative of real-world conditions, we only used volunteers (i.e. not group members) in our validation set. In the coming week, we will continue to collect more training videos, as well as evaluate how the diversity of training data affects the accuracy of prediction.

In order to ensure that the Jetson’s CPU is not maxed out, we are exploring the idea of connecting the sensors and buttons to the Raspberry Pi instead. Communication between the Pi and the Jetson would then be done through the GPIO pins on both devices. This week, we managed to successfully get the buttons to work on the Pi and also to set up and receive data from the ultrasonic sensor. The next step is to incorporate all these peripherals together with the Pi and the Jetson, and more specifically, to start implementing the Watson text to speech API on the buttons such that a pre set phrase is said when a button is pressed.

In order to power up the Jetson Nano, an estimated minimum of 20W (5V/ 4A) should be supplied. The power bank that we currently have would simply not be strong enough. Furthermore, a USB to barrel jack connector was used to connect the power bank to the Jetson as seen in the picture below. This could be another reason why the Jetson was not receiving enough power as a result of the power limitation from a USB port.

USB to Barrel Jack connection

Therefore, we have made an order for a more powerful power bank capable of charging laptops. It also has the added benefit of a DC output jack which can be fed straight into the barrel jack of the Jetson. This eliminates the need for the Jetson to be powered through a USB port.

The next main challenge is to fit and incorporate all this technology into Waldo, the toy elephant. This week, we began the process of trying to fit all our sensors and devices into Waldo. This process started with ‘operating’ on Waldo to gain a greater understanding of the amount of limited space we were going to be dealing with. Here’s a group photo of team with Waldo, shortly before the ‘operation’ took place.

Stay tuned for an improved and technologically advanced version of Waldo in the week to come!