The technology, developed by Fujitsu Laboratories, utilises deep learning to recognise the positions and connections of adjacent joints in complex movements or behaviour in which multiple joints move in tandem. This makes it possible to achieve greater accuracy in recognising, for example, when a person performs a complex task like removing objects from a box.

Fujitsu says the technology has achieved the world's highest accuracy in behaviour recognition, with significant gains over the results achieved using conventional technologies which don't make use of information on neighbouring joints.

Fujitsu says the technology can be used to perform checks of manufacturing procedures or unsafe behaviour in public spaces. Fujitsu scientists have presented the details of this technology at the 25th International Conference on Pattern Recognition (ICPR 2020), which is being held online until Friday 15 January.

“In recent years, advances in AI technology have made it possible to recognise human behaviour from video images using deep learning,” said Fujitsu Lab researchers Yuya Obinata Takuma Yamamoto, who presented a paper on the technology at the conference.

“In general, human behaviour recognition utilising AI relies on temporal changes in the position of each of the skeletal joints, including in the hands, elbows, and shoulders, as identifying features, which are then linked to simple movement patterns such as standing or sitting.

“With this new time series behaviour-recognition technology we have developed, we have successfully realised highly-accurate image recognition using a deep learning model that can operate with high-accuracy even for complex behaviours in which multiple joints change in conjunction with each other.”

Complex movements like unpacking involve hand, elbow, and shoulder joints moving in tandem with the arm bending and stretching. Fujitsu has now developed a new AI model for such processes, using a graph consisting of edges connecting adjacent joints based on the structure of the human body with the joint position as a node, or vertex.

By training this model in advance using the time series data of joints, the connection strength (weight) with neighbouring joints can be optimised, and effective connection relationships for behaviour recognition can be acquired.

“With conventional technologies, it was necessary to accurately grasp the individual characteristics of each joint,” said the researchers. “With an AI model that has already been trained, the combined features of the adjacent joints that are linked can be extracted, making it possible to achieve highly-accurate recognition for complex movements.”

The technology was evaluated against the world standard benchmark in the field of behaviour recognition using skeleton data, and in the case of simple behaviours such as standing and sitting in the open data set, the accuracy rate was maintained at the same level as that of conventional technology that does not use information on neighbouring joints.

But in the case of complex behaviours like a person unpacking a box or throwing an object, the accuracy rate improved greatly, to achieve an overall improvement of more than 7% over the conventional alternative to reach the world's highest recognition accuracy.

By adding the newly developed AI model for recognising complex behaviours obtained with this technology to the 100 basic behaviours already accommodated by Fujitsu's behavioural analysis technology Actlyzer, Fujitsu says it will become possible rapidly deploy new, highly-accurate recognition models.