As a part of the Tech4PositiveFutures initiative, Capgemini’s Applied Innovation Exchange London have partnered with Pollenize, a UK-based Community Interest Company (CIC), to explore the opportunity of using machine learning and artificial intelligence to get live plant health data direct from pollinators resembling bees.

Bees play a vital role as pollinators, facilitating plant reproduction, maintaining biodiversity, and supporting food production in ecosystems.

The bee waggle dance is a extraordinary communication behavior utilized by honeybees to share information concerning the direction and distance of useful resources resembling food or recent hive locations to their fellow hive mates. This waggle (or dance) involves a series of high frequency movements performed in an 8-figure pattern. Scientists call this a waggle run.

These runs may be decoded to know what information is communicated. Scientists have discovered that the waggle angle, waggle length and sound emitted all indicate what the bee is attempting to pass on.

1. The angle at which the bee is waggling indicates where the pollinating flower is with respect to the hive — which way should I am going.
2. The length of waggle indicates how far the flower is from the hive —how far should I fly.
3. The sound emitted from the bee indicates the profitability of the food — how tasty the pollen is.

Surprised? Amazed? Interested? Yeah, me too.

At first, we separated our task into two principal areas: detecting of the waggle and decoding it. Given our timeframe and resources available, we focused mainly on constructing a sturdy detection piece, paving the way in which for future decoding methods. Over the course of the study we explore three different approaches, all summarized below. Read on to see what all the excitement is about! 😅

1. Recording the live hive data

Our partner Pollenize recorded several hours of live hive footage. By live hive, I’m referring to the actual fact the hives are placed outside and are usually not in-built a lab setting; bees have direct contact with the outside.

2. Spatio-Temporal Clustering

The initial dataset provided by Pollenize didn’t contain waggle annotations. The dearth of annotations led our team to give attention to unsupervised learning frameworks where inputs don’t require goal values. We used elements from a model proposed within the Automatic Evaluation of Bees’ Waggle Dance research paper and added additional modules of our own. We referred to this model as spatio-temporal clustering because it detects bee waggle events over time and space.

The model breaks down each video frame by frame, to detect areas of high vibration or potential waggles. To generate these potential waggles, the OpenCV python package is used to remove the background of each frame. From there, we compute the pixel-wise difference of frames; areas which have essentially the most difference correspond to areas of high vibrations. We then use OpenCV contour evaluation to detect the biggest contours of every difference frame and save them as potential waggle events. At this point, the set of potential waggles is comparatively noisy. Noise within the dataset could generate a big pixel difference in areas where bees aren’t waggling. To beat this, we use spatio-temporal clustering. In layman’s terms, we are attempting to attract relationships between the placement and time of various potential waggle events. If separate potential waggle events occur in an analogous location and timestamp as others, they can be deemed to be a part of a cluster; each event is assumed to be performed by the identical bee. Bees with no neighboring waggle events are regarded as outliers and are remove from the set of waggle events. This 3D spatio-temporal clustering was done using a built-in DBSCAN model.

The figure below illustrates the detection of potential waggle events by removing background and computing the pixel-wise difference.

From now, we have now the set of waggle events: the x and y coordinates and the frames at which they occurred. To go one step further, we decided to try to decode these predicted waggles by generating the direction of the communicated pollinating flower. Assuming that the waggle runs were performed along an axis (see above), we fit a linear function to each cluster; the slope of this function gave us an estimate of where the flower was with respect to the hive.

Despite performing well on lab generated hive data provide within the paper, this approach performed poorly on noisier live hive data. Hence, we decided to give attention to obtaining labelled data to have the option to experiment with more robust deep learning approaches.

3. Motion Recognition

Naturally, we desired to test out possibilities using a clearer video feed. Using labelled data provided by the University of California at San Diego (UCSD) we built an motion recognition model for bee waggle.

Motion recognition identifies and classifies actions or activities in a video without localizing temporal boundaries (on a video level). It extracts spatio-temporal features to represent actions and predicts the general motion category.

We used a pretrained Temporal Segment Network (TSN) provided by the MMAction2 open-source video modeling. The TSN was pretrained on Kinetics-400, a dataset of 400 human motion videos. We then finetuned it on our recent labelled bee waggle dataset, containing 700 video clips. That is seen as transfer learning.

Transfer learning is a machine learning technique that leverages knowledge gained from pre-training on one task/dataset to enhance the performance on a unique but related task. It allows the transfer of learned representations, features, or parameters from a source domain to a goal domain.

After training for 50 epochs our model returned a training set accuracy of 92% and a validation set accuracy of 89%. The video below illustrates the model in motion.

Despite showing fairly promising results, the model accommodates bias. The dataset was using a sliding window, which inevitably signifies that information form training set has leaked into the validation set. The unique dataset contained 142 seconds of video with 32 waggle events. Given, this somewhat small dataset, we decided that we wanted generate as much data as possible. In a future scenario where more labelled data is obtainable, the difficulty can be resolved.

Our final approach to solving this bee waggle detection task was to construct out an motion detection model.

Motion detection localizes and temporally segments actions in a video (frame by frame). It identifies motion categories, in addition to the precise temporal boundaries or segments where actions occur.

The dataset provided by UCSD contained pixel annotations firstly and end frames of every waggle. Motion Detection models require more information concerning the location and the time at which the waggle is occuring. To construct out this elevated dataset, we implemented a CSRT tracker from the OpenCV python library. For each waggle we perform the next:

1. Constructed a bounding box across the waggling bee firstly frame
2. Initialize a CSRT tracker using this primary bounding box
3. Update the tracker and save the expected bounding box for each frame until the tip frame is attained

The video below illustrates a sample of the dataset we created.

At this point, we reached the tip of our project. If we had more time, we might have considered using an Actor-Centric Relation Network pretrained on the AVA-Kinetics dataset, provided within the MMAction2 toolkit.

The detecting of a bee waggle is a fancy task in itself, without even considering the unique decoding project. Acquiring top quality video feeds requires a high tech camera setup. Creating annotations requires extensive beekeeper knowledge (and patience 😉 ). We provided a novel evaluation of how unsupervised and supervised machine learning models may very well be used to detect the bee waggle dance. Our findings display potential in considering machine learning as a viable contender for addressing this task. Nonetheless, data quality must be the priority to make sure successful future research. We recommend specializing in creating top quality footage and annotations prior to investing time in resources into constructing an AI solution.

—

Access to code via Github

We’d like to offer credit to Heather Broccard-Bell from the Nieh Lab at UCSD for the labelled dataset we used throughout this research.