To combat the shortcuts and risk-taking, Lorenzo is working on a tool for the San Francisco–based company DroneDeploy, which sells software that creates each day digital models of work progress from videos and pictures, known within the trade as “reality capture.” The tool, called Safety AI, analyzes every day’s reality capture imagery and flags conditions that violate Occupational Safety and Health Administration (OSHA) rules, with what he claims is 95% accuracy.
That signifies that for any safety risk the software flags, there may be 95% certainty that the flag is accurate and pertains to a selected OSHA regulation. Launched in October 2024, it’s now being deployed on lots of of construction sites within the US, Lorenzo says, and versions specific to the constructing regulations in countries including Canada, the UK, South Korea, and Australia have also been deployed.
Safety AI is certainly one of multiple AI construction safety tools which have emerged in recent times, from Silicon Valley to Hong Kong to Jerusalem. Lots of these depend on teams of human “clickers,” often in low-wage countries, to manually draw bounding boxes around images of key objects like ladders, as a way to label large volumes of information to coach an algorithm.
Lorenzo says Safety AI is the primary one to make use of generative AI to flag safety violations, which implies an algorithm that may do greater than recognize objects akin to ladders or hard hats. The software can “reason” about what is occurring in a picture of a site and draw a conclusion about whether there may be an OSHA violation. This can be a more advanced form of research than the item detection that’s the present industry standard, Lorenzo claims. But because the 95% success rate suggests, Safety AI shouldn’t be a flawless and all-knowing intelligence. It requires an experienced safety inspector as an overseer.
A visible language model in the actual world
Robots and AI are inclined to thrive in controlled, largely static environments, like factory floors or shipping terminals. But construction sites are, by definition, changing somewhat bit each day.
Lorenzo thinks he’s built a greater method to monitor sites, using a sort of generative AI called a visible language model, or VLM. A VLM is an LLM with a vision encoder, allowing it to “see” images of the world and analyze what is occurring within the scene.
Using years of reality capture imagery gathered from customers, with their explicit permission, Lorenzo’s team has assembled what he calls a “golden data set” encompassing tens of 1000’s of images of OSHA violations. Having fastidiously stockpiled this specific data for years, he shouldn’t be apprehensive that even a billion-dollar tech giant will have the opportunity to “copy and crush” him.
To assist train the model, Lorenzo has a smaller team of construction safety pros ask strategic questions of the AI. The trainers input test scenes from the golden data set to the VLM and ask questions that guide the model through the means of breaking down the scene and analyzing it step-by-step the best way an experienced human would. If the VLM doesn’t generate the right response—for instance, it misses a violation or registers a false positive—the human trainers return and tweak the prompts or inputs. Lorenzo says that slightly than simply learning to acknowledge objects, the VLM is taught “how you can think in a certain way,” which implies it may well draw subtle conclusions about what is occurring in a picture.
