AI Learns By Watching Videos

Mark Hamilton, an MIT Ph.D. student, has developed DenseAV, an algorithm that learns language solely by associating audio and video signals. Inspired by scenes from “March of the Penguins,” Hamilton sought to create a system able to understanding human language “from scratch.” DenseAV learns language by predicting audio from video and vice versa, identifying objects based on auditory cues. Through training, the algorithm identifies relevant pixels in video streams corresponding to spoken words.

Interestingly, it distinguishes between words and sounds, exhibiting a nuanced understanding of language. This self-supervised approach, detailed on this research, holds promise for various applications, including instructional video evaluation and decoding non-written languages like dolphin communication. Read more.

Apple’s WWDC 2024 event dropped a dozen major announcements, with AI taking center stage.

First, Apple introduced its recent AI system, Apple Intelligence, for iPhone, iPad, and Mac. This offering leverages generative models with user data to boost writing, image creation, and Siri functionality. On-device processing prioritizes user privacy for basic AI tasks, with dedicated servers handling complex queries.

Siri received a context-aware upgrade (currently U.S. English beta), allowing it to fetch flight details or interact with app content and leveraging LLMs for improved understanding. Siri will integrate with OpenAI’s ChatGPT, enabling more dynamic query handling. Apple also rolled out AI features across Mail, Messages, Photos, and other apps, including email summarization, custom emoji creation, and AI-powered image cleanup.

iOS 18 brings customization galore with a revamped Control Center, app locking, RCS messaging, and even satellite SMS. Apple TV+ gains “inSight” for media identification, while a brand new Passwords app manages logins.

iPadOS 18 welcomes a long-awaited Calculator app with Apple Pencil support, while macOS 15 allows iPhone mirroring for enhanced multitasking. The health-focused watchOS 11 personalizes widgets and adds recent tracking features. VisionOS 2 upgrades the Vision Pro with spatial photos, an ultrawide Mac display, gesture controls, and travel mode.

And last but not least, the subsequent AirPods Pro update enables you to nod or shake your head to interact with Siri, while also improving voice quality with Voice Isolation.

Apple’s “Apple Intelligence” incorporates a brand new “Private Cloud Compute” feature to safeguard user data processed on cloud servers. Craig Federighi, SVP for Software Engineering emphasized on-device processing for enhanced privacy, leveraging A17+ and M-series chips for executing generative AI models locally.

In instances requiring cloud processing, Apple’s dedicated servers, powered by proprietary silicon, employ Swift’s intrinsic security tools to make sure data integrity. Federighi underscored that only pertinent data is transmitted, with no storage or accessibility to Apple, prioritizing user privacy.

Enhancing transparency, Apple plans to publicly release the server code for independent validation. While specifics were scant, the keynote underscored Apple’s commitment to privacy and security because it ventures into generative AI. Independent scrutiny of the server code in the longer term will validate these assurances.