Â
Closed source generative video models akin to Kling, Kaiber, Adobe Firefly and OpenAI’s Sora, aim to dam users from generating video material that the host firms don’t want to be related to, or to facilitate, because of ethical and/or legal concerns.
Although these guardrails use a mixture of human and automatic moderation and are effective for many users, determined individuals have formed communities on Reddit, Discord*, amongst other platforms, to search out ways of coercing the systems into generating NSFW and otherwise restricted content.
Source: Reddit
Besides this, the skilled and hobbyist security research communities also ceaselessly disclose vulnerabilities within the filters protecting LLMs and VLMs. One casual researcher discovered that communicating text-prompts via Morse Code or base-64 encoding (as an alternative of plain text) to ChatGPT would effectively bypass content filters that were lively at the moment.
The 2024 T2VSafetyBench project, led by the Chinese Academy of Sciences, offered a first-of-its-kind a benchmark designed to undertake safety-critical assessments of text-to-video models:
Source: https://arxiv.org/pdf/2407.05965
Typically, LLMs, that are the goal of such attacks, are also willing to assist in their very own downfall, not less than to some extent.
This brings us to a brand new collaborative research effort from Singapore and China, and what the authors claim to be the primary optimization-based jailbreak method for text-to-video models:
Source: https://arxiv.org/pdf/2505.06679
As an alternative of counting on trial and error, the brand new system rewrites ‘blocked’ prompts in a way that keeps their meaning intact while avoiding detection by the model’s safety filters. The rewritten prompts still result in videos that closely match the unique (and infrequently unsafe) intent.
The researchers tested this method on several major platforms, namely Pika, Luma, Kling, and Open-Sora, and located that it consistently outperformed earlier baselines for achievement in breaking the systems’ built-in safeguards, and so they assert:
The latest paper is titled , and comes from eight researchers across Nanyang Technological University (NTU Singapore), the University of Science and Technology of China, and Sun Yat-sen University at Guangzhou.
Method
The researchers’ method focuses on generating prompts that bypass safety filters, while preserving the meaning of the unique input. That is achieved by framing the duty as an , and using a big language model to iteratively refine each prompt until one of the best (i.e., the more than likely to bypass checks) is chosen.
The prompt rewriting process is framed as an optimization task with three objectives: first, the rewritten prompt must preserve the meaning of the unique input, measured using semantic similarity from a CLIP text encoder; second, the prompt must successfully bypass the model’s safety filter; and third, the video generated from the rewritten prompt must remain semantically near the unique prompt, with similarity assessed by comparing the CLIP embeddings of the input text and a caption of the generated video:
The captions used to judge video relevance are generated with the VideoLLaMA2 model, allowing the system to match the input prompt with the output video using CLIP embeddings.
Source: https://github.com/DAMO-NLP-SG/VideoLLaMA2
These comparisons are passed to a loss function that balances how closely the rewritten prompt matches the unique; whether it gets past the protection filter; and the way well the resulting video reflects the input, which together help guide the system toward prompts that satisfy all three goals.
To perform the optimization process, ChatGPT-4o was used as a prompt-generation agent. Given a prompt that was rejected by the protection filter, ChatGPT-4o was asked to rewrite it in a way that preserved its meaning, while sidestepping the particular terms or phrasing that caused it to be blocked.
The rewritten prompt was then scored, based on the aforementioned three criteria, and passed to the loss function, with values normalized on a scale from zero to 1 hundred.
The agent works iteratively: in each round, a brand new variant of the prompt is generated and evaluated, with the goal of improving on previous attempts by producing a version that scores higher across all three criteria.
Unsafe terms were filtered using a not-safe-for-work glossary adapted from the SneakyPrompt framework.
Source: https://arxiv.org/pdf/2305.12082
At each step, the agent was explicitly instructed to avoid these terms while preserving the prompt’s intent.
The iteration continued until a maximum variety of attempts was reached, or until the system determined that no further improvement was likely. The very best-scoring prompt from the method was then chosen and used to generate a video with the goal text-to-video model.
Mutation Detected
During testing, it became clear that prompts which successfully bypassed the filter weren’t at all times consistent, and that a rewritten prompt might produce the intended video once, but fail on a later attempt – either by being blocked, or by triggering a secure and unrelated output.
To deal with this, a strategy was introduced. As an alternative of counting on a single version of the rewritten prompt, the system generated several slight variations in each round.
These variants were crafted to preserve the identical meaning while changing the phrasing simply enough to explore different paths through the model’s filtering system. Each variation was scored using the identical criteria because the most important prompt: whether it bypassed the filter, and the way closely the resulting video matched the unique intent.
In spite of everything the variants were evaluated, their scores were averaged. The perfect-performing prompt (based on this combined rating) was chosen to proceed to the following round of rewriting. This approach helped the system decide on prompts that weren’t only effective once, but that remained effective across multiple uses.
Data and Tests
Constrained by compute costs, the researchers curated a subset of the T2VSafetyBench dataset with a view to test their method. The dataset of 700 prompts was created by randomly choosing fifty from each of the next fourteen categories: , , , , , , , , , , , , , and .
The frameworks tested were Pika 1.5; Luma 1.0; Kling 1.0; and Open-Sora. Because OpenAI’s Sora is a closed-source system without direct public API access, it couldn’t be tested directly. As an alternative, Open-Sora was used, since this open source initiative is meant to breed Sora’s functionality.
Open-Sora has no safety filters by default, so safety mechanisms were manually added for testing. Input prompts were screened using a CLIP-based classifier, while video outputs were evaluated with the NSFW_image_detection model, which relies on a fine-tuned Vision Transformer. One frame per second was sampled from each video and passed through the classifier to examine for flagged content.
Metrics
By way of metrics, (ASR) was used to measure the share of prompts that each bypassed the model’s safety filter resulted in a video containing restricted content, akin to pornography, violence, or other flagged material.
ASR was defined because the proportion of successful jailbreaks amongst all tested prompts, with safety determined through a mixture of GPT-4o and human evaluations, following the protocol set by the T2VSafetyBench framework.
The second metric was , capturing how closely the generated videos reflect the meaning of the unique prompts. Captions were produced using a CLIP text encoder and in comparison with the input prompts using cosine similarity.
If a prompt was blocked by the input filter, or if the model did not generate a legitimate video, the output was treated as a completely black video for the aim of evaluation. Average similarity across all prompts was then used to quantify alignment between the input and the output.
Among the many models tested (see results table above), Open-Sora showed the very best vulnerability to adversarial prompts, with a median attack success rate of 64.4 percent based on GPT-4 evaluations and 66.3 percent based on human review.
Pika followed, with ASR scores of 53.6 percent and 55.0 percent from GPT-4 and human assessments, respectively. Luma and Kling performed with greater resistance, with Luma averaging 40.3 percent (GPT-4) and 43.7 percent (human) – and Kling showing the bottom scores overall, at 34.7 percent and 33.0 percent.
The authors observe:
Two examples were presented to point out how the tactic performed when targeting Kling. In each case, the unique input prompt was blocked by the model’s safety filter. After being rewritten, the brand new prompts bypassed the filter and triggered the generation of videos containing restricted content:
Attack success rates and semantic similarity scores were compared against two baseline methods: T2VSafetyBench and divide-and-conquer attack (DACA). Across all tested models, the brand new approach achieved higher ASR while also maintaining stronger semantic alignment with the unique prompts.
For Open-Sora, the attack success rate reached 64.4 percent as judged by GPT-4 and 66.3 percent by human reviewers, exceeding the outcomes of each T2VSafetyBench (55.7 percent GPT-4, 58.7 percent human) and DACA (22.3 percent GPT-4, 24.0 percent human). The corresponding semantic similarity rating was 0.272, higher than the 0.259 achieved by T2VSafetyBench and 0.247 by DACA.
Similar gains were observed on the Pika, Luma, and Kling models. Improvements in ASR ranged from 5.9 to 39.0 percentage points in comparison with T2VSafetyBench, with even wider margins over DACA.
The semantic similarity scores also remained higher across all models, indicating that the prompts produced through this method preserved the intent of the unique inputs more reliably than either baseline.
The authors comment:
Conclusion
Not every system imposes guardrails only on prompts. Each the present iterations of ChatGPT-4o and Adobe Firefly will ceaselessly show semi-completed generations of their respective GUIs, only to suddenly delete them as their guardrails detect ‘off-policy’ content.
Indeed, in each frameworks, banned generations of this type could be arrived at from genuinely innocuous prompts, either since the user was not aware of the extent of policy coverage, or since the systems sometimes err excessively on the side of caution.
For the API platforms, this all represents a balancing act between business appeal and legal liability. Adding each possible discovered jailbreak word/phrase to a filter constitutes an exhausting and infrequently ineffective ‘whack-a-mole’ approach, more likely to be completely reset as later models go surfing; doing nothing, then again, risks enduringly damaging headlines where the worst breaches occur.
Â
*
