We’re exploring using LLMs to handle these challenges. Our large language models like GPT-4 can understand and generate natural language, making them applicable to content moderation. The models could make moderation judgments based on policy guidelines provided to them.

With this technique, the means of developing and customizing content policies is trimmed down from months to hours. 

1. Once a policy guideline is written, policy experts can create a golden set of knowledge by identifying a small variety of examples and assigning them labels in response to the policy.  
2. Then, GPT-4 reads the policy and assigns labels to the identical dataset, without seeing the answers. 
3. By examining the discrepancies between GPT-4’s judgments and people of a human, the policy experts can ask GPT-4 to provide you with reasoning behind its labels, analyze the paradox in policy definitions, resolve confusion and supply further clarification within the policy accordingly. We will repeat steps 2 and three until we’re satisfied with the policy quality.

This iterative process yields refined content policies which are translated into classifiers, enabling the deployment of the policy and content moderation at scale.

Optionally, to handle large amounts of knowledge at scale, we will use GPT-4’s predictions to fine-tune a much smaller model.