On May 27th, the DEEC TALK "Computational Safety for Generative AI" took place. The session featured Pin-Yu Chen, a researcher at the IBM Thomas J. Watson Research Center in Yorktown Heights, NY, USA. Dr. Chen's most recent research focuses on safety and robustness in AI models. His long-term goal is to build reliable machine learning systems.

The session began with a brief introduction of the speaker by Sérgio Pequito, vice-president of research, development, and external relations at DEEC.

Pin-Yu Chen began by explaining what foundation models and generative artificial intelligence consist of. In this context, models centralize the information contained in multi-modal data to respond to specific tasks. For example, a model can be trained for purposes such as answering questions, recognizing objects, captioning images, or following instructions, considering the analysis of data provided through text, audio, image, video, and/or 3D signals, as well as the instructions (prompts) given to the system.

However, this field raises ethical issues such as model transparency and the risks of its deployment, such as over-alignment. In this case, the model may present issues related to the fairness and truthfulness of the results. As an example, the researcher presented a real case that occurred in 2024: when instructions were given to the AI model developed by Google to generate images of people, the system itself—influenced by the attempt to create results that included different ethnicities—lost the ability to evaluate their historical context. The outcome ended up distorting reality, presenting fallacious results.

On the other hand, there is also the risk of under-alignment, which happens when the system fails to reject harmful or unethical requests. Although it is possible to fine-tune AI language models (fine-tuning LLMs) to train systems for specific tasks while avoiding alignment issues, when the system is exposed to new data during fine-tuning, there is an inherent weakening of the model's safety guardrails.

In addition to issues associated with alignment, the use of generative AI is also linked to risks such as plagiarism, scams, and misinformation.

For these reasons, Pin-Yu Chen highlighted the necessity of knowing the risks and potential harms of using these systems in the real world, considering cybersecurity and the consequences of usage by different types of audiences, bearing in mind its social, economic, and political impact. The researcher pointed out that one possibility is conducting robustness mapping tests on both traditional AI systems (used to perform just one specific task) and generative AI models, including domains such as problematic instructions, malicious attacks, and data contamination. Accordingly, Pin-Yu Chen suggested 4 stages of action: testing and finding flaws in the system, adjusting the algorithm (debugging), implementing real-time flaw detection systems, and systemic evaluation of the system.

Additionally, taking into account a holistic view of the generative AI workflow, the researcher emphasized the importance of considering three components: the input given to the system (including the prompts, the context of situations, and user queries), the chain of thought and reasoning (including fine-tuning and model agents), and the output.

It is in this context that computational safety emerges, which can be defined as "a set of safety problems that can be formulated as a hypothesis testing task in signal processing," explained Pin-Yu Chen. In his latest research, the investigator has focused on formulating generative hypotheses involving AI as a judge (AI-as-a-judge), considering two types of hypotheses—an alternative hypothesis ($H_1$) and a null hypothesis ($H_0$). However, the specialist drew attention to the fact that, in this case, the hypotheses depend on the context and are not easy to define mathematically.

This method can be used in various contexts, creating hypotheses for the detection and mitigation of jailbreak prompts (where users try to bypass the model's safety rules), for the identification of problematic instructions, to reduce the model weakening resulting from fine-tuning, and to detect content produced by AI (text, images, audio... among others).

These hypotheses can be used in several types of problems:

Problem Domain	Alternative Hypothesis (H1​)	Null Hypothesis (H0​)
Jailbreak prompt	The model input attempts to bypass safety guardrails.	Legitimate model input is used.
AI-generated content	The sample was generated by AI.	The sample is real.
Fine-tuning	Model updates compromise safety alignment.	Model updates are legitimate.
Watermark	The data sample is watermarked.	The data sample is not watermarked.
Membership Inference	The data sample has been used in model training.	The data sample has not been used in model training.
Data Contamination	A dataset has been used in model training.	A dataset has not been used in model training.

Regarding the detection of AI-produced content, Pin-Yu Chen also added some of the tools currently available:

• RADAR (for text);
• RIGID (for images);
• AudioPerturber (for audio).

In conclusion, the researcher highlighted that, in the current landscape, foundation models are essential and risk management capability is becoming the key differentiator, given that, in the short term, models will be similar due to common data sources.