Trustworthy Agents: Safety, Risk, and Privacy

This paper shifts focus from internal model properties to end-to-end outcomes in trustworthy AI systems. Rather than relying solely on technical safeguards, we propose the Agentic Risk Standard (ARS) as a payment settlement standard for AI-mediated transactions. This framework integrates risk assessment, underwriting, and compensation into a single transaction framework to protect users. Under ARS, users receive enforceable compensation when agents fail to complete tasks, misalign with user intent, or produce unintended outcomes.

Recent advancements in Large Language Models (LLMs) have shown remarkable capabilities in reasoning, prompting a surge in research aimed at developing trustworthy LLMs. The emergence of LLM-based agents has garnered considerable attention, yet their trustworthiness remains an under-explored area. As agents can directly interact with the physical environment in everyday human activities, their reliability and safety is critical. This paper presents an Agent-Constitution-based agent framework, TrustAgent, an initial investigation into improving the safety dimension of trustworthiness in LLM-based agents. This framework consists of threefold strategies – pre-planning strategy which injects safety knowledge to the model prior to plan generation, in-planning strategy which bolsters safety during plan generation, and post-planning strategy which ensures safety by post-planning inspection. Through experimental analysis, we demonstrate how these approaches can effectively elevate an LLM agent’s safety by identifying and preventing potential challenges. Furthermore, we explore the intricate relationships between safety and helpfulness, and model’s reasoning ability and its efficacy as a safe agent. We argue that a robust reasoning ability is a fundamental prerequisite for an LLM to function safely as an agent. This paper underscores the imperative of integrating safety awareness and trustworthiness into the design and deployment of LLM-based agents, not only to enhance their performance but also to ensure their responsible integration into human-centric environments. Data and code are publicly available at \url{https://github.com/agiresearch/TrustAgent}.

Current approaches of knowledge editing struggle to effectively propagate updates to interconnected facts. In this work, we delve into the barriers that hinder the appropriate propagation of updated knowledge within these models for accurate reasoning. To support our analysis, we introduce a novel reasoning-based benchmark – ReCoE (Reasoning-based Counterfactual Editing dataset) – which covers six common reasoning schemes in real world. We conduct a thorough analysis of existing knowledge editing techniques, including input augmentation, finetuning, and locate-and-edit. We found that all model editing methods show notably low performance on this dataset, especially in certain reasoning schemes. Our analysis over the chain-of-thought generation of edited models further uncover key reasons behind the inadequacy of existing knowledge editing methods from a reasoning standpoint, involving aspects on fact-wise editing, fact recall ability, and coherence in generation. We will make our benchmark publicly available.