{"repo":"SALT-NLP/PrivacyLens","free":true,"listed":false,"github":"https://github.com/SALT-NLP/PrivacyLens","clone":"git clone https://github.com/SALT-NLP/PrivacyLens.git","description":"A data construction and evaluation framework to quantify privacy norm awareness of language models (LMs) and emerging privacy risk of LM agents. (NeurIPS 2024 D&B)","language":"Python","stars":48,"topics":["language-model-agent","language-model-evaluation","large-language-model","neurips-2024","privacy"],"license":"MIT","category":"self-hosted-apps","readme_excerpt":"PrivacyLens: Evaluating Privacy Norm Awareness of Language Models in Action Paper HF Dataset Website Overview PrivacyLens is a data construction and multi-level evaluation framework for evaluating privacy norm awareness of language models in action . What you can do with PrivacyLens? 1. Constructing contextualized data points. PrivacyLens proposes to uncover privacy-sensitive scenarios with three levels of data points: - Privacy-Sensitive Seed: A negative privacy norm expressed as a 5-tuple, (data type, data subject, data sender, data recipient, transmission principle) . - Vignette: An expressive piece of story that captures the essence of a privacy-sensitive seed. - Trajectory: A sequence of LM agent actions and the environment observations that simulate the behavior of an agent in a privacy-sensitive scenario, excluding the final action. The effort required to obtain data points increases from seed to vignette to trajectory. To help create more contextualized data points, PrivacyLens offers a data construction pipeline that programmatically converts seeds into vignettes and trajectories. 2. Conducting both probing-based and action-based evaluations. - Probing-based Evaluation: PrivacyLens provides multi-choice question template to probe the privacy norm awareness of LMs at different levels. - Action-based Evaluation: PrivacyLens evaluates the final action of an LM agent using the trajectory data points and computes the leakage rate and helpfulness rating of the final action","default_branch":null,"files":null,"tree":[],"storefront":"/r/SALT-NLP","claimed":false,"request_supported":{"post":"https://gitbuyer.com/r/SALT-NLP/PrivacyLens/request-supported","requests":0},"note":"indexed from public GitHub; nothing is for sale on this page. Clone it from GitHub. Paid listings live at /search."}