---
title: "A survey of tool use and workflows in alignment research"
description: "Before building anything we asked alignment researchers how they work and where a language model tool would earn its place. The survey is closed, the dual-use note still stands."
published: 2022-03-23
tags: ["Automating alignment research"]
importance: 5
docStatus: "finished"
audio: "https://pub-4ee2f71bc29541a7a6e8d9694f0a1b21.r2.dev/62d9c28aa7117a1815e9c568/audio.mp3"
crosspost:
  lesswrong: "https://www.lesswrong.com/posts/ebYiodG3MAEqskCDG/a-survey-of-tool-use-and-workflows-in-alignment-research-1"
author: "Jacques Thibodeau"
canonical: "https://jacquesthibodeau.com/a-survey-of-tool-use-and-workflows-in-alignment-research/"
---
Crossposted from the [AI Alignment Forum](https://alignmentforum.org/posts/ebYiodG3MAEqskCDG/a-survey-of-tool-use-and-workflows-in-alignment-research-1).

TL;DR: We are building language model powered tools to augment alignment researchers and accelerate alignment progress. We could use your feedback on what tools would be most useful. We’ve created **a short survey that can be filled out** [**here**](https://forms.office.com/r/mNBR7AKMBU)**.**

We are a team from the current iteration of the [AI Safety camp](https://aisafety.camp/) and are planning to build a suite of [tools](https://www.alignmentforum.org/tag/tool-ai) to help AI Safety researchers.

We’re looking for feedback on what kinds of tools would be most helpful to *you* as an established or prospective alignment researcher. We’ve put together a short survey to get a better understanding of how researchers work on alignment. We plan to analyze the results and make them available to the community (appropriately anonymized). The survey is [**here**](https://forms.office.com/r/mNBR7AKMBU). If you would also be interested in talking directly, please feel free to schedule a call [**here**](https://calendly.com/jacquesthibodeau/discussion-about-tool-use-and-workflows-in-alignment-research).

This project is similar in motivation to Ought’s [Elicit](https://elicit.org/), but more focused on human-in-the-loop and tailored for alignment research. One example of a tool we could create would be a language model that intelligently condenses existing alignment research into summaries or expands rough outlines into drafts of full Alignment Forum posts. Another idea we’ve considered is a brainstorming tool that can generate new examples/counterexamples, new arguments/counterarguments, or new directions to explore.

In the long run, we’re interested in creating seriously empowering tools that fall under categorizations like [STEM AI](https://www.alignmentforum.org/posts/fRsjBseRuvRhMPPE5/an-overview-of-11-proposals-for-building-safe-advanced-ai#:~:text=with%20other%20humans.-,6.%20STEM%20AI,-STEM%20AI%20is), [Microscope AI](https://www.alignmentforum.org/posts/fRsjBseRuvRhMPPE5/an-overview-of-11-proposals-for-building-safe-advanced-ai#:~:text=it%20actually%20is.-,5.%20Microscope%20AI,-Microscope%20AI%20is), [superhuman personal assistant AI](https://www.alignmentforum.org/posts/SQ9cZtfrzDJmw9A2m/my-overview-of-the-ai-alignment-landscape-a-bird-s-eye-view#:~:text=superhuman%20personal%20assistant), or plainly [Oracle AI](https://www.alignmentforum.org/tag/oracle-ai). These early tools are oriented towards more proof-of-concept work, but still aim to be immediately helpful to alignment researchers. Our prior that this is a promising direction is informed in part by [our own](https://universalprior.substack.com/p/making-of-ian?s=w) very fruitful and interesting experiences using language models as writing and brainstorming aids.

<!--kg-card-begin: html-->
<div class="admonition warning">
<div class="admonition-title">Dual-use risk</div>
<p>One central danger of tools with the ability to increase research productivity is <a href="https://www.alignmentforum.org/posts/YQhBhxFhChGExS5HE/dual-use-of-artificial-intelligence-powered-drug-discovery">dual-use</a> for capabilities research. Consequently, we’re planning to ensure that these tools will be specifically tailored to the AI Safety community and not to other scientific fields. We do not intend to publish the specific methods we use to create these tools.</p>
</div>
<!--kg-card-end: html-->

We welcome any feedback, comments, or concerns about our direction. Also, if you'd like to contribute to the project, feel free to join us at the #accelerating-alignment channel in the [EleutherAI channel](https://discord.gg/2kpBev9nCd).

Thanks in advance!

<!--kg-card-begin: html-->
<div class="admonition note">
<div class="admonition-title">Related posts</div>
<p>This post is part of the Accelerating Alignment research series. See also: <a href="https://jacquesthibodeau.com/a-descriptive-not-prescriptive-overview-of-current-ai-alignment-research/">A Descriptive, Not Prescriptive, Overview of Current AI Alignment Research</a></p>
</div>
<!--kg-card-end: html-->

## Sources

Every external link in this piece that has a captured card, with what that page said
when it was captured. The quoted lines below are not the author of this piece writing:
they are the linked page describing itself, recorded by `bun run link-cards` on the date
given, and kept so that a reader still has them if the original moves or goes away.

- **A survey of tool use and workflows in alignment research** — Logan Riggs, alignmentforum.org, 2022-03-23
  <https://alignmentforum.org/posts/ebYiodG3MAEqskCDG/a-survey-of-tool-use-and-workflows-in-alignment-research-1>
  Captured 2026-08-28.

  > TL;DR: We are building language model powered tools to augment alignment researchers and accelerate alignment progress. We could use your feedback on what tools would be most useful. We’ve created a short survey that can be filled out here. We are a team from the current iteration of the AI Safety camp and are…

- **Microsoft Forms** — forms.office.com
  <https://forms.office.com/r/mNBR7AKMBU>
  Captured 2026-08-28.

- **Home** — aisafety.camp
  <https://aisafety.camp/>
  Captured 2026-08-28.

  > Research Incubator

- **Tool AI — AI Alignment Forum** — alignmentforum.org
  <https://alignmentforum.org/tag/tool-ai>
  Captured 2026-08-28.

  > Tool AI is a type of Artificial Intelligence that is built to be used as a tool by the creators, rather than being an agent with its own action and goal-seeking behavior. Generally meant to refer to AGI, tool AI is a proposed method for gaining some of the benefits of the intelligence while avoiding the dangers of…

- **Calendly** — calendly.com
  <https://calendly.com/jacquesthibodeau/discussion-about-tool-use-and-workflows-in-alignment-research>
  Captured 2026-08-28.

- **Elicit: AI for scientific research** — elicit.org
  <https://elicit.org/>
  Captured 2026-08-28.

  > Use AI to search, summarize, extract data from, and chat with over 125 million papers. Used by over 2 million researchers in academia and industry.

- **An overview of 11 proposals for building safe advanced AI** — evhub, alignmentforum.org, 2020-05-29
  <https://alignmentforum.org/posts/fRsjBseRuvRhMPPE5/an-overview-of-11-proposals-for-building-safe-advanced-ai>
  Captured 2026-08-28.

  > This is the blog post version of the paper by the same name. Special thanks to Kate Woolverton, Paul Christiano, Rohin Shah, Alex Turner, William Saunders, Beth Barnes, Abram Demski, Scott Garrabrant, Sam Eisenstat, and Tsvi Benson-Tilsen for providing helpful comments and feedback on this post and the talk that…

- **My Overview of the AI Alignment Landscape: A Bird's Eye View** — Neel Nanda, alignmentforum.org, 2021-12-15
  <https://alignmentforum.org/posts/SQ9cZtfrzDJmw9A2m/my-overview-of-the-ai-alignment-landscape-a-bird-s-eye-view>
  Captured 2026-08-28.

  > Disclaimer: I recently started as an interpretability researcher at Anthropic, but I wrote this doc before starting, and it entirely represents my personal views not those of my employer Intended audience: People who understand why you might think that AI Alignment is important, but want to understand what AI…

- **Oracle AI — AI Alignment Forum** — alignmentforum.org
  <https://alignmentforum.org/tag/oracle-ai>
  Captured 2026-08-28.

  > Oracle AI is a regularly proposed solution to the problem of developing Friendly AI. It is conceptualized as a super-intelligent system which is designed for only answering questions, and has no ability to act in the world. The name was first suggested by Nick Bostrom. See also * Basic AI drives * Tool AI * Utility…

- **Making of #IAN** — Jan Hendrik Kirchner, universalprior.substack.com
  <https://universalprior.substack.com/p/making-of-ian?s=w>
  Captured 2026-08-28.

  > TL;DR: I fine-tuned a large language model on my personal notes and embedded the resulting model in my everyday workflow. Personal experience, Roam Research, AI Safety.

- **Join the EleutherAI Discord Server!** — discord.gg
  <https://discord.gg/2kpBev9nCd>
  Captured 2026-08-28.

  > The original open science AI research collective. We started the open source LLM movement and have been pushing the boundaries of science ever since. | 39112 members

- **Dual use of artificial-intelligence-powered drug discovery** — Vaniver, alignmentforum.org, 2022-03-15
  <https://alignmentforum.org/posts/YQhBhxFhChGExS5HE/dual-use-of-artificial-intelligence-powered-drug-discovery>
  Captured 2026-08-28.

  > H/T Aella. A company that made machine learning software for drug discovery, on hearing about the security concerns for these sorts of models, asked: "huh, I wonder how effective it would be?" and within 6 hours discovered not only one of the most potent known chemical warfare agents, but also a large number of…

## Terms used

The author's own definitions for the glossary terms this piece uses. These are his words,
not a standard reference.

- **STEM** — Science, technology, engineering and mathematics.