Previous: Another Piece to the Puzzle
A broad summary of what, chez the Rationalists, sets our High-Control Group alarm bells ringing- in addition to what has already been discussed:
A. Thought Reform & Epistemic Control
- Dissent from core ideas— especially AI risk— is often dismissed as irrational rather than debated openly.
- A small number of sources (namely Yudkowsky’s writings) are elevated as superior, while alternate worldviews are discouraged or openly condemned (see pt. 5 for examples of this).
B. Existential Fear as a Control Mechanism
- The belief that artificial intelligence could imminently destroy humanity creates an urgent moral imperative for members to dedicate themselves to Rationalist orgs & projects.
- If someone is convinced that failing to work on AI safety means billions of future lives could be lost, leaving the community or changing priorities can feel not just intellectually wrong but morally unforgivable.
C. Social Reinforcement, Group Identity, & Isolation from Outsiders
— Rationalist hubs (namely the Bay Area) function as a closed social ecosystem where members primarily interact with like-minded individuals, reinforcing their quite unusual worldviews.
– Social capital in these communities is often based on one’s ability to engage with and extend Rationalist ideas. Criticizing core tenets can result in social ostracization.
- Many Rationalists who relocate to the Bay Area live in (desperately overcrowded) all-Rationalist housing, which further increases their social dependence on the group.
D. Difficulty in Leaving or Criticizing the Group
- Some former members describe a feeling of losing confidence in their own reasoning after leaving, having been conditioned to believe that Rationalist thinking is the most rigorous form of epistemology.
- Leaving the Rationalist community can mean losing an entire social and professional network, particularly for those embedded in AI safety careers.
- Those who leave are often dismissed as failing to understand Bayesianism, suffering from cognitive biases, or being emotionally irrational; their critiques & reasons for leaving are not engaged with.
*****