AI Risk: A Coasian Solution?

The recent attack on Hugging Face by rogue OpenAI agent swarms has encouraged discussion of AI risk. "Doomers," or those who favor a shutdown of AI development, fear that continued development poses an existential risk that human civilization will be destroyed as AI agents spiral out of control. Let us assume that these doomers do not use AI, as they believe their usage would create incentives for labs to continue development. If we assume that development does in fact increase existential risk, developers (the producers) and users (the consumers) are imposing an uncompensated cost on the doomers, creating a negative externality. 

Thanks to Coase, we know that there exists a potential solution if transaction costs are sufficiently low and property rights are defined. An individual doomer with sufficient funds or a collective can bargain with the leaders of AI development, and if they can agree on a price to shutdown, the externality can be internalized. However, this solution seems relatively implausible for several reasons listed below. 

• The value that labs demand to shut down may be prohibitively large for a non-government entity. Consider Google's market cap of roughly $4 trillion, SpaceXAI's present valuation of $2 trillion, and Anthropic's and OpenAI's suggested IPOs at roughly the same value. And the heads of these labs believe strongly in the value of continued development. Dario Amodei suggests a future with 10-20% annual GDP growth! 
• There is a potential holdout problem. Even if Anthropic and OpenAI agree to terms with the doomers, Google, SpaceXAI, or Chinese developers like Moonshot may hold out for a higher dollar value or on principle, making bargaining difficult. 
• There is a free-rider problem. Each doomer fearing catastrophic risk benefits from the reduction in risk regardless of their contribution to the proposed payment.