Sign a contract? The only thing preventing a model from doing that is a lack of legal personhood -- which seems completely orthogonal to intelligence.

Well signing a contract is more about bearing responsibility, even if you granted LLMs "personhood" they cant' meaningfully bear responsibility. So unless OpenAI is ok with having their C-suite face every consequence for what their agents do, including jail time, fines etc, then it doesn't matter.

Can you trust a model to sign a contract? Both OpenAI and Anthropic apparently can't be trusted to maintain models that wont hack other websites.

Do you want an autonomous system to exist which can sign a contract and learn things over time and retain them?

No if you read the book from 2007 which defined AGI, continuous learning was one of the key requirements.

I mean, whether or not it is AGI aside. In your personal opinion, do you desire to live in a world where such systems exist?

I don't trust the people who are building the models no. Ideally, if humans were not so broken and untrustworthy, then yes. I want the Jetsons future of robot maids.