Yeah, my starting position would be that it was obvious to any competent physicist at the time (although there weren't that many) that the potential energy release from nuclear fission was a real thing - the "only" thing to do to weaponise it or use it for peaceful ends was engineering.
The analogy to "runaway X-risk AGI" is there's a similar straight line from ELIZA to Acausal Robot God, all that's required is a bit of elbow grease and good ole fashioned American ingenuity. But my point is that apart from Yud and a few others, no serious person believes this.
Yeah it's been decades since I read Rhodes' history about the atom bomb, so I missed the years a bit. My point is that even if we couldn't explain exactly what was happening there was something physically there, and we knew enough about it that Oppenheimer and co. could convince the US Army to build Oak Ridge and many other facilities at massive expense.
We can't say the same about "AI".