Excellent article Evan! Along the same lines of your post you may appreciate Alberto Romero's perspective in his 2021 article on Medium "What No One is Thinking: AGI Will Take Everyone by Surprise". Romero posits: "Could we create an AGI without knowing it? Or even without having the ability to eventually know it? Our measurement tools aren’t infinite nor boundless. We could create an AI whose limitations can’t be assessed." Source: - https://towardsdatascience.com/what-no-one-is-thinking-agi-will-take-everyone-by-surprise-a76903474c79