We know lots of things about the mechanical functions of both human brains and LLMs, but that doesnt really help because we don’t know what causes consciousness in the first place. We dont know if internal memory is required, a sensory feedback loop, specific brain structures, or something else entirely.
This is not a very strong argument. By the same logic you could claim that biological thought, reflection and emotion are impossible because its just clumps of fat squirting chemicals and electrical signals at each other. The fact of the matter is we dont know what causes consciousness, so we cant know if it could form from sufficiently complex statistical interactions.
I know, its really a shame about all those low paying STEM jobs. I remember back when I was a kid I used to tell everyone I wanted to be an engineer when I grew up, but all the moneys in literature and poetry so thats what I had to do. Damn you, capitalism!
They dont have a choice. Until he gives an illegal order they are compelled to obey.