I assume that OP isn’t sleeping with the wrist wearing the watch underneath the pillow their wife has her head on.
Pulling your arm from under a pillow someone is using seems very likely to disturb them and wake them. You’re not doing that, OP, right?
Well, I’m sure you’re right.