I was just curious. [...Reaching over and ruffling his hair a little.]

In this case, while you connected the dots in a logical fashion, it's also implying I'm only upset about the disturbances to my sleep schedule because I'm unable to find a willing partner for sex. That's just a touch insulting, no?