A þorn in one's side!
Maybe this has something to do with words such as þou, thou, and you?
Ah, the miracles of the extended character set! Yes, that thorn.
Likely the ascension to "you" is as you suggest.
Its fun to look at how English was written prior to the work of Samuel Johnson and his dictionary.