The reason that people date is to find out about each other.
I am a big believer of asking. Obviously he is too. His penis was hard and he thought that you might want to know so you could decide what to do. A clumsy way to ask if you wanted to have sex. You didn't. You both found out something. He just wanted to have sex, you didn't want to.
Next, he believed that you might be impressed that he was drinking expensive whiskey. You were not. And didn't seem to want to drink and fuck. Another learning experience.
So, overall, you have little in common. It should be easy for you to move past the experience.