He wants to see where I work, meet my colleagues
It’s your life and your office. Ask him what he’s curious about and, if you’re not comfortable inviting him, don’t. Your personal and professional lives are different and how you handle them is up to you alone. He’s probably confused about how he feels, and you’re giving him a lot of importance for no apparent reason. If his opinion doesn’t matter, and you don’t owe him an explanation, why is this even a problem? What do you want to confront him about? Do you want him to be more than just a friend? Ask yourself why this bothers you, and what his presence in your life means, before figuring out how you want to deal with it. Also, recognise the fact that no one has the right to control your life in any way. It’s your prerogative alone to give anyone that kind of power.