I've seen quite a few people in here who have had issues like this with family or friends, and I'm still surprised by it. Wasn't everyone taught that whole deal about "If you can't say anything nice...."? Besides, with everything going on today, a nice hobby seems to be the last thing that would raise a red flag.
This also makes me appreciate my family even more. No one, and I do mean no one, in my circle has any interest in trains, yet none have ever said a word against them. I have a life long friend who is the all work, no play type, yet he was more than happy to spend a whole day walking around the Allentown show, talking to me while I looked at trains. A few years ago I tore down my layout due to some home remodeling. While feeling a little low, I mentioned in a family email that I might give up the trains. Siblings and siblings-in-law all chimed in that I should rebuild. Again, none of them care about trains, but they all know that I enjoy them. My mother and father in law even bought me a set of locomotives while wandering an antiques shop. Wish my Dad had been around long enough to see me get into trains. He never had time for them, but he was a guy that always dug Lionel trains. Indirectly, he is the reason I got into them.
Sorry if I'm rambling, I just thought I'd point out that there are non-train people that still understand that it is important to those of us that do like them.