B&O actually has two museums, one in Ellicott City, and one Downtown. The one in Ellicott City has a lot of (only?) model train bits. Having driven past it for years, some how I never made it in...
DEFINITELY check out Avalon area at Patapsco. It is super easy to get there (95 south to 195 south to rt 1 south make a right, make a left, park). There are a few different ways to check it out. An old girlfriend and I put different coins on the rails on the viaduct and watched a train run over them. We collected them once the trail passed, and had our very own flattened coins. I think the conductor might have been cranky about that...