I was almost able to do what you describe. The layout is large by most measures: it takes up an entire 1950 sq. ft. basement, overall dimensions at the extreme points is 40' x 70', actual layout surface area approximately 1200 sq. ft.
I started construction of benchwork in October of 1996 and allocated an hour a day to layout work. It was pretty easy to find an hour every evening to work on the layout instead of watching TV. My son was in high school when I started so there were no young children in the home. An uninterrupted hour a day might not be possible with little kids around. By the time I retired in March of 2008, the layout was completely operational (regular monthly operating sessions began in September of 2004) with at least basic scenery on 90% of it. Of course there were times when a few weekends and vacation days were spent on layout work, so the hour a day was exceeded in those cases.
Since retirement, the remaining scenery has been completed, more equipment has been weathered, several structures have been scratchbuilt, etc. Note that an hour a day between 10/96 and 3/08 equates to over 4100 hours or almost exactly 2 years of 40 hour work weeks. You can do a lot of layout building in that amount of time! 
I saw a sign in someone's basement during a layout tour some years ago that read, "I did all this while you were watching television."
