Keep your mill and log loading area on each side of the divider but rearrange your sidings so they connect through the center of the divider with two parallel tracks. you now have a loads in and empties out setup. Two sets of cars, one empty and one loaded start out sitting on the two parallel storage tracks, each roughly halfway through the divider. The empty cars get picked up on the mill side and are trundled around on your loop and get pushed into the "empties" siding on the log loading side. They pop out on the mill side and are uncoupled. These empties can then picked up by the mill engine and the cycle repeats.
A separate set of permanently loaded cars are then picked up by the engine on the loading side and trundled around to the mill, pushed into the mill and through the divider and pop out on the loading side of the divider. They are then uncoupled . Conceal the opening for the two tracks in the divider by using a tunnel of building or trees.
This is not my original idea. John Armstrong and Ed Rappe both used this many years ago.