I think one of your limiting factors is your connection pads seem to be too generous. I think there's where a lot of space could be gained. Look at the space I leave for power and LED connections and the space you leave. I think just shrinking those would do wonders for the size of the board, you'd have a much better chance if fitting stuff in a more compact manner. The rest of your layout seems to be reasonably space efficient. However, I think you'll find you can also squeeze the components a bit tighter than you have when you get actual components on the board.
What I do to size things if I'm not sure of the spacing is to print a 1:1 image of the board and then set the components on the image to see if I have any conflicts.
