One of the benefits of the Bluetooth control is that the interface on the phone apps should be much easier to make improvements on as time goes by.
Addressing the issue of needing to look at the screen for basic control, a possibility is something like the Amazon fire TV (or fire stick) app, which has a large pad in the middle. I could foresee something like this where you swipe up for a 1 smph increase, and down for a 1 smph decrease, double tap to change direction, or whatever.