Ha! I already killed that colossus. I watched a YT video and the guy says DON'T control the horse. Just make the horse run and then keep your thumb away from the L stick and only worry about firing arrows. The controls are so bad that the key is to not use them. Killed him in no time as soon as I got that tip.
They have an intentional lag. So you have to know when to expect a button to press to have any results. I prefer controls that actually react when you do, but that's just me.