Actually the time it takes to go out of your way to pick up SJ is about the same. The time from Chozo's Magmoor elevator > SJE > Tallon's Magmoor Elevator is about the same as Arboretum > SJE > Arboretum. Except the latter has the advantge of having SJ for Flaahgra and for climbing Arboretum.
And the other method I suggested makes it so you can go straight from Space Jump to phendrana, instead of going spacejump > flaahgra > magmoor.