No need to remove the ENTIRE exhaust, but you do need to remove the muffler.
However - to get the driveshaft to slide forward, you must loosen the driveshaft center support. Doesnt have to be removed, just loosened. For easy access, the FSM says to drop the entire exhaust. But if you remove the heat shield nuts, there is just barely enough room to slide tools up there to reach the 13mm hex head bolts and loosen each a few turns. Once that is done, the driveshaft will slide to the front enough to separate from the diff.
If removing the axle shafts, consider replacing the rear wheel bearings, if you have the tools for it. And if you havent replaced the main fuel tank hose that exits the center of the tank, it will never be easier than when the diff is removed...