Could be your prop will not open in reverse. Try going into fwd whislst still tied up to open up the blades. Then use a spring to get the stern out. Watch out for the proverbial 'prop walk'. You could, of course walk/warp the boat around and go out fwd.
If you are making way in reverse then the prop must be opening ok more likely to be prop walk which causes the stern to kick one way or the other depending on left or right hand rotation of prop, if you are aware that it will happen then its not so bad but if the wind is pushing your stern in the same direction as your prop is pushing you then it can make things difficult. there was an article in last month PBO with some good tips on how to overcome it such as remaining tied up with the engine in reverse to build up a flow of water past the rudder before casting off.
Try gunning the engine in reverse to get the boat moving and then knocking it out of gear to see if it will steer. Using the engine in bursts can sometimes reduce the degree of prop walk. But every boat is different and usually it is a matter of trial and error.
Depends a lot on the boat and prop - but usually having a prop that works more effectively in reverse helps.
In my case the recent change from a 2 bladed fixed to a 3 bladed Gori folding made a great improvement, but the Gori is unlike any other folding propellor on the market.
I have found that this works with Aeolus if I need to turn to starboard in astern (boat has bias towards turing to port - forward and astern - due port hand side prop). As we now also have a folding prop (like to stack all the odd against myself here), this "burst" mode also ensure that the blades spin out smartly. But the key seems to be getting some way on first, without water flowing past the rudder you have no streerage.