Doesn't quite work that way round. Usually the limiting factor is the size of prop you can swing - for motoring performance the bigger diameter the better. Broadly speaking the larger diameter the prop, the lower the required shaft speed and therefore the greater the reduction ratio. The reduction ratio then varies according to the maximum engine revs - the higher the revs the greater the reduction ratio. Shaft speeds on small installations are usually in the range of 1200-1800. Most modern engines run at max 3000-3600, so they tend to be fitted with reduction ratios between 2:1 and 2.6:1 although some (like Yanmars) may offer 3:1 so that you can run large diameter low pitch props on heavy displacement boats (for their size).
The choice of prop size (that is dia*pitch) is usually the last thing you do after you have decided on HP and reduction ratio. Props are cheap compared with getting the engine/box combination wrong.
Suggest you put your boat's data into a programme such as Propcalc www.castlemarine.co.uk with your preferred choice of engine then change the ratios to see what size props are suggested. This will give you a ballpark to see if your prop is suitable for your installation. However, it is much more sensible to get a recommendation from the engine/propeller supplier to suit your boat.
Just to give you an example, on my boat when I re-engined I had a choice of a 2:1 and a 2.6:1 - the former would swing a 13" prop, the latter a 15" - which is what I chose, partly because it is a heavy long keel boat with a big deadwood and partly because the previous engine also ran the 15" prop, but with lower HP and a 3.1:1. So I could reuse the prop.
Its dependant upon the max prop size you can fit and the rpm of the engine. A larger slower rotating prop is generally more efficient than a smaller faster one. The Castle Marine prop calculator is a pretty good place to start to look at different combinations. A marine engineering company or prop supplier should be able to help.