If the only load across the solar panel is your multimeter, you will get an output of about 19 - 20 volts in bright sunlight. Once you put a load (the battery to be charged) on the panel, the voltage will drop back to something less worrying - probably about 13 - 14 volts, depending on the state of charge of the battery.
Whats "really bright sunlight"? I can just about remember sunshine, but anything more is lost in the mists of time.
The voltage that you measure across your solar panel is very dependent on the load on the panel. You will see a higher voltage using a high impedence digital multimeter than with a lower impedence analogue meter. The true open circuit voltage will probably be higher still. However, this is pretty academic, because as soon as you connect any sort of load (such as a battery), the voltage will drop dramatically.
About the only type of situation where the high voltage might be a problem is if you are driving a very low current, very voltage sensitive device from a large solar panel. However, in normal boaty use, you are using the panel as a battery charger, and all this is irrelevant.
You should be more concerned about the current that the panel can produce - a large panel charging a small battery can fry the battery, if left to its own devices. As a rule of thumb, if the panel rated output (in amps) is more than 10% of the battery stated capacity (in amp-hours), then you need a regulator to protect the battery.
P.S. if you see any of this "really bright sunlight" stuff, let me know
Most solar panels have higher than expected output for a few days/weeks when they are new. After an initial normal degradation they go to the nominal levels indicated by the manufacturer.
Don't know why that is, but there was a warning when I bought mine to make sure you don't have voltage sensitive equipment connected for the first two weeks.