That looks like "old" ablative paint - keeps the critter and plants from growing on the bottom if the boat is left in the water. It doesnt look like its in all that bad of shape.
I think you have a very reasonable choice of just smoothing down the rough spots with sand paper and leaving it as is for a while and spend your time on the rest of the boat.
When I bought my boat (a 1990 - the PO had it iin a slip for a bunch of years), the bottom looked worse than yours - had some old ablative paint with some marine fresh water growth on the paint as the PO had not maintained the paint the last few years he owned the boat. I just sanded "the top layer" leaving most of the ablative paint on but aso making the bottom smooth. In some places, the ablative paint is still there, in some places, the white gell coat is now visible. Not pretty.. but when Im sailing, only the fish have to look at it.
This year, the boat spent two months at the first of the season in a slip and then six weeks on a mooring along with a bunch of day sailing and a one week trip.. In the places where I have the old ablative paint, the hull comes out of the water fairly clean after the long stays in the water. But on the places where there isnt paint, I get growth (not bad but its always at fairly cold mountain lakes). I had plans for a big job of stripping down the paint and then barrier coat, ect. But now Im just thinking about applying "touch up" ablative paint over the existing ablative paint.