I used ( in order) 600, 1500 and 3000 grit sandpaper and then a heavy duty polishing compound applied with a cloth.
I would use a drill head attachment to apply the polishing compound, however the one I have isn't up to the job and buying a new one for this small job would be a waste. I will say that the surface is not visibly smooth, and still has light scratches present, could this be an issue?
Your problem is that you missed out at least two stages after the 600 grit. 1500 is too fine to cut through the 600grit finish. And that was probably coarser than necessary, as I've never needed more than a few strokes of 1000grit to cut through the worst of the grot on really bad lamps.
So, before wasting time and polishing materials, buy a sheet each of 800, 1000 and 1200grit wet and dry paper -
not sandpaper which is for woodwork.
Start with the 800, preferably on a flexible block, in a criss-cross pattern across the lens using plenty of soapy water. Once you've done a few strokes in each direction, wipe it dry and check you have an even finish across the surface. If you don't, repeat the process. Then you do the same with the 1000 and 1200 grit papers, finishing with your 1500.
That should give a smooth, dull, but nearly clear surface for you to finish with coarse(and then fine if you have it) rubbing compound. There's no need for anything fancy here, the materials used for improving paint are plenty good enough - I use old fashioned Farecla G3 and G10 because I have them. Headlights aren't big enough to really justify buying power tool attachments, when you can do the whole process, by hand, in about 10 minutes per lamp on the car.
If the lens has lots of tiny cracks, you should consider replacing it, because they are caused by UV light and can't be corrected.