Chemical paint stripper, sanding or scraping will take it down to metal unless you are awfully careful... or at least look like you tired to clean it up, which looks suspicious. That's why I asked about the type of paint... Try thinner or better yet, alcohol --first. If it was some enamel out of a can, those two may work. By the way, some body shops use vodka to clean light overspray.
Of course, if you use the chemical stripper, you can start from scratch...