Should I be worried if a couple of sites that I don’t want to be associated with are linking to me?

Should I be worried if a couple of sites that I don't want to be associated with are linking to me?

Recently I found two porn websites linking to my site. I disavow those links and wrote to admins asking them to remove those links but… what can I do if someone, (my competition), is trying to harm me with bad backlinks?
Jay, Spain