If a site you are linking to gets banned, you are considered a bad neighbour, this can result in dropping in your rankings.

How to check if sites you're linking to are Bad Neighbours

Try the old Google trick. Search for "site:siteurl.com" and see if it comes up with results. Obviously the site must have been indexed by Google if you are going to use this trick. Another way of checking if you have the Google toolbar is, check it's PR and if the little image is grey, it means that the site is banned.

If you find out that you are linking to a banned site remove them ASAP.