Maybe try using Metacrawler? I usually have really good luck with that one.