For downloading http or https files or pages, both curl and wget work well, and I'm surprised to hear that curl would be giving you trouble. What exactly is the problem that you're seeing?
For looking at a downloaded HTML page and looking for links that match a particular pattern, I often find that regex's work well.
however wget worked for me!!! and I should be able to write code to follow the links from there.