Crawling a Search Engine?

[nomaam]

I am using VB.NET 2002 and am working in a windows form

 

I am trying to "Crawl" or index a search engine that has a database of just files, not websites. When you perform a search your results will be links to files to download. What i would like to do is create a program to index all files within the search engine's database.

 

My question is how do i go about doing this the most effective way? I can see just trying every possible search string then placing the results in a database and filtering out any duplicate addresses i get, but this seems like a poor method. The restrictions on the search engine are a minimum of 3 characerts, so i can't just put in a "*" or "***" to get all files, it must be 3 characters other than special characters.

 

Could anyone recommend a better approach?

 

Thanks.