downloading files from internet

Rajeev Chavan · Post by **Rajeev Chavan** » Fri Apr 28, 2006 5:54 am

Hi
I need to download files from internet.Can anyone pls suggest a method how it can be done ?
Thanks

Post by **Antonio Linares** » Fri Apr 28, 2006 8:27 am

If the files are located on a FTP server you may use samples\gallery\icopyfil\icopyfil.prg

Rajeev Chavan · Post by **Rajeev Chavan** » Fri Apr 28, 2006 2:37 pm

Hi
Thanks Antonio.I am not sure whether they are on ftp server.I have to paste the complete path in URL address bar and the file gets downloaded.I have to provide the date for which the file is required and the file should be downloaded.Pls advice how this can be done
Thanks

Post by **Antonio Linares** » Fri Apr 28, 2006 3:47 pm

Rajeev,

Please show a URL sample like the ones that you are using.

Rajeev Chavan · Post by **Rajeev Chavan** » Sat Apr 29, 2006 6:16 am

Hi Antonio
The URL below will download EOD prices of shares
http://www.nseindia.com/content/histori ... 06bhav.csv

James Bott · Post by **James Bott** » Sat May 06, 2006 10:04 pm

Rajeev,

Because IE recognizes the CSV extension, it displays the data in the browser rather than prompting the user for a download filename, then saving the data to a file.

Assuming you are not the one putting the data up on the net, then you have to find another way to collect the data. FW had a web client class you can use to read a web page. Below is the code to read the data into a varialble (cHTML). The data will come in with some header lines which you will then have to strip off, then you can just save cHTML to a file.

This code was tested with Harbour. I don't know if it will work with Clipper.

Regards,
James

Code: Select all | Expand

/*
Purpose: Example of how to get the HTML code for a web page
Date   : 01/04/05
Notes  : You must create a main window to use a socket.

         I had to fix a bug in the TWebClient class, new method. It was returning nil
         instead of self. TWEBCLIE.PRG was dated 11/11/1999 (came with FWH 2.4 July 2003).

         The function ProcessPage does not return the code since it is collected in a socket
         which may still be running when the function exits. You have to process the code
         by assigning a function call to oWeb:oSocket:bClose

         James Bott, jbott@compuserve.com
*/

#include "fivewin.ch"


#define FALSE .F.
#define TRUE  .T.


function main()
   local oWnd
   define window oWnd
   activate window oWnd ;
      on init ProcessPage("http://www.nseindia.com/content/historical/EQUITIES/2006/APR/cm24APR2006bhav.csv")
return nil


function ProcessPage(cURL)
   local oWeb
   local cHTML:=""  // contains HTML code
   local cSite:=""
   local cPage:=""

   if left(upper(cURL),7) = "HTTP://"
      cURL:= right(cURL,len(cURL)-7)
   endif

   cSite:= left(cURL, at("/",cURL)-1 )
   cPage:= right(cURL,len(cURL)-at("/",cURL))

   //msgInfo(cSite,"cSite")
   //msgInfo(cPage,"cPage")

   oWeb := TWebClient():New()

   oWeb:oSocket:Cargo := FALSE
   oWeb:bOnConnect = {|oWClient| oWClient:oSocket:Cargo := TRUE}
   oWeb:bOnRead    = {|cData| if(valtype(cData) == "C", cHTML += cData, )}

   oWeb:Connect(cSite)

   //msgInfo(cHTML)

   // Keep trying to connect
   while ! oWeb:oSocket:Cargo
      WaitMessage()
      SysRefresh()
   enddo

  // msgInfo("Connected to site")

   oWeb:GetPage( cPage )

   // Assign function to process code
   oWeb:oSocket:bClose = {|self| ::end(), self:=Nil, Process(cHTML) }

   //oWeb:oSocket:close()

   sysrefresh()

   //msgInfo("page collected")
   //msgInfo(cHTML)

return nil


// Process the HTML code
function Process(cHTML)
   msgInfo(cHTML,"HTML code") // for testing only
   // Process the code here--save to file, extract data, etc.
return nil

// eof
//----------------------------------------------------------------------------//

Rajeev Chavan · Post by **Rajeev Chavan** » Tue May 09, 2006 7:14 am

Hi James
Thanks for the program supplied for downloading the info from web page.When I ran that program I got the info displayed in a window.When I clicked the close button of the window I got following error message "Error BASE/1209 String Overflow: + from Errorsys,line:0".Pls advice how to overcome this problem.Is it occuring because info is more than 65K . I have noticed the CSV file size is between 65K-68K.If this is the case pls advice how to overcome this.
Digressing from the subject.A few years back when I had posted my first query on fivewin newsgroup you had answered it and introduced me to the concept of "Paying it forward".Eventhough I have not contributed anything to fivewin I try to reply to posts on other forums which I visit.It does give mental satisfaction when you help other people.
With Best Wishes
Rajeev Chavan

Post by **Antonio Linares** » Tue May 09, 2006 8:21 am

Rajeev,

You need to migrate your application to FiveWin for Harbour (32 bits) where you can manage files as large as needed. In a week or less you may have it working with FWH.

James Bott · Post by **James Bott** » Thu May 11, 2006 11:05 pm

Rajeev,

Thanks for the program supplied for downloading the info from web page.When I ran that program I got the info displayed in a window.When I clicked the close button of the window I got following error message "Error BASE/1209 String Overflow: + from Errorsys,line:0".Pls advice how to overcome this problem.Is it occuring because info is more than 65K . I have noticed the CSV file size is between 65K-68K.If this is the case pls advice how to overcome this.

Well, the best answer is like Antonio says, upgrade to 32bit. You are going to have to do this sooner or later.

Otherwise, you can try processing the data one line at a time and saving it to a file, instead of pasting it all together into one large string. Oh, and you don't need to display the data, I just put that in there for testing.

Take a look at the TTxtFile class for ideas how to write the data to a file as it is being read.

I'm glad you have been paying it forward.

James

Rajeev Chavan · Post by **Rajeev Chavan** » Fri May 12, 2006 6:39 am

Hi Jeff
Will do as suggested by you.I am really getting confused should I go for Harbor or xHarbor ? Pls advice
Thanks
Rajeev Chavan

Post by **Antonio Linares** » Fri May 12, 2006 7:53 am

Rajeev,

xharbour is more popular, but both Harbour and xharbour work very well.

If you don't use xharbour extensions, then you may be able to build your application with both compilers and decide for yourself which one to use.

Rajeev Chavan · Post by **Rajeev Chavan** » Tue May 16, 2006 4:55 am

Hi Jeff
Thanks for the guidance.The problem I am getting is the variable would'nt hold more than 65K.Does TTxtFile class contain a way to overcome the 65K limit ? Pls advice.
Thanks
Hi antonio
Thanks for the info.I will try it
Rajeev Chavan

James Bott · Post by **James Bott** » Tue May 16, 2006 6:52 am

Rajeev,

>The problem I am getting is the variable would'nt hold more than 65K.Does TTxtFile class contain a way to overcome the 65K limit ?

Only by going to 32bit. 65K is the 16bit limit.

James

Rajeev Chavan · Post by **Rajeev Chavan** » Tue May 16, 2006 2:52 pm

Hi Jeff
What is the limit for 32bit ? In case the file I am downloading is huge
Thanks
Rajeev chavan

Tue May 16, 2006 2:53 pm

The limit is the available memory.

EMG