Wildcard Search

Re: Wildcard Search

Postby nageswaragunupudi » Tue May 05, 2009 9:00 am

Mr Otto

All my earlier programs contained similar code since clipper days. Open the dbf file with FOpen in shared mode scan the entire text and work back the record numbers and then proceed. ( I also read the dbf files in this raw mode to check integrity and for repairs. )

But those were the days when we did not have these advanced functions.

Now these functions are available. I did not compare the speeds but please consider what happens in these two different cases. Assume the dbf resides on a remote server. Our program is executed on the client PC. Our old logic reads the entire DBF file. May be fast, but it still reads the entire DBF data. OrdWildSeek reads only the index contents. In effect the quantity of data ordwildseek has to read is less than the data we read in our raw dbf read method.

Assume a DBF file with record length of 2048, containing 100,000 records. We read 205 MB of data in our old method.
Assume an index on field city with a field length of 20 bytes. Size of the index is only 2 MB. Isn't it faster to search in 2 MB of data than in 205 MB of data ?

Long time back, I read somewhere that indexes are fully read over the client in the beginning. I am not sure it it is so even in the present RDDs. In such a case, reading data in indexes from local memory ( or memory cache ) is faster than reading from a remote computer. I say I remember to have read, but I am not sure.

Further more, our coding effort is reduced a lot.
I can’t imagine that skipping through the file record by record is the solution.

I agree. For this, setting filter with WildMatch( ... ), as proposed by Mr Armando, is the best solution. It is working for me well. But this function is available in xharbour only. I am sure some function with similar functionality may be available in Harbour too. But I am not aware of it. Wish some Harbour pundits throw some light on this.
Regards

G. N. Rao.
Hyderabad, India
User avatar
nageswaragunupudi
 
Posts: 10295
Joined: Sun Nov 19, 2006 5:22 am
Location: India

Re: Wildcard Search

Postby Otto » Tue May 05, 2009 9:25 am

Hello Mr. Rao,

thank you for your answer.

>Assume a DBF file with record length of 2048, containing 100,000 records. We read 205 MB of data in our old method.
>Assume an index on field city with a field length of 20 bytes. Size of the index is only 2 MB. Isn't it faster to search in 2 MB of data than in 205 MB of data ?

I assumed that Dietmar needs a fulltext search I mean he does not know before searching in his case if the lawcase is in this lex or in that.
So an index can’t be used. To compare search speed we must assume that the basis data is for every method the same.

>Further more, our coding effort is reduced a lot.
This may be true but what counts I think is always the result. So if it is possible to get a more powerful solution one should try.
As searching the databases is a very important task form my application I am always searching for a better way.

I will try the temporary index . This would resolve 2 problems first search and then the presentation of the data in – in my case - xBrowse.
I only have to switch index.

Best regards,
Otto
********************************************************************
mod harbour - Vamos a la conquista de la Web
modharbour.org
https://www.facebook.com/groups/modharbour.club
********************************************************************
User avatar
Otto
 
Posts: 6046
Joined: Fri Oct 07, 2005 7:07 pm

Re: Wildcard Search

Postby nageswaragunupudi » Tue May 05, 2009 9:38 am

>
I assumed that Dietmar needs a fulltext search I mean he does not know before searching in his case if the lawcase is in this lex or in that.
So an index can’t be used. To compare search speed we must assume that the basis data is for every method the same.
>
If it is full text search, then obviously we need to scan the entire DBF and thats the way you recommended. Only ADS provides Full Text Search natively. ( I don't know if RMDBFCDX provides it too )

>
first search and then the presentation of the data
>
Setting filter does avoids first search and then presenting. This filter is working quite fast, surprisingly.
Regards

G. N. Rao.
Hyderabad, India
User avatar
nageswaragunupudi
 
Posts: 10295
Joined: Sun Nov 19, 2006 5:22 am
Location: India

Re: Wildcard Search

Postby nageswaragunupudi » Tue May 05, 2009 10:15 am

Just now I tested with a DBF containing 608664 records with record length of 733 bytes on a field of length 40.
I used the filter using wildmatch( ... )
Time to set the filter and refresh the browse is varying between 0.000 seconds to highest 0.015 seconds.
Regards

G. N. Rao.
Hyderabad, India
User avatar
nageswaragunupudi
 
Posts: 10295
Joined: Sun Nov 19, 2006 5:22 am
Location: India

Re: Wildcard Search

Postby nageswaragunupudi » Tue May 05, 2009 10:28 am

For full text search this is working for me

cFilt := "WildMatch( '*/14/*', DbRecordInfo( 9 ) )"
set filter to &cFilt
Go Top

Works if the expression is found in any field of the DBF. Fast enough still.
Regards

G. N. Rao.
Hyderabad, India
User avatar
nageswaragunupudi
 
Posts: 10295
Joined: Sun Nov 19, 2006 5:22 am
Location: India

Re: Wildcard Search

Postby Otto » Tue May 05, 2009 12:18 pm

Hello Rao,

Thank you very much for your help.
You are right that a filter is fast but to browse a filtered database in xBrowse or a listbox is slow.

I had this problem with a PPC solution.

http://forums.fivetechsupport.com/viewtopic.php?f=4&t=15482

I have now tested with conditional index and I find that speed is very ok also if you skip through the records in a browser.

Best regards,
Otto
********************************************************************
mod harbour - Vamos a la conquista de la Web
modharbour.org
https://www.facebook.com/groups/modharbour.club
********************************************************************
User avatar
Otto
 
Posts: 6046
Joined: Fri Oct 07, 2005 7:07 pm

Previous

Return to FiveWin for Harbour/xHarbour

Who is online

Users browsing this forum: Google [Bot] and 44 guests