Latest FWH upgrade, performance issues, larger EXE

[Diego Decandia]

Unfortunately I do not have time to extrapolate a simple test, but, as I said, no changes in the procedure was made and until 17.04, ie before the update to 18.02, everything proceeded well and fast.

I have reported this situation so that Darrell can find elements in common or not...

I look forward to more info from my client, luckily the only one I have upgraded...

[Antonio Linares]

Please report the FWH, Harbour and C compiler used versions

[Diego Decandia]

I use xHarbour.com, before I used FWH 17.04 and now FWH 18.02

[cdmmaui]

Dear Antonio,

FWH 2016-04 with Harbour 3.2 compiled with BCC 582

Sincerely,

[cdmmaui]

Hello,

Yes, the web application (CGI) was created with xBase++ 2.0 with a modified library. The application runs on Windows Server 2012 and Windows Server 2016.

Sincerely,

[Rick Lipkin]

Enrico

The makers of Aspack have not charged me anything for the use of their product since my initial purchase .. just today, I was notified about a version upgrade and I was able to download and install for free.

Rick Lipkin

[Enrico Maria Giordano]

Enrico

The makers of Aspack have not charged me anything for the use of their product since my initial purchase .. just today, I was notified about a version upgrade and I was able to download and install for free.

Rick Lipkin

Ok, great!

EMG

[TimStone]

If you use one of these compression utilities, will it still allow the errorsys program to work and display the sourcecode upon an error ?

[Antonio Linares]

Tim,

Yes

In fact, the only difference is the EXE size. The app consumes exactly the same memory as before (it expands when the EXE is loaded).

[nageswaragunupudi]

I am wondering if I need all the LIB in the link script. Can someone help and let me know if all the LIB below are ABSOLUTELY necessary? Can someone provide a list of CORE lib and then I can add the necessary LIB based on application requirements?

Whatever be the libs included in the link script, the linker includes only those modules (not entire library) containing the functions used by the application. So, removing a few libs or adding some more libs, even unwanted, to the link script does not change the size of the exe even by a single byte.

I have noticed that the EXE are over 1 MB larger than before.

There have been several improvements and additions which resulted in an increase of some library modules. Still, we will look into the internals of FWH libs and examine if it is possible to reduce the resultant application size.

I actually watched customer run application and load data (DBFCDX). The DBFs are very large but that has not been a problem until very recently. Our main purpose of the upgrade was to utilize XBROWSE and we had to remove XBROWSE because LISTBOX performed better with large DBFs.

You may recall that sometime back I advised you:
viewtopic.php?f=3&t=35592&p=211898&hilit=wbrowse#p211898

Also, one more suggestion.
Unless there is a need to use some exclusive features of xbrowse, it may not be worth converting from listbox (wbrowse) to xbrowse. WBrowse is the fastest browse.

At the same time, since your experiences and observations may send wrong signals to other users, I am obliged to clarify some points about the comparative performance of wbrowse and xbrowse.

WBrowse is fast. It does not mean XBrowse is slow. Its speeds come quite close to that of WBrowse and in some cases much faster than WBrowse, if used the right way.
In addition, XBrowse comes with a plethora of great features not available in WBrowse.

We are talking about large DBFs. In real life, we would be accessing the table from a remote server. I made a test using "CUSTBIGM.DBF" containing 1,000,000 records. Every field is indexed. I ran the program from my desktop, accessing the DBF located on my laptop, connected over home WiFi, which is slower than a wired LAN used in office environment. Also, accessing data from a normal PC is slower than accessing from a file server.

This is the output of the program:


Navigation of XBrowse is as fast as WBrowse and scrolling is a lot faster than WBrowse. On a wired lan it should be faster and on local disk it should be flying.

You were talking about load times. Now we test with even a larger DBF. File size is 2.8 GB with more than 13 million records. Again accessing the DBF over WiFi network, this is the performance. You may see that the browse loads instantaneously.



Some points to be kept in mind: XBrowse mostly depends on the RDD functions OrdKeyCount(), OrdKeyNo() and OrdKeyGoTo(), which in turn depend on the indexes. So, very large index files, large index key values to some extent impede the performance of xbrowse when large DBFs are accessed remotely. We need to keep this in mind while building and using XBrowse.

XBrowse also has some incompatibilities with ADS for the same reasons. However, we can provide support where necessary.

I may have the same problem as Darrell.
After going from 17.04 to 18.02, I provided an update to a customer and reported a significant slowdown in the application.

I do not have for now other info, nor ideas, since the update did not change any of the procedures that the customer has reported to me...
The impression is that the problem concerns do while loops, but it is difficult to be sure or to test changes from the customer.
The executable works locally, the DBF are on the server.

We are quite concerned. It is not easy to find the causes of such problems. We need your help and cooperation. Is the entire application is slow? or some parts are running slow? Can you identify and give some idea about what kind of interfaces are working slow?
Our own software, as well as my friend-programmers' software with their clients, are working well. I requested them to rebuild using 17.04 and compare at the users' sites. So far current version appears to work quite well.
We will get back when we notice some issues and in the meanwhile, we will be grateful if you can provide us some clearer clues.

Source Code of the above programs:

Code: Select all  Expand view

#include "fivewin.ch"

REQUEST DBFCDX

function BigDBF()

   local oDlg, oFont, oBrw, oLbx
   local nSecs, nRecNo := 1
   local atags := {}
   local cDbf  := "\\GNRDELL\C\TESTS\DATA\CUSTBIGM.DBF"

   SET DATE ITALIAN
   SET CENTURY ON
   SetGetColorFocus()
   FWNumFormat( "A", .t. )

   USE ( cDbf ) NEW SHARED ALIAS BIG1 VIA "DBFCDX"
   USE ( cDbf ) NEW SHARED ALIAS BIG2 VIA "DBFCDX"

   DEFINE FONT oFont NAME "TAHOMA" SIZE 0,-14
   DEFINE DIALOG oDlg SIZE 1000,500 PIXEL TRUEPIXEL FONT oFont ;
      TITLE NetName() + " using " + cDbf

   @ 00,000 SAY "XBROWSE" SIZE 500,20 PIXEL OF oDlg CENTER COLOR CLR_HRED,oDlg:nClrPane
   @ 00,500 SAY "WBROWSE" SIZE 500,20 PIXEL OF oDlg CENTER COLOR CLR_HRED,oDlg:nClrPane

   @ 65, 20 XBROWSE oBrw SIZE 470,-20 PIXEL OF oDlg ;
      ALIAS "BIG1" COLUMNS "ID","FIRST", "CITY", "SALARY" ;
      COLSIZES 70,120,120,90 ;
      CELL LINES NOBORDER FOOTERS AUTOSORT

   WITH OBJECT oBrw
      :nEditTypes    := EDIT_GET
      :lFastDraw     := .t.
      :bKeyCount     := { || BIG1->( LASTREC() ) }

      :bKeyNo        := <|n|
                        if n != nil
                           //( oBrw:cAlias )->( OrdKeyRelPos( n / oBrw:nLen ) )
                           ( oBrw:cAlias )->( OrdKeyGoTo( n ) )
                        endif
                        n  := ( oBrw:cAlias )->( OrdKeyRelPos() ) * oBrw:nLen
                        return n
                        >

      :bKeyNo        := { |n| If( n != nil, ( oBrw:cAlias )->( OrdKeyGoTo( n ) ), nil ), ( oBrw:cAlias )->( OrdKeyRelPos() ) * oBrw:nLen }

      :lHScroll      := .f.
      :bRecSelClick  := { || BIG1->( OrdSetFocus( 0 ) ), oBrw:cOrders := "", oBrw:Refresh(), oBrw:SetFocus() }
      :aCols[ 1 ]:cFooter := TRANSFORM( BIG1->(LASTREC()), "9,999,999" )
      :CreateFromCode()
   END

   @ 65,510 LISTBOX oLbx  FIELDS TRANSFORM( FIELD->ID,"9,999,999" ), ;
         FIELD->FIRST, FIELD->CITY,;
         TRANSFORM( FIELD->SALARY, "999,999.99" ) ;
      ALIAS "BIG2" ;
      HEADERS "ID","FIRST","LAST","SALARY" ;
      COLSIZES 70,120,120,90 ;
      SIZE 470,415 PIXEL OF oDlg FONT oFont

   oLbx:aJustify  := { .t., .f., .f., .t. }
   oLbx:aFooters  := { TRANSFORM( BIG2->( LASTREC() ), "9,999,999" ), "", "", "" }

   @ 25,044 GET nRecNo PICTURE "999999" SIZE 70,24 PIXEL OF oDlg RIGHT ;
      VALID ( BIG1->(DBGOTO(nRecNo)), oBrw:SetFocus(), .t. )

   @ 25,320 BTNBMP PROMPT "TOP"    SIZE 80,30 PIXEL OF oDlg FLAT ACTION ( oBrw:GoTop(),    oBrw:SetFocus() )
   @ 25,410 BTNBMP PROMPT "BOTTOM" SIZE 80,30 PIXEL OF oDlg FLAT ACTION ( oBrw:GoBottom(), oBrw:SetFocus() )

   @ 25,510 GET nRecNo PICTURE "999999" SIZE 70,24 PIXEL OF oDlg RIGHT ;
      VALID ( BIG2->(DBGOTO(nRecNo)), oLbx:Refresh(), oLbx:SetFocus(), .t. )

   @ 25,810 BTNBMP PROMPT "TOP"    SIZE 80,30 PIXEL OF oDlg FLAT ACTION ( oLbx:GoTop(),    oLbx:SetFocus() )
   @ 25,900 BTNBMP PROMPT "BOTTOM" SIZE 80,30 PIXEL OF oDlg FLAT ACTION ( oLbx:GoBottom(), oLbx:SetFocus() )

   oDlg:bLClicked := { || CursorWait(), BIG2->( OrdSetFocus( "CITY" ) ), oLBX:Refresh() }
   oDlg:bRClicked := { || CursorWait(), BIG1->( OrdSetFocus( "CITY" ) ), /*MsgInfo( "sorted" ),*/ oBrw:Refresh() }


   ACTIVATE DIALOG oDlg CENTERED
   RELEASE FONT oFont

return nil
 

Code: Select all  Expand view

#include "fivewin.ch"

REQUEST DBFCDX

function Main()

   local cDbf     := "\\GNRDELL\C\TESTS\DATA\CUST2GB.DBF"
   local cSize    := TRANSFORM( FSIZE( cDbf ), "999,999,999,999" ) + " Bytes"
   local cRecs

   SET DATE ITALIAN
   SET CENTURY ON
   FWNumFormat( "A", .t. )

   ? "START"
   USE (cDbf) NEW SHARED VIA "DBFCDX"
   cRecs    := TRANSFORM( LASTREC(), "999,999,999" ) + " Records "

   XBROWSER Alias() TITLE NetName() + " using " + cDbf + " with " + cRecs + cSize ;
      SETUP ( oBrw:aCols[ 1 ]:cEditPicture := "99,999,999", oBrw:aCols[ 1 ]:nWidth := 100 )

return nil
 

[Enrico Maria Giordano]

Still, we will look into the internals of FWH libs and examine if it is possible to reduce the resultant application size.

Thank you, it would be great!

Navigation of XBrowse is as fast as WBrowse and scrolling is a lot faster than WBrowse.

What is the exact reason for that?

EMG

[cdmmaui]

Dear Rao,

I apologize for the delayed response as I have been traveling.

By no means, I mean to say that there are problems with FWH. I have relied on FWH nearly from the beginning the FWH started. We have built one specific application that is nearly 30 years old and used by over 5,000 users in the global logistics / supply chain industry. Our application is heavy on data entry and EDI integration with many different parties including end users and government agencies throughout the world. We always would hesitate before we would upgrade to new versions of FWH as it would be detrimental to our business if changes were made to FWH that caused poor system performance or bugs. Our customers are not patient and would not tolerate lost time due to system bugs or lost functionality.

One our customers is using a DBF that has 328 fields with over 7,500,000 records. We did notice slow performance with XBROWSE with 30 fields displayed (I sent you notices and you assisted), we switched back to LISTBOX and performance seemed to improve. However, once we moved back to FWH 2016.10, we noticed drastic improvement in performance. I know there a lot improvements that we lost by going back but we had no choice.

One thing is for sure it that we must move to a web based application connected to SQL. That is why we choose to use xBase++ 2.0 with native SQL connectivity running on Microsoft Server with IIS. I will say xBase++ is very expensive but again we have to move with the times and the demands of our customers and the RIGHT NOW movement. I will say that we are using FWH for all of our automated backend applications with SQL connectivity to Microsoft SQL Server.

Again, I LOVE FWH and what is has done for me and my business. I have ONLY positive comments just noting my comments above.

Thank you again Rao and Antonio for your invaluable support.

Sincerely,

[Rick Lipkin]

Darrell

Just curious ... on your app that is slow with your customer .. are they running your program on a Microsoft 2008r2-2016 Network with predominantly W7-W10 Clients ?

If the answer is yes .. you may be having OptLock problems on those specific networks and clients .. Optlock problems do not occur on every network and client run-time .. the problem seems random by client .. I have seen no problems with cbf\cdx on some networks and horrible ( optlock issues ) with other network\client environments ..

Have you considered a possible Optlock issue .. you can search this forum .. if I recall there are several threads on this issue and possible remedies.

Not knowing your clients environment .. that is why I moved away from dbf\cdx years ago and exclusively write all my back-ends using ADO with either Ms Access for portable installs or MS Sql Server for Client\Server .. there are no record locks needed with ADO .. the RDMS is handled much differently .. I have never looked back .. and performance with ADO and SQL has never been an issue.

Just my 2 cents worth ... I understand your situation .. and I know from our conversations that it may not be feasible to re-write or convert a huge program to utilize ADO .. especially when you are moving forward with your CGI web app.

Let me know if you have eliminated or thought about looking for any optlock issues with your troublesome client.

Rick Lipkin

ps .. we can chat privately if you would like me to create you a quick ADO proof of concept with your huge .dbf\cdx