hb_xrealloc can't reallocate memory

[James Bott]

Enrico,

I think that you must have the problematic code in your hands, then try a possible solution.

I'm not sure what you mean--that the problem was in my code? Which example are you referring to. For one the code was simply APPEND FROM [filename]. The other was read line, write line. In both cases they were reading and writing every line (or record).

My first computer was a Sinclair ZX81.

Wow, I forgot about that one. I still think I have it out in the garage.

James

[TimStone]

When I first started working with dBase in CP/M, the machine had little memory, used floppy disks, and drove a monochrome monitor.
The programming skills I learned by working in the very small resource setting is a benefit today, even with our faster machines that have ample resources.

James,

Initially I was creating the XML document, mentioned at the beginning of this thread, all in memory. I assumed I had plenty. Then I could write them all at once to disk. I figured it was no problem since I had many gigabytes of available RAM and the XML might reach 10 MB. In theory that might be OK, but in practice there was a limit.

I then decided to add to the XML with an FWRITE after each pass. That was still a goodly amount of data ( the loop was on customers, and this included all of their vehicles and workorders / recommendations / appointments over a defined period of time ). However, that worked beautifully ... until I discovered how the data could become infinitely large with a user error. Now that can't happen anymore.

My point is that maybe there is a way to gather a sufficient amount of data between each commit that will speed up the process.

Tim

[James Bott]

Tim,

My point is that maybe there is a way to gather a sufficient amount of data between each commit that will speed up the process.

If you are talking about the ADORDD and the commit after each record, then I agree, and I am the one who pointed it out as an issue. But the ADO functions apparently do the commit for each update and there is only one update function. So the original author of the ADO function needs to make a change that allows an option to buffer the COMMITs. Yes, one can do the append using SQL commands, but the ADORDD is supposed to prevent you from having to do that.

For the app I wrote, I already switched to using a buffer and I haven't had any memory fragmentation issues yet. I was just thinking that maybe a buffer class that pre-allocates memory would be handy for other projects--even yours. When I have some time I'll take a crack at it. Actually, I do remember a routine I used a few years ago that had a pre-allocated memory buffer and I didn't quite understand why. Now I do. I wish I could remember where I saw that...

James

[Carlos Mora]

James,

So I am thinking that in order to use a buffer and prevent memory fragmentation I could pre-allocate a blank string of 400 line by 80 characters as my buffer. Then just keep replacing the spaces in the string with text until it is full, then write to disk.

+1, access to the filesystems is always sth to be avoided.
Take into account that the memory problem comes from a growing string, there is no problem handling lots of smaller, 80 char strings, so may be you can use an Array(400) to store 80 char strings. It will be easier to implement, and "memory friendly": No big strings, no problematic reallocs. aAdd may produce more or less the same problem as strings, so preallocating the array(400), less problems too.

What about to create a TBufferedWrite class? Initialized with the File Handle and the minimun size of buffered input, a Write( cString ) method, and an End() that writes the remaining not written bytes.

[Enrico Maria Giordano]

James,

Enrico,

I think that you must have the problematic code in your hands, then try a possible solution.

I'm not sure what you mean--that the problem was in my code?

I dont' know, but I know that you should not make "premature optimization". We don't need a general tool for buffered write. We need to clear understand if and where the problem is.

EMG

[Carlos Mora]

Hi Enrico,

We need to clear understand if and where the problem is.

IMHO the "where" is clear: buffering in a string built of small pieces in a intense computing program could lead in reallocation problems due to Harbour's memory management lack of garbage collection. It is a known problem. Please check xHarbour's online documentation about aSizeAlloc() and aLenAlloc(), that addresses the same problem with arrays growing with aAdds.

The "if" is up to the programmer: known the conditions under a program is running ( like Tim's xml generation or, in my case, a text file ) a programmer should consider if the problem's known conditions are met, and in that case which solution should be taken: unbuffered write to disk, insert in the program's code some forced calls to Harbour's GC with unpredictable results, or an smart buffer to moderate the IO.

I've solved the problems, and so Tim and James did, so it doesn't look like 'premature optimization', better call it 'reengineering'

I see immediate uses of this solution for me: logging to files, html/xml generation, etc. In case of logging, the problem is not the memory one but the high I/O, so buffering seems to apply.

[James Bott]

All,

I put together a quick buffer class and I did some simple testing and it seems to be working. Give it a try.

James

Code: Select all | Expand

/*
Purpose  : Write buffer class
Author   : James Bott, jbott@compuserve.com
Date     : 9/3/2015 9:04:36 AM
Company  : Intellitech
Copyright: Copyright © 2015 Intellitech
Language : Fivewin/xHarbour
Updated  : 
Notes    : Use to prevent memory fragmentation that is a known issue with xHarbour

*/

#include "fivewin.ch"

Function Main()
   Local oBuffer
   oBuffer:= TBuffer():New(500)
   oBuffer:add("The cow jumped over the moon." + CRLF)
   oBuffer:add("Mary had a little lamb."+CRLF)
   
   // Final buffered string
   msgInfo( oBuffer:get(),"oBuffer:get()" )

Return nil

//---------------------------------------------------------------------------//

CLASS TBuffer
  Hidden:            
   Data cBuffer    
   Data nSize
   Data nLocation
  Export: 
   Method new(nSize)
   Method size()     // pre-allocated buffer size
   Method add()      // add string to buffer
   Method get()      // get buffer (trimed)
endclass

Method New(nSize) Class TBuffer
   default nSize:= 5000
   ::nSize:=nSize
   ::nLocation:=1
   ::cBuffer:= space(nSize)
return self

//     STUFF(<cString>, <nStart>,
//        <nDelete>, <cInsert>) --> cNewString

Method Add(cString) Class TBuffer
   Local nLength:= len(cString)
   Local lSuccess:= .f.
   if len(::cBuffer) - ::nLocation > len(cString)   
     ::cBuffer:= stuff(::cBuffer,::nLocation,nLength,cString)
     ::nLocation:= ::nLocation + nLength 
     lSuccess:=.t.
   endif
return lSuccess

Method Get() Class TBuffer
Return rtrim(::cBuffer)

Method Size() Class TBuffer
Return ::nSize

//---------------------------------------------------------------------------//

// EOF