Well, we finally fixed the mystery crashing error in our new accounting software. It took a week of testing and debugging. At least we fixed a few other potential problems along the way.
The good news is, the bug was simple. Our company file grew bigger than 4 gigabytes during the conversion from Goldenseal, but one old line of code still used 32-bit file marks. It truncated the 64-bit address, tried to save a record into the wrong part of the file, and chaos ensued. One small change made it OK.
The bad news is, our company file should not be that big. It already took up 3 gigabytes, then grew to 11 gigs in June. Goldenseal only needs 64 megs for TurtleSoft’s data. The new accounting app will have bigger files, but not by that much.
In theory, the new database has a good system to keep the file small, similar to what’s in Goldenseal. When saving a record, it looks for the first gap that’s a snug fit, and puts it there. If no tight fits, it then looks for large gaps with enough space left over to hold another record. If none of those, then it tries gaps that waste a little space (currently 24 bytes, but we may tweak it later). Any of those keep the file size unchanged. If no suitable gaps at all, the record tacks onto the end. That makes the file bigger.
If we’re lucky, there’s a stupid error in the gap-tracking code. Fix that, and files will shrink to a reasonable size.
If not lucky, we’ll need to fix a subtle design problem. One that can affect any database.
The conversion from Goldenseal to the new app does something potentially nasty. It loops through each bank transaction, and adds the ID to its bank account. That makes the account record 4 bytes bigger.
New files start out tightly packed, with zero gaps. When we save the account, it’s slightly too big for its old space, so it adds to the end. Next save it’s even bigger, so to the end again. Repeat for 25,000 bank records and the file gets huge, with 25,000 gaps that gradually increase in size.
If that’s the case, something clever needs to happen during the conversion to keep things compact. There are a few options, but we’ll worry about that only if it’s needed.
Luckily, the gap-finding system rarely fails in normal daily use. Records come in all sizes, so most of them will find a gap that’s snug. There always will be some wasted space within the file, but not much.
When our staff wrote the database code back in 2016-2018, we also added reports to help debug what’s inside the file. Right now, we’re tweaking those to be more useful and accessible. It will help track down whatever it is that makes files too big. Also any future problems.
Dennis Kolva
Programming Director
TurtleSoft.com
