We just fixed the problem of huge files in our new accounting software. It was caused by a couple stupid bugs in the code that decides where to save changed records inside the file. Zapping one got the TurtleSoft company file from 11 gigabytes down to 1.5 gigs. The second fix shrank it to 85 megabytes: bigger than the current 64 megs in Goldenseal, but that’s expected. We added a few things.
Our staff is back to using the new accounting software for our own business, in parallel with Goldenseal. As soon as there’s a month without serious bugs, it’ll be time for a first release.
Why did TurtleSoft write its own database code? Well, if we were just starting now, we’d use something free and open-source, for sure. Let someone else write the code to manage data on disk.
That was the original plan for Goldenseal in the late 80s, also. Our first estimating & accounting templates hit the limits of MS Excel, so we needed to switch to a real app. Open source software didn’t exist back then, so it needed to build atop something commercial.
First, we tried a dozen different databases and other development platforms: FileMaker, FoxBase, Omnis, etc. All had fatal flaws (usually lack of spreadsheet-like breakdowns for estimates).
We finally decided to write the interface in C++, and license an object database called NeoAccess to manage files. Neo was popular at the time, especially on Macintosh. It worked pretty well, up until we started to test with real data. Uh oh, there were crashes, corrupted files, and weird error messages deep in NeoAccess code. Emails to their tech support went unanswered. Soon the company disappeared.
Searching the early Internet showed internal emails from AOL, Netscape and others also searching for solutions, and/or making plans to give up on Neo. We were in too deep, and had no better options. So our staff spent a year rewriting NeoAccess to be more sturdy.
Most of the work was just making the code readable, so we could figure what it did. Several definite bugs turned up. We probably fixed others accidentally. That made it good enough for the 1.0 release. A couple years later, there were more updates to Goldenseal’s database code. Those fixed the last of the leftover bugs.
Our new accounting software inherits the good parts from the original NeoAccess design. Plus the good parts of what we changed and added for Goldenseal. Plus a few improvements, based on 20+ years of experience afterwards.
One thing that’s different now is gap tracking. It helps keeps files compact, even though records change size and need to move elsewhere.
Neo used CNeoFreeList to track empty spaces. Actually, many of them, scattered within the file. We replaced them with a single DB_GapManager, doing the same things but easier to debug. We also added a DB_FileManager, a list of where each record was in the file. It made double-sure that records would not write on top of something else. Later, it also allowed recovery of data that was lost when a NeoAccess index became corrupted.
Tracking gaps is complicated. Removing a record may leave a gap of the same size, or it may widen a gap that it touches. Remove a record with gaps on either side, and three gaps must merge into one. The system worked, but it was too complicated.
For the new accounting software, DB_GapManager is gone. Computers are fast enough now that we just zip through lists of record locations and sizes, and compute gaps between them. Simpler and more reliable, once the stupid logic bugs are fixed.
DB_FileManager is also replaced. Goldenseal sometimes ran out of memory because the manager was so big, so we now divide the file into bite-size sectors. DB_SectorManagers each handle disk space for 16,000+ records. Our company file has 13 sectors. There’s room for 16,000+ more, enough to manage 256 million records. There’s a way it can go beyond that, if ever needed.
BTW Google still shows a few hits for CNeoFreeList, 20+ years after its death. It’s mostly users back in the Aughties, trying to fix error messages and crashes in various apps.
NeoAccess caused pain for many Mac software developers, big and small. Other people’s code is not always the best answer.
Dennis Kolva
Programming Director
TurtleSoft.com
