Last week, Goldenseal Pro saved record changes to disk for the first time. It was very exciting to enter data, quit, reopen the file, and see the changes still there. This may not seem like much, but it is a major milestone. It’s the last big connection needed between the new human interface code, and the business logic that runs our estimating and accounting software.
Since then, we have been working on the database code, which manages how records are stored on the hard drive. The new code we wrote last year has performed well while importing existing Goldenseal files, and then viewing records. However, it needs more bells and whistles to properly manage changes, deletions, and other daily usage.
We often complain about the NeoAccess database library used in Goldenseal, but it did have some very good features. We had to fix a few serious bugs in the late 1990s, but since then it has run thousands of company’s files, with few problems. As we write the code that replaces it, our goal is to keep the good features from the old database, but make it more reliable, and easier to maintain.
When Goldenseal 4.x opens any record, it reads the data from hard drive to RAM. It then stays “cached”, either until memory got low, or until our code deliberately purges the cache. Reducing disk accesses makes many operations run much faster.
Goldenseal Pro uses a similar cache system. The main difference is that we simplified the list of cached records, so it is easier to troubleshoot. The same list also handles “dirty” records, as described in the previous post.
Every database has to decide where to put new and changed records during a save. The easiest place is at the end of the file. Unfortunately, that eventually results in a huge file that is mostly holes. For example, FileMaker used to save everything at the end (and maybe still does). Files can grow to 10s or 100s of megabytes, and eventually need a compression step to get back to a reasonable size.
NeoAccess solved that problem with a list of gaps between records. That way, new or changed records could go into the empty spaces. Unfortunately, their code was hard to debug, and we suspected that it sometimes wrote records on top of existing ones (extremely bad news). In 2002 we replaced their code with a File Manager to track every record and every empty space. That system worked well, but it used an extremely large record that sometimes crashed, if memory was too low.
For Goldenseal Pro, we’ve split the original file manager into separate, smaller Sector Managers. Each tracks records in just one portion of the file (currently 32,000 records). There is also a separate Gap Manager that tracks all empty spaces. It decides where to put each record, when it is saved. The sector managers have been working great for over a year, and we are refining the gap manager now. It’s tricky, because the manager needs to combine gaps that touch, and some records work better with empty space before or after.
Goldenseal Pro stores the location of every record in two different places: in the main indexing system, and in the sector managers. That will allow the new database to recover from file corruption, by fixing the one that is wrong. There are places in Goldenseal 4.x files where a single bit change can totally trash the file. In Pro, recovery will be possible from almost all types of “bit rot”.
Dennis Kolva
Programming Director
TurtleSoft.com
