Goldenseal Pro Progress Report (Dec 21)

The past couple weeks, Turtlesoft’s staff has mostly worked on breakdown tables.  Tables are a big, complicated part of the interface for Goldenseal accounting software. Basically, it’s anything that looks like a list or spreadsheet. Last Spring, we estimated 5 weeks of programming time for tables on Macintosh, and another 3 weeks for Windows. So far, that seems about right.

In Goldenseal 4.96 and earlier, estimates, expenses and many other business transactions use 3 different layouts. There is an item breakdown with a list of Assemblies, a category breakdown where you type in line items, and a none layout with no table at all. Each transaction must use one of the three, so a popup button on the left switches between them. The system often confuses new users, and it was complicated to program.

Goldenseal Pro will be simpler. It will have just one main layout, plus an optional breakdown table. The breakdowns will slide out in a side drawer (Mac) or attached window (Win) when you click a button. Tables in Goldenseal Pro will combine the former category and item breakdowns. Just as functional, but less complicated.

The original plan was to release the first version of Goldenseal Pro without the Custom Layouts command, and then add it in the next update. We started to write code that reads existing layouts, but splits off the breakdown tables to a different window. That way, the old layouts still work in the new format. The code mostly works, but it’s an ugly hack.

Rather than waste more time on temporary code, we decided to rearrange the schedule a bit, and work on Custom Layouts next. That way, it will be possible to design better-looking layouts at the very beginning. This change delays the earliest possible release, but it doesn’t alter the overall completion date.

Back in the 90s when we started work on Goldenseal, Custom Layouts was one of the first things we finished. We used it to design the screen appearance, in parallel with the business and database code. I guess the same logic applies to Goldenseal Pro.

Custom Layouts acts much like a drawing program. Fortunately, there is already good sample code for that. We won’t have to re-invent anything.

This week, we also added a few things to the Goldenseal-to-Pro translator. It now merges category and item breakdown records, and also moves a few types of bank accounts and transactions. The conversion is necessary so we can use the Sample Company File for testing. It also will convert all user data, after Goldenseal Pro is released. So far, it looks like the transition will be completely seamless.

Dennis Kolva
Programming Director


Database Reliability- Part 4 (Dec 14)

The past few posts have described many ways for databases to go bad. All those potential problems give us plenty to think about, as we write code for our new accounting and estimating software. As a bonus, there are two more risks that we still haven’t covered.

The first problem is relatively minor: database files can easily expand to enormous size. FileMaker used to suffer from that. Things start to go south when a data record changes, and gets bigger. For example, when you add a new Material Purchase in Goldenseal, 3 accounts and 3 record arrays add a reference to it. That adds 4 bytes to their size, which makes them too big for their old spots within the file. It’s easiest to move them to the end of the file, but that leaves 6 empty gaps in the middle. Make that shift enough times, and the file will grow huge, even though it is mostly empty.

The NeoAccess database actually had a very good solution. It kept a list of empty spaces, and used it to save records in the most suitable gap, as close to the front of the file as possible. Unfortunately, their implementation was buggy, and too hard to fix. In 2002 we rewrote it using the same basic design, but with understandable code. It has worked well ever since. Goldenseal Pro uses the same approach, but with a few improvements.

The second problem is extremely serious: it’s relatively easy for a database to accidentally write data on top of other data. This is usually called “buffer overflow”, and it’s a general issue that all software faces. Overflows are the cause of most security holes that you read about in the news. They cause most data corruption, and the most serious of crashes.

Overflows are so dangerous that Goldenseal Pro has five different ways to prevent them.

  1. The Gap Manager that prevents file bloat, also knows where it’s safe to add records, without risk of damaging something else.
  2. After finding a gap, we next check the Sector Manager, which has a list of every record stored in that part of the file. The Manager looks on both sides of the gap, and makes doubly-sure there is no overlap with existing records.
  3. When Goldenseal accounting software saves records, it adds up the amount of data it writes, and makes sure the total matches exactly with the expected size. The same sanity-check happens again when it reads records from disk.
  4. Each record has about 20 bytes of “safety tags” at the beginning and end. When reading each record, we test the tags to make sure they are still OK.  If there ever is a bug that causes a record to overlap something else, it’s not hard to do forensics and discover the culprit. After we added the safety tags back in 2005, we soon tracked down the last remaining bug that was overwriting data.
  5. Within each record, there are additional safety tags for data that is most likely to cause a buffer overflow. If data is corrupted, the goal is to learn about it as quickly as possible. That way we can abort instead of damaging something else.

Most of these safety features are already present in current versions of Goldenseal. However, we are expanding and enhancing them for Goldenseal Pro. Disk storage is now so cheap that it makes sense to add an extra few megabytes, if it creates a more reliable database.


Meanwhile, back to Graphic Interface land. Since finishing the database work, our staff has focused on more advanced interface features. That includes breakdown tables, printing, “more info” boxes and the Find commands.

We also have started to simplify our accounting a bit. Goldenseal currently has 5 classes of bank accounts and transactions that are almost identical: for cash, checking, credit cards, loans, and savings. For Pro, they will all merge into one. Bank transactions will still look the same, so the only change is how you navigate to them.  Investment and escrow accounts are also very similar, but we haven’t decided yet whether to merge them, or keep them separate.

For Goldenseal estimating and accounting records, we currently have two types of breakdowns: category and item. There isn’t much difference between them, and they always have confused new users. Goldenseal Pro will merge them.

The tricky part for both these changes is converting existing data to the new format. Fortunately, a couple of previous updates made similar class changes, so we can reuse that code.

Dennis Kolva
Programming Director

Database Reliability- Part 3 (Dec 5)

The most important task for any database is to remember where each record is, within its file. To accomplish that, our accounting and estimating software uses a binary tree (b-tree) to store record locations. It has a different tree for each class of records. Every tree has a root (starting point) and nodes (branches). Nodes at the end are “leaf nodes”, which store the record locations.

A true binary tree has 2 branches at each node, which makes a tall tree (11 levels for 1,000 records, or 21 levels for a million). The NeoAccess database we’ve used for Goldenseal versions 1.0 to 4.9x came with a default of 8 branches per node. We upped it to 32. For unknown reasons NeoAccess also had two levels at each node, so it took 8 levels for 1,000 records, or 12 levels for a million. Still a rather complex tree.

Back in the 1980s and 90s, RAM and hard drives were cramped, expensive, and relatively slow. Because of that, it was best to keep nodes small. To save space, NeoAccess also didn’t create nodes until there were records to fill them. As a result, it scattered nodes randomly throughout the file (a bit more than 1 node for every 32 records). To save space, NeoAccess also stored very little data in each node. They only contained 4 or 8 bytes for each branch: just a file position, plus a record ID in leaf nodes.

We started to use Goldenseal to run TurtleSoft in 2000, just before releasing version 1.0. Unfortunately, a few NeoAccess nodes in our company file are corrupted, and have been for more than 10 years. When we try to look at the affected records, the node points to a very wrong file location. The OS is not happy with that, so it crashes.

We’ve spent dozens of hours stepping through code and looking at our raw file data, trying to find the errors. The problem is, our company file contains over 9,000 NeoAccess nodes, and they all look identical. If we could find the lost or damaged nodes in a raw file editor, we might be able to fix them. Unfortunately, it’s the proverbial needle in haystack.

Over the years, we also have looked at 30 or 40 corrupted Goldenseal files sent in by users.  When we scanned them with a raw file editor, about half contained data from other programs entirely. Our code has no access to anything beyond our file, so we didn’t put it there. Those files probably were damaged when the OS or hard drive firmware got confused, and wrote someone else’s data on top of ours. A few bad files were caused by TurtleSoft bugs, but we haven’t seen any of those since 2006. The rest were probably caused by small NeoAccess errors: possibly due to unknown bugs in their code, possibly from “bit rot” screwing up file addresses. Change one bit in the middle of a tree, and the whole thing stops working.

For Goldenseal Pro, our goal is to make the database less fragile, and more repairable. Your hardware (and the OS) will never be 100% perfect, but we can at least make it possible to diagnose and repair any database errors.

As a first step, Goldenseal Pro creates all its basic tree structures right at the beginning. Doing so has a cost. New, empty Goldenseal Pro files start out at 6+ megabytes (compared to 106K for original Goldenseal). A file that big would have been absurdly huge in the 1980s or 90s, but now it’s just a few millionths of a hard drive. The new setup puts the most important database structures in the same places in every file, and they won’t ever move. A uniform file structure will be much easier to diagnose and repair.

Goldenseal Pro also uses bigger nodes: with 128 to 4,096 branches, depending on how many items you are likely to have there. Almost all trees will only be one level deep. If they go two levels deep, they’ll hold a few million records before needing a third. The result is less clutter, and easier debugging.

The new nodes include text for their name and contents, so they are easily visible in the raw file. They also include a short text tag for each branch, and ‘safety tags’ at the beginning and end. All these ‘file navigation markers’ help us now during development and debugging. They also will help anyone looking at damaged files, in the future.

For extra security, all the important database structures are duplicated somewhere else in the file. At least in theory, we can rebuild anything that suffers damage.  Most likely we’ll wait for actual damaged files before writing the repair tools, since it’s hard to predict in advance, how data may go wrong.

All of this extra security adds more than 100 bytes per record, so Goldenseal Pro files will be significantly larger. At $60 to $300 per terabyte, we figure you probably won’t mind spending a penny or two for a few extra megabytes, to get more reliable data.

You still should have a good backup system, using multiple locations and methods. There are plenty of ways for data to die, and we can’t prevent all of them. However, we can at least make our files stronger and more survivable. As a bonus, it makes our programming and debugging easier.

Dennis Kolva
Programming Director