Month: April 2018

Goldenseal currently manages its accounting data with NeoAccess (a popular object database library in the 1990s). To locate records, it uses a binary tree.

You may have used binary search while looking for something in an old-fashioned paper phone book or dictionary. Open at the middle, and compare it to what you want. Go left or right depending, then open the middle of that half. Repeat with smaller and smaller divisions until you get to the right page.

Binary search is very efficient, especially when the data is huge. It takes 10 steps to cover a thousand items, 20 steps for a million, 30 for a billion, and so on.

It’s easy to set up a binary search in sorted items of identical size. However, that is rarely the case for database records. For those, it’s hard to find the middle, so they are better managed with a binary tree. The tree has a single root, which grows increasingly ‘branchy’ as you go up. To find something you start at the bottom, compare at each node, then move up through the tree. Eventually you arrive at a leaf node, which links to the actual record. This kind of navigational tree is also called an index.

NeoAccess was optimized for speed and small size. Each node included a list of file addresses for its children, and nothing more. Sparseness made sense, back when memory was expensive and processors were slow.

Unfortunately, a lean index is also very fragile. If a NeoAccess node is corrupted, everything above it becomes invisible. Those records are still on the hard drive, but they lack a path that says where they are. NeoAccess had no alternate route. Any damage was fatal.

These days, memory is cheap and processors are very fast. As a result, it’s nearly always better to build software for reliability rather than speed. Probably the best example for bullet-proof design is the Internet. It began as Arpanet, which was meant to keep running even after nuclear annihilation. Turns out, that kind of sturdiness helps, even in a world of small, normal, ordinary snafus.

Instead of a tree, the Internet is a web. Lots of connections from here to there. There is always a way to find something, even if one path breaks. Cell towers work the same way.

Goldenseal Pro still use a tree structure for normal finds. But it also includes more web-like connections and alternate routes. With their help, it should be able to rescue records, even if a tree branch is damaged. The accounting database can even be self-repairing.

Meanwhile, we still need to use the old NeoAccess code to read data from the current Goldenseal accounting software, so it can move into the new Pro format. Unfortunately, the Sales breakdowns in our TurtleSoft data have some corrupted index nodes. More than half can’t be found. Without them, we won’t know what people bought.

Fortunately, coming to the rescue is the File Manager, which we added to NeoAccess in 2002. Its main function is to prevent records from overwriting each other, but it also provides an alternate way to find record locations on the disk. We just added some code to it, so it can read and recover lost records when the tree can’t find them. Voilà, it rescued our missing data! The new code will also help any other users with damaged trees.

There is a cost for the extra connections, webbishness and redundancy. Our old TurtleSoft company file was 51 megabytes, and the new one weighs in at 127. It will probably grow even larger, as we add more safety features.

3x the file size for 3x the reliability seems a worthy trade.

Dennis Kolva
Programming Director
TurtleSoft.com

 

Up until now, our staff has been using the Sample Company File to test Goldenseal Pro (our new, updated accounting and estimating software). The sample file has fake data for a fake company. It was first set up in 1997 to test pre-alpha versions of Goldenseal. Having existing records makes it easier to run accounting commands, and make sure everything works properly.

When you open an older-format file with Goldenseal Pro, it first creates a new file in the new format. Then it reads each record from the old file, and copies it to the new one.

The work on Goldenseal Pro is almost far enough along to start using it for our own business. To get ready for that, we currently are working on the conversion, to get TurtleSoft’s accounting file into the new Goldenseal Pro format.

When a software company uses its own software for daily use, it’s called “eating your own dog food”. It does produce better software, since any problems are right there in your face. On the other hand, it also means that corrupted data can creep in, from using the accounting software in real life before it is fully debugged. Because of that accumulated cruft, converting our own file gives errors. Most of this week, we have been running the conversion until it complains, then stepping through to find out what’s wrong. Fix that problem, and move to the next.

Right now we are still working on a tough one. Since about 2002, our file has suffered from a corrupted NeoAccess index for sales breakdowns. Try to look at them, and Goldenseal crashes. The bad data may have been caused by “bit rot”, or it may have been from some NeoAccess bug that was fixed long ago.

Fortunately, the corruption only affects a few records from the 1990s, so it doesn’t bother us in daily use. Unfortunately, the conversion goes there when it moves records to the new file. So, we need to get past the corrupted index. Ideally, we should fix it in a way that handles any other data corruption that may be present in other users’ files. That means stepping through NeoAccess code many times, and trying out small changes. Right now it no longer crashes, but it still skips all the sale breakdowns that are after the bad index. We need those.

The NeoAccess database structure has many places where changing a single bit wreaks havoc. That’s a big flaw, because there are things like cosmic rays and radioactive decay that sometimes change a 1 to a 0. “Bit rot” is rare (and getting rarer as technology improves), but it still happens.

In this case, each index in the tree has a list of file addresses that point to its branches. If any of those change by accident, they’ll read from the wrong part of the disk and get garbage. It’s all downhill from there.

Goldenseal Pro already stores addresses in two different places, but the headache of trying to fix NeoAccess code made us decide to store each file address a 3rd time. It adds 8 bytes per record, but then we can compare two numbers at every read, and give a warning if different. After that, the code can look at both locations, and see which one makes sense.

The conversion from Goldenseal to Goldenseal Pro already futzes with a few things: converting bank accounts, merging breakdown classes, switching to 64-bit, and adding more safety tags to the data. As corrupted data turns up in our file, we add a fixer routine in the conversion code. It might as well tidy up along the way, in case other users have similar damage.

There will be a few weeks or a month when we run TurtleSoft from two company files: one in Goldenseal, and in Goldenseal Pro. We’ll taste the dog food as an appetizer, but still eat human food for the main course. That will be the perfect time for current Goldenseal users to run the conversion on their own data. If any new problems turn up, we’ll have plenty of time to fix them before the final release.

Dennis Kolva
Programming Director
TurtleSoft.com

Goldenseal Pro accounting software is now in the testing and completion phase. The interface is mostly set up, but there are many small details left to finish. Basically, our staff tries everything with the Sample Company File until it gives an error message or doesn’t work right. We fix it, and repeat. Most of our time will be spent doing that, from now until the final release.

For construction projects, there’s usually a “punch list” phase near the end. It starts with a site inspection, where you/the client/the architect writes up a list of problems to fix. Then, you fix them and take them off the list. Software is harder to inspect, so its development goes differently. Right now it’s more like checking each wall in a remodeling project, to see if everything is trimmed and painted. If not, time to finish it.

Some problems take a few minutes to solve, while others take hours. Rarely they take days, if the code needs serious refactoring or rewriting. The fixes probably average out to 3o minutes or an hour apiece. As a wild guess, there may be 500 or 1000 little details still to finish. Multiplying a guess by a guess gives a total that is about what the estimating spreadsheet says (it predicts completion in October).

As we update Goldenseal’s code, we are finding many places where it jumps through hoops in order to conserve memory, or give results a bit faster. Goldenseal was originally built for machines with 4 megabytes of RAM, and processors in the 16 megahertz range. It took some effort to get good performance with that hardware.

Modern computer have 1000x the RAM capacity and 100x faster processors. That means we can rewrite some things to have simpler code, and still perform well. Simpler code means easier maintenance, and less chance of mystery bugs. When a chunk of code needs rewriting anyhow to fit in with Cocoa, we refactor it now. Some things get flagged for later.  Sometimes it’s a judgement call between “if it ain’t broke don’t fix it”, and “do it right”.

At the moment, our staff is still working on bank transactions, but they are almost done. Goldenseal Pro has one little improvement: you can now have transactions for multiple bank accounts open at the same time, and switch between them easily. It actually makes the code simpler than our current approach of closing the window, and then reopening it with a different account.

Dennis Kolva
Programming Director
TurtleSoft.com

 

 

Goldenseal accounting software includes action commands to help run your business: Pay Bills, Write Payroll, Deposit Funds, Project/Sales Billing, Job Costing and Reconcile. They present a window with a list of items in a table. Click buttons to do stuff.  If you double-click, most show another window with more details for that item.

The PowerPlant framework that we used to build Goldenseal did not have a table class, so we had to manage and draw the action command tables with our own code. For various reasons, those tables were particularly hard to set up, and hard to maintain. Replacing them has been on the to-do list for a long time.  The action commands (and their tables) are the largest remaining task for Goldenseal Pro, so we started on them last week.

Fortunately, Apple’s Cocoa framework does have a table class. It also has an Interface Builder tool to create windows. It only took a few hours to use it, and set up the Reconcile window and its table. The same approach will work for all the action commands, and it ought to be much easier to maintain.

To test the Reconcile window, we need working bank transactions. That means finishing a design change that we started last year.

Goldenseal accounting software currently has seven types of bank accounts: Cash, Checking, Savings, Investments, Credit Cards, Loans and Escrows. They each have their own menu commands and windows. However, in the years since we first designed Goldenseal, deregulation happened. The former, strictly defined differences between commercial banks, savings banks, savings & loans and credit unions are long gone. These days, you might write checks or use a debit/credit card from almost any type of bank account, or even use bank-ish entities such as PayPal or Apple Pay. The distinctions have blurred.

It still makes sense to treat Investments separately (because they can be non-monetary, and may have job costs). Ditto for Escrows (because it’s not your money).  The others really aren’t that different any more, so we are merging cash, checking, savings, credit cards and loans into just plain Bank Accounts. They’ll have a popup field so you can still group them by type, but otherwise they’ll all look the same. It will make cash-basis accounting much simpler.

For bank transactions, Goldenseal currently has 5 different screen layouts (checks, payments, deposits, transfers out and transfers in). Switching between them is very complicated code, so we also plan to merge those into a single layout.

For new users, setting up the new bank account system was easy. For existing users, we need to handle many years of data that was entered from the old system. To preserve it, the update process automatically moves bank accounts and transactions from old format to new. It also goes through purchases and other transactions, and converts their links to the new payment and deposit accounts. The code for that was written last summer, but it needs real data before it can be tested. That’s one reason why we are so eager to start using Goldenseal Pro to run TurtleSoft.

Dennis Kolva
Programming Director
TurtleSoft.com