Month: June 2017

Goldenseal stores a lot of data. Many users have more than 100,000 records in their company file, and some have millions. The code that keeps track of everything is an important part of our ‘back end’. It needs to access records quickly, and store them reliably on disk.

For the original Goldenseal, we licensed an object database called NeoAccess. It ran on both Mac and Windows, and was very popular in the 1990s. Unfortunately, the source code had serious bugs that were never fixed, and we ended up rewriting a lot of it (especially after its developer folded their business).

For Goldenseal Pro, we looked at a couple dozen open-source and commercial database libraries, both object and relational. Using a 3rd-party library has its advantages: someone else did all the work, maybe they are an expert, and maybe they will keep it updated. However, as we found with NeoAccess, other people’s code also has disadvantages.  For one thing, it’s usually easier to fix one’s own bugs and design flaws. Every library also has a learning curve, plus quirks and limitations. There is no perfect way to manage a database.

After a few trials, we finally decided to write our own database code for Goldenseal Pro.  Since we already rebuilt half of someone else’s database, designing a whole new one didn’t seem too risky. We kept some of the design from NeoAccess, but replaced the parts that didn’t work well.

The main thing any database needs is a list of where each record is in the file. NeoAccess used a ‘binary tree’, which is theoretically the fastest way to find things, using the least amount of memory. However, modern computers are designed to move relatively big blocks of memory, so it makes sense to have a tree that is ‘wide’ (with more records in each branch, and fewer branches). A wide tree gives faster record access on modern machines, and also uses simpler, cleaner code.

We wrote the first part of the database last summer, then tested and debugged it over the winter. The past few months we finished the remaining ‘tree’ code, to handle larger numbers of records.

If you want to see what C++ source code looks like, here is our function that looks for an empty spot to index a new record, and creates a new index if the current one is full:

To keep the database code simpler, we take advantage of the fact that record IDs are always increasing, so we can just add them at the end and have a sorted list. We also just leave empty spaces when you delete a record, rather than rearrange the whole tree. Our primary goal is to make the code reliable and easy to maintain. So far, that approach has been successful.

Dennis Kolva
Programming Director
TurtleSoft.com

One of the lesser-known features of Goldenseal accounting software is the Audit Trail, tucked away inside the View-Security menu. It tracks when each record is created, changed or deleted. If multiple users sign on with different passwords, it also shows who did what. You can control exactly what gets logged with Options-Preferences-Data Entry. For security reasons, the Audit Trail can’t be changed, so it creates a permanent record of what happens to your business records.

The Audit Trail is a valuable accounting feature for larger companies. You can look back at all changes made to records, which makes it much easier to investigate possible fraud (or just carelessness). We also use it occasionally when diagnosing posting problems in user files.

On the flip side, the Audit Trail adds many records to the file, and increases file size by 10% to 30%. Many of our users do their own data entry (or have a relative do it), so it is unlikely they will ever need the trail for a security investigation. The feature is obscure enough that most people don’t even know it is running.

The conversion to Goldenseal Pro is a chance to redo some of our early design choices. After 17 years of user inputs, we have a much better picture of what is actually useful. The Audit Trail is one of those things we can probably do better.

It doesn’t make sense to remove the Audit Trail, because it helps some users, and it already works fine. However, when we import data from older files, we probably will skip the old Audit Trail entries. If they are ever needed for an investigation, they are still available in the old file. If moving them to the new file is important to you, please let us know.

We may also change our default settings, and leave the Audit Trail off for most users. If you don’t use multiple passwords, then you probably don’t need it.

Another option is to go in the opposite direction, and expand the Audit Trail into a full ‘journaling’ system. Some software programs (e.g. Photoshop) store a list of all changes made, so you can go back with Undo multiple times, or selectively apply some past changes but not others.

To make a journaling system work for accounting, the Audit Trail would end up using more than half of the file size. Having multiple undo would add complexity to our posting code, and it may have security issues.

We won’t change the Audit Trail by much in the first release of Goldenseal Pro, but we may revise it in a future update. If you have often wanted to undo deletions made days ago, please let us know. That is probably the best advantage we’d get from a more complete Audit Trail.

Dennis Kolva
Programming Director
TurtleSoft.com

Apple recently announced plans for High Sierra, the next OS update, due in Fall 2017. It will continue to run 32-bit Carbon applications like Goldenseal, but the phase-out of 32-bit support starts next year. 32-bit apps will still run under the subsequent, yet-unnamed Fall 2018 update, but it will complain about them.  Starting in 2019, 32-bit apps probably won’t launch at all.

Goldenseal Pro uses Cocoa, which is 64-bit. It will run fine on current and future Mac OS versions. It looks like we will finish it well before Fall 2018, so there will be at least a year of cushion.

The current Goldenseal is 32-bit. It runs fine on the current Mac OS (Sierra), and will be OK with the next two after that, and anything older. But it will have problems with new hardware starting in Fall 2019.

It took until August 2016 for Microsoft Office to switch from 32-bit to 64-bit. Over the next couple years, we will see how many other Mac programs (e.g. QuickBooks Desktop) can build 64-bit versions in time. Going from OS 9 to OS X killed off about 75% of Mac software companies, and moving from PPC to Intel zapped another 10%. My guess is that the carnage this time will be similar. Converting from Carbon to Cocoa is not easy.

If you are confused by the meaning of 32-bit vs 64-bit, here is some background. 32-bit integers can handle any whole number from 0 to 4 billion, while 64-bit integers go up to 16,000,000,000,000,000,000 (16 quintillion).

For accounting, 4 billion pennies is $40 million, which is probably good enough for residential construction. However, some unit costs need fractional cents, some amounts are negative, and some countries use currencies with values much smaller than $1 US. That means 32-bit numbers are too small for estimates and accounting.

If starting today, we would probably use 64-bit numbers for money amounts. Even with fractional pennies, it is big enough to handle the entire global GDP. However, when we started Goldenseal, 64-bit integers were poorly supported, and memory was much tighter. As a result, we developed a CMoney class that uses 32-bits for dollars, and 16-bits for pennies and fractions (48-bits total, or 6 bytes). It took a few programmer-months to get CMoney working. Now that the code is reliable, there is little reason to change it (and it does save about 5% on file size). Maybe if hyperinflation hits or we run out of things to do in the version 7.0 update, we’ll make the switch.

The biggest impact of 32-bit vs 64-bit is in file addressing.  A 32-bit OS maxes out with 4 gigabyte files, while a 64-bit OS can have huuuge files.

As part of our database overhaul last year, we already converted to 64-bit addressing in Goldenseal Pro. It adds approximately 20 bytes per record, which makes files about 10% bigger. The change doesn’t mean much for Goldenseal users, who rarely need more than 100 megabytes (.1 gigabyte). However, we use the same database code for an app that stores genome data, and it definitely will help there. The human genome is 3 billion base pairs, or 750 megabytes compressed. Start doing comparative genomics, and it’s easy to get files in the terabytes.

Kilobytes = 1 thousand bytes. Megabytes = 1 million.  Gigabyte = 1 billion. Terabyte = 1 trillion.

Dennis Kolva
Programming Director
TurtleSoft.com

Since we develop estimating software, you’d think it would be easy for us to estimate the time it takes to build our own products. Unfortunately, that is rarely the case.

For one thing, software features are less tangible than roofing or cabinets, so it’s hard to turn them into unit costs. Also, most of our work is new invention and done only once, so past history doesn’t help much. It’s kind of like being a kitchen and bath remodeler, but calculating the cost to build a boat. App development is also a design-build process, and we often proceed without firm specs. It’s as if the only blueprints for the boat are ‘don’t sink’ and ‘catch fish’.

Years ago, Turtle Creek Carpentry had the same problem for construction estimates. We started by calculating actual costs on all past jobs, so we could make ‘comparison estimates’ for similar projects. The same approach ought to help for Goldenseal Pro. We just looked at our labor data, and calculated the following (all values in programmer-months):

Goldenseal first prototypes:  6 p-m
Goldenseal 1.0 (Mac): 108 p-m
Goldenseal 1.0 (Windows): 12 p-m (subbed)
Goldenseal 1.01 to 4.96 feature & bug fix updates: 72 p-m
Goldenseal Multi-User: 6 p-m (partly subbed)
Goldenseal OS 9 to OS X (Mac): 3 p-m
Goldenseal PPC to Intel (Mac): 2 p-m
Goldenseal Multi-User for Intel (Mac): 1.5 p-m (subbed)

Goldenseal 4.96 already has about 18 programmer-years of work in it.  It’s a fairly big app.

Based on the last three Mac updates, we seriously thought that going from Carbon to Cocoa (Mac) and from QuickTime to MFC (Windows) would only take a few months apiece.  Many subcontractors agreed, but we all turned out to be too optimistic. This update is harder.

My personal ‘gut feeling’ estimate was that Goldenseal Pro would be ready by late 2017, but it seemed worth pausing for some calculations, to get a more accurate prediction. The best way to estimate more accurately is ‘divide and conquer’: split projects into smaller bits, and calculate those individually. It’s the basic approach behind Goldenseal estimates. The strategy works well even if the data is imperfect, because errors tend to average out. ‘Divide and conquer’ is also a common approach for sorting algorithms, house cleaning, conquering Gaul, etc.

We divided past Goldenseal programming into 23 categories, then divided those into Goldenseal 1.0, updates from 1.0 to 4.96, and work so far on Goldenseal Pro. It involved a lot of guessing, but probably no worse than our first construction estimating spreadsheet (which became MacNail 1.0).

The most important numbers are the time spent so far on Goldenseal Pro. There has been 13 months of non-GUI work: replacing obsolete code, converting from 32-bit to 64-bit, rewriting the database, etc. The GUI  (graphic user interface) has taken 8.5 months so far, but a lot of that time was spent learning Objective-C, Cocoa and MFC.  It probably has been only 5 or 6 months of actual programming.

We divided that work into categories, estimated % completion for each, then calculated time remaining. Excel currently predicts 11.08 months until a final release, then another 4.75 months for loose ends like networking, app store listing and Custom Layouts.  It’s very guessy. The spreadsheet has predicted anywhere from 7 to 13 months, depending on the assumptions made.

Regardless of how long it takes, we are committed to finishing Goldenseal Pro.  We owe it to our long-term users, which includes our own staff! Switching our own businesses to another system would be almost as much work as finishing Goldenseal Pro. The software includes 7 or 8 programmer-years of well-tested C++ business logic that doesn’t need updating, and it’s worth leveraging that investment.

Dennis Kolva
Programming Director
TurtleSoft.com