Goldenseal Pro Database (June 28)

Goldenseal stores a lot of data. Many users have more than 100,000 records in their company file, and some have millions. The code that keeps track of everything is an important part of our ‘back end’. It needs to access records quickly, and store them reliably on disk.

For the original Goldenseal, we licensed an object database called NeoAccess. It ran on both Mac and Windows, and was very popular in the 1990s. Unfortunately, the source code had serious bugs that were never fixed, and we ended up rewriting a lot of it (especially after its developer folded their business).

For Goldenseal Pro, we looked at a couple dozen open-source and commercial database libraries, both object and relational. Using a 3rd-party library has its advantages: someone else did all the work, maybe they are an expert, and maybe they will keep it updated. However, as we found with NeoAccess, other people’s code also has disadvantages.  For one thing, it’s usually easier to fix one’s own bugs and design flaws. Every library also has a learning curve, plus quirks and limitations. There is no perfect way to manage a database.

After a few trials, we finally decided to write our own database code for Goldenseal Pro.  Since we already rebuilt half of someone else’s database, designing a whole new one didn’t seem too risky. We kept some of the design from NeoAccess, but replaced the parts that didn’t work well.

The main thing any database needs is a list of where each record is in the file. NeoAccess used a ‘binary tree’, which is theoretically the fastest way to find things, using the least amount of memory. However, modern computers are designed to move relatively big blocks of memory, so it makes sense to have a tree that is ‘wide’ (with more records in each branch, and fewer branches). A wide tree gives faster record access on modern machines, and also uses simpler, cleaner code.

We wrote the first part of the database last summer, then tested and debugged it over the winter. The past few months we finished the remaining ‘tree’ code, to handle larger numbers of records.

If you want to see what C++ source code looks like, here is our function that looks for an empty spot to index a new record, and creates a new index if the current one is full:

SInt64 DB_RecordIndex::AddObjectToIndex(const DB_PersistentObject *aObject, CTCS_FileStream *inStream)
{
    TCS_FailNILMsgID(aObject, errID_BadObject);
    TCS_FailNILMsgID(inStream, errID_BadStream);
    SRecordIndexInfo info;
   
    if (mTreeLevel > 0)
    {                   // add to the last subindex
        DB_RecordIndex *lastIndex = FetchLastSubindex();
        DB_ObjectWatcher watcher (lastIndex);
        SInt64 subMark = lastIndex->AddObjectToIndex(aObject, inStream);
        if (subMark != 0) return subMark;    // was added to subindex
       
        if (mRecordArray.GetAvailableSpace() > 0 && !IsFilled())
        {            // still room for more subindexes here, so add one
            return AddSubindex(aObject, inStream);
        }
        else if (IsRootIndex())
        {            // Need to increase tree level by one
            mRecordArray.FetchFirstItem(info);
            subMark = ShiftItemsToSubindex(inStream, info.startID);
            IncrementTreeLevel();
            return AddSubindex(aObject, inStream);
        }
        else
        {         // subindex is full. Pass back to parent
            SetIsFilled();
            return 0;
        }
    }
    else if (mRecordArray.GetAvailableSpace() > 0 && !IsFilled())
    {            // base level index with room here.
        info.fileMark = aObject->getFileMark();
        info.itemID = aObject->GetDBID();
        mRecordArray.Append(info);
        MakeDirty();
        return info.fileMark;
    }
    else if (IsRootIndex())
    {          // the array is full, so it’s time to become a new parent!
        mRecordArray.FetchFirstItem(info);
        ShiftItemsToSubindex(inStream, info.itemID);
        IncrementTreeLevel();
        return AddSubindex(aObject, inStream);
    }
    else return 0;            // this base index is full.  Pass back to the parent
}
 
There are plenty of quirks to managing a database. We need to keep track of empty spaces, so the file stays compact. We also store each record’s location in two different places, so it will be possible to fix a corrupted index (something that NeoAccess could not do).
 

To keep the database code simpler, we take advantage of the fact that record IDs are always increasing, so we can just add them at the end and have a sorted list. We also just leave empty spaces when you delete a record, rather than rearrange the whole tree. Our primary goal is to make the code reliable and easy to maintain. So far, that approach has been successful.

Dennis Kolva
Programming Director
TurtleSoft.com

Goldenseal Accounting- Audit Trail (June 19)

One of the lesser-known features of Goldenseal accounting software is the Audit Trail, tucked away inside the View-Security menu. It tracks when each record is created, changed or deleted. If multiple users sign on with different passwords, it also shows who did what. You can control exactly what gets logged with Options-Preferences-Data Entry. For security reasons, the Audit Trail can’t be changed, so it creates a permanent record of what happens to your business records.

The Audit Trail is a valuable accounting feature for larger companies. You can look back at all changes made to records, which makes it much easier to investigate possible fraud (or just carelessness). We also use it occasionally when diagnosing posting problems in user files.

On the flip side, the Audit Trail adds many records to the file, and increases file size by 10% to 30%. Many of our users do their own data entry (or have a relative do it), so it is unlikely they will ever need the trail for a security investigation. The feature is obscure enough that most people don’t even know it is running.

The conversion to Goldenseal Pro is a chance to redo some of our early design choices. After 17 years of user inputs, we have a much better picture of what is actually useful. The Audit Trail is one of those things we can probably do better.

It doesn’t make sense to remove the Audit Trail, because it helps some users, and it already works fine. However, when we import data from older files, we probably will skip the old Audit Trail entries. If they are ever needed for an investigation, they are still available in the old file. If moving them to the new file is important to you, please let us know.

We may also change our default settings, and leave the Audit Trail off for most users. If you don’t use multiple passwords, then you probably don’t need it.

Another option is to go in the opposite direction, and expand the Audit Trail into a full ‘journaling’ system. Some software programs (e.g. Photoshop) store a list of all changes made, so you can go back with Undo multiple times, or selectively apply some past changes but not others.

To make a journaling system work for accounting, the Audit Trail would end up using more than half of the file size. Having multiple undo would add complexity to our posting code, and it may have security issues.

We won’t change the Audit Trail by much in the first release of Goldenseal Pro, but we may revise it in a future update. If you have often wanted to undo deletions made days ago, please let us know. That is probably the best advantage we’d get from a more complete Audit Trail.

Dennis Kolva
Programming Director
TurtleSoft.com

 

64-Bit vs 32-Bit (June 8)

Apple recently announced plans for High Sierra, the next OS update, due in Fall 2017. It will continue to run 32-bit Carbon applications like Goldenseal, but the phase-out of 32-bit support starts next year. 32-bit apps will still run under the subsequent, yet-unnamed Fall 2018 update, but it will complain about them.  Starting in 2019, 32-bit apps probably won’t launch at all.

Goldenseal Pro uses Cocoa, which is 64-bit. It will run fine on current and future Mac OS versions. It looks like we will finish it well before Fall 2018, so there will be at least a year of cushion.

The current Goldenseal is 32-bit. It runs fine on the current Mac OS (Sierra), and will be OK with the next two after that, and anything older. But it will have problems with new hardware starting in Fall 2019.

It took until August 2016 for Microsoft Office to switch from 32-bit to 64-bit. Over the next couple years, we will see how many other Mac programs (e.g. QuickBooks Desktop) can build 64-bit versions in time. Going from OS 9 to OS X killed off about 75% of Mac software companies, and moving from PPC to Intel zapped another 10%. My guess is that the carnage this time will be similar. Converting from Carbon to Cocoa is not easy.

If you are confused by the meaning of 32-bit vs 64-bit, here is some background. 32-bit integers can handle any whole number from 0 to 4 billion, while 64-bit integers go up to 16,000,000,000,000,000,000 (16 quintillion).

For accounting, 4 billion pennies is $40 million, which is probably good enough for residential construction. However, some unit costs need fractional cents, some amounts are negative, and some countries use currencies with values much smaller than $1 US. That means 32-bit numbers are too small for estimates and accounting.

If starting today, we would probably use 64-bit numbers for money amounts. Even with fractional pennies, it is big enough to handle the entire global GDP. However, when we started Goldenseal, 64-bit integers were poorly supported, and memory was much tighter. As a result, we developed a CMoney class that uses 32-bits for dollars, and 16-bits for pennies and fractions (48-bits total, or 6 bytes). It took a few programmer-months to get CMoney working. Now that the code is reliable, there is little reason to change it (and it does save about 5% on file size). Maybe if hyperinflation hits or we run out of things to do in the version 7.0 update, we’ll make the switch.

The biggest impact of 32-bit vs 64-bit is in file addressing.  A 32-bit OS maxes out with 4 gigabyte files, while a 64-bit OS can have huuuge files.

As part of our database overhaul last year, we already converted to 64-bit addressing in Goldenseal Pro. It adds approximately 20 bytes per record, which makes files about 10% bigger. The change doesn’t mean much for Goldenseal users, who rarely need more than 100 megabytes (.1 gigabyte). However, we use the same database code for an app that stores genome data, and it definitely will help there. The human genome is 3 billion base pairs, or 750 megabytes compressed. Start doing comparative genomics, and it’s easy to get files in the terabytes.

Kilobytes = 1 thousand bytes. Megabytes = 1 million.  Gigabyte = 1 billion. Terabyte = 1 trillion.

Dennis Kolva
Programming Director
TurtleSoft.com

 

Goldenseal Pro Completion Estimate (May 31)

Since we develop estimating software, you’d think it would be easy for us to estimate the time it takes to build our own products. Unfortunately, that is rarely the case.

For one thing, software features are less tangible than roofing or cabinets, so it’s hard to turn them into unit costs. Also, most of our work is new invention and done only once, so past history doesn’t help much. It’s kind of like being a kitchen and bath remodeler, but calculating the cost to build a boat. App development is also a design-build process, and we often proceed without firm specs. It’s as if the only blueprints for the boat are ‘don’t sink’ and ‘catch fish’.

Years ago, Turtle Creek Carpentry had the same problem for construction estimates. We started by calculating actual costs on all past jobs, so we could make ‘comparison estimates’ for similar projects. The same approach ought to help for Goldenseal Pro. We just looked at our labor data, and calculated the following (all values in programmer-months):

Goldenseal first prototypes:  6 p-m
Goldenseal 1.0 (Mac): 108 p-m
Goldenseal 1.0 (Windows): 12 p-m (subbed)
Goldenseal 1.01 to 4.96 feature & bug fix updates: 72 p-m
Goldenseal Multi-User: 6 p-m (partly subbed)
Goldenseal OS 9 to OS X (Mac): 3 p-m
Goldenseal PPC to Intel (Mac): 2 p-m
Goldenseal Multi-User for Intel (Mac): 1.5 p-m (subbed)

Goldenseal 4.96 already has about 18 programmer-years of work in it.  It’s a fairly big app.

Based on the last three Mac updates, we seriously thought that going from Carbon to Cocoa (Mac) and from QuickTime to MFC (Windows) would only take a few months apiece.  Many subcontractors agreed, but we all turned out to be too optimistic. This update is harder.

My personal ‘gut feeling’ estimate was that Goldenseal Pro would be ready by late 2017, but it seemed worth pausing for some calculations, to get a more accurate prediction. The best way to estimate more accurately is ‘divide and conquer’: split projects into smaller bits, and calculate those individually. It’s the basic approach behind Goldenseal estimates. The strategy works well even if the data is imperfect, because errors tend to average out. ‘Divide and conquer’ is also a common approach for sorting algorithms, house cleaning, conquering Gaul, etc.

We divided past Goldenseal programming into 23 categories, then divided those into Goldenseal 1.0, updates from 1.0 to 4.96, and work so far on Goldenseal Pro. It involved a lot of guessing, but probably no worse than our first construction estimating spreadsheet (which became MacNail 1.0).

The most important numbers are the time spent so far on Goldenseal Pro. There has been 13 months of non-GUI work: replacing obsolete code, converting from 32-bit to 64-bit, rewriting the database, etc. The GUI  (graphic user interface) has taken 8.5 months so far, but a lot of that time was spent learning Objective-C, Cocoa and MFC.  It probably has been only 5 or 6 months of actual programming.

We divided that work into categories, estimated % completion for each, then calculated time remaining. Excel currently predicts 11.08 months until a final release, then another 4.75 months for loose ends like networking, app store listing and Custom Layouts.  It’s very guessy. The spreadsheet has predicted anywhere from 7 to 13 months, depending on the assumptions made.

Regardless of how long it takes, we are committed to finishing Goldenseal Pro.  We owe it to our long-term users, which includes our own staff! Switching our own businesses to another system would be almost as much work as finishing Goldenseal Pro. The software includes 7 or 8 programmer-years of well-tested C++ business logic that doesn’t need updating, and it’s worth leveraging that investment.

Dennis Kolva
Programming Director
TurtleSoft.com

Goldenseal 4.96 Released (May 24)

We just released the latest update to the current Goldenseal apps. If your download access is current, follow these steps to get it:

1. Go to our shopping cart.
2. Click the My Account link at top left.
3. Sign in with your Email address and the password you used to create the account.  The password is case-sensitive (capitals & lower case matter!).
4. Click the Completed Orders link.
5. Click the Download Files link, and save the download on your hard drive.
6. You may need to decompress the download file, if your browser doesn’t do that automatically.
7.  MACINTOSH– drag the Goldenseal program (tree icon) into your Applications folder. WINDOWS– run the installer.

If you paid for updates on or after Oct 1, 2014 and it tells you your access expired, please contact us. We will extend your update privileges.

Goldenseal 4.96 includes many small bug fixes, plus fixes for a couple of not-so-small bugs in the Find command, and range selections in Reports.

For our international users, we also added an alternate way to set the currency symbol (we also will add that feature to Goldenseal Pro).

The current Goldenseal code is growing old, as are the machines we use to build it. Our main focus is now on completing Goldenseal Pro, so this may be the last update we make to the current app. However, if a serious problem turns up, we will release another minor update.

Dennis Kolva
Programming Director
TurtleSoft.com

 

Goldenseal 4.96 (May 16)

We’ve taken a short break from work on Goldenseal Pro, to finish a minor update for the current version of Goldenseal. No new features, but it includes many small bug fixes written since the last update. It’s going through stress testing now, and will be released next week.

To build Goldenseal 4.96, we can’t use anything newer than Snow Leopard (10.6) for Mac, and Vista for Windows. In fact, the final part of the Windows build requires Windows XP or older. Fortunately, we were able to dust off enough ancient equipment (and rusty brain cells) to make it work.

Dennis Kolva
Programming Director
TurtleSoft.com

 

Windows Main Window (May 6)

Running a project-based business is complicated, and we spend a lot of time thinking about how to make it easier.

The original Goldenseal design was inspired by FileMaker. There is an area on the right showing one record, and a smaller control area on the left with a ‘book’ for navigation (and a few other controls).  Each type of record has its own window: Customers, Estimates, Material Purchases etc.  Additional windows show reports, Pay Bills, Reconcile, Custom Layouts, and other commands. It’s easy to end up with dozens of windows scattered about the screen.

Goldenseal Pro is switching to a single-window interface. Instead of pull-down menus at the top of the screen, it has an outline (aka tree) view on the left that organizes all records, reports and actions into folders. Records still appear on the right as usual. Those two parts of the interface are already working on both Mac and Windows, and it’s a lot tidier than what we had before.

On the Mac, we have a control area at top right with the navigation controls (the ‘book’ and other controls that used to be on the left). We can easily pop in other controls: for Find command, reports, action dialogs like Pay Bills, and Custom Layouts.  Everything in Goldenseal Pro can now happen in the main window, with tabs to move between recent items. If you like clutter, you can still use separate windows, but they aren’t essential any more.

On Windows, we are still experimenting with the control area.  So far, there are 3 different options: a wide ribbon bar, a simpler tool bar, or a detachable/dockable tool bar.  None of them look exactly like the Mac layout, but the dockable bar is closest.  We are still testing them, and haven’t yet decided which to use.

One last bit of unfinished interface is the way we handle breakdown tables. They are bulky and optional, and we want a better way to show and hide them. The Mac version now has drawers that pop out from the side or bottom of the window, to show optional breakdowns.  There is nothing equivalent in Windows, so we are still fiddling with that. 

The MFC library comes with about 200 sample projects, demonstrating different bits of interface. We are building and trying all of them again. Maybe we will discover something that works better for the control area and tables.  It will probably be another week or two before we settle on a final design.

Inventing new interface is frustrating and slow. We are also working on Mac interface whenever we want to see tangible results, quickly.

Dennis Kolva
Programming Director
Turtlesoft.com

 

 

 

 

Goldenseal Pro Alpha 1 for Mac (Apr 27)

The first Goldenseal Pro Alpha is now available for Mac. It requires OS 10.11 (El Capitan) or 10.12 (Sierra). It will not run on older Mac OS versions.

To run the alpha version, follow these steps:

  1. Click the link above to download the software.
  2. It is compressed, so you may need to double-click to decompress it, if your browser does not do that automatically.
  3. Open the GS-Pro-Alpha1 folder, and double-click on the Goldenseal-Pro-Alpha1 app.
  4. Choose Open Business File from the File menu, and open Test-File.  The outline on the left contains all your accounts and transactions. Open the folders and click to see one or more types of records. They will appear in the main part of the window, with tabs at the bottom to navigate between them. You can also use separate windows like the current Goldenseal (right now the only one that works is Prospects, in the Records menu at the top of the screen).
  5. This is an ‘alpha’ version, which means that many things do not work yet. You are welcome to explore the app but some things will do nothing. Some may give error messages or crash the program. You probably will hear random beeps (we add them to unfinished code so we don’t forget). No need to report problems in this version.

To check how well your company file converts to the new format, follow these steps:

  1. Make a copy of your company file.
  2. If still running Goldenseal Pro, choose Quit from the Goldenseal Pro menu.
  3. Open the GS-Pro-Alpha1 folder, and double-click on the Goldenseal-Pro-Alpha1 app.
  4. Choose Open Business File from the File menu, and select the copy of your company file.
  5. Goldenseal Pro will ask you if you want to convert your old file, then will ask you where to save the new file, and what to name it.
  6. The conversion may take a minute or two.  When it finishes, choose Quit from the Goldenseal Pro menu.  If the conversion gives errors or can’t finish, please let us know so we can improve it before the final version.

Dennis Kolva
Programming Director
TurtleSoft.com

Goldenseal Pro Progress Report (Apr 18)

The Mac version of Goldenseal Pro has come to a good, temporary stopping point.

It shows most fields in the data entry screens, but we are still working on  breakdown tables.  They are by far the most complicated part of the interface. The Cocoa half of that code is ready, but the new tables are very different from the PowerPlant tables we used in regular Goldenseal.  We can probably delete 90% of our existing table code, and make a much simpler connection.

However, before we do anything that drastic, it’s time to see how tables work in the Windows version, using their MFC library.  So, we will switch to Windows programming for a while, and get it caught up to the Mac version again. Then we can design the new table classes so they work well on both platforms.

The Mac code is ready for an alpha version. We’ll release it next week, after we do more testing.  Nothing you can use to run your business yet, but it’s a chance for users to see (and comment on) the new interface.  Also, you can use it to convert your company file to the new format, and see how well the translator works.  That way we can see what breaks on real data, and fix it before the final release.

Dennis Kolva
Programming Director
TurtleSoft.com

 

Turtlesoft 30th Anniversary (Apr 10)

30 years ago this week, Turtlesoft sold its first copy of MacNail. It was a set of estimating templates for Macintosh Excel 1.03, spruced up from the spreadsheets we used for Turtle Creek Carpentry. MacNail came on one floppy disk. The label had a nail head that we whipped up in MacPaint, and printed at 72dpi dot matrix.

As we overhaul the Goldenseal source code this year, it’s fun to think about how much has changed since 1987.

Back then, the Mac Plus was an expensive, high-end machine that came with 1 megabyte of RAM. That was generous for the time (MS-DOS maxed at 640K), but programmers still had to be frugal. Our staff spent many hours thinking of ways to shrink the MacNail spreadsheets by 1 or 2K. It was a big deal, deciding whether to use 1 or 2 or 4 bytes for a value.

These days, programmers can count on a few 1000 megabytes of RAM. It’s often reasonable to use an 8-byte value, just in case that number gets too big in some weird future world. We wouldn’t want Goldenseal to fail when you start work on your 2-billionth project, right?

Back then, it took 3 minutes just to open the MacNail spreadsheet file.  Progress bars and spinning cursors were vital, to let people know the computer hadn’t frozen.  Having a big progress bar helped, so you could do something else across the room, and see when it was almost done.

These days, processors are 100x faster. This week we removed all the code that spins cursors, because modern machines are so fast they don’t need a rotating clock or hourglass. We kept the progress bars, but they’ll be a subtle little bar at the bottom, rather than a full-size window. There aren’t many things that will take more than a second or two.

When we first began programming Goldenseal in the mid 1990s, it took 7 minutes to rebuild the source code, even if just one word changed. Large changes had to run overnight, and complete project rebuilds took a whole weekend.

These days, Goldenseal rebuilds in 15 to 150 seconds, depending on how much changed. That means many more cycles of writing-and-testing per day. Practically speaking, programming is much less draining when you can make a change and just try it out, rather than triple-checking to make sure the logic is correct before daring to build it. Programmers can think more about large design issues, and less about punctuation and typos.

Back in the 80s, programming languages were still young. They broke a lot, and required complicated coding to accomplish anything.  Just drawing a button on the screen could easily take 30 lines of rather complex code.

These days, C++ and other languages have matured.  They’ve added well-tested libraries, so it’s possible to write code that runs fast, and won’t crash. There is still plenty of learning curve for each new library, but the Internet makes it relatively easy to find tutorials and answers.

Meanwhile, Goldenseal Pro is just starting to fill data into breakdown tables. Half the work seems to be throwing out all the old code that we don’t need any more.

Dennis Kolva
Programming Director
Turtlesoft.com