Goldenseal Pro Tables & Progress (Feb 13)

You probably have had one of those days when you go to the garage to fetch a screwdriver, but the light is burned out. So you head to the kitchen for a new bulb, but they’re all gone. So you drive to the store to pick up more, but while you are there, you accidentally lock the keys inside your vehicle. You try to pry the door open with a credit card, but what you really need is a screwdriver.

Anyhow, it has felt that way with the tables in our new accounting software. They are a major part of the interface, showing rows and columns of info much like a spreadsheet.

Back in December, breakdown tables were just starting to display in a cute pop-out drawer on the side. It filled in the correct columns, but needed an overhaul to the layouts before proceeding further.  Before doing that, it made sense to get all of the mode-switching to work for everything, not just layouts. That led to still further rabbit holes.

After 7 weeks of digressions, we are now back in the garage (metaphorically speaking) to fetch a screwdriver for Goldenseal Pro.

Since tables are present in all 6 basic modes, it makes sense to do them all at the same time. Print and report tables are easiest.  They are now filling in data, and looking pretty good. We’re currently working on the small tables found in lists, and some accounts. Then we’ll futz with layouts, then finish off the breakdown tables found in estimates, material purchases, and other business transactions.  Last and most difficult are the ‘action’ tables in Reconcile, Pay Bills and similar commands. The current app has an entirely different way to create them. Getting it to be more similar to the other tables has always been on the to-do list. That will happen soon.  It will make setup easier for us, and will allow users to modify them (slightly).

In an attempt to get a completion estimate for Goldenseal Pro, we looked at the draw schedules in the programming contracts from a couple years ago. Unfortunately, those were based on which menu commands worked. The actual project is proceeding more ‘horizontally’ than ‘vertically’. The current work on tables is spread out in 3 different project phases. It’s a lot like the difference between a ‘room by room’ estimate, and an estimate based on categories of work. They both get to the same place, but it’s hard to compare them.

About half the work we have done so far has been on major improvements to the basic interface. Those don’t show on the draw schedules at all.

The original plan was to start with a duplicate of the existing app (because easier to spec it that way for subcontractors). Once we had a working version, we would have made improvements in later updates. However, now that our staff is programming the whole project, it’s far more efficient to design and build a better interface, right from the beginning.  Doing it that way is slower to get to the first release, but faster to get to a really good version.

Dennis Kolva
Programming Director


Estimating Accuracy- Part 2 (Feb 6)

Our estimating software has its roots in an Excel spreadsheet started in Winter ’86. It was just a list of construction items and prices, with formulas to do math on them.  The first unit costs were the “rules of thumb” that we already used for our construction estimates.

The spreadsheet improved gradually, as we duplicated it for each new job. Its first big test was a $100K remodeling project in a log house that summer (about $250K today). There were many complications, so the estimate ran to 200+ line items. The work was so unusual that nobody else bid on it. Our price was significantly over their planning budget, but we got it anyhow.

It was the biggest project ever, for Turtle Creek Carpentry. We were nervous about the size, so we built another new spreadsheet to track expenses. It later grew into our current accounting software.

Real-time job costing was a roller-coaster. For the new log garage, the estimate had a *12/5 formula to convert square feet of wall to lineal feet of log. Unfortunately, it was zapped by a later copy/paste, so expenses were suddenly $4,000 higher than expected. Then, oops, interior paneling was listed twice. That put it back close to estimate.  Other errors turned up, but in the end, final costs matched the estimate within 2%.

The lessons learned from that job still apply to our current construction estimating software.

1). it’s possible to get surprisingly good accuracy, even when the data is imperfect. The trick is to split the project into small pieces, and calculate each separately. The small errors will nearly always cancel out. It’s important to avoid systematic errors (everything wrong in the same direction). However, that is much easier than getting everything perfect.

2). Having a detailed breakdown of every item is impressive. Later on, we won bids even as high bidder, just because the list was so complete. Detailed printouts are an excellent marketing tool. In those days, it wasn’t hard to look better than everyone else. Now, it’s almost a requirement.

3). Matching actual expenses to the estimate really helps improve accuracy. The better you track job costs, the more accurate your estimates become.

4). It’s important to group things. That first big estimate had a huge list of items with no subdivisions. It was overwhelming, and hard to see errors. We overlooked the doubly-listed paneling. So did the architect, and the client. Seeing subtotals for different categories is a good ‘sanity check’ that we added to later estimates. As a bonus, it’s much easier to track job costing by category, rather than by item. Then you don’t need to ask where every 2×4 went.

5). Don’t rely on hidden math. As the spreadsheets and software evolved, we gradually found ways to make the cost calculations more visible and obvious. It’s an ongoing process that we still are improving.

We seem to be thinking more about estimating basics lately, as work progresses on Goldenseal Pro. Some of that is because the rewrite stirs up ideas for how to improve the software. Some is because we still aren’t very good at estimating the time it takes for our programming work. Ideally, it would sure be nice to calculate in advance, exactly how long each new product or upgrade will take.

Anyhow, 4 of the 6 basic modes in Goldenseal Pro are now functioning moderately well, and the 5th is getting there. Still many small details to finish, but it’s progressing at a steady pace.

Dennis Kolva
Programming Director

Goldenseal Pro Colors & Resources (Feb 1)

We just added color lookup tables to Goldenseal Pro (the coming update to our accounting and estimating software). They are the last of the resources that needed updating.

Color lookup tables (aka CLUTs) are pretty much a vestige of 1990s computing. To save space, 8-bit colors were common back then. To choose an 8-bit color, you select from a palette of 256 choices, rather than click on a full 32-bit color wheel. 

In the current version of Goldenseal accounting software, we use an 8-bit color palette for text and graphics in data entry layouts, printed forms and reports.  The system CLUT is mostly bright colors, so we replaced it with a custom version that includes more pastels (for field backgrounds) and dark colors (for text and borders).

32-bit color is close to the limits for human vision, and it’s worth having for photographs and videos. However it’s overkill for tiny color blobs like text and field borders. For those, 8-bit is perfectly fine. For example, Microsoft Excel shows a limited palette of colors for text and cell backgrounds. It’s easier to use than the full color wheel.

However, these days it is easier to program the standard color-choosers, rather than write custom code to show a palette. Goldenseal Pro still needs to support 8-bit colors so existing layouts read in correctly, but we will convert them to 32-bit colors when saved in the new format. It does mean you’ll have more choices for colors, especially if you want bright ones.

As for resources, they are a design feature that was first made common in the early Macintosh. Resources store program data that can be changed without any coding. It means that programmers (or users) can switch text to a different language, change field details, or add keyboard shortcuts to menus, without needing to rebuild the source code.

Mac OS 9 and earlier used a program called ResEdit to modify resources. It let users “hack” app resources, so they could change certain details of their software. We also used resources to set up most of the interface for Goldenseal.

Unfortunately, there is no equivalent resource editor for Windows, nor for current Mac OS. To update resources in the current version of Goldenseal, we actually have to go back and run Mac OS 9 on ancient hardware.

Goldenseal Pro uses modern Cocoa and MFC resources. That means they are user-accessible again, in both Mac and Windows version. Users or developers can translate the text to a different language, fiddle with colors, or tweak a small detail for a data field. Most of the resources are simple text files, but some use XML (eXtended Markup Language, similar to the HTML language used for websites).

It takes some effort to change resources, but that is how it should be. We don’t want it to be too easy for inexperienced users to accidentally mangle something important within the software.

The update to Goldenseal Pro is a long slog, but it does bring significant improvements. Better colors and resources are just a small part of that.

Dennis Kolva
Programming Director


Estimating Accuracy- Part 1 (Jan 25)

A couple weeks ago, a support call came in from a potential buyer of our construction estimating software. They were confused, because the math seemed wrong for one material line item in an Assembly. When we checked it in our copy of the Sample Company File, it definitely was wrong. The line item total should be equal to unit cost times quantity plus waste factor, and it wasn’t.

It took a while to investigate this problem, and I’ll get to that later.

First of all, you can do the following to fix things:

  1. Choose Cost Items from the Costs menu.
  2. Choose Find All from the Edit menu.
  3. Choose Replace All from the Edit menu.
  4. In the Replace popup field, choose Adjust by Percentage.
  5. Type in a small percentage like .0001%. Or use a bigger number if you want to account for inflation.
  6. Click OK.
  7. Goldenseal will recalculate all Assemblies, and redo all the math correctly.

Actually, this adjustment fixes everything except for four items: all for unskilled labor in closet shelving. We still need to investigate why that is off. Right now, even after the Replace All update, it still calculates $6 to $12 too low per closet. Everything else is fine. There may be a reason why closets calculate that way, but if so, nobody here remembers why.

We did not find anything wrong in the code that calculates unit costs in Assemblies and Estimates. You also won’t see errors for materials if you’ve changed their purchase price, or if you used a percentage adjustment any time in the past. Those force the Assembly line items to recalculate.

Right now, we still don’t know exactly how the numbers got off in our construction estimating data. We have tried hard to duplicate the problem, but so far it never goes wrong in real time.

The Sample Company File has been around since the early, pre-release versions of Goldenseal. That’s why most of its contents have dates in 1998. We used it for daily testing. Back then, the file format changed frequently, so we exported the sample data to a text file, then imported it back into newer versions. It was faster than retyping the fake business records.

Most of the construction price data came from our older software (MacNail, HyperEstimator and BidMagic). Converting those to Cost Items and Assemblies required some hacking and export/import. As we improved the estimating process, there was more hacking and export/import. Then we moved data between the sample and starter files, which meant even more hacking and export/import. Lots of opportunities for errors to creep in.

Over the years we have discovered and fixed a few dozen unit cost problems. Users have also reported a few. I suspect that the rest of the data bugs have lingered there unnoticed since the late 90s or early 00s.

The errors are almost exactly balanced between positive and negative. In real world estimates, they probably canceled out. Most are just pennies, probably from rounding errors. The big ones are in obscure materials that we don’t update very often. Anyone actually using them probably entered a new price, which fixed the math.

Still, we sell data as well as software. It should be perfect. We are working on that now.

While testing the construction assemblies in our estimating software, it also seemed time to update the material pricing. In general, wood pricing is up, concrete is way up, while hardware and manufactured items are down slightly. There are still more prices to gather, but updated files will be available for download soon.

Digging around in our construction cost database stirred up some other issues that are worthy of discussion. I’ll cover them in a future post.

Dennis Kolva
Programming Director





2018 Payroll Tax Withholding Update (Jan 19)

We just released 2018 US tax tables for Goldenseal payroll software. This is our latest release date, ever.

IRS issued their preliminary 2018 tax formulas on Jan 11.  About a dozen states base their own tables on the federal ones. As of the 19th, all but Missouri, North Dakota and Vermont managed to catch up.  We are planning on another release whenever IRS publishes final numbers (and remaining states publish their info).

Using the 2018 withholding tables, there won’t be much change in payroll withholding amounts, for most employees.  The biggest change for most of our users is whether to incorporate, and take advantage of pass-through loopholes.

BTW, tax rates for rich people have really changed over the years. This year’s decrease was relatively minor.

Modern income taxes started after the 16th Amendment in 1913, with a top marginal tax rate of 7%. The top rate jumped to about 70% from 1917 to 1921, then declined to 25% from 1925 to 1931.

Great Depression rates were 63% to 79%. During World War II they gradually increased to 94%, then hung out in the 86% to 91% range until 1963. Through the 1970s they were about 70%. They shrank to 28% during the Reagan 1980s, then ranged between 31% and 39.6% since then.

Dennis Kolva
Programming Director



Goldenseal Pro Progress (Jan 11)

If Goldenseal Pro were a construction project, framing and utility rough-ins would now be done (mostly), and drywall would be starting. It’s that satisfying period when the interior transitions from a 2×4 maze, to actual rooms. Or, from a software point of view, the code is starting to look like a real estimating/accounting program.

Goldenseal’s software interface has six functional modes: data entry, finding records, reports, custom layouts, printing, and accounting actions. The first two are now working, and we just started on reports.  The six modes have very similar programming, so we will continue on them in that order until they’re all functional.

Work is progressing quickly enough that we probably will focus on just the Mac version for a while. The goal is to complete enough that we can start using it to run TurtleSoft. That’s a huge milestone, since we can then test daily under real-life conditions, with real business data. In the software world, this is known as “eating your own dog food”. Best estimate is a month or possibly two, to get that far.

After that, it shouldn’t take long for the Windows version to catch up. Working with the MFC library on Windows is easier than Cocoa on Mac, and the prep work in our existing code will already be done.

Our programming staff has definitely ascended far up the “learning curve” for the new language and libraries needed to build Goldenseal Pro. We are zipping right along!

Dennis Kolva
Programming Director



Goldenseal Pro Navigation (Jan 4)

Goldenseal currently uses pull-down menus to navigate through its data entry screens, reports and commands. With more than 300 menu choices, it’s easy to get lost or overwhelmed. Our accounting software risks being the virtual equivalent of a desk piled high with papers and whatevers.

Both Microsoft and Apple have improved their developer libraries enormously, in the past 15 years. They now include much better interface tools. We are using them to improve navigation for users, so it’s easier to prepare estimates and run the business.

One big improvement is the outline view, which organizes stuff into folders and subfolders. It’s how you navigate through files in both Mac and Windows. Goldenseal Pro uses an outline view to replace about half the former pull-down menus. It allows everything to happen in a single window, with folders and tabs to navigate. The outline is particularly handy when using the full accounting/job costing/payroll software, since that is more complex.

Sometimes it’s nice to put two windows side-by-side, so Goldenseal Pro also allows individual windows. As a bonus, they are simpler to program. We usually get them working first, then move code into the main window.

Another big improvement is the Windows ribbon bar, which organizes buttons and controls. It replaces the other half of Goldenseal’s pull-down commands. There’s no need for a menu bar at all, in the Windows version of our accounting and estimating software.

The Cocoa library for Macintosh doesn’t have a ribbon bar. However, we just finishing building a top bar that works the same way. It’s actually tidier than the Windows version, and equally functional. Goldenseal Pro for Mac will still have a menu bar, but it will be brief and optional. If Apple ever decides to remove the menu bar and make the Mac more iPhone-like, we’ll be ready.

The new interface in Goldenseal Pro ought to make life easier, especially for beginners. For example, consider a user who’s working on an estimate, and wants to change the layout of a printed form. In old Goldenseal, they need to choose Custom Layouts from the Options menu, then choose Printed Forms, then choose Estimates from a popup menu. Then choose a form from another popup. In new Goldenseal, they click the Layout button, then choose a form. Two steps instead of four, and much more obvious.

Setting up the new ribbon bars made it apparent that there are six different ‘modes’ in Goldenseal software. You can do data entry, find records, customize layouts, view reports, print forms, and use one of the action commands like Reconcile or Pay Bills. All but a few of the pull-down commands map to one of those. Goldenseal currently uses different types of windows for each mode, with different tools and controls.  Goldenseal Pro is similar, but they all can fit into the same window.

As we start to program the modes, it’s also apparent that they are very similar. They all read a layout resource, and use it to position fields and tables on the screen. Programming-wise, that means we are entering a project phase that finishes almost everything. We already have code that loads data entry fields, and it won’t be too hard to adapt it to the other functions.

Dennis Kolva
Programming Director







Goldenseal Pro Progress Report (Dec 21)

The past couple weeks, Turtlesoft’s staff has mostly worked on breakdown tables.  Tables are a big, complicated part of the interface for Goldenseal accounting software. Basically, it’s anything that looks like a list or spreadsheet. Last Spring, we estimated 5 weeks of programming time for tables on Macintosh, and another 3 weeks for Windows. So far, that seems about right.

In Goldenseal 4.96 and earlier, estimates, expenses and many other business transactions use 3 different layouts. There is an item breakdown with a list of Assemblies, a category breakdown where you type in line items, and a none layout with no table at all. Each transaction must use one of the three, so a popup button on the left switches between them. The system often confuses new users, and it was complicated to program.

Goldenseal Pro will be simpler. It will have just one main layout, plus an optional breakdown table. The breakdowns will slide out in a side drawer (Mac) or attached window (Win) when you click a button. Tables in Goldenseal Pro will combine the former category and item breakdowns. Just as functional, but less complicated.

The original plan was to release the first version of Goldenseal Pro without the Custom Layouts command, and then add it in the next update. We started to write code that reads existing layouts, but splits off the breakdown tables to a different window. That way, the old layouts still work in the new format. The code mostly works, but it’s an ugly hack.

Rather than waste more time on temporary code, we decided to rearrange the schedule a bit, and work on Custom Layouts next. That way, it will be possible to design better-looking layouts at the very beginning. This change delays the earliest possible release, but it doesn’t alter the overall completion date.

Back in the 90s when we started work on Goldenseal, Custom Layouts was one of the first things we finished. We used it to design the screen appearance, in parallel with the business and database code. I guess the same logic applies to Goldenseal Pro.

Custom Layouts acts much like a drawing program. Fortunately, there is already good sample code for that. We won’t have to re-invent anything.

This week, we also added a few things to the Goldenseal-to-Pro translator. It now merges category and item breakdown records, and also moves a few types of bank accounts and transactions. The conversion is necessary so we can use the Sample Company File for testing. It also will convert all user data, after Goldenseal Pro is released. So far, it looks like the transition will be completely seamless.

Dennis Kolva
Programming Director


Database Reliability- Part 4 (Dec 14)

The past few posts have described many ways for databases to go bad. All those potential problems give us plenty to think about, as we write code for our new accounting and estimating software. As a bonus, there are two more risks that we still haven’t covered.

The first problem is relatively minor: database files can easily expand to enormous size. FileMaker used to suffer from that. Things start to go south when a data record changes, and gets bigger. For example, when you add a new Material Purchase in Goldenseal, 3 accounts and 3 record arrays add a reference to it. That adds 4 bytes to their size, which makes them too big for their old spots within the file. It’s easiest to move them to the end of the file, but that leaves 6 empty gaps in the middle. Make that shift enough times, and the file will grow huge, even though it is mostly empty.

The NeoAccess database actually had a very good solution. It kept a list of empty spaces, and used it to save records in the most suitable gap, as close to the front of the file as possible. Unfortunately, their implementation was buggy, and too hard to fix. In 2002 we rewrote it using the same basic design, but with understandable code. It has worked well ever since. Goldenseal Pro uses the same approach, but with a few improvements.

The second problem is extremely serious: it’s relatively easy for a database to accidentally write data on top of other data. This is usually called “buffer overflow”, and it’s a general issue that all software faces. Overflows are the cause of most security holes that you read about in the news. They cause most data corruption, and the most serious of crashes.

Overflows are so dangerous that Goldenseal Pro has five different ways to prevent them.

  1. The Gap Manager that prevents file bloat, also knows where it’s safe to add records, without risk of damaging something else.
  2. After finding a gap, we next check the Sector Manager, which has a list of every record stored in that part of the file. The Manager looks on both sides of the gap, and makes doubly-sure there is no overlap with existing records.
  3. When Goldenseal accounting software saves records, it adds up the amount of data it writes, and makes sure the total matches exactly with the expected size. The same sanity-check happens again when it reads records from disk.
  4. Each record has about 20 bytes of “safety tags” at the beginning and end. When reading each record, we test the tags to make sure they are still OK.  If there ever is a bug that causes a record to overlap something else, it’s not hard to do forensics and discover the culprit. After we added the safety tags back in 2005, we soon tracked down the last remaining bug that was overwriting data.
  5. Within each record, there are additional safety tags for data that is most likely to cause a buffer overflow. If data is corrupted, the goal is to learn about it as quickly as possible. That way we can abort instead of damaging something else.

Most of these safety features are already present in current versions of Goldenseal. However, we are expanding and enhancing them for Goldenseal Pro. Disk storage is now so cheap that it makes sense to add an extra few megabytes, if it creates a more reliable database.


Meanwhile, back to Graphic Interface land. Since finishing the database work, our staff has focused on more advanced interface features. That includes breakdown tables, printing, “more info” boxes and the Find commands.

We also have started to simplify our accounting a bit. Goldenseal currently has 5 classes of bank accounts and transactions that are almost identical: for cash, checking, credit cards, loans, and savings. For Pro, they will all merge into one. Bank transactions will still look the same, so the only change is how you navigate to them.  Investment and escrow accounts are also very similar, but we haven’t decided yet whether to merge them, or keep them separate.

For Goldenseal estimating and accounting records, we currently have two types of breakdowns: category and item. There isn’t much difference between them, and they always have confused new users. Goldenseal Pro will merge them.

The tricky part for both these changes is converting existing data to the new format. Fortunately, a couple of previous updates made similar class changes, so we can reuse that code.

Dennis Kolva
Programming Director

Database Reliability- Part 3 (Dec 5)

The most important task for any database is to remember where each record is, within its file. To accomplish that, our accounting and estimating software uses a binary tree (b-tree) to store record locations. It has a different tree for each class of records. Every tree has a root (starting point) and nodes (branches). Nodes at the end are “leaf nodes”, which store the record locations.

A true binary tree has 2 branches at each node, which makes a tall tree (11 levels for 1,000 records, or 21 levels for a million). The NeoAccess database we’ve used for Goldenseal versions 1.0 to 4.9x came with a default of 8 branches per node. We upped it to 32. For unknown reasons NeoAccess also had two levels at each node, so it took 8 levels for 1,000 records, or 12 levels for a million. Still a rather complex tree.

Back in the 1980s and 90s, RAM and hard drives were cramped, expensive, and relatively slow. Because of that, it was best to keep nodes small. To save space, NeoAccess also didn’t create nodes until there were records to fill them. As a result, it scattered nodes randomly throughout the file (a bit more than 1 node for every 32 records). To save space, NeoAccess also stored very little data in each node. They only contained 4 or 8 bytes for each branch: just a file position, plus a record ID in leaf nodes.

We started to use Goldenseal to run TurtleSoft in 2000, just before releasing version 1.0. Unfortunately, a few NeoAccess nodes in our company file are corrupted, and have been for more than 10 years. When we try to look at the affected records, the node points to a very wrong file location. The OS is not happy with that, so it crashes.

We’ve spent dozens of hours stepping through code and looking at our raw file data, trying to find the errors. The problem is, our company file contains over 9,000 NeoAccess nodes, and they all look identical. If we could find the lost or damaged nodes in a raw file editor, we might be able to fix them. Unfortunately, it’s the proverbial needle in haystack.

Over the years, we also have looked at 30 or 40 corrupted Goldenseal files sent in by users.  When we scanned them with a raw file editor, about half contained data from other programs entirely. Our code has no access to anything beyond our file, so we didn’t put it there. Those files probably were damaged when the OS or hard drive firmware got confused, and wrote someone else’s data on top of ours. A few bad files were caused by TurtleSoft bugs, but we haven’t seen any of those since 2006. The rest were probably caused by small NeoAccess errors: possibly due to unknown bugs in their code, possibly from “bit rot” screwing up file addresses. Change one bit in the middle of a tree, and the whole thing stops working.

For Goldenseal Pro, our goal is to make the database less fragile, and more repairable. Your hardware (and the OS) will never be 100% perfect, but we can at least make it possible to diagnose and repair any database errors.

As a first step, Goldenseal Pro creates all its basic tree structures right at the beginning. Doing so has a cost. New, empty Goldenseal Pro files start out at 6+ megabytes (compared to 106K for original Goldenseal). A file that big would have been absurdly huge in the 1980s or 90s, but now it’s just a few millionths of a hard drive. The new setup puts the most important database structures in the same places in every file, and they won’t ever move. A uniform file structure will be much easier to diagnose and repair.

Goldenseal Pro also uses bigger nodes: with 128 to 4,096 branches, depending on how many items you are likely to have there. Almost all trees will only be one level deep. If they go two levels deep, they’ll hold a few million records before needing a third. The result is less clutter, and easier debugging.

The new nodes include text for their name and contents, so they are easily visible in the raw file. They also include a short text tag for each branch, and ‘safety tags’ at the beginning and end. All these ‘file navigation markers’ help us now during development and debugging. They also will help anyone looking at damaged files, in the future.

For extra security, all the important database structures are duplicated somewhere else in the file. At least in theory, we can rebuild anything that suffers damage.  Most likely we’ll wait for actual damaged files before writing the repair tools, since it’s hard to predict in advance, how data may go wrong.

All of this extra security adds more than 100 bytes per record, so Goldenseal Pro files will be significantly larger. At $60 to $300 per terabyte, we figure you probably won’t mind spending a penny or two for a few extra megabytes, to get more reliable data.

You still should have a good backup system, using multiple locations and methods. There are plenty of ways for data to die, and we can’t prevent all of them. However, we can at least make our files stronger and more survivable. As a bonus, it makes our programming and debugging easier.

Dennis Kolva
Programming Director