Category: Goldenseal news

We just released 2018 US tax tables for Goldenseal payroll software. This is our latest release date, ever.

IRS issued their preliminary 2018 tax formulas on Jan 11.  About a dozen states base their own tables on the federal ones. As of the 19th, all but Missouri, North Dakota and Vermont managed to catch up.  We are planning on another release whenever IRS publishes final numbers (and remaining states publish their info).

Using the 2018 withholding tables, there won’t be much change in payroll withholding amounts, for most employees.  The biggest change for most of our users is whether to incorporate, and take advantage of pass-through loopholes.

BTW, tax rates for rich people have really changed over the years. This year’s decrease was relatively minor.

Modern income taxes started after the 16th Amendment in 1913, with a top marginal tax rate of 7%. The top rate jumped to about 70% from 1917 to 1921, then declined to 25% from 1925 to 1931.

Great Depression rates were 63% to 79%. During World War II they gradually increased to 94%, then hung out in the 86% to 91% range until 1963. Through the 1970s they were about 70%. They shrank to 28% during the Reagan 1980s, then ranged between 31% and 39.6% since then.

Dennis Kolva
Programming Director
TurtleSoft.com

 

 

If Goldenseal Pro were a construction project, framing and utility rough-ins would now be done (mostly), and drywall would be starting. It’s that satisfying period when the interior transitions from a 2×4 maze, to actual rooms. Or, from a software point of view, the code is starting to look like a real estimating/accounting program.

Goldenseal’s software interface has six functional modes: data entry, finding records, reports, custom layouts, printing, and accounting actions. The first two are now working, and we just started on reports.  The six modes have very similar programming, so we will continue on them in that order until they’re all functional.

Work is progressing quickly enough that we probably will focus on just the Mac version for a while. The goal is to complete enough that we can start using it to run TurtleSoft. That’s a huge milestone, since we can then test daily under real-life conditions, with real business data. In the software world, this is known as “eating your own dog food”. Best estimate is a month or possibly two, to get that far.

After that, it shouldn’t take long for the Windows version to catch up. Working with the MFC library on Windows is easier than Cocoa on Mac, and the prep work in our existing code will already be done.

Our programming staff has definitely ascended far up the “learning curve” for the new language and libraries needed to build Goldenseal Pro. We are zipping right along!

Dennis Kolva
Programming Director
TurtleSoft.com

 

 

Goldenseal currently uses pull-down menus to navigate through its data entry screens, reports and commands. With more than 300 menu choices, it’s easy to get lost or overwhelmed. Our accounting software risks being the virtual equivalent of a desk piled high with papers and whatevers.

Both Microsoft and Apple have improved their developer libraries enormously, in the past 15 years. They now include much better interface tools. We are using them to improve navigation for users, so it’s easier to prepare estimates and run the business.

One big improvement is the outline view, which organizes stuff into folders and subfolders. It’s how you navigate through files in both Mac and Windows. Goldenseal Pro uses an outline view to replace about half the former pull-down menus. It allows everything to happen in a single window, with folders and tabs to navigate. The outline is particularly handy when using the full accounting/job costing/payroll software, since that is more complex.

Sometimes it’s nice to put two windows side-by-side, so Goldenseal Pro also allows individual windows. As a bonus, they are simpler to program. We usually get them working first, then move code into the main window.

Another big improvement is the Windows ribbon bar, which organizes buttons and controls. It replaces the other half of Goldenseal’s pull-down commands. There’s no need for a menu bar at all, in the Windows version of our accounting and estimating software.

The Cocoa library for Macintosh doesn’t have a ribbon bar. However, we just finishing building a top bar that works the same way. It’s actually tidier than the Windows version, and equally functional. Goldenseal Pro for Mac will still have a menu bar, but it will be brief and optional. If Apple ever decides to remove the menu bar and make the Mac more iPhone-like, we’ll be ready.

The new interface in Goldenseal Pro ought to make life easier, especially for beginners. For example, consider a user who’s working on an estimate, and wants to change the layout of a printed form. In old Goldenseal, they need to choose Custom Layouts from the Options menu, then choose Printed Forms, then choose Estimates from a popup menu. Then choose a form from another popup. In new Goldenseal, they click the Layout button, then choose a form. Two steps instead of four, and much more obvious.

Setting up the new ribbon bars made it apparent that there are six different ‘modes’ in Goldenseal software. You can do data entry, find records, customize layouts, view reports, print forms, and use one of the action commands like Reconcile or Pay Bills. All but a few of the pull-down commands map to one of those. Goldenseal currently uses different types of windows for each mode, with different tools and controls.  Goldenseal Pro is similar, but they all can fit into the same window.

As we start to program the modes, it’s also apparent that they are very similar. They all read a layout resource, and use it to position fields and tables on the screen. Programming-wise, that means we are entering a project phase that finishes almost everything. We already have code that loads data entry fields, and it won’t be too hard to adapt it to the other functions.

Dennis Kolva
Programming Director
TurtleSoft.com

 

 

 

 

 

 

The past couple weeks, Turtlesoft’s staff has mostly worked on breakdown tables.  Tables are a big, complicated part of the interface for Goldenseal accounting software. Basically, it’s anything that looks like a list or spreadsheet. Last Spring, we estimated 5 weeks of programming time for tables on Macintosh, and another 3 weeks for Windows. So far, that seems about right.

In Goldenseal 4.96 and earlier, estimates, expenses and many other business transactions use 3 different layouts. There is an item breakdown with a list of Assemblies, a category breakdown where you type in line items, and a none layout with no table at all. Each transaction must use one of the three, so a popup button on the left switches between them. The system often confuses new users, and it was complicated to program.

Goldenseal Pro will be simpler. It will have just one main layout, plus an optional breakdown table. The breakdowns will slide out in a side drawer (Mac) or attached window (Win) when you click a button. Tables in Goldenseal Pro will combine the former category and item breakdowns. Just as functional, but less complicated.

The original plan was to release the first version of Goldenseal Pro without the Custom Layouts command, and then add it in the next update. We started to write code that reads existing layouts, but splits off the breakdown tables to a different window. That way, the old layouts still work in the new format. The code mostly works, but it’s an ugly hack.

Rather than waste more time on temporary code, we decided to rearrange the schedule a bit, and work on Custom Layouts next. That way, it will be possible to design better-looking layouts at the very beginning. This change delays the earliest possible release, but it doesn’t alter the overall completion date.

Back in the 90s when we started work on Goldenseal, Custom Layouts was one of the first things we finished. We used it to design the screen appearance, in parallel with the business and database code. I guess the same logic applies to Goldenseal Pro.

Custom Layouts acts much like a drawing program. Fortunately, there is already good sample code for that. We won’t have to re-invent anything.

This week, we also added a few things to the Goldenseal-to-Pro translator. It now merges category and item breakdown records, and also moves a few types of bank accounts and transactions. The conversion is necessary so we can use the Sample Company File for testing. It also will convert all user data, after Goldenseal Pro is released. So far, it looks like the transition will be completely seamless.

Dennis Kolva
Programming Director
TurtleSoft.com

 

The past few posts have described many ways for databases to go bad. All those potential problems give us plenty to think about, as we write code for our new accounting and estimating software. As a bonus, there are two more risks that we still haven’t covered.

The first problem is relatively minor: database files can easily expand to enormous size. FileMaker used to suffer from that. Things start to go south when a data record changes, and gets bigger. For example, when you add a new Material Purchase in Goldenseal, 3 accounts and 3 record arrays add a reference to it. That adds 4 bytes to their size, which makes them too big for their old spots within the file. It’s easiest to move them to the end of the file, but that leaves 6 empty gaps in the middle. Make that shift enough times, and the file will grow huge, even though it is mostly empty.

The NeoAccess database actually had a very good solution. It kept a list of empty spaces, and used it to save records in the most suitable gap, as close to the front of the file as possible. Unfortunately, their implementation was buggy, and too hard to fix. In 2002 we rewrote it using the same basic design, but with understandable code. It has worked well ever since. Goldenseal Pro uses the same approach, but with a few improvements.

The second problem is extremely serious: it’s relatively easy for a database to accidentally write data on top of other data. This is usually called “buffer overflow”, and it’s a general issue that all software faces. Overflows are the cause of most security holes that you read about in the news. They cause most data corruption, and the most serious of crashes.

Overflows are so dangerous that Goldenseal Pro has five different ways to prevent them.

1. The Gap Manager that prevents file bloat, also knows where it’s safe to add records, without risk of damaging something else.
2. After finding a gap, we next check the Sector Manager, which has a list of every record stored in that part of the file. The Manager looks on both sides of the gap, and makes doubly-sure there is no overlap with existing records.
3. When Goldenseal accounting software saves records, it adds up the amount of data it writes, and makes sure the total matches exactly with the expected size. The same sanity-check happens again when it reads records from disk.
4. Each record has about 20 bytes of “safety tags” at the beginning and end. When reading each record, we test the tags to make sure they are still OK.  If there ever is a bug that causes a record to overlap something else, it’s not hard to do forensics and discover the culprit. After we added the safety tags back in 2005, we soon tracked down the last remaining bug that was overwriting data.
5. Within each record, there are additional safety tags for data that is most likely to cause a buffer overflow. If data is corrupted, the goal is to learn about it as quickly as possible. That way we can abort instead of damaging something else.

Most of these safety features are already present in current versions of Goldenseal. However, we are expanding and enhancing them for Goldenseal Pro. Disk storage is now so cheap that it makes sense to add an extra few megabytes, if it creates a more reliable database.

*******************

Meanwhile, back to Graphic Interface land. Since finishing the database work, our staff has focused on more advanced interface features. That includes breakdown tables, printing, “more info” boxes and the Find commands.

We also have started to simplify our accounting a bit. Goldenseal currently has 5 classes of bank accounts and transactions that are almost identical: for cash, checking, credit cards, loans, and savings. For Pro, they will all merge into one. Bank transactions will still look the same, so the only change is how you navigate to them.  Investment and escrow accounts are also very similar, but we haven’t decided yet whether to merge them, or keep them separate.

For Goldenseal estimating and accounting records, we currently have two types of breakdowns: category and item. There isn’t much difference between them, and they always have confused new users. Goldenseal Pro will merge them.

The tricky part for both these changes is converting existing data to the new format. Fortunately, a couple of previous updates made similar class changes, so we can reuse that code.

Dennis Kolva
Programming Director
TurtleSoft.com

The most important task for any database is to remember where each record is, within its file. To accomplish that, our accounting and estimating software uses a binary tree (b-tree) to store record locations. It has a different tree for each class of records. Every tree has a root (starting point) and nodes (branches). Nodes at the end are “leaf nodes”, which store the record locations.

A true binary tree has 2 branches at each node, which makes a tall tree (11 levels for 1,000 records, or 21 levels for a million). The NeoAccess database we’ve used for Goldenseal versions 1.0 to 4.9x came with a default of 8 branches per node. We upped it to 32. For unknown reasons NeoAccess also had two levels at each node, so it took 8 levels for 1,000 records, or 12 levels for a million. Still a rather complex tree.

Back in the 1980s and 90s, RAM and hard drives were cramped, expensive, and relatively slow. Because of that, it was best to keep nodes small. To save space, NeoAccess also didn’t create nodes until there were records to fill them. As a result, it scattered nodes randomly throughout the file (a bit more than 1 node for every 32 records). To save space, NeoAccess also stored very little data in each node. They only contained 4 or 8 bytes for each branch: just a file position, plus a record ID in leaf nodes.

We started to use Goldenseal to run TurtleSoft in 2000, just before releasing version 1.0. Unfortunately, a few NeoAccess nodes in our company file are corrupted, and have been for more than 10 years. When we try to look at the affected records, the node points to a very wrong file location. The OS is not happy with that, so it crashes.

We’ve spent dozens of hours stepping through code and looking at our raw file data, trying to find the errors. The problem is, our company file contains over 9,000 NeoAccess nodes, and they all look identical. If we could find the lost or damaged nodes in a raw file editor, we might be able to fix them. Unfortunately, it’s the proverbial needle in haystack.

Over the years, we also have looked at 30 or 40 corrupted Goldenseal files sent in by users.  When we scanned them with a raw file editor, about half contained data from other programs entirely. Our code has no access to anything beyond our file, so we didn’t put it there. Those files probably were damaged when the OS or hard drive firmware got confused, and wrote someone else’s data on top of ours. A few bad files were caused by TurtleSoft bugs, but we haven’t seen any of those since 2006. The rest were probably caused by small NeoAccess errors: possibly due to unknown bugs in their code, possibly from “bit rot” screwing up file addresses. Change one bit in the middle of a tree, and the whole thing stops working.

For Goldenseal Pro, our goal is to make the database less fragile, and more repairable. Your hardware (and the OS) will never be 100% perfect, but we can at least make it possible to diagnose and repair any database errors.

As a first step, Goldenseal Pro creates all its basic tree structures right at the beginning. Doing so has a cost. New, empty Goldenseal Pro files start out at 6+ megabytes (compared to 106K for original Goldenseal). A file that big would have been absurdly huge in the 1980s or 90s, but now it’s just a few millionths of a hard drive. The new setup puts the most important database structures in the same places in every file, and they won’t ever move. A uniform file structure will be much easier to diagnose and repair.

Goldenseal Pro also uses bigger nodes: with 128 to 4,096 branches, depending on how many items you are likely to have there. Almost all trees will only be one level deep. If they go two levels deep, they’ll hold a few million records before needing a third. The result is less clutter, and easier debugging.

The new nodes include text for their name and contents, so they are easily visible in the raw file. They also include a short text tag for each branch, and ‘safety tags’ at the beginning and end. All these ‘file navigation markers’ help us now during development and debugging. They also will help anyone looking at damaged files, in the future.

For extra security, all the important database structures are duplicated somewhere else in the file. At least in theory, we can rebuild anything that suffers damage.  Most likely we’ll wait for actual damaged files before writing the repair tools, since it’s hard to predict in advance, how data may go wrong.

All of this extra security adds more than 100 bytes per record, so Goldenseal Pro files will be significantly larger. At $60 to $300 per terabyte, we figure you probably won’t mind spending a penny or two for a few extra megabytes, to get more reliable data.

You still should have a good backup system, using multiple locations and methods. There are plenty of ways for data to die, and we can’t prevent all of them. However, we can at least make our files stronger and more survivable. As a bonus, it makes our programming and debugging easier.

Dennis Kolva
Programming Director
TurtleSoft.com

When writing new code for our estimating and accounting software, we have a rule of thumb. When it first works, it’s 1/3 done. When it works for everything the programmer can think of to break it, it’s 2/3 done.  After it survives testing by people actually using it, it (eventually) reaches 100%.

We built the new database for Goldenseal Pro more than a year ago, but it was bare bones, and really just the first 1/3 of the work. It was sufficient for initial interface testing, and to make sure the basic approach was valid. The past couple weeks we added the rest of what a database needs, so it’s now at the 2/3 mark. The final 1/3 will be polishing, tweaking and testing, done gradually over the next few months. The final test will be using it to handle our TurtleSoft business accounting for a while.

Our primary goal for the new database code is to make it extremely reliable. We suspect that there are still a few subtle bugs remaining in the NeoAccess database that runs Goldenseal 4.9x, and we want to avoid that for Goldenseal Pro.

One specific way to increase reliability is to simplify the basic structure of the database.

NeoAccess used something called a binary tree (or b-tree) to store the location of each record within the file. B-trees are optimized to access records via binary search, which is the most efficient way to find anything. It only takes log-2 time to find any record in a sorted list. That means a binary search in a thousand records only takes 10 steps, a million records only takes 20 steps, and a billion takes 30. Binary search is amazingly fast when you are indexing, say, all humans on Earth.

The problem is, b-trees are complicated. The branches need to be balanced and trimmed, and there are many ways to go wrong. If there are still bugs left in our modified version of NeoAccess, they are probably buried in the tree management code. It’s just too complex, so we never touched it.

For Goldenseal Pro we use simple lists, rather than b-trees. They are slower to search, but on our scale it’s only a matter of microseconds.  B-trees are gross overkill for small businesses with mere hundreds or thousands of records. Simple linear searches are easy to debug, and much less fragile. Even when a b-tree is appropriate, it’s far better to use a well-tested library, rather than “roll your own” code as in NeoAccess.

For large numbers of records, we still use a tree structure. But it is a very “broad” tree that rarely grows more than two levels deep. It’s more like a shrub or low-maintenance ground cover. Theoretically, the new database won’t be as efficient as a true binary tree. However, we much prefer increased simplicity and reliability, over a tiny performance gain.

It’s possible to do binary searches without a b-tree, and binary search is even built into the C++ standard library that we use. However, binary searches are harder to debug, so we will only add them in places where linear search is visibly slow. That’s part of the tweaking process.

************************

Another part of making code reliable is to write cautious code. That means we frequently “sanity check”, and give an error message if anything is amiss. It makes our development process much easier, since most bugs give a message that takes us right to the problem. If any bugs survive into the final version, users will get a warning, rather than a hidden problem that bites them later.

The main reason that Goldenseal 4.9x rarely crashes, is because it sanity checks first. There are 6,700 places in our code where we confirm that something really exists, and 1,600 places where we check for a reasonable value. There are more thousands of sanity checks built into the code logic, but we can’t count those easily.

Along with our usual cautious coding, Goldenseal Pro includes several layers of sanity checking to make sure the database stays healthy. I’ll talk more about that in a future post.

Dennis Kolva
Programming Director
TurtleSoft.com

 

TurtleSoft started out in 1987 with MacNail, a set of Excel templates for construction estimating. Later we added accounting and scheduling features. The MacNail software was extremely popular, and it still has a few hundred die-hard users. Unfortunately, we could only do so much with Excel macros and spreadsheets. MacNail was complicated to use, and difficult to support.

In 1989 we released a second estimating program, built atop Apple’s HyperCard. It had a much nicer human interface, but severe limitations on the back end. Among other problems, HyperCard introduced us to “race conditions”. That’s where the app has intermittent bugs, depending on which message path completes first (unpredictably). They are extremely hard to track down.

In the early 90s, Microsoft replaced the Excel macro language with Visual Basic, and Apple halted development on HyperCard. It was clearly time for us to move on.

For the next software generation, we looked at many database programs to build upon: FileMaker, Omnis, Double Helix, Sirius Developer, Prograph. Each was better in some ways than what we had already, but in some ways each was worse. Rather than build a mediocre app on top of some other software, we finally decided to write a “real” application in C++. Coding from the ground up takes longer, but allows full control. As it happens, it also prevented us from riding a platform to its death, since all but FileMaker are now long gone.

We were too inexperienced to write our own database code, so we tested several C++ database libraries, and settled on one called NeoAccess. It was quite popular in the mid 1990s. The database performed well, and wasn’t too difficult to build upon. Unfortunately, when Goldenseal was almost completed, we discovered that NeoAccess had serious bugs. Files eventually became corrupted, and all data was lost. The developers at NeoLogic never fixed the bugs, and eventually stopped answering emails. 

AOL 4.0 and Netscape Communicator also used NeoAccess. AOL abandoned it in their version 5.0. Netscape retired their software prematurely (and was soon swallowed by AOL). Dozens of other companies went through similar drama, thanks to an unreliable database engine.

We also considered abandoning NeoAccess, but the alternatives were just as buggy. So we put about 2 programmer-years into rewriting their code to make it more reliable. We learned a lot in the process, which is why we decided to build our own database for Goldenseal Pro. You might say that bugs that you create yourself are much easier to fix, than ones made by other people.

The first step in writing reliable software is to make the code maintainable. That means simple logic, clear comments, and easily readable code that anyone can understand. NeoAccess was completely the opposite, and that was its primary flaw. To fix it, we stepped through their code hundreds of times, and gradually rewrote it to make more sense. A few of the bugs were simple logic errors that finally stood out when the code was less cryptic. Some of the bugs were in code that was so convoluted that we just gave up, and rewrote it entirely.

BTW most open-source code is equally unreadable. Whenever you hear in the news about a major security breach, the root cause was probably some important open-source code that nobody understands. If other programmers can’t read it, then they can’t fix it or improve it.

There are more specific methods we are using to make the database more reliable in Goldenseal Pro. NeoAccess taught us a lot about database design, both from its good parts and its bad. I’ll cover them in more detail, in a future post.

Meanwhile, the past couple weeks we have been working on the new database code for Goldenseal Pro. It is mostly working, but probably still needs another week to finish.

Dennis Kolva
Programming Director
TurtleSoft.com

 

Last week, Goldenseal Pro saved record changes to disk for the first time. It was very exciting to enter data, quit, reopen the file, and see the changes still there. This may not seem like much, but it is a major milestone. It’s the last big connection needed between the new human interface code, and the business logic that runs our estimating and accounting software.

Since then, we have been working on the database code, which manages how records are stored on the hard drive.  The new code we wrote last year has performed well while importing existing Goldenseal files, and then viewing records. However, it needs more bells and whistles to properly manage changes, deletions, and other daily usage.

We often complain about the NeoAccess database library used in Goldenseal, but it did have some very good features. We had to fix a few serious bugs in the late 1990s, but since then it has run thousands of company’s files, with few problems.  As we write the code that replaces it, our goal is to keep the good features from the old database, but make it more reliable, and easier to maintain.

When Goldenseal 4.x opens any record, it reads the data from hard drive to RAM. It then stays “cached”, either until memory got low, or until our code deliberately purges the cache. Reducing disk accesses makes many operations run much faster.

Goldenseal Pro uses a similar cache system. The main difference is that we simplified the list of cached records, so it is easier to troubleshoot. The same list also handles “dirty” records, as described in the previous post.

Every database has to decide where to put new and changed records during a save. The easiest place is at the end of the file. Unfortunately, that eventually results in a huge file that is mostly holes. For example, FileMaker used to save everything at the end (and maybe still does). Files can grow to 10s or 100s of megabytes, and eventually need a compression step to get back to a reasonable size.

NeoAccess solved that problem with a list of gaps between records. That way, new or changed records could go into the empty spaces. Unfortunately, their code was hard to debug, and we suspected that it sometimes wrote records on top of existing ones (extremely bad news). In 2002 we replaced their code with a File Manager to track every record and every empty space. That system worked well, but it used an extremely large record that sometimes crashed, if memory was too low.

For Goldenseal Pro, we’ve split the original file manager into separate, smaller Sector Managers. Each tracks records in just one portion of the file (currently 32,000 records). There is also a separate Gap Manager that tracks all empty spaces. It decides where to put each record, when it is saved. The sector managers have been working great for over a year, and we are refining the gap manager now. It’s tricky, because the manager needs to combine gaps that touch, and some records work better with empty space before or after.

Goldenseal Pro stores the location of every record in two different places: in the main indexing system, and in the sector managers. That will allow the new database to recover from file corruption, by fixing the one that is wrong.  There are places in Goldenseal 4.x files where a single bit change can totally trash the file. In Pro, recovery will be possible from almost all types of “bit rot”.

Dennis Kolva
Programming Director
TurtleSoft.com

 

 

 

 

 

 

At heart, Goldenseal is a database program. It stores your company data on a hard drive so you can use it later. One important part of that process is knowing which data is “dirty”. In programmer talk, that means a record has changed, so it needs to be saved. After it’s saved, it becomes “clean” again. It’s hard to say whether the origins of the term are from laundry, or religion!

The old NeoAccess database had a true/false “dirty bit” in each record, to mark whether it had changed. It also had a “busy bit” for items currently in use. At every save, it looked through every record stored in RAM, and saved anything that was dirty but not busy. The system usually worked OK, but it was extremely hard to troubleshoot. If something didn’t work right, it meant a half an hour in the debugger, stepping through many hundreds of code lines.

We ended up adding internal lists of busy and dirty records, just so we could see what was going on more easily. Those lists worked so well that they now replace the old NeoAccess system entirely, in our new database code.

When you first look at an existing record, it starts out clean. Then it turns dirty the moment you change anything. To make that work, each data entry field has to send a message, and change the record status to dirty. Unfortunately, the existing code is a hodge-podge. There is makeDirty, setDirty and MakeDirty (case matters), and there are bits stored in 3 different places. It’s partly because different programmers reinvented the wheel, partly because NeoAccess was ugly, and partly because PowerPlant didn’t separate view and controller functions.

When we “refactor” something like this, it’s a bit like knocking out support members one by one, to see if the building collapses. If it does, we hit undo and decide whether to keep it, or rebuild it. If not, we probably can remove that code, or merge it with something else.

This all started because we need a quick way to decide whether the Save button should be grayed-out, or not. However, the dirty status has important consequences in the multi-user version, where more than one person might change the same record. It’s worth spending a day or two to redesign the system, so it’s more understandable.

Dennis Kolva
Programming Director
TurtleSoft.com