Author: Dennis Kolva

Our staff is well into the miscellaneous phase of the Goldenseal Pro project. That means tidying up small details until it’s all shiny.

First thing, we enhanced the database verification reports so they catch a few more potential errors. The reports will make the final testing go much faster. Rather than waiting around to see if file error messages pop up, we can run Verify File a few times a day. It will notice any problems right away.

It’s quite possible there won’t be any more database issues. The data conversion from version 4.96 to version 5.0 already futzes with many thousands of records, and it may already have hit every possible problem. Really, database code is not that huge a task. It’s much simpler than construction estimating or payroll. A database has to locate records on disk, make sure they don’t write on top of each other, and manage gaps (otherwise the file becomes an enormous hunk of Swiss cheese). Most of our database code is just sanity checking, and duplicating info so it’s possible to recover if data is corrupted by the outside world.

Since we are already converting data from old format to new, we decided to simplify bank accounts and transactions in the new accounting software. Goldenseal Pro only has 3 types of bank accounts instead of the current 7. The update process converts any old records, and also converts links between records. As we started using the Reconcile command, one missed link turned up. We fixed it.

Right now, Passwords are on the plate. I personally designed the Goldenseal interface, with little sliders to set the access each employee has to different parts of the data. It looks nice, but it was so non-standard that it took more than a month of programming. Too much time for a minor feature that many companies don’t even use.  In Goldenseal Pro it won’t look quite as nice, but it will use existing code. One more day to finish it, instead of a month.

Our staff started work on the Custom Layouts view more than a year ago. Some things work, but we got stuck on a few things and moved on to other code. Our staff is revisiting it, now that we’re more experienced with Cocoa.  It’s the last major interface element. The first release doesn’t absolutely need Custom Layouts, but it will make Passwords easier to finish.

Dennis Kolva
Programming Director
TurtleSoft.com

The past week, our staff has focused on the database code that runs our new accounting software. First thing was to update the Verify File command to work for Goldenseal Pro. It reads every record, runs a few sanity tests, then reports any errors. When we ran the verify, the Sample Company File showed 4 errors, and our Turtlesoft file had 8.

As it turned out, almost every error was caused by something different.  They were all very obscure problems. Hard to find and hard to figure out, so it took a full week to fix them all.

Along the way, we added a few other commands that test the integrity of the database structures. They helped with the bug fixes. Even better, they’ll show any future file errors before they become serious problems.

Like almost all databases, Goldenseal Pro locates records with a tree structure. Searches in a tree start at the root, then travel up through branches to the leaf nodes: one for each record. Actually, Goldenseal Pro uses a few thousand branches at each node, so it looks more like crabgrass than an actual tree. It’s faster that way.

With such fat nodes, it only takes two levels to handle anything under a few million records. None of our test files are even close to that big, so we temporarily turned one class into a binary tree, with just two records per node.  For 900 records it built a 10-level binary tree, and that tested out fine. So we are now confident that our accounting database can handle at least a few nonillion records. That’s a thousand billion billion billion, or a 1 followed by 30 zeros.

Our current company file has a bad NeoAccess node, which breaks several thousand Sales records from the early 1990s. Fortunately, we found a way to recover all but 8 of them, when converting from the old format to the new.

There are times when it feels insane to be writing our own database code, rather than using something well-established and open-source. So, I recently downloaded the MySQL source code to see what we are missing. It is very old C and C++ code. Hard to read and understand, and probably very hard to debug. Many potential security flaws and time bombs. Its style is similar to the NeoAccess code that we struggled with, partly rewrote, then finally abandoned. However, MySQL has several thousand source files rather than a couple hundred. It’s probably even more convoluted.

I think we made the right choice. It’s nice to have a report that explains every bit within the accounting file.

Dennis Kolva
Programming Director
TurtleSoft.com

Our staff is making steady progress on the Mac version of Goldenseal Pro.

The Reconcile command is now functional, with a table that shows a list of unreconciled bank transactions. As usual, it was a struggle to get the table working properly. The NSTableView class does not forgive small errors. Fortunately, we can duplicate it for Pay Bills, Write Payroll, and a few other commands, so they will go much faster. Even better, it’s the last type of table that’s needed for Goldenseal Pro.

We tested the Reconcile command with our TurtleSoft bank account. It has over 10,000 transactions, which take almost a minute to process. First thing we did was to rewrite the search code. Now it eliminates most records quickly, and only takes a few seconds.

During the test, there were a few database error messages. Not unexpected, since this is the “shakeout cruise” to find problems.

Goldenseal uses a record cache to manage how long data sticks around in RAM. It helps to give good performance, without consuming a lot of memory. The new version has been working pretty well, but the test gave a couple of obscure cache error messages. We figured out why, and fixed it. The fix made things worse. Fixing those was even worse: crashing instead of error messages. For a while it felt like an episode of House MD, with the patient gradually dying from some unknown, obscure disease. Fortunately, after an overhaul of the way we create records, the record cache is finally cured. Just like House MD.

There also were serious errors in a few of the bank transactions. Most likely, we missed some data structures that need revising for 64-bit. Having a partly-defective file is a good opportunity to soup up the Verify File command, so our staff is working on that now. The goal is to get a more detailed problem report, so we know exactly what to fix. Kinda like building an MRI machine for files.

Along the way, we have been fiddling with custom fields, pasted-in images and a few other details. Most of the work is in familiar code, so it’s much more pleasant than working with the Cocoa library.

Dennis Kolva
Programming Director
TurtleSoft.com

We are starting to use our own data to test Goldenseal Pro accounting software. There’s lots of small stuff to do, to get it ready for running TurtleSoft.

First step was the sign-on dialog. It’s now a bit better than the current version, with an option to view or hide the password as you type it.

Ever since we switched to Apple’s most recent memory management system, the main window has crashed randomly after switching tabs. We finally narrowed it down to one line of code, but were still baffled. Something was breaking that never should have. It finally turned out to be an old Apple bug in the NSTabView class. We posted details to the Cocoa developer list, and folks gave some suggestions that eventually led to a work-around. Tabs are working properly again. We switched them to newer Apple code, so they even have niftier animation now.

There have been database errors in bank accounts that were converted from old format to new. Duplicate ID numbers. The whole system to give record IDs has always been a problem, ever since Goldenseal 1.0. Its code is confusing. Bug fixes over the years have only made it more confusing. So, we tossed it all and rewrote everything from scratch. It’s simpler now, and will be much easier to maintain.  Software design is much easier after you’ve made every possible mistake already.

We are also about half-way through the Reconcile command. That uses a table. They have been the hardest programming problem so far, in the Mac update. Fortunately, several types of tables are now working, so we can duplicate that code. Our staff futzed around with tables all Winter, but they will go faster now.

I’m not even going to try to estimate completion. Our staff will be finishing features and fixing bugs until it’s done.

Dennis Kolva
Programming Director
TurtleSoft.com

All the basics are now working for the tables in lists and accounts. Menus pop up, check boxes change, and text saves. Our staff is double-checking all of the tables now, to find any small details that may need extra code.

Even better, our staff can use the same basic code for other types of tables. We aren’t masters of NSTableView yet, but at least we aren’t baffled by it any more. The home stretch is in sight.

In the current Goldenseal, the C++ table classes handle mouse clicks, screen drawing, and everything in between. For Goldenseal Pro, NSTableView takes over all the mouse stuff and screen drawing. So, the remaining work will be in the special details in the middle, plus deleting lots of obsolete code.

The first step was to entirely remove the old messaging system from Goldenseal.

There are lots of ways for users to interact with an app: they can click on something, or double-click, or click and drag. They can type stuff from the keyboard. Most actions need to be passed along elsewhere in the code, to actually do something. Most platforms do that by sending a message. Apple’s Cocoa framework is extremely message-heavy. Microsoft’s MFC framework and the PowerPlant framework also use messages, but less so.

The current version of Goldenseal has a few hundred different messages that go traveling around. Unfortunately, the whole message thing turned out to be one of those ideas that sounds great in theory, but never lives up to its promise.

The biggest problem has been that messages are hard to debug. All code starts out with bugs, but what counts is how easy it is to fix them. With messages, the code doesn’t directly explain how it works. One side sends a message and the other side receives it, but there’s no way to be sure it actually gets to the right place. To check it requires 2o or 50 steps in the debugger, instead of just one or two.

Even worse, messages can be sent at the wrong time, or in the wrong order. Debugging that kind of error is a nightmare. Sometimes we have to log every single message, and then try to deduce what’s wrong.

As our staff fixed bugs in Goldenseal between 2001 and present, we solved many problems by replacing messages with simple, direct connections. Often, a complete rewrite was easier than trying to debug and fix the messages.

The original plan was to replace the remaining messages gradually.  But then we discovered an extremely old bug in the Work Crews command. It’s not a biggie, but enough to inspire extra skepticism for the existing code. If we’re going to test every last detail, then there’s little reason to keep any remaining message-based code.  Better to delete it all, and reduce clutter.

BTW the message system inside the Cocoa framework drives us crazy. It’s one of the major reasons why this conversion is taking so long. I’ll complain more specifically in a future post.

Dennis Kolva
Programming Director
TurtleSoft.com

Our staff is back to working on Goldenseal Pro. Right now we are focused on the little tables inside lists. It’s going well, with steady progress each day.

Working with Cocoa’s NSTableView class has been extremely frustrating the past few months, but I think we are finally over the hump.

About a year ago, it only took a few weeks to get the various tables to show data. Back then we were very optimistic. Unfortunately, creating a data entry interface turned out to be much, much harder than just displaying some text.

Apple has two different ways to set up tables: with cells or with views. Last year, we set up all the tables with cells. They are simpler, and we couldn’t find any sample code that used views. Unfortunately, since then we discovered that Apple has deprecated cell-based tables. That means they are officially obsolete, and will eventually stop working.

Because of that, we switched the list tables over to view-based. After everything is working properly there, it shouldn’t be too painful to go back and switch over the other tables, also. After all this effort, we really don’t want to start out with any outdated code in Goldenseal Pro.

During the past month, our staff spent many hours in Excel, fiddling with the spreadsheets that calculate pricing and page details for SmartKnives.com. It was interesting to be writing Excel macros again! That is how TurtleSoft first began, back in the late 80s.

It was also interesting to work with the new Cocoa-based rewrite of Excel for Macintosh. It seems very clunky and awkward, compared to the previous Carbon versions. Maybe we aren’t the only ones who have struggled with NSTableView.

Dennis Kolva
Programming Director
TurtleSoft.com

There’s not much progress to report.

While finishing Goldenseal Pro, TurtleSoft has paid staff and bills by selling rare/retired Swiss Army knives and Leatherman tools on eBay. We have piles of them, purchased from airport confiscations in the Aughties. Cleaning and sorting multi-tools is a soothing break from programming.

A few weeks ago eBay decided to eliminate shipping of knives outside the US. That was about half our sales.  So we are finishing up the shopping cart on our SmartKnives website, a year earlier than planned.

Setup work was already close to completion, so it shouldn’t cause too long a delay. We ought to be back to full programming speed in a few weeks.

Dennis Kolva
Programming Director
TurtleSoft.com

Our staff is making decent progress on member tables (the relatively small tables found in a few accounts, and many lists). Simple ones now work properly. Next were the ‘to/from’ tables used in Tax Tables and a few other places. They are more complicated, but most features work in those now, also.

We also made a breakthrough on the breakdown tables used in Estimates and many other transactions. The current Goldenseal uses two types of tables: category breakdowns let you type in a list of items, while item breakdowns have a list of Assemblies or Cost Items. The difference is not obvious, and it often confuses users. Their code also overlaps by quite a bit. Category breakdowns still need to show allowances and bids, while item breakdowns sometimes use type0ins.

Last year, we decided to combine them into just one type of table. Merging them was not too hard, but they still seemed a bit too complicated.

For example, in Estimates there are 37 possible types of line items. It makes for a very long list of choices in the pop-up menu. Overwhelming. It was also difficult to find short phrases to describe each option properly. The merged version is even worse.

The breakthrough was to add one more column to the breakdown tables. Each line item now starts with a Type column.  It splits up the list, so the Cost Area options are shorter and more logical. That should make data entry easier for beginners (or tired experts).

A year ago, we estimated that tables would take 2 months of programming time. So far, our staff has spent about 5 months on them. There is probably another month or two left, before they are completely done and polished. It was a gross under-estimation, but that has been the trend for this project. Working with the Cocoa library is hard.

Dennis Kolva
Programming Director
TurtleSoft.com

Last week, our staff finished the back-end code to run member tables. They are the little tables with a list of checkbooks, rental units, tax table steps or other setup details. The interface is also mostly done, but it’s still not as responsive as we’d like. When you click in a cell, sometimes it shows a type-in field, as it’s supposed to. Sometimes it does nothing, or waits a bit.

Apple’s Cocoa library uses messages for almost everything. Every window and field and button works by sending messages around to other things. Right now we are setting breakpoints in all possible messages, so we can understand exactly what’s being sent when.

Unfortunately, this is not the first time we’ve had to deal with message problems.

Back in 1987, Apple introduced a program called HyperCard. At the time, it was by far the fastest way for non-programmers to whip up something interesting. We used it to create a simple construction estimator called Hyper*Estimator.

HyperCard also worked by sending messages. It was our first exposure to a programming problem called race conditions. A race happens when two messages are sent, then travel through the CPU with no guarantee of which arrives first. The winner may be entirely random. Or worse, one message may win 99.9% of the time. That creates a hard-to-reproduce bug in the rare cases when the other message wins. Even worse, the race results can be different on different machines. Debugging is so much harder when it breaks for users but not for us.

Hyper*Estimator took about 8 months to program. Then it required another 3 or 4 frustrating months to fix the obscure bugs caused by race conditions. Programming in HyperCard was fun, but fixing intermittent bugs was exactly the opposite.

The current version of Goldenseal uses the PowerPlant library, and it also included messages. However, its message system is written in regular C++ code. That meant we can step through it, and always know what happens when. Even then, we use messages sparingly. Usually there are better ways to get things done.

It’s still too early to say whether Cocoa messages have race conditions. It’s not something that Apple discusses in their specs. Regardless, the messaging is buried in machine language, so we can’t see what sent each message, or why. It’s very much a “black box”. The best we can do is experiment, and gradually guess at what’s inside.

That’s a nuisance, but Cocoa is no more opaque than HyperCard was. We eventually conquered that, and we’ll conquer this one too. At least for Cocoa it’s just the first click that’s a problem. After that it’s in our code, and fully under our control.

Dennis Kolva
Programming Director
TurtleSoft.com

Our staff is making good progress on the new interface for breakdown tables. To edit a line item, it now shows a separate window with regular fields.  This design change gets us past some frustrating road blocks. It also seems like a big improvement over the current interface.

Breakdown tables are difficult. They’ve been a struggle in Goldenseal Pro, just as they were for the original Goldenseal. That’s partly because the bar is set very high for them. A big percentage of data entry happens in breakdown tables, so they need to look good, and run very smoothly.

For example, for the original Goldenseal, we spent a full week designing the little button that appears when you click in a table cell that shows a list of choices. For Goldenseal Pro we spent another full week for the same thing. The standard Cocoa popup button didn’t look good in a cell, and it’s not easy to change it.

Users need responsive action after a click in a table cell, and they’ll want to use arrow keys or tab keys to navigate. Cocoa’s NSTableView just isn’t designed to work that way.

Meanwhile, Goldenseal uses tables for more than just transaction breakdowns. Bank accounts include a little table for checkbooks, and Real Estate accounts have a table for rental units. About 30 lists also have tables. They handle stuff like payroll tax brackets, shipping costs by weight, and depreciation by year.

Fortunately, most of those tables are very simple- just 1 to 3 columns, and usually just text or numbers. Even better, they don’t get used very often. For those, we probably can still enter data directly into cells.

Dennis Kolva
Programming Director
TurtleSoft.com