Goldenseal Pro Progress- Tab Order & Open Source (Jul 13)

When viewing records in accounting software, it really helps if you can tab from field to field. That way, users can keep their hands on the keyboard while doing data entry, with no need to click in each field.

For the current version of Goldenseal, we wrote our own tabbing code, back in the late 1990s. It wasn’t too hard. Each window already stores a list of fields in front-to-back order. When you hit the tab key, it finds the current field, looks through the list for the next editable field, and selects it. For shift-tab, it looks in the other direction. Tabbing in tables is a bit more complicated, but not by much. The entire system took about one programmer-day to build.

Last week, we started to set up tabbing in Goldenseal Pro for Macintosh. The Cocoa framework includes an automatic tab system. They even brag that “you don’t have to write any code” to make it work. We turned it on, but it didn’t work.

Next attempt was Cocoa’s manual tabbing system. It uses what’s called a linked list. Each field is linked to the next field in the list, until you loop back from the last to the first. We set that up in Cocoa, but it still didn’t work. We read Apple’s documentation and checked Stack Overflow, but found no answers. We searched for a sample project with tabbing, and didn’t find any.

Next, we tried stepping through the tab process in the debugger, to see where it was failing. Unfortunately, Cocoa is very much a “black box”. Unlike the other frameworks we have used, it has no visible source code. The debugger only shows machine instructions, which are usually too low-level to be helpful.

When that failed, we went back to our previous approach. Go through the list of fields, and find the next one. To make that work we have to intercept the tab key, and then highlight the next field. Those both turned out to be difficult tasks in Cocoa.  Eventually it worked, but it sure wasn’t easy. Altogether, it probably took a programmer-week. The only good news is that some of what we had to learn will be useful for other things.

Apple’s Cocoa framework is the only major framework that hides its source code. It’s definitely an impediment. We have been working with Cocoa since September 2016, and it’s definitely the hardest framework our staff has ever experienced.  A while back we calculated an October 2018 completion date, but that probably will be pushed back a few months. The only good news is that Cocoa is hard for everyone, not just us. When the Mac goes entirely 64-bit in 2019, a lot of our competitors will be gone.

Ironically, Apple’s Unix operating system code (Darwin) is open source.  That means its source code is not only visible, but editable by qualified developers. Unfortunately that doesn’t help us, since our type of programming entirely relies on Cocoa.

Speaking of open source, it’s an option we are considering for Goldenseal’s code. 

Dennis Kolva
Programming Director
TurtleSoft.com

 

Goldenseal Pro Progress- Right Clicks & Extra Windows (July 2)

Goldenseal accounting software was designed for a one-button mouse. It’s all that was available for the Mac back then.

Goldenseal Pro is modern, so we can also use the scroll wheel and right-button. Last week we started on the first of the right-click commands. They are a much better way to access some details. For example, when you click the popup button next to a clairvoyant field in the current app, you’ll see a few options down at the very bottom of the menu. They let you enter ‘none’, edit the selected item, or add a new one. A few specialty fields add other options.

In Goldenseal Pro, you access those commands with a right-click. It’s much easier to find the commands (especially when the list is long). Right-click menus are easy to program, so we’ll be adding them in other useful places.

When you click the popup to edit or add a record, the current Goldenseal opens a modal dialog. It’s a separate window that takes control until you hit OK or Cancel. Modern versions of Mac and Windows both use drop-down panels instead. Panels work the same as the old modal dialogs, but they are attached to the current window. That makes them easier to find, and more logically attached to what you are already doing. Goldenseal Pro uses panels for almost everything that used to be in a modal dialog.

There are many places in Goldenseal where we show linked records. For example, in a Project account you might click a button to see Material Purchases for it. When you pay for a purchase, it shows the Bank Transaction that pays for it.

The current Goldenseal opens a separate window for linked records. The end result is often a bunch of different windows scattered about. Our own TurtleSoft file usually has 10 to 20 windows open. It makes the screen very cluttered. Eventually we close them all, but the clutter soon returns.

For Goldenseal Pro, we can use panels to show linked records, or tabs in the main window, or separate windows. As we start to use Goldenseal Pro to run our own company, we’ll be experimenting with different setups. It may be something that we control via Preferences, or by right-click options.

That part of the interface programming is going well. Unfortunately, the bad news is that Apple has decided to discontinue drawers as part of the Mac interface. The code still works, but it is deprecated and will stop working in some future version. Drawers are an additional type of window that slides out from the side, to show extra details. They are (were) perfect for showing breakdowns. We are still looking for a substitute.

Dennis Kolva
Programming Director
TurtleSoft.com

 

 

 

Goldenseal Pro Progress- Lists #2 (Jun 23)

When Apple launched OS X in 2001, they provided two different ways for developers to write apps.  One was called Carbon. It was an extended version of OS 9, accessed with the C++ programming language. The other was Cocoa. It was first developed for the Next computer, using a different language (Objective-C).  Software developers could use parts from both, but usually it made more sense to stick with one or the other.

TurtleSoft released Goldenseal accounting software for OS 9 in late 2000. It only took a few months to rewrite it so it would run on OS X, using Carbon. When Apple switched to Intel chips, it took another few months to rewrite the Goldenseal code for that.

Apple plans to release OS 10.14 (Mohave) this coming Fall. It will be the last version that supports 32-bit code. Unfortunately, Apple did not update Carbon from 32-bit to 64-bit, and does not plan to. Many Macintosh apps will need to either rewrite to Cocoa, or fade away after Fall 2019.

TurtleSoft started work on the conversion to Cocoa in September 2016. It soon became clear that this update would take more than a few months. All of the interface code needs to be rewritten completely in a new programming language. The only good news is that is has also been a chance to update and improve the way Goldenseal runs.

In Fall 2016, one of the first things we worked on was the list interface. It’s what you see when you edit categories, sales tax rates, billing rates, and other minor setup details. It’s relatively simple, so it was a good place to start.

The list window has a table that shows a list of list items. You can then select one, and edit it in a separate window. There are also buttons to add and delete items.

Back then, our staff was brand-new to Cocoa programming. We were working through tutorials and programming books to learn Objective-C and Cocoa. For lists, we adapted one of the sample projects from The Big Nerd Ranch. Unfortunately, the official Cocoa way to handle tables is complex. Miss one step and it won’t work. We ended up making a 20-step checklist to use for tables, since it was so easy to miss something.

Since then, we have built a half-dozen other types of tables. The official system ended up being just too complicated. We abandoned it, and wrote them more simply. It’s more like 4 or 5 steps now, instead of 20.

Last week we started to fix a few flaws in the list interface. Revisiting the original system was a return to nightmare. Our staff soon gave up, and rewrote the list tables completely. Now, they are similar to our other tables. They still look great, but are much easier to maintain. Still a few things to fiddle with, but we will soon be moving on to other interface details.

Dennis Kolva
Programming Director
TurtleSoft.com

Goldenseal Pro Progress- Lists (Jun 13)

Our new Goldenseal Pro accounting software is moving into the home stretch. Everything big is done, but there are still plenty of medium and small things to finish.

Right now, our staff is working on lists. Goldenseal includes about 70 of them. They handle all sorts of little details: job cost categories, payroll tax tables, sales tax rates, etc.

In general, lists are simpler than regular business records. Some are  just a name, and the others have only a few fields. Because of their simplicity, lists were the first things we worked on, back in 2016 when we first started the conversion to Cocoa (on Macintosh) and MFC (on Windows). Since then, the rest of the software interface has gotten ahead of the lists, so it’s time to catch up.

The current version of Goldenseal has lists scattered all over the place. Most are in submenus, under 12 different menu commands. It’s not always easy to find them. Originally, we planned to just duplicate the interface in our current accounting and estimating software. However, both Windows and Mac are moving away from top-bar menus, and putting controls inside the window instead. So, Goldenseal Pro still does the same stuff, but in just one window rather than many.

Goldenseal Pro already uses an outline view for navigation (similar to how you can organize your desktop files). With it, you can find accounts and business transactions more easily. We just put the lists there too. Now they are located near the business transactions that use them.

Back in 2016, the database code was still primitive. The list interface we wrote back then could show list records, but it didn’t make new ones, or save changes. Now that works, also. All that’s left are a few list quirks that need special treatment.

Dennis Kolva
Programming Director
TurtleSoft.com

 

 

Goldenseal Pro Progress (June 1)

There is a decades-old programmer’s joke that goes:

There are two hard things in computer programming: cache invalidation, naming things, and off-by-one errors.

This week the database code for our new accounting software finally started to work properly for very large files. It took almost a month, or twice as long as expected. Along the way we faced both of those three types of problems.

A cache is data in temporary storage, so it’s quicker to access.  Invalidation is just programmer talk for keeping the cache synchronized with the original data. Basically, the cache determines how long we keep accounting records stored in RAM, and when we save back to disk.

Our staff wrote the caching system for Goldenseal Pro almost 2 years ago. It worked great with small files. Then we started testing with our own company file, which is big enough to require six ‘sector managers’ to manage record locations. Anything put into the extras would eventually give errors. After dozens of debugger runs, it narrowed down to the cache system. Changes weren’t being saved.

Meanwhile, naming things is a chronic issue for all computer programmers. Poor naming is the main reason why code becomes impossible-to-understand ‘black boxes’. Good code tells a story, and explains exactly what it’s doing at every step. Getting it that way takes time and effort.

It wasn’t obvious why the cache system wasn’t saving sectors. Stepping through it, everything seemed OK. After a couple frustrating days of debugging, our staff rewrote the whole thing.  Same basic approach and almost identical code, but with names that made more sense.  Renaming everything was a good way to see how the cache interacted with other parts of the software. Finally, it became clear that sectors were indexed differently from everything else. After changing one line of code, it worked fine again. One minute to fix it, and three days to find it.

Then there are the off by one errors, sometimes called ‘off by on’. For anything involving numbers, it’s easy to get close but no cigar. It doesn’t help that there are math quirks. For example, the end of a record is at start+length-1, not start+length. It’s easy to forget that. Even worse, libraries written by true programmer nerds start counting at zero, while those written by mortals start at one. NeoAccess considered the file end to be the last byte of data, while C++ libraries put ‘end of file’ at one after the last byte. Etcetera.

Earlier this week, our TurtleSoft file was converting OK, but the file size was over 43 gigabytes. Obviously, a gap-finding problem. It took most of a day to track it down to code that called ResetToEnd on a list, thinking that it selected the last item. Nope, it was one past the last item. That name makes sense if you want to add something at the end, but not if you want to access the item at the end. Off by one.

Adding a ResetToLastItem function solved the problem, and file size dropped instantly by 800x.

Anyhow, we’re back to interface programming.

Dennis Kolva
Programming Director
TurtleSoft.com

 

Goldenseal Pro Progress- Black Boxes (May 25)

Remodelers rarely build kitchen appliances from scratch. Store-bought models run better than anything you could ever cob up on your own.

The same thing applies to software apps. It’s hard enough to write code that does construction estimating and accounting. There’s no sense reinventing wheels, so we use libraries written by other people for most of the basics. 

The good news is that there are plenty of software libraries out there. They’ll do just about anything. Most of them are free or cheap.

The bad news is that they are free or cheap. There is little incentive for their developers to polish things. It’s very rare to find a library that is fully reliable, well documented, and easy to use. Even if the source code is available, it’s often an ugly mess that someone whipped out on a caffeine high. Subtle bugs and security flaws probably lurk in there, but there’s no way to know for sure.

When you can’t view or fix the interior workings, it’s called a black box. Maybe it uses C++, or magnets, or squirrels.  There’s no way to know for sure. Whatever it is in there, about the best you can do is learn the quirks of the black box, and work around its flaws.

The NeoAccess database was definitely a black box. A dank basement with spiderwebs. For Goldenseal Pro, we considered switching to MySQL or some other database library. However, we soon discovered that most open-source software libraries are also black boxes, and just as ugly inside. Better the toxic danger zone you know, than the one you don’t.

Anyhow, last week our staff finally decided to completely rewrite the cache-management code. The first attempt, adapted from NeoAccess, was just too confusing. Chasing down its bugs was like wack-a-mole.

Basic design was not hard. Getting it to work on new files wasn’t too bad either. Then we went back and tested the conversion from older Goldenseal files to the new format. For a while it felt like this bathroom repair.  The conversion still uses NeoAccess to read the old file, and it was not happy with the changes.

It was time to put on the hazmat suits, climb in there, and update the NeoAccess cache system for a 64-bit world. We only need to run it one last time, but it’s an important one time. Our existing Goldenseal users will want to keep their old accounting data, just as much as we do.

Jiggling the cache system turned up several obscure bugs. It’s possible they weren’t hurting anything, but now that code actually does what it claims to do. After considerable effort, the conversion started working properly again. Previous testing was giving a few mystery database errors, but most of them disappeared. Sometimes it’s easier to just rebuild stuff, rather than fix it.  That is one problem with black boxes: sometimes the weird stuff you do to work around their flaws becomes its own problem, even after you replace the black box with something better.

We’re still finishing the File Manager code that kicks in when there are more than 32,000 records, but this latest spate of database work is nearing completion. Then we can get back to interface.

Most ‘black box’ libraries were someone’s one-time project, then they moved on to something else. Our software is exactly the opposite. We have been working on the same code base since 1995, and probably will still be working on it in 2035. So, the goal is to make Goldenseal Pro the opposite of a black box!

Dennis Kolva
Programming Director
TurtleSoft.com

 

 

 

Goldenseal Pro- File Manager (May 17)

There are many ways for a database to go bad. Losing records is a big problem, of course. Even worse is when one record writes over the top of another one. That corrupts at least one record. Sometimes the damage expands further.

A more subtle problem happens when records change: something that happens often in accounting and estimating software. Usually records get bigger, which means they no longer fit into their old place. It’s easiest to move them to the end of the file, but then there’s a gap at their former location. Do that enough times, and you end up with a huge file that is mostly empty space.

The NeoAccess database kept a ‘free list’ of empty spaces that was meant to prevent both those problems. New and changed records filled in the gaps, and kept the file compact.  However, there were occasional mystery bugs, and we suspected the free list was at fault.  If it ever forgot to remove a gap, then a record would be trashed when a second was written to the same place. We tried to debug the free list code but it was too confusing.

In 2002, we added the File Manager to help manage file contents. It kept its own list of gaps that duplicated the free list, and also stored the location of every record in the file.  That way there were two ways to make sure records were added safely. Goldenseal could find a gap, then double-check the neighborhood to make sure the space was really free.

At first the File Manager was just a diagnostic tool, but it soon replaced the old free list. Mystery over-write bugs disappeared. The Manager added an extra 10 bytes per record, but increased reliability was well worth the cost.

Unfortunately, the File Manager had a subtle flaw. It was by far the biggest record in the database (in the TurtleSoft file, it’s 2.5 megabytes).  If users were low on RAM, it was the first thing that wouldn’t fit. Nearly always, that led to a crash or a freeze, but the database would still be fine. However, if there was exactly enough RAM to load the Manager, but not enough to save it, it would die midway through the file save, and corrupt the database. It didn’t happen often but it was awful when it did.

Goldenseal users haven’t reported this problem in the past few years, probably because computers now have so many gigabytes of RAM. However, it’s still a design flaw.

In Goldenseal Pro we fix it by using multiple, smaller File Managers. There’s one for each sector in the file, enough to hold 32,000 records. Managers are accessed via a very broad, 2-level tree that is similar to the ones used to index records. It’s safer and a teeny bit faster than the old system, with less data to read from the drive.

The old File Manager kept growing larger as more records were added. Periodically, it had to be relocated in the file. That was when it was most likely to die, and kill the file with it.  In Goldenseal Pro, each File Manager is a fixed size. When it fills, we just add another one. That also makes it safer.

Essentially, Goldenseal Pro stores the location of every record in two different places: one sorted by record ID, and one sorted by location in the file. There’s also a separate list of file gaps that helps keep the file compact. The system is more complicated than the previous File Manager, but not by much. The code is much more understandable, so we’ll be able to repair damage to either half of the record storage.

Right now, the system can handle a billion records. If any users ever get close to that amount, we’ll add code to allow a 3-level tree. That will jump it into the trillions.

Dennis Kolva
Programming Director
TurtleSoft.com

 

Memory & Speed (May 10)

When our staff first started work on Goldenseal accounting software, computers had 1 to 4 megabytes of RAM. That was sparse. Programmers had to work hard to conserve memory. It’s probably why the NeoAccess database library used such a skinny tree to locate records on disk. It only consumed 9 or 10 bytes per record.

Back then, processors ran at 16 to 32 megahertz. Speed was another big concern. We originally chose the NeoAccess database because it was fast, thanks to record caching.  A cache means that each record sticks around in RAM for a while, after it is opened. Reading data in RAM is many times faster than the hard drive, so a cache saves time for anything used more than once. Of course, there is only so much room in RAM, so a cache must be managed carefully. Our programmers thought hard about when to write records to disk, and when to remove items from the cache.

These days, computers have gigabytes of RAM (a thousand times more than the 90s). Processors run at a few gigahertz (a hundred times faster). Programmers can spend less time worrying about memory and clock cycles. They can focus more on reliability and maintainability.

Our staff has been using the Sample Company File for testing since late 2016. It’s a small file. A few weeks ago we converted our own TurtleSoft file to the new format, in prep for using it daily. With 10,000s of records instead of 100s, it’s a chance to stress-test and fine-tune the database code.

NeoAccess had only one way to locate a record on disk: climb through the index tree, and find its ID and file mark (4 bytes each). It was fast, and saved memory. Unfortunately, if any bits were damaged along the way, the record was lost forever (or read from the wrong part of the file, producing garbage).  In 2002 we added a File Manager to store record addresses in a second place. It definitely reduced database errors. Unfortunately, it had its own flaws, and never quite lived up to its potential. More about that next week.

As mentioned a couple weeks ago, Goldenseal Pro is more ‘webby’, with extra links, and ways to recover damaged data. To make that happen, it now uses about 80 bytes for each record location, instead of 10. We may even add more. The extra links, padding and redundancy make Goldenseal Pro files about triple the size. However, they are still a tiny fraction of a modern terabyte drive. For a few extra cents of drive space, it’s well worth the increase in reliability.

For speed, Goldenseal Pro still uses a record cache. However, it needed a complete rewrite. The old NeoAccess code worked OK, but it was very hard to understand and debug. Their cache also relied on their indexing system, which we’ve replaced. Back in 2002 we added code to help with cache diagnostics. It has been running in parallel for 16 years, and now it takes over completely.

Managing the cache is tricky. Some records are used more than once, so the cache needs ‘reference counts’ to keep them open until everything is finished.  Mess up the count, and the program will crash (if the record is removed prematurely) or leak memory (if never removed). Memory leaks don’t cause immediate problems, but eventually they use up all available RAM and crash. Even worse, memory leaks cause a crash at some random later time, which makes them hard to debug.

Working on database code is probably the hardest possible programming. It’s complicated.  Fortunately, there also is plenty of smaller stuff to work on, when programmer brain cells are not cranked up to 11.

Dennis Kolva
Programming Director
TurtleSoft.com

 

Goldenseal Pro Progress- Back to Databasics (May 3)

Back in 1999, we started using real data to test beta versions of Goldenseal accounting software. There was an unpleasant surprise. Intermittently, the software gave mystery NeoAccess errors, then crashed with a corrupted database. The only recourse was to trash the file and start over. We reported the bugs to NeoAccess support, but by then they had stopped answering emails.

We considered switching to a different database library, but couldn’t find anything better. Out of desperation, our staff spent half a year rewriting the NeoAccess code to make it more reliable. We found and fixed at least two serious bugs. Most likely, the rewrites accidentally fixed a few more. After that, the code ran well enough for a version 1.0 release.

The database still had rare problems, so we added a bunch of diagnostic commands to better understand what was happening inside the file. With their help, we tracked down a few more subtle bugs, and squashed the last of them in 2004.

Now that we are testing Goldenseal Pro with our own real data, we’re seeing database bugs again. The software adds many thousands of records just fine, but sometimes it has failed after deleting or changing them. This time around, bugs are entirely expected, since it’s brand new code getting its first stress test. The schedule has 2 weeks allocated for database bug fixes, and so far that seems about right.

Fortunately, the new database code is much cleaner and more understandable. Bug fixes usually take hours instead of months.

Up until now, we have been able to step through the database code in the debugger to find and fix problems. However, as the amount of data increases, the bugs grow more obscure (and harder to duplicate). So, we just added some diagnostic commands again. Being able to see interior details helped catch the deletion bug. Right now Goldenseal Pro is working fine, although there are almost certainly more bugs lurking.

You might wonder why we don’t just write bug-free code to start with. And yeah, we agree, that would sure be nice. Having perfect code in the first run would save everyone a lot of time.

What prevents that is the same reason that construction projects have punch lists. Why first drafts have tyops and bad grammar. Why cars aren’t perfect, even after 130 years of engineering. Complex things are difficult, and creators can’t anticipate every possible problem. Frequently, errors need to occur at least once, before you even know they’re a problem. Not to mention, just plain old random omissions and mistakes.

Since we can’t write perfect code, we spend a lot of time testing and debugging. The software gets closer and closer to perfect.

Right now Goldenseal Pro is in that rough polishing stage. Polishing is a slow, grinding process that takes time before it produces something shiny.

Dennis Kolva
Programming Director
TurtleSoft.com

 

Trees vs Webs (Apr 26)

Goldenseal currently manages its accounting data with NeoAccess (a popular object database library in the 1990s). To locate records, it uses a binary tree.

You may have used binary search while looking for something in an old-fashioned paper phone book or dictionary. Open at the middle, and compare it to what you want. Go left or right depending, then open the middle of that half. Repeat with smaller and smaller divisions until you get to the right page.

Binary search is very efficient, especially when the data is huge. It takes 10 steps to cover a thousand items, 20 steps for a million, 30 for a billion, and so on.

It’s easy to set up a binary search in sorted items of identical size. However, that is rarely the case for database records. For those, it’s hard to find the middle, so they are better managed with a binary tree. The tree has a single root, which grows increasingly ‘branchy’ as you go up. To find something you start at the bottom, compare at each node, then move up through the tree. Eventually you arrive at a leaf node, which links to the actual record. This kind of navigational tree is also called an index.

NeoAccess was optimized for speed and small size. Each node included a list of file addresses for its children, and nothing more. Sparseness made sense, back when memory was expensive and processors were slow.

Unfortunately, a lean index is also very fragile. If a NeoAccess node is corrupted, everything above it becomes invisible. Those records are still on the hard drive, but they lack a path that says where they are. NeoAccess had no alternate route. Any damage was fatal.

These days, memory is cheap and processors are very fast. As a result, it’s nearly always better to build software for reliability rather than speed. Probably the best example for bullet-proof design is the Internet. It began as Arpanet, which was meant to keep running even after nuclear annihilation. Turns out, that kind of sturdiness helps, even in a world of small, normal, ordinary snafus.

Instead of a tree, the Internet is a web. Lots of connections from here to there. There is always a way to find something, even if one path breaks. Cell towers work the same way.

Goldenseal Pro still use a tree structure for normal finds. But it also includes more web-like connections and alternate routes. With their help, it should be able to rescue records, even if a tree branch is damaged. The accounting database can even be self-repairing.

Meanwhile, we still need to use the old NeoAccess code to read data from the current Goldenseal accounting software, so it can move into the new Pro format. Unfortunately, the Sales breakdowns in our TurtleSoft data have some corrupted index nodes. More than half can’t be found. Without them, we won’t know what people bought.

Fortunately, coming to the rescue is the File Manager, which we added to NeoAccess in 2002. Its main function is to prevent records from overwriting each other, but it also provides an alternate way to find record locations on the disk. We just added some code to it, so it can read and recover lost records when the tree can’t find them. Voilà, it rescued our missing data! The new code will also help any other users with damaged trees.

There is a cost for the extra connections, webbishness and redundancy. Our old TurtleSoft company file was 51 megabytes, and the new one weighs in at 127. It will probably grow even larger, as we add more safety features.

3x the file size for 3x the reliability seems a worthy trade.

Dennis Kolva
Programming Director
TurtleSoft.com