Category: Goldenseal news

Our staff is back at work on Goldenseal Pro. The first thing we faced was a new problem while editing list items. The OK and Cancel buttons used to work fine, but suddenly they didn’t.  It turned into a journey down the rabbit hole of Cocoa memory management.

Memory is a general problem for all programmers. Apps need to use many small chunks of RAM: thousands, sometimes millions of them. If you don’t delete them when done, it’s a memory leak. Leaks cause the app to gradually use up more and more memory space. Eventually it will get slow. Pile up enough leaks, and it will crash. Web browsers often do that.

Preventing leaks is not easy. Delete something too early, and the program will fail when you access a long-gone bit of memory. Usually the app crashes from that, but sometimes it reads corrupted data and causes a hard-to-fix bug. Wait too long, and there’s no way to go back to tidy up.

The C and C++ programming languages are considered “hard” because they require programmers to manage raw memory.  On the other hand, most newer programming languages do the memory management for you. Usually that involves periodic garbage collection: automatically deleting objects from memory, when they are no longer needed.

Apple’s Cocoa framework started out by requiring programmers to manage their own memory, just like C++. It then added its own memory manager in 2010. Rather than garbage collection, Cocoa uses something called ARC (automatic reference counting).

We turned on ARC when we first started programming Goldenseal Pro for Mac. Sometime in the past year or two it was accidentally turned it off, possibly during an Xcode update. The setting is buried in a list of 200+ other build settings, so it’s easy to overlook.

Removing ARC caused no noticeable problems, since the app was never used long enough to show leaks. Finally we noticed the setting this March, and turned ARC back on. That broke the buttons, and a few other things.

Our staff has worked with ARC long enough now that I feel confident in saying it’s the worst of both worlds. Yet another complaint about Cocoa.

It’s possible to write code in Python, Java, Javascript or other garbage-collected languages, and simply not worry about memory. It’s all taken care of for you. During run time there may be weird multi-second delays as the app cleans up memory, but it’s a small price to pay for convenience.

It’s also possible to use C or C++, and manage memory reliably. Many tricks exist to make it easier. We used them for the original Goldenseal, and it almost never leaks or crashes.  Recent versions of C++ provide even better tools for managing memory. We use them in any new code we write.

With its in-between approach, ARC is definitely not mindless. It makes you decide whether to make connections “strong” or “weak”. It sometimes deletes things unexpectedly. It also is hard to debug. There are tricks to help track down memory problems, but nothing as simple as in C++.

Despite all that, the buttons are working again. Then for a while, the app crashed when editing a second list item. That is fixed too. Both were general “object lifetime” problems caused by ARC. They also applied to code in other places, so all those are also fixed now. Sometimes it’s one step backwards, two steps forward.

Dennis Kolva
Programming Director
TurtleSoft.com

 

 

Our staff took a short break from Goldenseal Pro, the past couple weeks. Instead, we’ve been finishing up the shopping cart for SmartKnives.com.  It’s almost done. A contractor did most of the work, but we needed to import inventory data, and tweak the appearance.

As app developers, it has been interesting to visit the world of e-commerce and website development.  Things are very different there.

When building an app like Goldenseal Pro, programmers spend most of their time in source code. For us it’s a couple thousand text files, mostly in the C++ programming language. Some of the files are 25 years old. We make life easier by writing clean code, and adding comments to explain what’s going on. When something doesn’t work, there’s a debugger that shows what is happening, line by line. Most of the code is very well polished.

App developers also use libraries, usually written by some big entity like Microsoft or Apple. Some libraries show their source code, and some don’t. It’s always harder to work with code written by other people. It’s even harder when the source code isn’t visible. But, lots of people use the same libraries. For most problems, there usually is an answer on Stack Overflow or a developer forum.

Web development is much crazier.

It’s possible to create simple web pages by just writing some html and css text files. But, to accomplish anything complicated, programmers must use layers upon layers upon layers of other people’s code. Most of it is open-source. Most is in a state of utter chaos.

App developers occasionally make hacks- an ugly bit of code that does the job. It’s the duct tape of the programming world. Eventually we refactor them into good, solid code.

E-commerce code is almost entirely hacks. It’s an empire built entirely of duct tape. No debuggers. Lots of poorly written code that is rarely fixed. Lots of guessing.

It will be good to get back to the comfort zone of Goldenseal Pro source code. We’ll probably appreciate the Cocoa libraries more.

Dennis Kolva
Programming Director
TurtleSoft.com

When we first designed Goldenseal accounting software, banking was heavily regulated. There were clear distinctions between different types of banks, so it made sense to have different types of bank accounts. Then, deregulation and the Internet gradually made it all blurry.

Goldenseal Pro uses a new file format. When you open an old-format file, it automatically converts the data. The process is an opportunity to do other conversions, so it also condenses the original 7 types of bank accounts down to just 3. Investment accounts are special, because they can be job costed. Ditto for Escrows, since they aren’t assets. Everything else is now just a plain old Bank account. We added a popup menu that you can still use to distinguish the old types (plus new banking types that have arisen since then).

Up until now, our staff has done most testing on expense transactions. They are relatively simple, which made them a good starting point. Bank transactions are more complicated. The biggest problem is that every account gets a separate window. That is different from everything else, which just has one class per window. There is a lot of interface code that does something special for bank transactions. It’s working now, but it took a few days to find all the quirks. The old system to display bank transactions was rather clunky, but it’s much better now.

We also revised the indexes so the Reconcile command is much faster. When we used Reconcile for our own bank statements, we discovered that start balances were slightly off. That tracked down to a few dud transactions in the early Aughties. The original Goldenseal ignored them, but Goldenseal Pro didn’t. The change is annoying because it throws off the start balance for all later periods. We’re still deciding how best to handle old, corrupted data like that. Turtlesoft sometimes used beta versions for daily data entry, so it’s possible that other business users won’t run into the same problems.

Bank transactions are probably the third-most difficult thing in Goldenseal. Estimates are more complicated, and Payroll is more complicated yet. We are getting close to fixing all the banking bugs. Then it’s on to the tougher stuff.

Dennis Kolva
Programming Director
TurtleSoft.com

Besides the crashing bug that we fixed recently, there has also been a long-time problem with window layouts. They functioned, but didn’t resize in a pretty way.

Apple has at least 4 different methods to position things on the screen. They started with plain old x and y coordinates, plus a springs and struts model to move things when you resize the window. Then they added constraints, a more complicated version of springs and struts. Finally they added anchors, yet another method for window setup. It’s a confusing jumble.

Last week we decided to finally get it right. But, for a while, everything we tried just made things worse. At one point, switching between tabs caused the whole window to jump around on the screen. Then we discovered setTranslatesAutoResizingMaskIntoContraints, a poorly-documented bit of magic that solves most conflicts between the different systems. After a bit more tweaking, the main window finally works like it should.

In the last post I compared Apple’s Cocoa library to Serius Developer, an unusable platform we tried briefly in the early 90s. Serius was close to fradulent, but I didn’t mean to imply that Cocoa is that way.

I think the main problem is that Apple has developed the Cocoa framework primarily to build their own apps.  Making it work well for other software companies is a relatively low priority. So, Cocoa is great for building apps that are similar to something from Apple. The more you diverge, the harder it gets.

Goldenseal Pro uses NSOutlineView to replace most of the current menu commands. It only took a week to set up, and it has worked great ever since. Apple uses that class for the Finder, so they must have really polished it. It was one of the first things we programmed 2 years ago, and the ease gave us a false sense of optimism.

Getting NSTabView to work properly was much harder. Apple uses it for Preferences, but our tabs are more complicated. Our staff has spent a couple months getting them right.

The NSTableView class that we use for breakdown tables has been a nightmare that consumed almost 3/4 of a year. Apple built a spreadsheet app, but after exploring its package I’m pretty sure that Numbers doesn’t use NSTableView.  With hindsight, we probably should have adapted our existing table code for breakdowns, and ignored NSTableView.

At any rate, our staff has fixed all the big problems. Goldenseal Pro is stable enough that we just started to use it to run our own business. There are plenty of small bugs still lurking, but they don’t take long to fix. Daily use is now daily testing. Eating our own dog food.

When the app stops being frustrating, we will release a public beta version. Best guess: in a month or so.

Dennis Kolva
Programming Director
TurtleSoft.com

 

Goldenseal Pro has a new, single-window appearance. To move between different functions, you click on on tabs rather than separate windows. We’ve used it enough now to really like the change.

Unfortunately, the tab view has had a serious bug: crashing after a few tab switches. It started about a year ago, when we updated to Apple’s newest memory management system. About once a month, we’ve spent a day or two trying to fix it, without success.

Our staff spent most of the past week attacking it again. We finally cracked it Wednesday, with help from folks on the Cocoa Developer Forum. The root cause was some unexpected behavior in Apple’s NSTabViewController class, complicated by the fact that we can’t see Cocoa source code.

We have been using Apple’s Cocoa framework for 2.5 years now. It is probably the most difficult programming environment our staff has ever used. Unfortunately, there are no viable alternatives for building software that runs on future versions of Macintosh OS. We are getting there, but it has been a real struggle.

As an attempt to quantify the difficulty of programming environments, I’ve decided to invent the Serius Scale. It’s a measure of programmer pain.

0 on the Serius Scale is an absolute joy for programmers. Productive days that create useful code, quickly. 10 on the scale is a tool that causes maximum frustration, and never results in anything useable. Right now, I’d rate Cocoa at about a 6, maybe a 7. Apple really could do better.

The scale is based on Serius Developer, a software development tool that came out in 1989. It had an amazing demo that included a built-in spreadsheet. We were desperate for a better platform for our construction estimating and accounting software, so we plunked down $2000 for it.

Unfortunately, we soon discovered that their spreadsheet tool was hard-coded to run the demo, and nothing else. Other features were similar. There was no way to get it to do what we wanted. Serius was a totally useless black box. Its developer later moved on to start a Ponzi-like Internet company, then got into space aliens. So, I’ll rate his software a 10 on the eponymous Serius Scale.

Before that, TurtleSoft started out in 1987 with simple construction estimating templates for Excel. They took less than a year to develop, and sold like hotcakes. That was a fantastic time to be in software.

I’d give early versions of Excel a 1 or 2 on the Serius scale. The macro language had its quirks, but it was pretty decent for its day. Things improved in Excel 2.2 and 3.0, but then went seriously downhill. Excel 4.0 added a bug that randomly changed our code. Then the macro language was replaced with Visual Basic. That’s why we were so desperate to find a better way to produce software.

Back then, we also shipped a few products based on Apple’s HyperCard. It was a fun app that could whip up something functional, very quickly. At the beginning, I probably would have given it a pain-free 0 or 1. Unfortunately, after using it to build real software, we discovered race conditions. HyperCard worked by passing messages around, and they weren’t always received in the same order. It made hard-to-find, intermittent bugs. Apple never fixed them. Instead, HyperCard languished, and was never updated for OS X.  I guess the net result is a high Serius number. It just never lived up to its potential.

After fiddling with FileMaker, Omnis, Double Helix and a half-dozen other platforms, we finally decided to use plain old C++ for Goldenseal. It was built upon the PowerPlant framework that came with Metrowerks CodeWarrior. Basic, but easy to understand. About a 2 on the Serius scale. Metrowerks was a small, competent company. Very pleasant to work with. Life was good again, for a while.

Unfortunately, Metrowerks was bought and strip-mined by Motorola. They sold off the Intel version of CodeWarrior a month before Apple switched from Motorola chips to Intel. A quick end for CodeWarrior and PowerPlant.

Unfortunately, this post has been using “unfortunately” a lot. We would like to spend our time writing code for estimating, accounting, payroll, and other functions that users need. But, looking back, at least 75% or 80% of our programmer days have been spent on OS and hardware issues. Too much time struggling with tools that are high on the Serius Scale.

The good news is, we are making progress. It’s hard, but not impossible.

Dennis Kolva
Programming Director
TurtleSoft.com

Our staff is well into the miscellaneous phase of the Goldenseal Pro project. That means tidying up small details until it’s all shiny.

First thing, we enhanced the database verification reports so they catch a few more potential errors. The reports will make the final testing go much faster. Rather than waiting around to see if file error messages pop up, we can run Verify File a few times a day. It will notice any problems right away.

It’s quite possible there won’t be any more database issues. The data conversion from version 4.96 to version 5.0 already futzes with many thousands of records, and it may already have hit every possible problem. Really, database code is not that huge a task. It’s much simpler than construction estimating or payroll. A database has to locate records on disk, make sure they don’t write on top of each other, and manage gaps (otherwise the file becomes an enormous hunk of Swiss cheese). Most of our database code is just sanity checking, and duplicating info so it’s possible to recover if data is corrupted by the outside world.

Since we are already converting data from old format to new, we decided to simplify bank accounts and transactions in the new accounting software. Goldenseal Pro only has 3 types of bank accounts instead of the current 7. The update process converts any old records, and also converts links between records. As we started using the Reconcile command, one missed link turned up. We fixed it.

Right now, Passwords are on the plate. I personally designed the Goldenseal interface, with little sliders to set the access each employee has to different parts of the data. It looks nice, but it was so non-standard that it took more than a month of programming. Too much time for a minor feature that many companies don’t even use.  In Goldenseal Pro it won’t look quite as nice, but it will use existing code. One more day to finish it, instead of a month.

Our staff started work on the Custom Layouts view more than a year ago. Some things work, but we got stuck on a few things and moved on to other code. Our staff is revisiting it, now that we’re more experienced with Cocoa.  It’s the last major interface element. The first release doesn’t absolutely need Custom Layouts, but it will make Passwords easier to finish.

Dennis Kolva
Programming Director
TurtleSoft.com

The past week, our staff has focused on the database code that runs our new accounting software. First thing was to update the Verify File command to work for Goldenseal Pro. It reads every record, runs a few sanity tests, then reports any errors. When we ran the verify, the Sample Company File showed 4 errors, and our Turtlesoft file had 8.

As it turned out, almost every error was caused by something different.  They were all very obscure problems. Hard to find and hard to figure out, so it took a full week to fix them all.

Along the way, we added a few other commands that test the integrity of the database structures. They helped with the bug fixes. Even better, they’ll show any future file errors before they become serious problems.

Like almost all databases, Goldenseal Pro locates records with a tree structure. Searches in a tree start at the root, then travel up through branches to the leaf nodes: one for each record. Actually, Goldenseal Pro uses a few thousand branches at each node, so it looks more like crabgrass than an actual tree. It’s faster that way.

With such fat nodes, it only takes two levels to handle anything under a few million records. None of our test files are even close to that big, so we temporarily turned one class into a binary tree, with just two records per node.  For 900 records it built a 10-level binary tree, and that tested out fine. So we are now confident that our accounting database can handle at least a few nonillion records. That’s a thousand billion billion billion, or a 1 followed by 30 zeros.

Our current company file has a bad NeoAccess node, which breaks several thousand Sales records from the early 1990s. Fortunately, we found a way to recover all but 8 of them, when converting from the old format to the new.

There are times when it feels insane to be writing our own database code, rather than using something well-established and open-source. So, I recently downloaded the MySQL source code to see what we are missing. It is very old C and C++ code. Hard to read and understand, and probably very hard to debug. Many potential security flaws and time bombs. Its style is similar to the NeoAccess code that we struggled with, partly rewrote, then finally abandoned. However, MySQL has several thousand source files rather than a couple hundred. It’s probably even more convoluted.

I think we made the right choice. It’s nice to have a report that explains every bit within the accounting file.

Dennis Kolva
Programming Director
TurtleSoft.com

Our staff is making steady progress on the Mac version of Goldenseal Pro.

The Reconcile command is now functional, with a table that shows a list of unreconciled bank transactions. As usual, it was a struggle to get the table working properly. The NSTableView class does not forgive small errors. Fortunately, we can duplicate it for Pay Bills, Write Payroll, and a few other commands, so they will go much faster. Even better, it’s the last type of table that’s needed for Goldenseal Pro.

We tested the Reconcile command with our TurtleSoft bank account. It has over 10,000 transactions, which take almost a minute to process. First thing we did was to rewrite the search code. Now it eliminates most records quickly, and only takes a few seconds.

During the test, there were a few database error messages. Not unexpected, since this is the “shakeout cruise” to find problems.

Goldenseal uses a record cache to manage how long data sticks around in RAM. It helps to give good performance, without consuming a lot of memory. The new version has been working pretty well, but the test gave a couple of obscure cache error messages. We figured out why, and fixed it. The fix made things worse. Fixing those was even worse: crashing instead of error messages. For a while it felt like an episode of House MD, with the patient gradually dying from some unknown, obscure disease. Fortunately, after an overhaul of the way we create records, the record cache is finally cured. Just like House MD.

There also were serious errors in a few of the bank transactions. Most likely, we missed some data structures that need revising for 64-bit. Having a partly-defective file is a good opportunity to soup up the Verify File command, so our staff is working on that now. The goal is to get a more detailed problem report, so we know exactly what to fix. Kinda like building an MRI machine for files.

Along the way, we have been fiddling with custom fields, pasted-in images and a few other details. Most of the work is in familiar code, so it’s much more pleasant than working with the Cocoa library.

Dennis Kolva
Programming Director
TurtleSoft.com

 

We are starting to use our own data to test Goldenseal Pro accounting software. There’s lots of small stuff to do, to get it ready for running TurtleSoft.

First step was the sign-on dialog. It’s now a bit better than the current version, with an option to view or hide the password as you type it.

Ever since we switched to Apple’s most recent memory management system, the main window has crashed randomly after switching tabs. We finally narrowed it down to one line of code, but were still baffled. Something was breaking that never should have. It finally turned out to be an old Apple bug in the NSTabView class. We posted details to the Cocoa developer list, and folks gave some suggestions that eventually led to a work-around. Tabs are working properly again. We switched them to newer Apple code, so they even have niftier animation now.

There have been database errors in bank accounts that were converted from old format to new. Duplicate ID numbers. The whole system to give record IDs has always been a problem, ever since Goldenseal 1.0. Its code is confusing. Bug fixes over the years have only made it more confusing. So, we tossed it all and rewrote everything from scratch. It’s simpler now, and will be much easier to maintain.  Software design is much easier after you’ve made every possible mistake already.

We are also about half-way through the Reconcile command. That uses a table. They have been the hardest programming problem so far, in the Mac update. Fortunately, several types of tables are now working, so we can duplicate that code. Our staff futzed around with tables all Winter, but they will go faster now.

I’m not even going to try to estimate completion. Our staff will be finishing features and fixing bugs until it’s done.

Dennis Kolva
Programming Director
TurtleSoft.com

 

 

All the basics are now working for the tables in lists and accounts. Menus pop up, check boxes change, and text saves. Our staff is double-checking all of the tables now, to find any small details that may need extra code.

Even better, our staff can use the same basic code for other types of tables. We aren’t masters of NSTableView yet, but at least we aren’t baffled by it any more. The home stretch is in sight.

In the current Goldenseal, the C++ table classes handle mouse clicks, screen drawing, and everything in between. For Goldenseal Pro, NSTableView takes over all the mouse stuff and screen drawing. So, the remaining work will be in the special details in the middle, plus deleting lots of obsolete code.

The first step was to entirely remove the old messaging system from Goldenseal.

There are lots of ways for users to interact with an app: they can click on something, or double-click, or click and drag. They can type stuff from the keyboard. Most actions need to be passed along elsewhere in the code, to actually do something. Most platforms do that by sending a message. Apple’s Cocoa framework is extremely message-heavy. Microsoft’s MFC framework and the PowerPlant framework also use messages, but less so.

The current version of Goldenseal has a few hundred different messages that go traveling around. Unfortunately, the whole message thing turned out to be one of those ideas that sounds great in theory, but never lives up to its promise.

The biggest problem has been that messages are hard to debug. All code starts out with bugs, but what counts is how easy it is to fix them. With messages, the code doesn’t directly explain how it works. One side sends a message and the other side receives it, but there’s no way to be sure it actually gets to the right place. To check it requires 2o or 50 steps in the debugger, instead of just one or two.

Even worse, messages can be sent at the wrong time, or in the wrong order. Debugging that kind of error is a nightmare. Sometimes we have to log every single message, and then try to deduce what’s wrong.

As our staff fixed bugs in Goldenseal between 2001 and present, we solved many problems by replacing messages with simple, direct connections. Often, a complete rewrite was easier than trying to debug and fix the messages.

The original plan was to replace the remaining messages gradually.  But then we discovered an extremely old bug in the Work Crews command. It’s not a biggie, but enough to inspire extra skepticism for the existing code. If we’re going to test every last detail, then there’s little reason to keep any remaining message-based code.  Better to delete it all, and reduce clutter.

BTW the message system inside the Cocoa framework drives us crazy. It’s one of the major reasons why this conversion is taking so long. I’ll complain more specifically in a future post.

Dennis Kolva
Programming Director
TurtleSoft.com