Turtlesoft Blog

How to classify things is a problem in both estimating and accounting software. It’s particularly difficult for job cost accounting. Expenses come in randomly, and you have to match them to vendors, jobs and categories.

One important principle in interface design is that people can recognize something much faster than they can recall it. It’s one reason why buffets are popular. No need to remember General Tso’s name. Just scoop and eat. 

Typically, software uses popup buttons for list choices. You click, see the list, and then choose what you like. However, that approach doesn’t work well for long lists (over 30 items). They get too hard to navigate. Popups also require a mouse-click, which is inconvenient when data entering from a keyboard. 

Bruce Tognazzini, the author of Apple’s Human Interface Guidelines, proposed a solution to both problems in the early 1990s. It’s a text field you can tab into. But, when you type, it shows a scrolling list, and jumps to the nearest match. Tab out or hit the enter key, and it selects that item. If you’re not sure what to type, you can also click and scroll until you see what you want. He called it a disambiguating field. At the time, FileMaker had a lookup field that was sorta like that, but his version was better.

In our older MacNail software, we didn’t even have popup menus. Users had to type in a number or letter code to do job costing. Finding a better way to handle long lists of jobs or categories was one reason why we abandoned Excel, and built a stand-alone app in C++. We copied Apple’s specs but called them clairvoyant fields, slightly less vague than disambiguating.

Programming clairvoyants required a text field, plus a temporary table for the list, plus a popup button to choose items with a mouse click. Interactions between them were complicated. Users could do many things with the mouse or keyboard, and the code had to handle them all. Lists might contain thousands of items, so it was challenging to get decent performance on 1990s hardware. We spent a full programmer-year getting clairvoyant fields to work right.

Since that time, both Macintosh and Windows have added a combo box control. It’s almost identical to the original disambiguating field, so we were delighted that we could use them for Goldenseal Pro. Kinda like, we had to build them from scratch back in the 90s, but now you can buy them at Walmart.

It took a few hours to get combo boxes working on Windows. A few days for Macintosh, but we ran into a problem. If you typed something not in the list, it just added that text. OK for some uses, but not for job cost accounting. You don’t want to start posting expenses to the Snith job just because you made a typo.

Apple’s web page for combo boxes said it was possible to restrict them to existing items. Our staff couldn’t find sample code to do that, and couldn’t figure how to make them work right. Eventually we built a replacement from a text field, temporary table and popup button, just like before.

Doing it that way was still just as complicated. Three or four times we got frustrated, and went back to the Cocoa combo box to try again. Then we kept reverting, and gradually perfected the built-from-scratch version.

A few weeks ago, we posted to Apple’s Cocoa-dev mailing asking for help.  Unfortunately, nobody there knew how to make them work right, either. Almost everyone insisted we should use popup buttons instead of a combo box. Out of desperation, we actually tried popups as suggested, but most of their old flaws still remain.

So, the dream of easy Mac combo boxes was just a dream. Windows has a simple yes/no option to restrict to existing, and you’d think that Cocoa would do the same. But, nope.

Building combo boxes from scratch has taken some time. Fortunately, after a couple more weeks of effort, they are working well. Unfortunately, they will look slightly different between Mac and Windows, and we may need to give them different names.

Dennis Kolva
Programming Director
TurtleSoft.com

 

I recently came across an article explaining why it’s hard to estimate programming projects. Still, our users want to know when our Pro update will be finished, and we do too. It’s worth a try, even if the effort is guaranteed to be flawed.

March was the last time we estimated a completion date for Goldenseal Pro, and that predicted mid-October. We just updated the data. The revised spreadsheet also calculates mid-October, but only for the Mac version of our new accounting software. It adds another 2.5 months to finish the Windows version.

It’s still very much a guesstimate. The work in our code is going about as expected. However, occasionally the Cocoa framework throws a curve ball. When that happens, it can take a few extra days or weeks, to finish something trivial. If there are a lot more curve balls, the work could easily drag on a few months longer. It’s an unknown unknown. We really won’t know what we don’t know until it’s over.

Some of the prep work for the Mac version will probably make the Windows update go faster. It’s even possible that Windows part of the estimate is too conservative. But the MFC framework does throw a few curve balls of its own. Those are the main uncertainty.  However, on the whole it is just less complicated to program for Windows. It definitely will take less time than the Mac.  Working with the Cocoa framework is difficult.

Apple has a Cocoa listserv, mostly viewed by developers building software for Mac and iPhone. We have posted a few questions there when stumped. Usually, the other programmers are stumped also. The list does not get much traffic. Likewise, Stack Overflow and other programmer exchanges have few recent posts for Macintosh Cocoa. Nearly all recent questions are for Swift and iOS.

Apple is ending support for Carbon and 32-bit apps in 2019. You would think there would be more developers in our position: rewriting and updating existing Carbon apps to use Cocoa. On the other hand, there were many Mac programs that did not survive the transition to OS X. Moving to Cocoa is even harder, so many developers may have decided to retire. In all honesty, if we had known 3 years ago how difficult Cocoa was, we’d have thrown in the towel then, also. But, our staff is now close enough to completion that it would be absurd not to finish.

I did remodeling work for an accounting professor who had a clever explanation for why people under-estimate projects. Essentially, one big difference between humans and other mammals is that we look further ahead. Building a house or a dam might take 10 years to break even, and no other species would make that gamble. Beavers and apes do a bit of long-range planning, but only on a span of months.

Human brains could have developed stronger foresight, but evolution is random. It uses the first thing that works, even if inferior. So, maybe humans brains just adjusted by under-estimating the cost of big projects, and over-estimating the rewards. Delusion has the same result as wisdom.

Anyhow, Goldenseal Pro looks much better than the current version. It will be good in the long run. It’s good that we were deluded.

Dennis Kolva
Programming Director
TurtleSoft.com

 

 

 

This week we are finishing up the last interface details in the main window for Goldenseal Pro. We like how it looks, so it’s time to release the second alpha version. To download it, click here.  It runs on Mac OS 10.11 (El Capitan) and newer.

We still are not using it to run TurtleSoft, and this version is definitely not ready for actual use. But there are two things you can do with it, that will help us (and you).

First of all, please check out the new interface, and let us know what you think of it. There is still time for us to make changes or improvements. It’s easier now, than later.

Secondly, please try converting your company file to the new format, and let us know if there are any problems. That way, we can fix them before final release.

Follow these steps to run the conversion:

1. Just to be extra careful, make a backup copy of your current company file.
2. Add .glds to the end of your company file’s name, if not already there.
3. Download the alpha version if you haven’t already done so.
4. Double-click on the zip file to decompress it (if your browser didn’t do that automatically).
5. Open the GS-Pro-Alpha folder, and double-click on the GS-Pro-19-Jul-2018 application icon.
6. If your security settings give a warning about an unidentified developer, right-click on the app icon, and choose Open.
7. When it asks you to open a file, locate your current company file, and open it.
8. The app will ask you if you want to convert to the new format. Click OK.
9. The app will ask you to save the new file. Give it any new name, choose a convenient location, and save it.  Your current file will not be changed.
10. It will take a minute or two to convert your data to the new file format. When the conversion finishes, you’ll see a report of what happened, and any problems that occurred.
11. Choose Quit from the Goldenseal menu.
12. Double-click on the new file to open it. Take a look at the new interface. Let us know what you think of it!
13. If you experienced any problems with the conversion, please let us know!  We will fix them, and make sure your file can be converted before the final release.

Dennis Kolva
Programming Director
TurtleSoft.com

When viewing records in accounting software, it really helps if you can tab from field to field. That way, users can keep their hands on the keyboard while doing data entry, with no need to click in each field.

For the current version of Goldenseal, we wrote our own tabbing code, back in the late 1990s. It wasn’t too hard. Each window already stores a list of fields in front-to-back order. When you hit the tab key, it finds the current field, looks through the list for the next editable field, and selects it. For shift-tab, it looks in the other direction. Tabbing in tables is a bit more complicated, but not by much. The entire system took about one programmer-day to build.

Last week, we started to set up tabbing in Goldenseal Pro for Macintosh. The Cocoa framework includes an automatic tab system. They even brag that “you don’t have to write any code” to make it work. We turned it on, but it didn’t work.

Next attempt was Cocoa’s manual tabbing system. It uses what’s called a linked list. Each field is linked to the next field in the list, until you loop back from the last to the first. We set that up in Cocoa, but it still didn’t work. We read Apple’s documentation and checked Stack Overflow, but found no answers. We searched for a sample project with tabbing, and didn’t find any.

Next, we tried stepping through the tab process in the debugger, to see where it was failing. Unfortunately, Cocoa is very much a “black box”. Unlike the other frameworks we have used, it has no visible source code. The debugger only shows machine instructions, which are usually too low-level to be helpful.

When that failed, we went back to our previous approach. Go through the list of fields, and find the next one. To make that work we have to intercept the tab key, and then highlight the next field. Those both turned out to be difficult tasks in Cocoa.  Eventually it worked, but it sure wasn’t easy. Altogether, it probably took a programmer-week. The only good news is that some of what we had to learn will be useful for other things.

Apple’s Cocoa framework is the only major framework that hides its source code. It’s definitely an impediment. We have been working with Cocoa since September 2016, and it’s definitely the hardest framework our staff has ever experienced.  A while back we calculated an October 2018 completion date, but that probably will be pushed back a few months. The only good news is that Cocoa is hard for everyone, not just us. When the Mac goes entirely 64-bit in 2019, a lot of our competitors will be gone.

Ironically, Apple’s Unix operating system code (Darwin) is open source.  That means its source code is not only visible, but editable by qualified developers. Unfortunately that doesn’t help us, since our type of programming entirely relies on Cocoa.

Speaking of open source, it’s an option we are considering for Goldenseal’s code. 

Dennis Kolva
Programming Director
TurtleSoft.com

 

Goldenseal accounting software was designed for a one-button mouse. It’s all that was available for the Mac back then.

Goldenseal Pro is modern, so we can also use the scroll wheel and right-button. Last week we started on the first of the right-click commands. They are a much better way to access some details. For example, when you click the popup button next to a clairvoyant field in the current app, you’ll see a few options down at the very bottom of the menu. They let you enter ‘none’, edit the selected item, or add a new one. A few specialty fields add other options.

In Goldenseal Pro, you access those commands with a right-click. It’s much easier to find the commands (especially when the list is long). Right-click menus are easy to program, so we’ll be adding them in other useful places.

When you click the popup to edit or add a record, the current Goldenseal opens a modal dialog. It’s a separate window that takes control until you hit OK or Cancel. Modern versions of Mac and Windows both use drop-down panels instead. Panels work the same as the old modal dialogs, but they are attached to the current window. That makes them easier to find, and more logically attached to what you are already doing. Goldenseal Pro uses panels for almost everything that used to be in a modal dialog.

There are many places in Goldenseal where we show linked records. For example, in a Project account you might click a button to see Material Purchases for it. When you pay for a purchase, it shows the Bank Transaction that pays for it.

The current Goldenseal opens a separate window for linked records. The end result is often a bunch of different windows scattered about. Our own TurtleSoft file usually has 10 to 20 windows open. It makes the screen very cluttered. Eventually we close them all, but the clutter soon returns.

For Goldenseal Pro, we can use panels to show linked records, or tabs in the main window, or separate windows. As we start to use Goldenseal Pro to run our own company, we’ll be experimenting with different setups. It may be something that we control via Preferences, or by right-click options.

That part of the interface programming is going well. Unfortunately, the bad news is that Apple has decided to discontinue drawers as part of the Mac interface. The code still works, but it is deprecated and will stop working in some future version. Drawers are an additional type of window that slides out from the side, to show extra details. They are (were) perfect for showing breakdowns. We are still looking for a substitute.

Dennis Kolva
Programming Director
TurtleSoft.com

 

 

 

When Apple launched OS X in 2001, they provided two different ways for developers to write apps.  One was called Carbon. It was an extended version of OS 9, accessed with the C++ programming language. The other was Cocoa. It was first developed for the Next computer, using a different language (Objective-C).  Software developers could use parts from both, but usually it made more sense to stick with one or the other.

TurtleSoft released Goldenseal accounting software for OS 9 in late 2000. It only took a few months to rewrite it so it would run on OS X, using Carbon. When Apple switched to Intel chips, it took another few months to rewrite the Goldenseal code for that.

Apple plans to release OS 10.14 (Mohave) this coming Fall. It will be the last version that supports 32-bit code. Unfortunately, Apple did not update Carbon from 32-bit to 64-bit, and does not plan to. Many Macintosh apps will need to either rewrite to Cocoa, or fade away after Fall 2019.

TurtleSoft started work on the conversion to Cocoa in September 2016. It soon became clear that this update would take more than a few months. All of the interface code needs to be rewritten completely in a new programming language. The only good news is that is has also been a chance to update and improve the way Goldenseal runs.

In Fall 2016, one of the first things we worked on was the list interface. It’s what you see when you edit categories, sales tax rates, billing rates, and other minor setup details. It’s relatively simple, so it was a good place to start.

The list window has a table that shows a list of list items. You can then select one, and edit it in a separate window. There are also buttons to add and delete items.

Back then, our staff was brand-new to Cocoa programming. We were working through tutorials and programming books to learn Objective-C and Cocoa. For lists, we adapted one of the sample projects from The Big Nerd Ranch. Unfortunately, the official Cocoa way to handle tables is complex. Miss one step and it won’t work. We ended up making a 20-step checklist to use for tables, since it was so easy to miss something.

Since then, we have built a half-dozen other types of tables. The official system ended up being just too complicated. We abandoned it, and wrote them more simply. It’s more like 4 or 5 steps now, instead of 20.

Last week we started to fix a few flaws in the list interface. Revisiting the original system was a return to nightmare. Our staff soon gave up, and rewrote the list tables completely. Now, they are similar to our other tables. They still look great, but are much easier to maintain. Still a few things to fiddle with, but we will soon be moving on to other interface details.

Dennis Kolva
Programming Director
TurtleSoft.com

Our new Goldenseal Pro accounting software is moving into the home stretch. Everything big is done, but there are still plenty of medium and small things to finish.

Right now, our staff is working on lists. Goldenseal includes about 70 of them. They handle all sorts of little details: job cost categories, payroll tax tables, sales tax rates, etc.

In general, lists are simpler than regular business records. Some are  just a name, and the others have only a few fields. Because of their simplicity, lists were the first things we worked on, back in 2016 when we first started the conversion to Cocoa (on Macintosh) and MFC (on Windows). Since then, the rest of the software interface has gotten ahead of the lists, so it’s time to catch up.

The current version of Goldenseal has lists scattered all over the place. Most are in submenus, under 12 different menu commands. It’s not always easy to find them. Originally, we planned to just duplicate the interface in our current accounting and estimating software. However, both Windows and Mac are moving away from top-bar menus, and putting controls inside the window instead. So, Goldenseal Pro still does the same stuff, but in just one window rather than many.

Goldenseal Pro already uses an outline view for navigation (similar to how you can organize your desktop files). With it, you can find accounts and business transactions more easily. We just put the lists there too. Now they are located near the business transactions that use them.

Back in 2016, the database code was still primitive. The list interface we wrote back then could show list records, but it didn’t make new ones, or save changes. Now that works, also. All that’s left are a few list quirks that need special treatment.

Dennis Kolva
Programming Director
TurtleSoft.com

 

 

There is a decades-old programmer’s joke that goes:

There are two hard things in computer programming: cache invalidation, naming things, and off-by-one errors.

This week the database code for our new accounting software finally started to work properly for very large files. It took almost a month, or twice as long as expected. Along the way we faced both of those three types of problems.

A cache is data in temporary storage, so it’s quicker to access.  Invalidation is just programmer talk for keeping the cache synchronized with the original data. Basically, the cache determines how long we keep accounting records stored in RAM, and when we save back to disk.

Our staff wrote the caching system for Goldenseal Pro almost 2 years ago. It worked great with small files. Then we started testing with our own company file, which is big enough to require six ‘sector managers’ to manage record locations. Anything put into the extras would eventually give errors. After dozens of debugger runs, it narrowed down to the cache system. Changes weren’t being saved.

Meanwhile, naming things is a chronic issue for all computer programmers. Poor naming is the main reason why code becomes impossible-to-understand ‘black boxes’. Good code tells a story, and explains exactly what it’s doing at every step. Getting it that way takes time and effort.

It wasn’t obvious why the cache system wasn’t saving sectors. Stepping through it, everything seemed OK. After a couple frustrating days of debugging, our staff rewrote the whole thing.  Same basic approach and almost identical code, but with names that made more sense.  Renaming everything was a good way to see how the cache interacted with other parts of the software. Finally, it became clear that sectors were indexed differently from everything else. After changing one line of code, it worked fine again. One minute to fix it, and three days to find it.

Then there are the off by one errors, sometimes called ‘off by on’. For anything involving numbers, it’s easy to get close but no cigar. It doesn’t help that there are math quirks. For example, the end of a record is at start+length-1, not start+length. It’s easy to forget that. Even worse, libraries written by true programmer nerds start counting at zero, while those written by mortals start at one. NeoAccess considered the file end to be the last byte of data, while C++ libraries put ‘end of file’ at one after the last byte. Etcetera.

Earlier this week, our TurtleSoft file was converting OK, but the file size was over 43 gigabytes. Obviously, a gap-finding problem. It took most of a day to track it down to code that called ResetToEnd on a list, thinking that it selected the last item. Nope, it was one past the last item. That name makes sense if you want to add something at the end, but not if you want to access the item at the end. Off by one.

Adding a ResetToLastItem function solved the problem, and file size dropped instantly by 800x.

Anyhow, we’re back to interface programming.

Dennis Kolva
Programming Director
TurtleSoft.com

 

Remodelers rarely build kitchen appliances from scratch. Store-bought models run better than anything you could ever cob up on your own.

The same thing applies to software apps. It’s hard enough to write code that does construction estimating and accounting. There’s no sense reinventing wheels, so we use libraries written by other people for most of the basics. 

The good news is that there are plenty of software libraries out there. They’ll do just about anything. Most of them are free or cheap.

The bad news is that they are free or cheap. There is little incentive for their developers to polish things. It’s very rare to find a library that is fully reliable, well documented, and easy to use. Even if the source code is available, it’s often an ugly mess that someone whipped out on a caffeine high. Subtle bugs and security flaws probably lurk in there, but there’s no way to know for sure.

When you can’t view or fix the interior workings, it’s called a black box. Maybe it uses C++, or magnets, or squirrels.  There’s no way to know for sure. Whatever it is in there, about the best you can do is learn the quirks of the black box, and work around its flaws.

The NeoAccess database was definitely a black box. A dank basement with spiderwebs. For Goldenseal Pro, we considered switching to MySQL or some other database library. However, we soon discovered that most open-source software libraries are also black boxes, and just as ugly inside. Better the toxic danger zone you know, than the one you don’t.

Anyhow, last week our staff finally decided to completely rewrite the cache-management code. The first attempt, adapted from NeoAccess, was just too confusing. Chasing down its bugs was like wack-a-mole.

Basic design was not hard. Getting it to work on new files wasn’t too bad either. Then we went back and tested the conversion from older Goldenseal files to the new format. For a while it felt like this bathroom repair.  The conversion still uses NeoAccess to read the old file, and it was not happy with the changes.

It was time to put on the hazmat suits, climb in there, and update the NeoAccess cache system for a 64-bit world. We only need to run it one last time, but it’s an important one time. Our existing Goldenseal users will want to keep their old accounting data, just as much as we do.

Jiggling the cache system turned up several obscure bugs. It’s possible they weren’t hurting anything, but now that code actually does what it claims to do. After considerable effort, the conversion started working properly again. Previous testing was giving a few mystery database errors, but most of them disappeared. Sometimes it’s easier to just rebuild stuff, rather than fix it.  That is one problem with black boxes: sometimes the weird stuff you do to work around their flaws becomes its own problem, even after you replace the black box with something better.

We’re still finishing the File Manager code that kicks in when there are more than 32,000 records, but this latest spate of database work is nearing completion. Then we can get back to interface.

Most ‘black box’ libraries were someone’s one-time project, then they moved on to something else. Our software is exactly the opposite. We have been working on the same code base since 1995, and probably will still be working on it in 2035. So, the goal is to make Goldenseal Pro the opposite of a black box!

Dennis Kolva
Programming Director
TurtleSoft.com

 

 

 

There are many ways for a database to go bad. Losing records is a big problem, of course. Even worse is when one record writes over the top of another one. That corrupts at least one record. Sometimes the damage expands further.

A more subtle problem happens when records change: something that happens often in accounting and estimating software. Usually records get bigger, which means they no longer fit into their old place. It’s easiest to move them to the end of the file, but then there’s a gap at their former location. Do that enough times, and you end up with a huge file that is mostly empty space.

The NeoAccess database kept a ‘free list’ of empty spaces that was meant to prevent both those problems. New and changed records filled in the gaps, and kept the file compact.  However, there were occasional mystery bugs, and we suspected the free list was at fault.  If it ever forgot to remove a gap, then a record would be trashed when a second was written to the same place. We tried to debug the free list code but it was too confusing.

In 2002, we added the File Manager to help manage file contents. It kept its own list of gaps that duplicated the free list, and also stored the location of every record in the file.  That way there were two ways to make sure records were added safely. Goldenseal could find a gap, then double-check the neighborhood to make sure the space was really free.

At first the File Manager was just a diagnostic tool, but it soon replaced the old free list. Mystery over-write bugs disappeared. The Manager added an extra 10 bytes per record, but increased reliability was well worth the cost.

Unfortunately, the File Manager had a subtle flaw. It was by far the biggest record in the database (in the TurtleSoft file, it’s 2.5 megabytes).  If users were low on RAM, it was the first thing that wouldn’t fit. Nearly always, that led to a crash or a freeze, but the database would still be fine. However, if there was exactly enough RAM to load the Manager, but not enough to save it, it would die midway through the file save, and corrupt the database. It didn’t happen often but it was awful when it did.

Goldenseal users haven’t reported this problem in the past few years, probably because computers now have so many gigabytes of RAM. However, it’s still a design flaw.

In Goldenseal Pro we fix it by using multiple, smaller File Managers. There’s one for each sector in the file, enough to hold 32,000 records. Managers are accessed via a very broad, 2-level tree that is similar to the ones used to index records. It’s safer and a teeny bit faster than the old system, with less data to read from the drive.

The old File Manager kept growing larger as more records were added. Periodically, it had to be relocated in the file. That was when it was most likely to die, and kill the file with it.  In Goldenseal Pro, each File Manager is a fixed size. When it fills, we just add another one. That also makes it safer.

Essentially, Goldenseal Pro stores the location of every record in two different places: one sorted by record ID, and one sorted by location in the file. There’s also a separate list of file gaps that helps keep the file compact. The system is more complicated than the previous File Manager, but not by much. The code is much more understandable, so we’ll be able to repair damage to either half of the record storage.

Right now, the system can handle a billion records. If any users ever get close to that amount, we’ll add code to allow a 3-level tree. That will jump it into the trillions.

Dennis Kolva
Programming Director
TurtleSoft.com