Month: February 2022

Our staff is just finishing up the loading of data entry screens from plain text layouts. The new code is much simpler. It will be easier to maintain.

For a while there was a bug that made some old-style layouts go haywire. Weird colors, everything out of place. In the last post I forgot to mention one more problem with binary data: byte orders. That’s what caused the error.

There are two ways for computers to store multi-byte numbers. Big-endian systems order the bytes from biggest to smallest. Little-endian goes small to big. It’s just a hardware thing. Neither is better or worse. For most things it makes no difference.

Intel chips are little-endian. Ditto for MS-DOS, Windows, and anything else in Wintel world. Motorola chips are big-endian. So was the Macintosh, back when it ran on 680×0 and PPC.

When the Mac switched to Intel in 2006, all Mac apps had to to add code to swap bytes around when reading old data with the new chip. We already did that in 2001 for the Windows version, so moving the Mac app to little-ends was not painful. The biggest problem was the switch of programming tools, from Metrowerks CodeWarrior to Apple’s Xcode.

A Goldenseal company file can use either byte order, or it may have some of each. Each record stores that info along with its version number. Any files started on pre-2007 Macs probably have both types of ordering.

The update to TurtleSoft Pro makes every record little-endian. Fortunately, the new M1 and M2 Apple chips are also little-ended, so we probably won’t need to swap bytes for a long time in the future.

However, there is one pocket of big-endian data that still remains: layouts for screens, reports and printed forms. They started out as old-style Mac resources: big-endian. Thanks to their packed binary format, it’s not easy to change them over. Instead, we’ll keep reading the old format for a while, but gradually convert them to text.

Last December our staff added one innocent line of code that gave the wrong endian marker to layouts coming from the database. It scrambled all the multi-byte formatting, including colors and screen positions. Oops.

When binary data is off by one byte, things also get weird. For a while we thought that was the cause. Finally, we ran older versions and narrowed it down to the day when the bug started. That made it easier to spot the culprit.

Byte order is one more reason to switch data to text. ASCII and UTF-8 use single bytes, which are the same for any endian. If Apple decides to move the Mac to some new big-endian chip in 2038, the update will be less painful.

I’m pretty sure there are no more huge changes left to do before finishing TurtleSoft Pro. So, our staff is back to normal programming. The process is simple but repetitive. Use the new app in parallel with the current Goldenseal. Get it to work. Fix any problems. Rinse, repeat.

Dennis Kolva
Programming Director
TurtleSoft.com

When we first started work on Goldenseal accounting software, 1 megabyte of RAM was the norm. Apps could only use a couple 100K of that. A few bytes here and there was a big deal, so programmers used the smallest possible memory sizes. Colors came from a palette with 256 choices (1 byte). Fonts used a 2 byte FontID (max 16,000 fonts). Pictures and other resources also used 2 byte IDs.

Larger data was packed tight to conserve space. Records inside a Goldenseal file have data that’s as dense as possible. Our old layout formats are similar.

Binary data like that is touchy. When reading or writing records, being off by just one byte will wreck havoc.

These days, memory is cheap, whether in RAM or storage. There is no reason to scrimp on bytes. Since text is easier to read, data storage and programming often use that instead of binary bytes. Modern system resources have names and file paths, rather than ID numbers. Fonts have names. Lists often use “hash tables”, based on text instead of a number.

A few years ago we shifted most of our resources to text files. The code finds them with a file name rather than an ID number. We also are half-done converting screen layouts from binary to text.

In theory, we could do the same with estimating and accounting records. The company file could be one huge text file. Switching over would not be hard: we already have code to export and import from text. It’s how we saved sample data during early development. Plain text sure would be easier to debug.

Unfortunately, text would also be easy for anyone else to debug. Just open the file with a text editor, search for your name, give yourself a raise, and save changes. Any other fraud would be just as easy. The current binary format is not invincible, but it takes much more effort to hack it. We haven’t heard of any cases where that happened.

On top of that, text is bigger. It’s not bullet-proof, either. Lose a tab character separating values (or add a new one), and the data after it still gets trashed.

To keep binary data reliable in Goldenseal, we’ve added all sorts of fail-safes over the years. It makes programming easier now. If we make a change and forget something, it soon gives an error message that tells us exactly which code line had the problem. Likewise, if a file has damaged data, it will complain. And there are extra markers inside the file, now, to help with forensics.

Goldenseal has run the TurtleSoft business for 22 years. In that time, our company file developed a couple of data errors. There’s a corrupted NeoAccess index that lost access to a block of sales breakdowns. It happened about 15 years ago. The converter found most of them via DB_FileManager, but one record was corrupted, and a few are gone forever. There also was a Contact Log record with a bad length marker in some text. That zapped all the data after the error. It was a 1993 phone call, imported from FileMaker in 1999 or 2000. We were able to fix that record by re-saving it within Goldenseal before the import.

As our staff gets closer to completion, we’ll post a beta version so users can convert their current Goldenseal files. If the import reports problems, there will be time to fix them.

It will be interesting to see how well other files have done. Hard drives and SSDs have grown more reliable over the years, so most “bit rot” probably will be in older records.

Dennis Kolva
Programming Director
TurtleSoft.com

Our staff made several big changes to the guts of our code recently. Layouts, text and money all have new formats. There’s a good reason for doing it now.

Goldenseal has been out for more than 20 years. During that time it evolved. More features, bug fixes, safer programming. Often, that meant changing the data that’s stored for each record.

To handle that, every record has a version number. It started at 1, and is now up to 9 for a few classes. When Goldenseal reads a record from storage, it checks the version number so it can read the proper amount of data. When it saves changes, it updates to the latest version. For both reads and writes, the code double-checks the size of the data. It needs to be exact, otherwise it may overlap neighboring data in RAM or on the drive. That is really bad news.

The code has branches to handle all possible versions. It has grown complicated.

When TurtleSoft Pro opens an existing Goldenseal file, it saves everything over to a new file, with a new database format. From then on, you use the new file and leave the old one behind. In TurtleSoft Pro, everything has the same version (at least for a while). We’re starting over at 10.

This project is pretty much a software gut-rehab. We might as well upgrade the foundation and framing while everything else is being redone. This is a once-in-decades opportunity to make big changes.

After increasing the size of CMoney, we had to look at the code for every data class. Some had comments about things to add in the next big update, so we added them. We also found and fixed a few subtle bugs. Nothing dangerous, but things that might make the database less reliable. The complicated branching for versions made it easy to screw up.

For now, TurtleSoft Pro still must keep support for old versions. It needs to translate Goldenseal files, which may have a mix of everything from 1 to 9. However, in a few years we can remove the converter. Also, lots of old code to support it. Less clutter means fewer headaches, and fewer mistakes. The change won’t be a problem for late adopters, since we can keep an older app handy for the conversion. It’ll just become a two-step process.

One thing we added this week is a text field to store a file path for pictures or other files. Actually, two of them, in every class.

Right now, Goldenseal uses ID numbers for fonts, pictures, and resources. TurtleSoft Pro will use font names and file paths for them, instead. It’s part of an overall trend in computer programming: text instead of binary data. That’s a topic for the next post.

Dennis Kolva
Programming Director
TurtleSoft.com