Month: July 2024

When printing, the new accounting software currently puts everything onto paper or PDF, but it looks ugly. Right now our staff is working to make their appearance as close as possible to Goldenseal’s. It means another deep dive into Custom Layouts, the user interface that lets you change the appearance of just about anything in the app.

One update is in date formats. Goldenseal uses this dialog:

It’s complicated for users. Complicated to program. In fact, the sample date at the bottom sometimes won’t update properly: a very old bug that we just noticed.

The new software uses the Qt method, which is similar to Microsoft Excel. Just some text: M/d/yy = 7/26/24 and MMM d, yyyy = July 26, 2024, etc. It’s easier for users, and simple to program. No need to futz with radio buttons.

There are other details still unfinished in Custom Layouts: money and number formats, table setup, fancy font details. Some of those would take weeks or months of work to look like Goldenseal. For example, it has this complex dialog to set up report tables:

The new accounting software saves layouts as text, internally. We plan to avoid a ton of interface work by exporting and importing those. Then users (and our staff) can make fancy changes in a word processor or spreadsheet app.

We just need to figure out how to make the process as obvious as possible. Or at least less scary than that report table dialog. It was a monster to program, and not easy to use.

Dennis Kolva
Programming Director
TurtleSoft.com

 

 

 

We just fixed the problem of huge files in our new accounting software. It was caused by a couple stupid bugs in the code that decides where to save changed records inside the file. Zapping one got the TurtleSoft company file from 11 gigabytes down to 1.5 gigs. The second fix shrank it to 85 megabytes: bigger than the current 64 megs in Goldenseal, but that’s expected. We added a few things.

Our staff is back to using the new accounting software for our own business, in parallel with Goldenseal. As soon as there’s a month without serious bugs, it’ll be time for a first release.

Why did TurtleSoft write its own database code? Well, if we were just starting now, we’d use something free and open-source, for sure. Let someone else write the code to manage data on disk.

That was the original plan for Goldenseal in the late 80s, also. Our first estimating & accounting templates hit the limits of MS Excel, so we needed to switch to a real app. Open source software didn’t exist back then, so it needed to build atop something commercial.

First, we tried a dozen different databases and other development platforms: FileMaker, FoxBase, Omnis, etc. All had fatal flaws (usually lack of spreadsheet-like breakdowns for estimates).

We finally decided to write the interface in C++, and license an object database called NeoAccess to manage files. Neo was popular at the time, especially on Macintosh. It worked pretty well, up until we started to test with real data. Uh oh, there were crashes, corrupted files, and weird error messages deep in NeoAccess code. Emails to their tech support went unanswered. Soon the company disappeared.

Searching the early Internet showed internal emails from AOL, Netscape and others also searching for solutions, and/or making plans to give up on Neo. We were in too deep, and had no better options. So our staff spent a year rewriting NeoAccess to be more sturdy.

Most of the work was just making the code readable, so we could figure what it did. Several definite bugs turned up. We probably fixed others accidentally. That made it good enough for the 1.0 release. A couple years later, there were more updates to Goldenseal’s database code. Those fixed the last of the leftover bugs.

Our new accounting software inherits the good parts from the original NeoAccess design. Plus the good parts of what we changed and added for Goldenseal. Plus a few improvements, based on 20+ years of experience afterwards.

One thing that’s different now is gap tracking. It helps keeps files compact, even though records change size and need to move elsewhere.

Neo used CNeoFreeList to track empty spaces. Actually, many of them, scattered within the file. We replaced them with a single DB_GapManager, doing the same things but easier to debug. We also added a DB_FileManager, a list of where each record was in the file. It made double-sure that records would not write on top of something else. Later, it also allowed recovery of data that was lost when a NeoAccess index became corrupted.

Tracking gaps is complicated. Removing a record may leave a gap of the same size, or it may widen a gap that it touches. Remove a record with gaps on either side, and three gaps must merge into one. The system worked, but it was too complicated.

For the new accounting software, DB_GapManager is gone. Computers are fast enough now that we just zip through lists of record locations and sizes, and compute gaps between them. Simpler and more reliable, once the stupid logic bugs are fixed.

DB_FileManager is also replaced. Goldenseal sometimes ran out of memory because the manager was so big, so we now divide the file into bite-size sectors. DB_SectorManagers each handle disk space for 16,000+ records. Our company file has 13 sectors. There’s room for 16,000+ more, enough to manage 256 million records. There’s a way it can go beyond that, if ever needed.

BTW Google still shows a few hits for CNeoFreeList, 20+ years after its death. It’s mostly users back in the Aughties, trying to fix error messages and crashes in various apps.

NeoAccess caused pain for many Mac software developers, big and small. Other people’s code is not always the best answer.

Dennis Kolva
Programming Director
TurtleSoft.com

 

Well, we finally fixed the mystery crashing error in our new accounting software. It took a week of testing and debugging. At least we fixed a few other potential problems along the way.

The good news is, the bug was simple. Our company file grew bigger than 4 gigabytes during the conversion from Goldenseal, but one old line of code still used 32-bit file marks. It truncated the 64-bit address, tried to save a record into the wrong part of the file, and chaos ensued. One small change made it OK.

The bad news is, our company file should not be that big. It already took up 3 gigabytes, then grew to 11 gigs in June. Goldenseal only needs 64 megs for TurtleSoft’s data. The new accounting app will have bigger files, but not by that much.

In theory, the new database has a good system to keep the file small, similar to what’s in Goldenseal. When saving a record, it looks for the first gap that’s a snug fit, and puts it there. If no tight fits, it then looks for large gaps with enough space left over to hold another record. If none of those, then it tries gaps that waste a little space (currently 24 bytes, but we may tweak it later). Any of those keep the file size unchanged. If no suitable gaps at all, the record tacks onto the end. That makes the file bigger.

If we’re lucky, there’s a stupid error in the gap-tracking code. Fix that, and files will shrink to a reasonable size.

If not lucky, we’ll need to fix a subtle design problem. One that can affect any database.

The conversion from Goldenseal to the new app does something potentially nasty. It loops through each bank transaction, and adds the ID to its bank account. That makes the account record 4 bytes bigger.

New files start out tightly packed, with zero gaps. When we save the account, it’s slightly too big for its old space, so it adds to the end. Next save it’s even bigger, so to the end again. Repeat for 25,000 bank records and the file gets huge, with 25,000 gaps that gradually increase in size.

If that’s the case, something clever needs to happen during the conversion to keep things compact. There are a few options, but we’ll worry about that only if it’s needed.

Luckily, the gap-finding system rarely fails in normal daily use. Records come in all sizes, so most of them will find a gap that’s snug. There always will be some wasted space within the file, but not much.

When our staff wrote the database code back in 2016-2018, we also added reports to help debug what’s inside the file. Right now, we’re tweaking those to be more useful and accessible. It will help track down whatever it is that makes files too big. Also any future problems.

Dennis Kolva
Programming Director
TurtleSoft.com

 

 

Thanks to rain and hot weather, our staff has fixed many problems in breakdown tables and elsewhere. It’s time to test on our own business again. Except, we can’t. Converting TurtleSoft’s company file to the new format crashes part way through.

It’s something that has happened before. In the past we fixed things by tweaking how often changes are saved to disk. Sadly, that wasn’t enough, this time. All it did was change where the crash occurs.

Conversion is a complex process. First it reads from the Goldenseal file, using NeoAccess database code. Then it writes to the new file, using the new database setup. Some data gets tweaked along the way. For example, breakdowns change from two types to one. Bank accounts and transactions change from seven types to three. Other small fixes and improvements happen.

Once all data is moved over, there’s a second round of tweaks. Breakdowns and banking records may have new ID numbers, so anything that links to them needs to change. That process happens entirely within the new code, and the crash happens there. The Sample file and other test files always sail through, but not our data.

The most likely culprit was memory use. Our file has 20+ years of data, so it’s big. Many things need to stay in RAM during the conversion.  However, Activity Monitor only shows 65 megs max, which is tiny compared to some other apps. It stays about the same, so no big memory leaks.

We tried saving changes much more frequently. That made the conversion run very slowly, but it went to completion. Yippee! Sadly, when we opened the file, it gave warnings for bad data and crashed in an entirely different place.

That is very bad news. Corrupted data usually happens when data writes onto the wrong place in RAM or on disk, and zaps something else. Corruption is hard to diagnose, because errors show up long after the bad write that caused them.

We stepped through the code many times, and found a few suspects: things that might screw up saves in rare cases. Fixing those made the software generally more reliable, but it didn’t stop the crashes or bad data.

Early versions of Goldenseal had corruption bugs. To fix them, we first added tons of sanity-checking code, to give error messages for anything suspicious. We also added menu commands to work with raw data inside the file. Verify File is the most useful, but there are others in the View–Security menu. Those helped track down each bug, one by one. The last happened if text was too long in one of the Preferences fields, caught in 2005.

The new accounting software still has the same sanity-checking code, but not the diagnostic tools. Looks like we need to hook them up again, to see what’s going on inside. It may be something funky with our data, or something rare that we just happen to hit sometimes.

If any Goldenseal users want to be data guinea pigs, send us a compressed copy of your company file. We’ll check how well it survives the conversion process. Maybe having more examples will narrow down the cause of this problem. Eventually we’ll stumble upon it, but sooner is better.

Dennis Kolva
Programming Director
TurtleSoft.com