Month: February 2025

Construction Price Updating (Feb 22, 2025)

TurtleSoft started in 1987 with construction estimating software written in Microsoft Excel. Our new accounting/estimating software will be the 4th generation since then.

Selling software in a “vertical market” like construction is tough. There are plenty of special problems to deal with, and a small pool who want to buy it. We don’t even get the full construction market: most of our users are a certain size of remodeler. Not too big, not too small, not too specialized.

In future years, we plan to focus more on general-purpose small business software, and not spend as much time on features that are specific to construction. However, there is one task where we can make a huge improvement for our builder and remodeler users: keeping material prices up to date.

The obvious place for us to fetch material prices is from the Home Depot or Lowes websites. Each Cost Item record would need to store a page URL. Then we could add code to get on the web, and find the price inside the HTML on that page.

Alas, Home Depot and Lowes really do not want anyone to scrape prices from their site. Item prices are not visible at all in their HTML code. Instead, it is “obfuscated”. Some Javascript must fill in the price while the page loads, probably by doing math on numbers that are scattered in the source text.

In theory, we probably could find a way to hack their systems. But then it’s an arms race: every time they make a minor change, we must spend many hours to figure it out again.

Smaller chains have more potential. We checked Menards in the Midwest, and their website pages do have pricing that we could access and import. If there are enough other sources to cover most of the US and Canada, it will be worth our time to develop a price-grabber system.

84 Lumber is about the only small chain that has survived around here, and their website does not even list prices. But there may be others that can work.

To check if your own local supplier can be used for automatic updates, follow these steps:

      1. Open a page for any specific material. Check the listed price.
      2. View the HTML source code for the page (View–Developer–Page Source in Chrome, Tools–Browser Tools–Page Source in Firefox).
      3. Search for the price text. It may be listed in several places.
      4. Check a few other products.
      5. Are all the prices near to some marker text that is the same on all pages? If so, please let us know.

Dennis Kolva
Programming Director
TurtleSoft.com

File Size (Feb 11, 2025)

There’s a saying for construction work: cheap/fast/good, pick any two.

It also applies when writing software for accounting. TurtleSoft is not rolling in money like the early to mid 1990s, so this time around we had to go for cheap/slow/good rather than cheap/fast/bad.

One issue we still need to decide on is file size. For that, the choice is more like fast/small/reliable, pick any two.

The NeoAccess database we used for Goldenseal was fast & small, and not reliable. We had to rewrite their code in order to make software good enough to sell. It made files bigger. Some of that was because we added sanity-check data to the indexes that kept track of where each record is located. We also added a second place to store record addresses: a list sorted by file location instead of a binary tree. That way, if the Neo code lost track of a record, we could still find it and repair the damage.

For the new accounting software, we used the same ideas, but added a few more to make the database even more reliable. It has a price. An empty Goldenseal file starts out at 106K. The new app currently starts at 14 megs, over 100x bigger.

Much of the extra bulk comes from a design decision. Any record database needs to keep lists of where records are located in the file: it’s called an index. There are over 200 record classes, and each gets its own index.

NeoAccess started a new index the first time you used a class. It went at the end of the file. Once an index had 8 records, another index went at the end (later we upped that to 32). Their approach was frugal with both RAM and disk space, but the result was indexes scattered throughout the file. They were hard to locate, and fragile. One stray bit change in an index could destroy a big chunk of the file.

The new app starts with 200+ empty indexes at the beginning of the file, even if you don’t ever use them. Doing that adds to file size, but it makes repair easier. We know exactly where everything is. Right now the indexes are big enough that the “tree” of indexes is not very deep. We can shrink them, but it will make things slower and a bit less reliable.

Another index tracks the contents of each file sector. More empty space at start-up. We also can shrink that, with similar consequences.

Goldenseal is a monster. It does all sorts of things: construction estimating, accounting, project management, rentals, sales, contract writing. Most users don’t need all that. We really need to pivot to smaller, cheaper apps that just do one or two things. Those users probably won’t like to start with a 14 meg file.

This is one of several quandries we are still working on, in the time left before first release. There are many things we can do to reduce file size, but it’s a matter of weighing the costs.

Dennis Kolva
Programming Director
TurtleSoft.com