Threaded Mode | Linear Mode

Thomas Okken · (This post was last modified: 08-22-2019 06:19 PM by Thomas Okken.)

To elaborate a bit about the bug in the earlier test builds...

These new versions have an architecture-independent state file format, in order to allow sharing state files between Free42 instances even if they're running on different platforms. In order to achieve platform independence, I had to stop saving data structures by simply dumping their in-memory representations to the state file, since the details of those in-memory representations depend on data sizes (32- or 64-bit integers), endianness (not as much of an issue as it used to be, but the world isn't 100% little-endian yet), and alignment (partly governed by hardware requirements, partly the result of compiler implementation choices).

The most complex data structures used by Free42 are programs, and in order to write those in an architecture-independent way, the obvious choice was to use the same "raw" format that is already used for importing and exporting user-code programs, and which is basically the HP-42S native format.

One side effect of this change is that the number of I/O operations involved in loading and saving programs from and to the state file increases dramatically, because the state file loader reads programs a byte at a time. Even though there is some buffering, the read operations were slowing state file loading down by a lot on Android, since on that platform, the reads were performed by Java code, while the core of course is native code, and Java/native calls are very CPU-intensive.

One side effect of all these changes is that the core has to make many more I/O calls while reading and writing state, specifically, lots of small ones instead of a relatively small number of large ones, and on Android, that caused a significant slowdown. The reason is that the actual I/O calls used to be performed in the shell, while everything else involving the state happened in the core, and since the Android version has a Java shell, that meant lots of CPU-intensive JNI calls.

In order to prevent the I/O-related slowdown, I moved all the state file read and write operations into the core, eliminating the shell_read_* and shell_write_* functions in the core/shell interface.

This core/shell API change worked fine in terms of speed, but unfortunately, I missed three places in the code where bytes read from the raw stream are "pushed back," causing the push-backs to fail silently, with the result that in certain cases, it's as if a byte is dropped from the program.

Two of those three cases are obscure, and guaranteed never to affect Free42 state files. The first handles the zero byte that terminates numbers in HP-42S programs; in HP-41 raw files, that byte may not be there, while in HP-42S files, it always is. When the program reader finds a non-zero byte after a number, it pushes it back so it can be treated as the first byte of the next instruction.

The second case handles ALPHA GTO or XEQ with a zero-length label, GTO "" or XEQ "", encoded as 1D F0 or 1E F0, respectively. When the program loader sees that F0 byte, it discards the 1D or 1E byte, and treats the F0 as the start of a new instruction. I'm not sure why I implemented it that way, but it was probably to match the behavior of the real 42S.

Neither of these two cases are a problem when reading a state file, because Free42 will never generate the problematic byte sequences in the first place.

The third one is different, though: it is guaranteed to cause problems. This is the case of ALPHA strings.

When reading an ALPHA string, the program loader has to examine the string's second byte, i.e. the first byte of the actual text, to see if its most significant bit is set. (This is how the HP-42S encodes instructions that aren't HP-41C functions but that do have arguments, so they can't be encoded using the XROM mechanism.) When the program loader finds that the second byte does not have its high bit set, it pushes it back so that it can then continue handling the entire string just like any other string; this code is shared between handling plain strings and functions with ALPHA arguments.

The failing pushback in this third case causes the first character of any ALPHA string to be lost, and, and here is the really serious problem, it causes the first byte following the string to be gobbled up by the string, turning every string into an unintentional byte grabber, seriously corrupting any affected program in the process.

Yikes.

Fortunately, I became aware of the problem before the new state file logic was released on my main page, but even the small number of people who downloaded the test versions in order to check out the new state file switching/import/export logic should not have been bitten by this bug; the new state file logic is several weeks old at this point, and I thought it was bug-free by now. My apologies to everyone who was affected, and many thanks to Vincent Weber for sending the bug report that caused me to identify the problem and enabled me to fix it.