John Chamberlain
Developer Diary
 Developer Diary · You Heard It Here First · 24 November 2003
Secrets of Saveless Development
Yesterday I wrote about the great innovation of Jeff Hawkins' Palm OS: no saving. Today I have some tips for developers on how to implement saveless applications using record-based data management. If you are writing for the Palm a lot of the techniques are spelled out for you by Palm record management system API and its philosophy, but if you in a standard OS and want to create a saveless system you have to devise your own system. It is wise to see how the Palm system works when writing your own.

When the RMS (Record Management System) of the Java MIDP API for handheld devices was created its authors were influenced by the Palm system. In some ways the MIDP is like Palm but even simpler to use. It's simple yet effective design is one of the key reasons why MIDP is turning out so successful today.

The underlying principle of both systems is that of being record-based. By this I mean that files are stored as a series of heterogeneous records each typically 80-250 bytes in length. The record begins with a code indicating its type. When data is changed a new record which copies the old one and includes the changes is added on to the end of the database and the old one is marked as deleted. Periodically the database is sorted and the deleted records physically disappear. By using a system like this the application makes minimal disk writes and is able to save changes easily and incrementally as the user works.

An important forerunner of today's record-based systems is the BIFF format created by the developers of Microsoft Excel. Even though Excel does not save incrementally it uses a record-based format. Because each record has length and type information in it, it is difficult to corrupt the format and even if a single record is corrupted the rest of the file is safe. The BIFF format is the reason why Excel early on established a reputation for rock-solid reliability. I encourage anyone working on a record-based format to study the way BIFF works as an example and source of ideas.

If you are writing a standalone system it is easier than you might think to use record-based storage. Nowadays many Java developers resort to JDBC databases to store data for their application but later suffer for their decision because JDBC can be brittle and little bugs tend to gnaw at reliability. If you create your own record-based system you have control over the environment and are not at the mercy of a bug in a database driver. In the long run you may find for many applications a roll-your-own record system will actually be simpler than fussing around with SQL and setting up database connections.

A big advantage of using a record-based system is that you can use journalling: maintaining a running list of every change to the data ever made. This is the Rolls Royce of data management because it allow absolute auditing. You can track every change ever made to the data and roll back the changes to a fixed point in time if necessary. To do this you maintain at least two record bases. One is the current list containing the current set of records and the other is the master list. When a change is executed the same record is added to both the master and current list. The difference is that deleted records are removed from the current list whereas nothing is ever removed from the master list. Periodically a snapshot of the current list can be stored thus making it easy to roll forward or backward from that point during an audit or repair.

The biggest challenge to implementing a journalled record-based system is that if the format changes, and inevitably it does, you have old incompatible records in the system. To solve this you can either retroactively convert the older records to the newer format or you can create new record types and maintain backwards compatibility for the old one, although this can be onerous to support from a code complexity point of view. In general there is big premium for not having to make any changes to the record formats once the system is implemented.

Record-based systems have been around for a long time, but now they have only really come into their own because they enable a beautiful simplification of the user's life by eliminating the need for the user to have to save constantly. There is a lesson to be learned there.

Developer Diary · · bio · Revised 24 November 2003 · Pure Content