When the RMS (Record Management System) of the Java MIDP API for handheld devices was created its authors were influenced by the Palm system. In some ways the MIDP is like Palm but even simpler to use. It's simple yet effective design is one of the key reasons why MIDP is turning out so successful today.

The underlying principle of both systems is that of being record-based. By this I mean that files are stored as a series of heterogeneous records each typically 80-250 bytes in length. The record begins with a code indicating its type. When data is changed a new record which copies the old one and includes the changes is added on to the end of the database and the old one is marked as deleted. Periodically the database is sorted and the deleted records physically disappear. By using a system like this the application makes minimal disk writes and is able to save changes easily and incrementally as the user works.

An important forerunner of today's record-based systems is the BIFF format created by the developers of Microsoft Excel. Even though Excel does not save incrementally it uses a record-based format. Because each record has length and type information in it, it is difficult to corrupt the format and even if a single record is corrupted the rest of the file is safe. The BIFF format is the reason why Excel early on established a reputation for rock-solid reliability. I encourage anyone working on a record-based format to study the way BIFF works as an example and source of ideas.

If you are writing a standalone system it is easier than you might think to use record-based storage. Nowadays many Java developers resort to JDBC databases to store data for their application but later suffer for their decision because JDBC can be brittle and little bugs tend to gnaw at reliability. If you create your own record-based system you have control over the environment and are not at the mercy of a bug in a database driver. In the long run you may find for many applications a roll-your-own record system will actually be simpler than fussing around with SQL and setting up database connections.

A big advantage of using a record-based system is that you can use journalling: maintaining a running list of every change to the data ever made. This is the Rolls Royce of data management because it allow absolute auditing. You can track every change ever made to the data and roll back the changes to a fixed point in time if necessary. To do this you maintain at least two record bases. One is the current list containing the current set of records and the other is the master list. When a change is executed the same record is added to both the master and current list. The difference is that deleted records are removed from the current list whereas nothing is ever removed from the master list. Periodically a snapshot of the current list can be stored thus making it easy to roll forward or backward from that point during an audit or repair.

The biggest challenge to implementing a journalled record-based system is that if the format changes, and inevitably it does, you have old incompatible records in the system. To solve this you can either retroactively convert the older records to the newer format or you can create new record types and maintain backwards compatibility for the old one, although this can be onerous to support from a code complexity point of view. In general there is big premium for not having to make any changes to the record formats once the system is implemented.