A Scheme Shell

# 8 Implementation

Scsh is currently implemented on top of Scheme 48, a freely-available Scheme implementation written by Kelsey and Rees [S48]. Scheme 48 uses a byte-code interpreter for portability, good code density, and medium efficiency. It is R4RS-compliant, and includes a module system designed by Rees.

The scsh design is not Scheme 48-specific, although the current implementation is necessarily so. Scsh is intended to be implementable in other Scheme implementations -- although such a port may require some work. (I would be very interested to see scsh ported to some of the Scheme systems designed to serve as embedded command languages -- e.g., elk, esh, or any of the other C-friendly interpreters.)

Scsh scripts currently have a few problems owing to the current Scheme 48 implementation technology.

- Before running even the smallest shell script, the Scheme 48 vm must first load in a 1.4Mb heap image. This i/o load adds a few seconds to the startup time of even trivial shell scripts.

- Since the entire Scheme 48 and scsh runtime is in the form of byte-code data in the Scheme heap, the heap is fairly large. As the Scheme 48 vm uses a non-generational gc, all of this essentially permanent data gets copied back and forth by the collector.

- The large heap size is compounded by Unix forking. If you run a four-stage pipeline, e.g.,

  > `
  > (run (| (zcat paper.tex.Z)
  > (detex)
  > (spell)
  > (enscript -2r)))`

  then, for a brief instant, you could have up to five copies of scsh forked into existence. This would briefly quintuple the virtual memory demand placed by a single scsh heap, which is fairly large to begin with. Since all the code is actually in the data pages of the process, the OS can't trivially share pages between the processes. Even if the OS is clever enough to do copy-on-write page sharing, it may insist on reserving enough backing store on disk for worst-case swapping requirements. If disk space is limited, this may overflow the paging area, causing the `fork()` operations to fail.

Byte-coded virtual machines are intended to be a technology that provides memory savings through improved code density. It is ironic that the straightforward implementation of such a byte-code interpreter actually has high memory cost through bad interactions with Unix `fork()` and the virtual memory system.

The situation is not irretrievable, however. A recent release of Scheme 48 allows the pure portion of a heap image to be statically linked with the text pages of the vm binary. Putting static data -- such as all the code for the runtime -- into the text pages should drastically shorten start-up time, move a large amount of data out of the heap, improve paging, and greatly shrink the dynamic size. This should all lessen the impact of `fork()` on the virtual memory system.

Arranging for the garbage collector to communicate with the virtual memory system with the near-standard `madvise()` system call would further improve the system. Also, breaking the system run-time into separate modules (e.g., bignums, list operations, i/o, string operations, scsh operations, compiler, etc.), each of which can be demand-loaded shared-text by the Scheme 48 vm (using `mmap()`), will allow for a full-featured system with a surprisingly small memory footprint.