University of Cambridge Cambridge, England
For the past 9 months, the author has been working half-time on an experimental X program to provide a windowing front-end for conventional interactive packages. He estimated that the project would take 12-18 months, and it is just on schedule, so he is approximately half-way through. This paper describes his experience with X, rather than the project itself.
The project is jointly funded by the University of Cambridge Computing Service, who are paying for his time, and the Numerical Algorithms Group, who have paid for the HP 9000/340 workstation that he is using. It is an experimental project, because both parties are interested in discovering whether such uses of X are practicable. However, the intention is to produce a usable product, if that is feasible.
An HP workstation was chosen because it was the most reliable and cost-effective of the mainstream UNIX workstations. It also has the important advantage that HP are the first major supplier to release Motif, which is clearly the variant of X most likely to become an industry standard. The intention is to write the application so that it will run under any reasonable UNIX system that supports Motif; this should include the R/6000 series, when it is available, and may even include the next release of AIX/386.
There is a disadvantage to the use of an HP workstation: the version of X that runs under HP/UX 6.5 is X11.2 with a large number of X11.3 features added, because HP froze the product to concentrate their efforts on what was to become Motif (which is now available under HP/UX 7.0). This means that the HP/UX 6.5 X Toolkit and Widget set have a large number of bugs, and are very painful to use. It is possible that the remarks about reliability and coding standards in this paper are unfair to later versions of X (i.e. X11.3 and X11.4).
As will be described later, the two main problems with adding X interfaces to existing packages are that the X Protocol is essentially unusable over a wide-area network and that the package may need extensive redesign to support X properly. The application is designed to provide a moderately user-friendly, windowing user interface that will communicate via a private protocol to a traditional line-mode, interactive package. The protocol is designed for possible use over a wide-area network and is simple enough that adding it to a well-structured package should be simple, and requires no redesign of the package.
To put the rest of this paper into perspective, the windowing interface has the following properties:
The X Windowing System was designed as a research project as part of Project Athena at the Massachusetts Institute of Technology. It was not originally intended to become a commercial product, and some of its authors regret that it has done so. Most of its serious design faults, as a commercial product, are because computer science research and commercial development have very different aims. Some others are less justifiable.
Both its user interface and its internal design follow the now conventional independent-window, event-driven model, that was first developed at Xerox Palo Alto Research Center and later became commercially successful on the Apple Macintosh. One essential feature of this is that interactions are usually started by the user, unlike traditional IBM full-screen systems (such as ISPF and most MS-DOS applications), where the user responds to prompts from the application. This difference affects the whole design.
The design of the X Windowing System is too complex to go into here, but the following of its assumptions are worth mentioning.
The X Server is the program that controls the display, and communicates with the applications (the X Clients) using only the X Protocol. It needs high-speed access to reasonably fast backing store for reading in the font descriptions that it needs, and to save copies of windows that are not in use because they have been overlaid by others. The fonts and backing store may be on a local hard disk or on a machine connected by a LAN; the X Server does not need immediate access to these, but delays of more than a few seconds become irritating to the user.
One application that almost all users will need is a Window Manager, which is used to move windows around, resize them, create new ones and so on. There may be an arbitrary number of interactive applications (e.g. a CAD/CAM package) and non-interactive ones (e.g. a clock). At the lowest level, these are all treated the same by the X Server, but it is usually simpler to think of the Window Manager as part of the X Windowing System.
The input part is at a very much lower level: the X Server sends the X Client a sequence of time-stamped events , which are such things as key presses, key releases and mouse movements. All interpretation of these (including such things as auto-repeat on key presses and the upper-casing of shifted letters) is done by the X functions linked into the X Client. The X Server does not interpret events in any way, and does not display characters until the X Client sends it some output.
This model causes many problems, but the most serious is that the X system is usable only if the round trip from the X Server to the X Client and back to the X Server takes less than about 0.1 seconds. Measured packet rates for simple activities are 4.5 packets a second, and each of these has to be processed completely before the user sees the effect of his action on the display. This is why the X Protocol is unsuitable for use over a wide-area network, and is the second main reason for the author's project.
There are several other problems with the X Protocol, but the only one that is relevant here is the one to do with how separate X Clients affect one another. As `real' X applications were written, it became clear that certain permitted actions were harmful to other X Clients; in an attempt to solve this problem, the ICCCM (Inter-Client Calling Conventions Manual) was written. This could be regarded as the Geneva Conventions of X, and will probably be obeyed to the same degree.
At present, existing Widget sets, the X Toolkit and Xlib are all very UNIX-specific and the X Host must run a version of UNIX or a system that looks very much like it. The UNIX emulation in VAX VMS is adequate, and so are many of the other UNIX-like small-machine operating systems. OS/2 may or may not be suitable, but MVS, CMS, MS-DOS and Apple Macintoshes are quite out of the question. These restrictions apply to the current X libraries, and could be relaxed with only a small amount of redesign.
Modern hardware is sufficiently fast that the extra CPU requirement for running X on the host is not a serious problem, provided that the overheads of the process-switching caused by input from the X Server do not cause trouble. In practice, this means that X applications will run acceptably on a single-user workstation or very lightly loaded multi-user system, but not on a traditional multi-user system; MVS is particularly unsuitable, because of its large process-switching overhead. I/O and disk space use are not likely to be problems in any serious configuration.
Most existing systems will run out of memory before any other resource. Experience indicates that a single-user system needs at least an extra 1 MB of real store to run X applications, and an extra 2 MB if X is being used seriously (i.e. there are more than a couple of displayed windows). This memory is just for the X interface from the applications, and does not include the store needed by the applications themselves or that for controlling the display.
The CPU requirements for the X Server can be higher than for the X Host, mainly because the mouse and keyboard have to be controlled in real time. Several reports have said that it is not practicable to run more than a few X Servers on a multi-user system, and that individual workstations or X terminals are almost essential. Most existing X terminals are rather slow, because they use older, cheaper CPU chips like the Intel 80186 or Motorola 68010, but the next generation should be better. Modern commercially-viable UNIX workstations are almost always adequate, but older models may not be.
The memory, I/O and disk requirements are closely related, as implied above. The X Server needs fairly high bandwidth (at least 10 Kbytes/second) access to a hard disk which holds the fonts and the images of windows that have been overlaid by others. The further away the disk is, the more important it is for the X Server to have a large amount of main memory, so that it can cache fonts and window maps. Practical X Servers will have either a small (20 MB) local hard disk, or a large amount of memory and an Ethernet (or equivalent) connexion to the font and backing-store server.
A reasonable rule for workstations is that running an X Server needs at least an extra 2 MB if the workstation has a reasonably fast hard disk and virtual store. This memory should be doubled if the workstation is diskless or does not have virtual store, and doubled again if X is being used seriously (i.e. there are more than a couple of windows). If this memory is not available, X may run extremely slowly and the system will be very difficult to use.
The next level up from Xlib is the X Toolkit, which is sometimes called the X Intrinsics. This is a part of the standard X system, and must be available in all implementations, though there may be other toolkits as well. It is a library of functions for calling Xlib at a slightly higher level and for manipulating the data structures that are used for building Widget sets. Naturally, its design has major effects on the sort of Widgets that can be written to use it.
One of the X Toolkit's main components is the X Resource Manager, which is a sort of option decoder and database facility, intended for the control of customisation parameters. For reasons that are far too complex to go into here, its design and restrictions have significant effects on the design of other levels of the X libraries, in ways that are not obvious. An example of this will be mentioned later.
The main X interface, as seen by the applications programmer, is a particular Widget set. This provides the facilities to build various kinds of windows (e.g. main windows, pop-up menues, forms etc. ), and the fields that are to be placed in them (e.g. push-buttons, messages, updatable text etc.) While the Athena Widgets and some others are part of the standard X distribution, all Widget sets are regarded as optional extras as far as the specification of X is concerned.
Except for Xlib, the X libraries are all written in terms of a single application. It very rapidly became apparent that they were not adequate for running multiple independent applications on the same display, and some extra mechanism was designed. This extra mechanism was added at a fairly late stage and does not fit well with the the original design of X; the author cannot comment on how well it works, because it has only just been released and is not yet fully implemented in Motif, though it is in X11.4.
This mechanism is defined in the ICCCM (mentioned above), and is largely the definition of some interfaces between an application and the Window Manager it is running under. This takes the form of the application supplying hints to the Window Manager about the sizes of window that it wants, their placing and whether they are iconised. Naturally, the effect of these depends both on the level of X being used and on the particular Window Manager, and it is not yet clear how well they will work.
There are some short glossaries, but they are not adequate for practical purposes. X uses a very large number of special-purpose expressions, whose meaning is not clear to the newcomer. The glossaries contain only about 30 of the important terms, and even then the descriptions are unhelpful to outsiders. It is possible that some of the `Introduction to X' books would help with this; even if they do, the reference manuals need better glossaries.
The most serious problem is that the indices contain the main references only, which makes it almost impossible to find which functions and data structures affect each other, without learning the whole manual. This is a common fault of UNIX documentation, and is one of the main reasons that UNIX is such a difficult system to learn. If the X system is to be usable by ordinary programmers with a reasonable amount of effort, the indices need to be improved to at least the standard of MVS and CMS ones.
There are essentially no descriptions of the limitations and restrictions, which means that it is almost impossible to decide what is a bug and what is a design restriction. While this is true of almost all software suppliers and programming systems, X appears to be particularly bad. However, IBM, OSF and others should note that it is a major cause of wasted programming effort (50 in this case)!
One of the `high-level language' features of C is the ability to define new data types, and X does this very heavily. Partly because C is not a strongly-typed language, it is essential for the programmer to know the real type of all data structures. None of this is documented, and the programmer has to search through the headers using grep or an equivalent. This is tedious, rather than difficult or time-consuming, but should not be necessary.
While the design of the X facilities does not impose a particular windowing style deliberately , it makes a great many assumptions about it. It is true that most restrictions on style are imposed by the particular Widget set, but there are some that run much deeper. The author has had to perform several major redesigns to fit his original interface into the model that was assumed by the X system, and he may have to do it several times more. The extra facilities in Motif and X11.4 will help, but he does not yet know how much.
Note that these assumptions are usually possible to evade, and so it could be argued that X does not impose them. However, doing something against X policy can take 20 more code, 100 more debugging time and even then not achieve exactly the effect the programmer wanted. From the point of view of writing a maintainable application at an economic rate, X's assumptions have to be regarded as almost compulsory.
There are two cases of windowing applications that are likely to cause very little trouble. The first is the simple application, where there are a fixed number of windows with fixed layout; almost all existing X applications (e.g. xterm and xclock ) fall into this category. The second is the application that takes over the whole display and treats it as a simple graphics terminal driven via the X Protocol; many CAD/CAM packages fall into this category. The serious problems arise when trying to write complex applications that behave in a civilised manner to other X Clients or are written using the X Toolkit.
Those starting new X projects without previous X experience are very strongly advised to start implementing their user interface before they finalise its design! This is exactly the opposite to good software engineering practice.
The need to experiment with X before designing an application is to find out the critical unwritten assumptions; these are different between Xlib and the X Toolkit, and are probably slightly different between every Widget set. One of the reasons that the distributed X tools work well together is that the complex ones were almost all written by people who were closely involved with the development of X, rather than outsiders working solely from the documentation. The following are examples of the sort of assumption made by the X Toolkit, that can cause trouble to this type of application; they should not be taken as a complete list, because there are many others.
The author's application has several top-level windows, which are sometimes handled independently and sometimes as a unit (depending on circumstances beyond his control), and this causes him quite serious problems with the two kinds of input focus. Even though he has redesigned this area several times, much of the user interface is constrained by the facilities available in the X Toolkit. Calling Xlib functions directly provides the necessary function, but often causes the X Toolkit to become confused.
Similar remarks apply about controlling when windows are resized, converted into icons, or even completely hidden. A reasonable user interface is that top-level windows simply disappear when closed, except for the last, which turns into an icon; naturally, it must be possible to pop up any top-level window from any other. This is extremely difficult to achieve with the current X system, and may or may not be possible with Motif, though it should be when ICCCM is implemented fully (i.e. probably in X11.4).
Another restriction in this area is that the same Widget tree is used both for the layout of the Widgets and for searching the X Resource database; clearly, this is the normal case, but the X Toolkit makes it almost compulsory. If an application's window does not fit into this model, it must go to great trouble to disable many of the X Resource Manager's actions. Because X Resources include both user preferences (like the colour of text) and critical options (like limits on the sizes of data), the application cannot ignore this problem.
Another aspect of this problem is that the X Resource Manager is designed to return a value if at all possible. A common requirement is for the application to set a default, unless the user has overridden that particular value. This is extremely difficult to do using the X Resource Manager, because the application cannot distinguish between the values returned by the X Resource Manager for generic defaults and specific ones. The normal (and unfriendly) action is for the application to force the value regardless.
One of the nastiest aspects of the author's program is his implementation of a Help callback, which is not supported by the HP Widgets. The reason for this is that there is no way to extend the set of global actions caused by the user pressing a key or a mouse button, and it has to be done for each Widget individually. Motif Widgets (but not Athena ones, even in X11.4) do support a Help callback, so he can clean up his code, but still do not permit global extensions; if he wanted to support a global Dump or Halt key, he would still have to do it on all Widgets individually. This lack of extensibility is an X Toolkit design restriction.
The ICCCM addresses the problem of how X Clients should use global facilities to avoid interfering with one another, but this is only part of the problem. X assumes that certain facilities should never be used globally, and makes it almost impossible to do so without modifying all X Clients. There is a serious risk that some software vendors may `solve' this problem by insisting that their programs have complete control of the display, and that any other suppliers' X Clients running on the same display must obey their conventions.
Debugging large C programs is notoriously difficult, because of the nature of the C language, but some of the features of the X libraries and most widget sets make this problem worse. Unfortunately, many of these features are at the design level and are be present in Motif and X11.4 as well. The author guesses that he has spent his debugging time for the following reasons:
While the author is an extremely experienced programmer, he is not that good! His code contains hundreds of places (sic) where he has found it necessary to do something strange because the obvious facility did not work; he regrets not having had access to Motif for long enough to discover how many of the problems are present in that. Whether or not they are, the effort needed to debug large X applications must be one of the main obstacles to using X.
The X libraries cause so many diagnostics from lint that it runs out of space before printing the ones caused by the application. With considerable effort, it is possible to extract some useful information from this output, but it is not easy. This problem is principally a case of poor coding standards, and may well be better in Motif; it is definitely better in X11.4. It cannot be solved completely in either, because of the design of the X libraries.
A more serious problem is that the X libraries use type punning everywhere, and many of their `functions' are actually macros that include an explicit cast (often to the generic X pointer type caddr t ). This means that the type-checking facilities of ANSI C are almost completely disabled for such functions, and this causes a great deal of wasted effort in locating simple typing errors. This is unchanged in Motif and X11.4.
The design of the X libraries also relies upon the compiler not enforcing the ANSI standard in several respects; the most serious of these is that they pun data and function pointers very heavily. It is essential to ignore very large numbers of warnings from ANSI C compilers (often one per line of source), which means that the messages that indicate programming errors are easy to miss. This is unchanged in Motif and X11.4, though there has been some improvement in the standard of the code.
It is fairly common for the application to fail with an X Toolkit error (and occasionally a protocol error from the X Server), or for the X Toolkit to issue a warning. The diagnostics give a brief description of the problem, but give no hint as to what operation was being performed or which X function was called by the application. Debugging is made even more difficult for the programmer because X Toolkit errors cause the program to stop without taking a core dump, and so a debugger cannot be used to provide a traceback.
When the failing call has been located, there is still more work to do. The design of the X libraries requires a large number of function calls to set up parameters, followed by one to take the appropriate action. In most cases, the cause of a failure is not in the action call but is a side effect of an erroneous parameter setting (sometimes many actions previously). Tracing a problem back to its original cause can be a very slow process.
One of the reasons that the above design can cause so much trouble is that the X libraries contain very little checking, which also increases the number of core dumps. When the parameters are checked, it is usually when they are used and not when they are set by the programmer, and this is normally deep in the X Toolkit doing some apparently unrelated activity. This may possibly be a little better in Motif and X11.4, but is unlikely to be much better because the overall design is unchanged.
Traditional (i.e. 1960s) software engineering practice was to provide two facilities to help locate such problems. The first was a function to check the internal consistency of the X libraries' data structures, and the second was one to dump them in a human-readable format. These functions have not been provided, and are not present in Motif or X11.4 either. In the author's view, they could reduce the debugging time for complex X applications by a factor of two to three. IBM and OSF please note!
The first problem is that the X libraries do not define the scopes that these data structures must have; some of them are copied as structures, but sometimes their address is saved for use when X performs the relevant action. Data structures of the second kind are assumed to remain in existence until the X libraries have finished with them, which may be an arbitrary amount of time later. If they were allocated on the stack, or their space has been freed, the symptoms are usually very peculiar and often unrepeatable.
The second problem is that the X libraries do not define which data structures are read-only and which are updated in place. This rarely causes serious trouble, unless the programmer uses the same data structure more than once, and even then may work in some cases. The author has been caught by this only once or twice, and believes that it is not a major problem in the way the previous one is.
Both of these problems could be solved by better documentation. Motif and X11.4 may be better, but the author has looked at both sets of documentation and did not notice any improvement. Of course, it is possible that they has removed the problems from the code and therefore does not need to document them, but this seems unlikely.
It does not take long to write 10,000 lines of code on a modern system, but that is not the point. The amount of code is one of the reasons that the store requirements for X are so excessive, but the real disadvantage is the reduction in maintainability. A large proportion of that code (say 1,000 lines) is to cover up bugs and restrictions in the X system; unless it is possible to remove almost all of it in Motif, the author doubts that his work will be worth using.
The following are the main problems with X, and whether the author believes that they will be better in Motif and X11.4:
The above is not very hopeful, and implies that the X system (including Motif and X11.4) still has a long way to go before it is a reasonable software tool. However, as described above, quite a lot of its problems are straightforward to solve. Several of the above could be resolved by better documentation, and it would not be difficult for an organisation like OSF to write some debugging tools. We shall see what happens in the next year or two.