Some useful additions to the RPC concept

I start to get this feeling that my body hates me. Why am I always so tired ? Don’t I sleep well enough ? Well, I can’t do much about pollens in the air and warm nights, but improving the rest didn’t help. Is it because I am a bit hypothyroidic then ? But medicine has yet to find a curable cause to that, although there still is a last chemical to study in my blood. Or is it a normal side-effect of one of my allergies ? I’m starting to lose hope. How much time left before I have to postpone all good stuff to holidays ? But that’s not happened yet, and while I still can write and design stuff, while I still have some energy left to rebel against that senseless fatigue that tries to fill my life with chaotic thoughts and a depressing absence of in-depth intellectual activity, here are some additions to the RPC system which I am currently pondering.

Task completion notification and results

RPC is, by design, a member of the family of asynchronous programming primitives. You send some work to do to a system service, the function returns immediately, and you are free to do everything you want from then on. This programming model is traditionally useful for building massively parallel and highly responsive software, since one needs not to have threads blocked and care about how long they will remain so. The inevitable counterpart, however, is that it is harder to return results, or even notify the caller that some chunk of work has completed.

The traditional solution that is proposed to this problem is callbacks. When it’s done, the asynchronous thread “returns” by calling a function of the caller to notify it of the completion of the requested work and its result. Callbacks, however, are not the easiest thing in the world to manage for clients, since they mean stray threads that have to be dealt with and properly synchronized with the rest of the program’s code. RPC makes it worse : while it is possible to use it to implement a form of inter-process callbacks, it also means that service clients have to mess with the complex part of the RPC system, service broadcasting. This, in turn, is not a highly desirable outcome : clients spawning and killing RPC entry points all the time hurts system performance, is a potential attack vector, and violates the important rule that OS structure should work towards making the life of user-mode software easier even when it means making the life of system developers harder.

An interesting high-level approach to this problem is futures : when an asynchronous function is run, it returns a “future” object, that looks much like a pointer to the result but will cause any function that attempts to touch it to block until said result is actually available. However, while this is an extremely elegant way to give asynchronous calls a more synchronous feel, I sense several problems with it. First, as with many other high-level programming constructs, future implementations tend to be highly language-specific, and mean a lots of work for the OS who tries to use them as a core primitive. A more pernicious side-effect of futures is that since they are designed to feel a lot like synchronous results, programs risk way too often to accidentally trigger the lookup mechanism and block.

Once these elegant approaches have been discussed, less elegant approaches remain. After all, one may argue, since RPC supports pointers and references by design, RPC routines could solely return results through those. The main issue, then, would be that the caller of the RPC routine does not know when it will complete, and when it will be able to access and trust the value of the processed variables. However, solving the task completion notification problem, alone, is easier than solving the full task completion notification and result retrieval one, and could perhaps be done with an ad-hoc solution.

Let’s imagine that anytime a program makes an RPC call, it would get as a result some sort of “task identifier” that uniquely identifies the request. Let’s add to that at least one of these functionalities is then added :

  • Threads can synchronously wait for the completion of a task when they have nothing else to do
  • Threads can check the status of a task (with the usual warning that polling is wasteful)
  • Threads can set up a parameter-less callback to be notified of task completion (though I don’t like this latter one, as it basically requires a re-implementation of RPC)

Now, this starts to look an awful lot like a tweaked UNIX signalling-like small extra IPC mechanism, that could just as well used in other places (such as for notifying user processes of external events). Task completion detection itself could work by automatically considering a task completed when the execution of the associated thread is over, as a default, but letting RPC services take back manual control on that mechanism if they need to (such as in task-serializing RPC services like disk drivers). There is one last problem to tackle, though.

Over time, as RPC calls are made, the OS will have to manage an increasing amount of task identifiers, linked with the boolean information of whether the associated task has terminated or not. In effect, this looks suspiciously like a memory leak : as system uptime grows, the amount of managed task identifiers also grows, possibly to unacceptable levels. Clearly, something must be done about this problem.

Among the various options that I could think of, the only one that could work fairly well would be to have task identifiers “time out” some time after a task has completed, which would naturally regulate their population. But then, the associated delay would have to be well chosen, and that in turn brings extra complexity to the implementation (what if, say, aggressive power management rules makes task run extremely slowly, like one second every 15 minutes, when a device is not in use ?). If someone has a better idea in mind, I’m all ears.

Task cancellation and error management

Sometimes, users start some big request in a software, then realize that they have made a mistake and want to cancel this request. To provide this option easily, applications need in turn system services to provide the same cancellation feature. Conversely, service requests also like to be able to return before having completed their work if they run into errors. This is particularly a problem with synchronous APIs, which need extra threads to be spawned and a way for API functions to return an “invalid” result when they have been cancelled.

Now, what if there was a way to send a UNIX-like signal to an RPC-driven daemon, that it is able to periodically poll during long tasks (I really can’t think of a more elegant way) so as to abort and return more early when a request is cancelled ? Reminds you of something ? Well, that is for a reason. The task state storage mechanism mentioned above, if adopted, could be used to manage an extra task state, the “Cancelled” state, that services and client could probe to adapt their behaviour as they see fit. That would be, as far as I can tell, one more reason to embrace it.

(And as a bonus feature, since that mechanism is kernel-managed, all pending RPC calls could automatically be put in an error state in the sad event that the associated server would freeze or crash, a useful if a bit frustrating error reporting feature)

RPC batches

Some system services are naturally suitable for a manipulation via a small amount of short and expressive requests, or can be easily adjusted to deal with those. One can think, as an example, of the file system. Others system services, however, are cursed to be forever spammed with endless amounts of service requests, unless they decide to embrace some sort of scripting API, which is slightly at odds with the philosophy of the RPC mechanism, that normally directly exposes the programming interface of services to clients.

Due to this, Alfman recently suggested here that a performance optimization be brought in the RPC mechanism so as to allow clients to send large amounts of RPC requests all at once. And I have to agree in principle, although it is the kind of ideas that I would tend to keep for a v2 of this operating system, since they are not essential to its core operation and only represent a nice extra if added on top of a working OS.

Conclusion

So, what do you think about these ideas and their possible inclusion in this OS ? Do you like or dislike some, and why ? Do you have ideas on those points that I don’t know how to address yet ? As usual, do not hesitate to express yourself in the comments.

Advertisements

One thought on “Some useful additions to the RPC concept

  1. Alfman June 18, 2012 / 9:31 am

    “Callbacks, … mean stray threads that have to be dealt with and properly synchronized with the rest of the program’s code. RPC … also means that service clients have to mess with the complex part of the RPC system …”

    I think you are assuming that you have to combine async callbacks and multithreaded code together at the same time. I agree it could become buggy and very frustrating, but don’t forget the pure event loop model only needs a single thread. No synchronisation is necessary at all.

    This kind of got out of hand below, but I bring it up because it might help with your concerns…

    Epoll allows extremely efficient event loops since it consolidates all kinds of events into a single syscall, and unlike select or poll, doesn’t suffer from additional implicit overhead as more events are monitored. Below I use epoll in my async library to retrieve a large batch of (unrelated) events from the kernel and use the “PTR” member to map directly to a callback function. I can then call these in a very tight loop.

    I pray this displays ok (but I’m not highly confident it will)

    void async_loop() {
    // what if an eariler event causes the deletion of an object in a later event?
    while(async_halt==false || async_halt_pend > 0) {
    // execute any defered callbacks (this includes objects which are cast as defered)
    // async threads can be cast into async_defers
    while(async_defer_queue_head!=NULL) {
    // remove from list first, then process
    ASYNC_DEFER*defer = async_defer_queue_head;
    async_defer_queue_head = defer->next;
    // Execute callback
    defer->next = (ASYNC_DEFER*)-1;
    (*defer->callback)(defer);
    }
    // as long as there are any defered callbacks, we don’t want to block via epoll
    // Allow all event types to get processed before blocking
    async_fd_pending_count = epoll_wait(async_fd_epoll_handle, async_fd_pending_queue, sizeof(async_fd_pending_queue)/sizeof(struct epoll_event), -1);
    while(async_fd_pending_count>0) {
    async_fd_pending_count–;
    ASYNC_FD*async = (ASYNC_FD*) async_fd_pending_queue[async_fd_pending_count].data.ptr;
    (*async->callback)(async, async_fd_pending_queue[async_fd_pending_count].events);
    // The callbacks may modify the event list (due to deletions).
    }
    } // while ! halt
    } // function async_loop

    Deferred callbacks above are process-local callbacks that don’t involve the kernel. When calling epoll with a batch size greater than 1, there are some very subtle issues that can arise, for instance if the 1st callback cancels a 2nd event that’s already been triggered in the same epoll event batch as the 1st event, it might free resources in use by the 2nd event, then the 2nd callback would not only be unexpected, but might act on invalid resources. Luckily the library does handle this case in the cancellation routine by cancelling callbacks posted to the global “async_fd_pending_queue” structure.

    The beauty in such a generic event loop with generic callbacks is that there’s only one blocking call in the whole program which gets shared by separate tasks that don’t need to have any knowledge about each other. And with epoll, one syscall can transfer dozens of unrelated events in one shot.

    void timer_callback(ASYNC_TIMER*async) {
    printf(“Timer Callback\n”);

    async_timer_schedule(async, 3000);
    //async_thread_destroy(&thread);
    }

    void test() {
    async_timer_create(&timer1, &timer_callback);
    async_timer_schedule(&timer1, 500);

    async_file_create(&file, &filebase, “async.test”, &file_cb, NULL, ASYNC_FILE_WRITE | ASYNC_FILE_CREATE | ASYNC_FILE_TRUNCATE);

    async_process_create(&process, “echo ECHO!”, &process_cb, NULL);

    async_dir_create(&dir, “/home”, &dir_cb, NULL, ASYNC_DIR_STAT);

    async_address_create(&remote_address);
    async_address_set(&remote_address, “127.0.0.1”, 5555, ASYNC_ADDRESS_AUTO);
    async_tcplisten_create(&lsock, &lsock_cb);
    async_tcplisten_listen(&lsock, &remote_address);
    async_address_destroy(&remote_address);

    async_address_set(&remote_address, “x.y.z.z”, 80, ASYNC_ADDRESS_AUTO);
    async_tcpsocket_create(&sock, &socket_cb);
    async_tcpsocket_setup_read(&sock, &socket_read_cb);
    async_tcpsocket_setup_write(&sock, &socket_write_cb);
    async_tcpsocket_connect(&sock, NULL, &remote_address);

    async_resolve_create(&resolve, &resolve_cb);
    async_resolve_lookup_host(&resolve, “av.com”, ASYNC_ADDRESS_IP4);
    }

    This test program prints “Timer Callback” at 0.5s, 3.5s, 6.5s, etc.
    It simultaneously writes output to async.test (mechanics not shown)
    It simultaneously runs an external shell script in parallel.
    It simultaneously lists the contents of /home/
    It simultaneously sets up an echo server on 127.0.01:5555
    It simultaneously connects to an HTTP server.
    It simultaneously resolves “av.com”.

    None of these are blocking calls. and all code runs atomically (ie no interruptions inside functions). Furthermore any of the callbacks (such as tcplisten) can establish/control other async resources. The library obviously targets C, and is a bit awkward as a result, but the syntax would be better in C++. You could have completely different programs in different namespaces running under the same async process and so long as they are well behaved they wouldn’t interfere with each other – zero synchronisation running asynchronously under one thread. Obviously these async primatives are a bunch of wrapper functions for linux calls in async mode (or falling back using MT blocking under the hood when async primitives aren’t available such as readdir).

    Granted, I’m working with a multitude of linux event passing mechanisms, while your working with RPC, but I don’t think they’re different in the abstract, are they?

    “An interesting high-level approach to this problem is futures”

    Very interesting, but I’m not sure what it gains you either. One still must setup async, in which case one should already know when an object is ready. Conversely with blocking code, one will always syscall into the kernel (even if only momentarily). So the object would be testing conditions that are already implied on each access.

    “Task cancellation and error management. Sometimes, users start some big request in a software, then realize that they have made a mistake and want to cancel this request.”

    This turns out to be very buggy under linux. There used to be lots of race conditions in the file system, while kernel devs have diligently fixed them, many were fixed by taking a shortcut of blocking the cancellation request instead of unblocking & cancelling the request. File system requests are usually fast so it doesn’t matter too much, but it’s why disk/NFS IO errors cause so much havoc – attempts to cancel blocked requests themselves get blocked. We have to wait for the timeout whether we want to or not.

    “So, what do you think about these ideas and their possible inclusion in this OS ? Do you like or dislike some, and why ?”

    To be honest, I’d like to see the choices as example code. While you’ve probably specified everything already, I have a short term memory and I don’t want to assume to know what it’s going to look like. Does this example approximate what you have in mind?

    #include “RPC”
    #include “SampleService”

    void BlockingTest() {
    SampleService service(“rpcservicename”); // optional parameter to override default
    SampleServiceRet ret = service.call(1,2,”abc”); // blocking call by default
    if (ret.error) {…}
    cout << ret.heresmydata << ' ' << ret.moredata << eol;
    }

    void callback(SampleServiceRet ret, void*context) {
    // spawned in a new thread or existing thread depending on async mode
    if (ret.error) {…}
    cout << ret.heresmydata << ' ' << ret.moredata << eol;
    }

    void MTTest() {
    RPC::async_mode(MT); // cause async calls to return with new threads
    service.call_async(1,2,"abc",&callback,NULL); // non-blocking call
    }

    void STTest() {
    RPC::async_mode(ST); // cause async calls to return in same thread.
    service.call_async(1,2,"abc",&callback,NULL); // non-blocking call
    }

    I think we could technically come up with a way to pass back more parameters in the callback, but when would that be desirable considering that the blocking version only returns one parameter anyways?

    "Threads can set up a parameter-less callback to be notified of task completion (though I don’t like this latter one, as it basically requires a re-implementation of RPC)"

    Is this example better than what you were thinking? I'm not a fan of unix signals because they require additional syscalls to get data, but I don't see a reason your RPC couldn't pass the data strait away in the callback like above?

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s