[Food for thought] Not everything is a file, but a universal filesystem does make sense

I’m fairly critical of the traditional UNIX idea that everything can be expressed in terms of files. Files are unstructured units of permanently accessible and sequentially organized data, which can be shrinked, grown and seeked through at will. Real-world computer hardware, on the other hand, is often better described in terms of fixed-size data structures of ill-known contents, which notify software of undergoing stuff through events (or, in hardware parlance, interrupts), and which predominantly communicate through streams of data even when they masquerade themselves as chunks of RAM.*

For this reason, computer hardware would be best abstracted as a special kind of object, which aside of methods and “normal” members also has event sources and sinks (think Qt signals and slots, C# delegates) and stream I/O (which can be emulated through virtual methods that pump data in and out, or return specially crafted stream objects which can be used to this end). This is not something which is easily expressed in terms of files.

That being said, I do see some value in having a universal way to index every system resource in a computer system. It makes sense, in particular, to have a universal way to express concepts such as “I have permission to access this resource”, “A resource has just been plugged in/taken out”, or “Here’s an extensive list of all the available system resources, which can be parsed hierarchically or searched by criteria”.

So, in the end, it seems to me that the most relevant option would be to have a virtual filesystem of objects, representing elementary system resources or organisational units thereof, which may or may not feature the appropriate open/read/write/seek interface required to call them a file. Does someone know if another OS has already attempted to use such an abstraction, or have additional thoughts on this concept?

* Try to address memory-mapped hardware using inappropriate memory alignment or integer sizes, or while forgetting CPU caching on, and you’ll see what I mean.

15 thoughts on “[Food for thought] Not everything is a file, but a universal filesystem does make sense

  1. Alfman April 16, 2013 / 5:43 am

    Hadrien,

    “I’m fairly critical of the traditional UNIX idea that everything can be expressed in terms of files.”

    I agree, the trouble with security that assumes everything is a file is that not everything is actually a file. Some mappings are even missing in linux, such as network interfaces. Security designed for file systems (ugo+rwx) offers bad granularity in other contexts. File system security is clumsy for pseudo/adhoc devices that are not persistent files, udevd can be configured to assign permissions in an inconsistent way, but that falls short of the security unification unix designers sought when the device files were actually static.

    I’ve always preferred the “everything is an object” conception (even files). The inverse is untrue, objects are not (necessarily) files.

    “Does someone know if another OS has already attempted to use such an abstraction, or have additional thoughts on this concept?”

    I may be criticized for saying this, but as I recall the WinNT APIs have much better kernel integration for security descriptors at all levels of the OS. MSDN used to have excellent examples of these. I’ll be damned if I can find them again, I find MSDN much less useful now that it’s been redesigned to promote windows 8. This link is not really what I was looking for, but one can dynamically create arbitrary access lists and pass them to any kernel or file system objects that are made accessible to userspace via kernel APIs. I liked that kernel security could be exported into userspace applications and worked across domains independently from individual file systems.

    http://msdn.microsoft.com/en-us/library/windows/desktop/ms717798%28v=vs.85%29.aspx

    “So, in the end, it seems to me that the most relevant option would be to have a virtual filesystem of objects, representing elementary system resources or organisational units thereof, which may or may not feature the appropriate open/read/write/seek interface required to call them a file.”

    I agree with your approach. I’d call it a database of objects though. It could contain both instances of objects as well as classes that can be dynamically instantiated into objects on demand. We usually think of objects as entities that are acted upon, but in an os context like this an object could also be a “service”, like crypto primatives or PNG encoding/decoding. Instead of calling DLL’s, application’s would use these userspace services. Any application could export it’s own services to other applications. Sorry if I’m straying a bit into what *I* want :)

    The power of “devices are files” is that common command-line tools are often able to access them (even though ioctrls break the ideology on linux). It would be very useful if we had command line tools to query/instantiate/access these objects.

  2. Hadrien April 16, 2013 / 7:12 am

    I may be criticized for saying this, but as I recall the WinNT APIs have much better kernel integration for security descriptors at all levels of the OS. MSDN used to have excellent examples of these. I’ll be damned if I can find them again, I find MSDN much less useful now that it’s been redesigned to promote windows 8. This link is not really what I was looking for, but one can dynamically create arbitrary access lists and pass them to any kernel or file system objects that are made accessible to userspace via kernel APIs. I liked that kernel security could be exported into userspace applications and worked across domains independently from individual file systems.

    http://msdn.microsoft.com/en-us/library/windows/desktop/ms717798%28v=vs.85%29.aspx

    I’m not sure I understand what it is that you are praising. Is it that kernel-mode security mechanisms can also be used in user space ? If so, how is that unique to Windows NT ?

    (I do agree that these days, WinNT should receive more attention from OS geeks, as AFAIK it’s one of the few well-documented OSs that is not UNIX-based remaining in the computing world)

    I agree with your approach. I’d call it a database of objects though. It could contain both instances of objects as well as classes that can be dynamically instantiated into objects on demand. We usually think of objects as entities that are acted upon, but in an os context like this an object could also be a “service”, like crypto primatives or PNG encoding/decoding. Instead of calling DLL’s, application’s would use these userspace services. Any application could export it’s own services to other applications. Sorry if I’m straying a bit into what *I* want :)

    Myself, I tend to imagine that kind of task done through various kinds of IPC, where in a way processes would themselves be considered much like objects. But implementing it as first-class system-handled objects might address a concern which I’ve been having about for some time with IPC-based approaches: how to replace a program which provides a given system service with an equivalent one, without posing requirements on the replacement such as having a given file name, and without letting any piece of malware do it either.

    In this context, system services could state “I implement named resource XYZ, with this public member/methods/event/streams/whatever interface” to the operating system, while applications would state “I seek named resource XYZ, with this public member/methods/event/streams/whatever interface”. The operating system would play the role of connecting the dots. System service naming would be irrelevant, which would give additional flexibility in filesystem organization.

    A concern which remains, though, is that of voluntary and involuntary namespace collision. If services are looked up by file name, this kind of issue is implicitly avoided, since file names are contractually guaranteed by the file system to be unambiguous. However, with object databases, nothing prevents two processes A and B to claim implementing the same resource, and the system would have no way to tell which is the right one.

    Conversely, if someone wants to implement a new service, how does he find a unique name for it, without breaking the “service are independent from their implementation” abstraction by putting personal information in it ?

    Perhaps the former problem could be handled like we do file associations today, by keeping track of some “permanent” system resources even when their implementation is not running, and allowing several processes to implement them by letting users directly choose which implementation they want to use. As for the later problem, that could be a good place for UUIDs, the problem being that UUIDs suck as identifiers because they are hard to read, hard to memorize, and have no meaning.

    The power of “devices are files” is that common command-line tools are often able to access them (even though ioctrls break the ideology on linux). It would be very useful if we had command line tools to query/instantiate/access these objects.

    Many modern object-oriented programming languages have the concept of type introspection, allowing one to query various information about an object (methods, members…) in a standard way, which would be perfectly appropriate for this purpose.

    (Speaking of which, I’ve been thinking for some time that command line OS scripting should probably aim to use a well-known, general-purpose and interpreter-friendly language like Python, instead of always forcing yet another obscure syntax and set of programming practices upon us.)

  3. Alfman April 16, 2013 / 10:01 am

    Hadrien,

    “I’m not sure I understand what it is that you are praising. Is it that kernel-mode security mechanisms can also be used in user space ? If so, how is that unique to Windows NT ?”

    It’s been over a decade since I’ve really worked on Win32s, so I hesitate to attempt any kind of detailed answer, but each call to access every resource had optional security descriptors, this permitted the OS and applications to use security checks completely different from the process UID/GID checks the linux kernel does. For instance it is much easier for services (such as IIS) to validate kernel security in the context of individual requests without giving the entire service permission to the all resources that any request may need.

    For instance, apache is often given full permission to all the client websites running under it. This is a security risk because the apache credentials are passed onto the PHP interpreter which, in many configurations, permits one website to access another. One can often access the website data files (and therefore mysql passwords) of other clients. I’ve confirmed this is still a common problem. PHP has a userland security model to try and catch script access to other user websites, but this is a clear shift away from kernel enforcement. There are alternatives, like running apache as root and letting it drop permissions before running PHP, but running apache as root is less secure than using windows security descriptors.

    Also, linux RWX permission bits are way too coarse and file specific. Windows security descriptors permeate the whole OS, from files to sockets to processes and even threads (this is probably what makes them “heavy weight”, but it’s still worth mentioning).

    “However, with object databases, nothing prevents two processes A and B to claim implementing the same resource, and the system would have no way to tell which is the right one.” … “Conversely, if someone wants to implement a new service, how does he find a unique name for it, without breaking the “service are independent from their implementation” abstraction by putting personal information in it ?”

    These are valid points, I don’t fully know the best practices but perhaps you could look to the java and .net frameworks which have the same issue with class namespaces. The linux filesystem doesn’t make shared object linking entirely unique either (ambiguity between /usr/lib/ /usr/local/lib /lib …). In any case I don’t see why the database couldn’t enforce unique constraints in the first place? Then only the first application would get a name. I’d only make entries persistent if an application requests it to be, but then ideally the OS could launch the application when it’s not running. Also, it might be worthwhile to have persistent entries require a “TestObjectInstance” call that needs to pass in order for the persistent exclusion to be enforced by the kernel, you wouldn’t want a deleted application from blocking a new application from hooking the service.

    “Speaking of which, I’ve been thinking for some time that command line OS scripting should probably aim to use a well-known, general-purpose and interpreter-friendly language like Python, instead of always forcing yet another obscure syntax and set of programming practices upon us.”

    I have very limited experience with python, but I agree. I use perl for my shell scripts, which may be a mistake because it’s a bit archaic, but my choice was easily decided by the fact that it runs everywhere without installing any additional languages. Also I think a GUI tool would be very useful as well for viewing/accessing the OS object DB.

  4. Hadrien April 16, 2013 / 8:40 pm
    However, with object databases, nothing prevents two processes A and B to claim implementing the same resource, and the system would have no way to tell which is the right one.” … “Conversely, if someone wants to implement a new service, how does he find a unique name for it, without breaking the “service are independent from their implementation” abstraction by putting personal information in it ?

    These are valid points, I don’t fully know the best practices but perhaps you could look to the java and .net frameworks which have the same issue with class namespaces.

    I don’t know anything about Java and only know little about .Net, but as far as I can tell, the standard way to manage a large framework is to make the namespace hierarchy as deep and clear-cut as possible, so as to reduce the likelihood of collisions occuring, and then perform integration tests (“okay, let’s link everything into a single binary !”) regularly so as to check that no namespace collision occurs.

    This works, as long as one is actually able to perform said integration tests, which is the case for centrally managed projects like frameworks. But what we are talking about here is people in an OS ecosystem independently deciding to create a new system resource type for their personal use, which is an example of decentralized decision making.

    The linux filesystem doesn’t make shared object linking entirely unique either (ambiguity between /usr/lib/ /usr/local/lib /lib …).

    Actually, I’m pretty sure that predictability is internally enforced through a hidden LD_LIBRARY_PATH-like variable, which enforces a sequential order through which library paths should be probed. On a given system, when a given library name is requested, it is always the same one that will get linked to your executable. Not always that which you expected, though…

    By the way, couldn’t that library path mechanism be used for security exploits, such as by setting a fake library path that replaces system GUI libraries with malicious variants that log your input data?

    In any case I don’t see why the database couldn’t enforce unique constraints in the first place? Then only the first application would get a name. I’d only make entries persistent if an application requests it to be (…)

    It could, but it only replaces the duplicate identifier problem with a race problem. As an example, on an OS where the startup of system services which are not immediately needed is delayed, a malicious program could claim to implement a system resource early on and become the effective master of that resource, with all the negative consequences that brings. Innocent software which doesn’t know better could crash the system in a similar way, by overriding system resources with unrelated home-made designs.

    (…) but then ideally the OS could launch the application when it’s not running.

    I had this one in mind too when proposing this persistent resource design. Also, it would be awesome if system services could be dynamically restarted without losing resource availability even for a fraction of a second, but I think there might be better way to manage that kind of scenario.

    Also, it might be worthwhile to have persistent entries require a “TestObjectInstance” call that needs to pass in order for the persistent exclusion to be enforced by the kernel, you wouldn’t want a deleted application from blocking a new application from hooking the service.

    Didn’t think about that one, though, you are right that as soon as resources become permanently visible, one must ensure that they are removed as soon as the last service which provided them is removed from the service.

    Not sure if I like the specific idea of instanciating a test object though. Wouldn’t it require to start up all system services at the time where the object database is initialized ?

  5. Alfman April 17, 2013 / 5:00 am

    Hadrien,

    This works, as long as one is actually able to perform said integration tests, which is the case for centrally managed projects like frameworks. But what we are talking about here is people in an OS ecosystem independently deciding to create a new system resource type for their personal use, which is an example of decentralized decision making.

    Yes, the official frameworks themselves are centrally managed and collision-free, but that’s not what I meant. 3rd party classes can conceivably collide if developers (intentionally or accidentally) share the same namespaces. For example, .net uses a GAC – global assembly cache to store/index all assemblies. I believe any assembly can contain any combination of classes & namespaces, One assembly can have companyA.projectA.classA, another assembly can have companyA.projectA.classB, I do not know what would happen if any assemblies contained overlapping classes.

    Actually, I’m pretty sure that predictability is internally enforced through a hidden LD_LIBRARY_PATH-like variable, which enforces a sequential order through which library paths should be probed. On a given system, when a given library name is requested, it is always the same one that will get linked to your executable.

    My point was that the file system doesn’t enforce the uniqueness of names in these corner cases, but if having a deterministic selection process is good enough here, then maybe you could also implement a deterministic selection process as well even if it’s not the expected result?

    By the way, couldn’t that library path mechanism be used for security exploits, such as by setting a fake library path that replaces system GUI libraries with malicious variants that log your input data?

    Funny thing happened to me once when I installed a package from repos, and then installed a newer official version from source using “make install”. There were breakages and it took me a while to strace the problem and discover that ldd wasn’t linking the correct shared object files. I don’t know the best way to prevent this from happening, I almost think having a clear run time error could be more helpful than accidentally using the wrong one – it’d certainly help pinpoint the problem much faster. Like you say, the wrong version might not be detected at all, and that could be dangerous.

    It could, but it only replaces the duplicate identifier problem with a race problem. As an example, on an OS where the startup of system services which are not immediately needed is delayed, a malicious program could claim to implement a system resource early on and become the effective master of that resource, with all the negative consequences that brings.

    Yes I see. in this case I guess you could either explicitly grant these apps permission to host services, or consider using signed executables.

    Another idea is to consider the problem from the opposite angle, that of the consumer. Whenever a conflict is detected, a GUI could pop up and ask which instance should be used. This choice could be saved persistently until the services changed again.

    This problem deserves more thought.

    Not sure if I like the specific idea of instanciating a test object though. Wouldn’t it require to start up all system services at the time where the object database is initialized ?

    I was thinking more along the lines of running the test only when the database is modified and a conflict is created. This would be relatively rare. At that point, detect if the old object is still valid before entering the new one.

    This is OT, but the ideal file systems in my head have dependency satisfaction similar to foreign key integrity within relational databases (I’ve always wanted to unify databases and filesystems). A foreign key record cannot be deleted or updated independently from the records which reference it. Foreign keys can be declared “on update cascade” or “on delete cascade” such that changing the record here will cause all referencing records to get changed as well. If you had a file system with dependency enforcement, then you could be confident that the object database would be updated in real time along with application updates.

  6. Hadrien April 20, 2013 / 8:33 am

    Yes, the official frameworks themselves are centrally managed and collision-free, but that’s not what I meant. 3rd party classes can conceivably collide if developers (intentionally or accidentally) share the same namespaces. For example, .net uses a GAC – global assembly cache to store/index all assemblies. I believe any assembly can contain any combination of classes & namespaces, One assembly can have companyA.projectA.classA, another assembly can have companyA.projectA.classB, I do not know what would happen if any assemblies contained overlapping classes.

    Just tried redeclaring the System.Console class in a test program. What happens is that the compiler spits a “/home/leonidas/Test/Main.cs(17,17): Warning CS0436: The type `System.Console’ conflicts with the imported type of same name’. Ignoring the imported type definition (CS0436) (Test)”, and then uses the homemade version. Next, I should try to create a library with my pointless class redefinition and add it to the GAC, so as to see what happens. Will do that once I’ve learned how it works in C# :)

    My point was that the file system doesn’t enforce the uniqueness of names in these corner cases, but if having a deterministic selection process is good enough here, then maybe you could also implement a deterministic selection process as well even if it’s not the expected result?

    (…)

    Funny thing happened to me once when I installed a package from repos, and then installed a newer official version from source using “make install”. There were breakages and it took me a while to strace the problem and discover that ldd wasn’t linking the correct shared object files. I don’t know the best way to prevent this from happening, I almost think having a clear run time error could be more helpful than accidentally using the wrong one – it’d certainly help pinpoint the problem much faster. Like you say, the wrong version might not be detected at all, and that could be dangerous.

    If filesystem timestamps as secure, perhaps the best option would then be to check all library paths, display a warning if multiple libraries share the same name, and then use the library which has the oldest “creation date” timestamp.

    This conservative approach ensures that when the conflict occurs, what worked before continues to work, and it’s only the new stuff causing the conflict that breaks. This is a good idea, because the conflicting library could be important, and any action that revolves around preventing software to use it temporarily or permanently could break the system.

    Similar mechanisms could be used with the resource mechanism that we’re discussing, by checking out the FS timestamp of the process that implements them.

    Of course, that’s only a short-term fix: conflicts should then be taken care of quickly, since if library files are moved around or updated later for some reason, the filesystem timestamps could lose their meaning.

    It could, but it only replaces the duplicate identifier problem with a race problem. As an example, on an OS where the startup of system services which are not immediately needed is delayed, a malicious program could claim to implement a system resource early on and become the effective master of that resource, with all the negative consequences that brings.

    Yes I see. in this case I guess you could either explicitly grant these apps permission to host services, or consider using signed executables.

    Another idea is to consider the problem from the opposite angle, that of the consumer. Whenever a conflict is detected, a GUI could pop up and ask which instance should be used. This choice could be saved persistently until the services changed again.

    This problem deserves more thought.

    A problem which I see with these approaches is that they implicitly assume either

    • That there is a power relationship between services, i.e. one has authority over the other. This can work for OS services versus user-implemented ones, but doesn’t work as soon as we have to discriminate two user-implemented services.
    • That the user has enough expertise to know which implementation of the service is right. Apart from the aforementioned timestamp heuristic, I’m not sure which data he could base such an decision on.
    • Perhaps it should be a good idea to monitor which services a process implements through the permission infrastructure, and notify the user of a possible conflict as soon as a conflicting process is installed. This brings in desirable information locality: the emergence of a warning is directly correlated to the installation of the new software, and thus what happens becomes more obvious to the user.

      I was thinking more along the lines of running the test only when the database is modified and a conflict is created. This would be relatively rare. At that point, detect if the old object is still valid before entering the new one.

      Would the permission-based, install/update-time scenario that I described above help there?

      This is OT, but the ideal file systems in my head have dependency satisfaction similar to foreign key integrity within relational databases (I’ve always wanted to unify databases and filesystems). A foreign key record cannot be deleted or updated independently from the records which reference it. Foreign keys can be declared “on update cascade” or “on delete cascade” such that changing the record here will cause all referencing records to get changed as well. If you had a file system with dependency enforcement, then you could be confident that the object database would be updated in real time along with application updates.

      Since I’m not so familiar with the world of databases, this may be a stupid question, but would it be like a file record having awareness of all symlinks that point towards it, so that these can be updated when the file is moved, and deleted when the file itself is deleted?

  7. Alfman April 20, 2013 / 11:28 am

    Perhaps it should be a good idea to monitor which services a process implements through the permission infrastructure, and notify the user of a possible conflict as soon as a conflicting process is installed. This brings in desirable information locality: the emergence of a warning is directly correlated to the installation of the new software, and thus what happens becomes more obvious to the user.

    Would the permission-based, install/update-time scenario that I described above help there?

    I agree. It would make the process more predictable. However in the long term, assuming you’ll get a software repo in place, presumably *it* should always know which dependency is the right one because the software repo should have all the correct (and current) dependencies listed it’s metadata. This meta-data could be installed alongside the software to tell the OS exactly what dependencies are expected at runtime.

    Since I’m not so familiar with the world of databases, this may be a stupid question, but would it be like a file record having awareness of all symlinks that point towards it, so that these can be updated when the file is moved and deleted along with the file?

    Yes, that would be an acceptable comparison. In an ACID relational database, the data is never allowed to violate any of the constraints you’ve defined for it.

    If I were building the foundations for a new software platform, I’d probably start with a DB and add in FS-like functionality rather than the other way around. I look at all the hundreds of configuration files under /etc and think of how wasteful it is that they need their own syntaxes & parsers and how difficult it is to update configurations in an automatic way (not to mention relational integrity benefits mentioned above, the DB/FS could enforce valid configuration file paths, etc). To many, the horror that is the windows registry will immediately pop into mind, but it’s important to realise that the windows registry is not, and was never intended to be, a usable database. Real databases are pleasant to use.

    I’ve actually given this quite a bit of thought over the years, the way I see it is that unix concept of stdin/stdout could be extended to include datatable equivalents. Instead (or in addition to) outputting text streams, an application might output datarows which could be manipulated very much like the text streams are under linix (pipes, file redirection, grep, etc). So, for example, ‘ifconfig’, ‘ls’, or ‘ps’ output would be a real datatable that could be saved and / or manipulated directly without the need for any parsing. The command line tools could made familiar enough to make regular linux users at home:

    ps > ps.out # ps.out would be a saved as a datatable
    cat ps.out # would output the datatable (which the shell could display correctly)
    vi ps.out # editor for datatables
    select * from ps.out where user=’xyz’ # regular sql expressions could be used

    All of the powerful database operations would be at our fingertips directly from the system command shell. I could go on for a while on this topic but I won’t; I sort of feel guilty for going off topic all the time given your limited time.

  8. Hadrien April 20, 2013 / 2:24 pm

    I agree. It would make the process more predictable. However in the long term, assuming you’ll get a software repo in place, presumably *it* should always know which dependency is the right one because the software repo should have all the correct (and current) dependencies listed it’s metadata. This meta-data could be installed alongside the software to tell the OS exactly what dependencies are expected at runtime.

    I don’t like the idea of having core system functionality rely on the existence of a centralized software repository. Having such a repo around is useful, but there are always scenarios where one will want to go outside of the repository system. One can, as an example, think of when software can’t be included in the repository for licensing reasons, or when some software in the repo becomes so outdated that installing a new version from another source is needed. In these situations, assumptions that software will always be distributed in a centralized fashion can become a source of major suffering.

    Perhaps a better alternative would be to use a combination of both resource names and UUIDs. In this scenarion, when a developer creates a new resource, he gives it a name, but also generates an UUID at the same time. When other software based on that resource is developed, it can use a constant string based on the resource name to refer to the UUID (think ‘const string [RESOURCE]_UUID = “[UUID]”;’). This way, conflicts only occur when two software provide not just two distinct resources bearing the same name, but two versions of the same resource. Yet for everyday reference, a clean resource name is also available.

    To account in a decentralized fashion for the full scenario which you propose, in which software A may be looking for software B’s implementation of resource C, we could imagine also giving software an UUID, and adding such the implementor’s UUID to the resource metadata. This way, software A could either perform a lookup for a resource having the UUID of C if it doesn’t care about the implementation, or perform a lookup for a resource having the UUID of C and the implementor UUID of B. One could also imagine making the dependency optional: if B’s implementation of C is available, software A will use it, but if it isn’t, A can satisfy itself with any other implementation of C.

    Regarding relational databases and databases in general, do you know of a good theoretical course on the matter which I could use to get up to speed with the subject, preferably without immediately tying myself to a specific technology such as SQL? That would make some good bedtime reading, and it certainly wouldn’t hurt to add this to my computing culture.

    (And, for now, time to build evil world domination plans and eat some freshly baked chocolate-chestnut cake)

    P.S : Got around making a toy library with my System.Console wannabe, and what happens is a textbook example of compilation error caused by a namespace collision:

    “Main.cs(132,17): error CS0433: The imported type `System.Console’ is defined multiple times
    /usr/lib/mono/4.0/mscorlib.dll (Location of the symbol related to previous error)
    /home/leonidas/Test/Override.dll (Location of the symbol related to previous error)”

    According to MSDN, there is a way to specify which class you want in the compiler command line, though.

  9. Alfman April 20, 2013 / 9:36 pm

    Hadrien,

    I don’t like the idea of having core system functionality rely on the existence of a centralized software repository. Having such a repo around is useful, but there are always scenarios where one will want to go outside of the repository system…

    Haha, I actually deleted a paragraph about this for the sake of brevity, but since you bring it up, I agree with this too. If I were to re-envision the software repos, it would be much less centralised, possibly even built it into a P2P system like bittorrent (yes I remember your previous remark about how they’ve become closed, but it is still a good baseline for discussion) . If I install software from vendor A, I could run a search for their torrent from my software manager. Alternatively I could click on torrent links on the developer’s website. When I install the software, the entire installation would be fully automatic just like a traditional repo. Not only would the software manager perform the initial install, it could schedule and automate upgrades as well. And cryptographic signatures would ensure that the upgrades could only be published by the original developer.

    Up to this point, there’s no centralised control whatsoever, every developer would be on equal footing without needing anyone else’s permission to publish software. Also it gives all developers an insanely scalable & reliable P2P distribution platform. Alas this opens up the door for malware, to which I can think of a couple couple compromised solutions:

    1. Provide one or more managed/centralised database of audited torrents. Like other traditional repos today, software developers would have to plead their case to be included. In addition to having a database of vetted software, this could offer a nice system for searching / browsing the software.

    1.a. You and others could run automated low security database that automatically verifies some basic information about the publisher, like website and email address “poor man’s authentication”, thereby proving that software is being published by them and not an impostor.

    2. Torrent links on an HTTPS website should give some confidence that the torrent is legit, as would SHA sums. https://mysite/MyApp.tosp could contain everything needed to securely install and upgrade an app assuming you trusted “mysite”.

    3. Sign the software using traditional CA certs, would give confidence that the publisher’s identity is what they say it is. I’m not a fan of this model though because it effectively puts CA’s directly in control of everyone’s software.

    And of course the risks should be somewhat mitigated by running untrusted apps in a sandbox such that fraudulent malware will have much less impact than it would in conventional unsandboxed OSes.

    Perhaps a better alternative would be to use a combination of both resource names and UUIDs. In this scenarion, when a developer creates a new resource, he gives it a name, but also generates an UUID at the same time…

    What you go on to describe is virtually identical to Active-X! It was rather difficult to work with at times, problems were impossibly difficult to track down. Take a look at the CLSID branch in regedit. I don’t think it’s much better than a friendlier scheme like this: “vendor URI” + “namespace” + “classname”. This way full collisions would be the vendor’s own fault rather than an unfortunate software configuration on the end user’s machine, the OS would always use the latest version. In the case of partial collisions (namespace+classname only) , the system could output an easy to understand message showing the conflicting vendor URIs, which would be easy to understand. Also, the vendor URI could have authentication methods to prevent fraud.

    Regarding relational databases and databases in general, do you know of a good theoretical course on the matter which I could use to get up to speed with the subject, preferably without immediately tying myself to a specific technology such as SQL?

    Not off the top of my head. I’d happily send you my books on the subject if you were closer, haha. I honestly think it’ll be difficult to find any relational database primers that don’t cover the topic from an SQL perspective. To my knowledge, all relational databases have an SQL interface, and conversely non-SQL databases tend to be non-relational. One piece of advice, having a good SQL IDE makes all the difference in the world, using a database from the commandline is painful. Sure you can follow along with the books, but the novelty wares off quickly and you’ll find yourself cursing that the command line tools don’t include a table editor (like a spreadsheet). All the best SQL IDEs I’ve seen are commercial. I’d use a web interface like phpMySql over going strait commandline. Since your working with MS dev tools, you could try the express/student version of MS SQL, it has a good IDE.

    Got around making a toy library with my System.Console wannabe, and what happens is a textbook example of compilation error caused by a namespace collision:

    Can you produce a runtime error by trying to install and use two conflicting assemblies in the GAC? (Conflicting with each other, and not necessarily system assemblies).

  10. Hadrien April 21, 2013 / 8:29 am

    What you go on to describe is virtually identical to Active-X! It was rather difficult to work with at times, problems were impossibly difficult to track down. Take a look at the CLSID branch in regedit. I don’t think it’s much better than a friendlier scheme like this: “vendor URI” + “namespace” + “classname”. This way full collisions would be the vendor’s own fault rather than an unfortunate software configuration on the end user’s machine, the OS would always use the latest version. In the case of partial collisions (namespace+classname only) , the system could output an easy to understand message showing the conflicting vendor URIs, which would be easy to understand. Also, the vendor URI could have authentication methods to prevent fraud.

    Not sure I follow. Would you suggest using vendor-specific namespaces in the end, even if it would prevent the existence of multiple implementations of the same resource?

    Not off the top of my head. I’d happily send you my books on the subject if you were closer, haha. I honestly think it’ll be difficult to find any relational database primers that don’t cover the topic from an SQL perspective. To my knowledge, all relational databases have an SQL interface, and conversely non-SQL databases tend to be non-relational.

    Well, know of a relational DB primer that covers SQL then? :)

    One piece of advice, having a good SQL IDE makes all the difference in the world, using a database from the commandline is painful. Sure you can follow along with the books, but the novelty wares off quickly and you’ll find yourself cursing that the command line tools don’t include a table editor (like a spreadsheet). All the best SQL IDEs I’ve seen are commercial. I’d use a web interface like phpMySql over going strait commandline. Since your working with MS dev tools, you could try the express/student version of MS SQL, it has a good IDE.

    Actually, even if I’m coding in C#, I’m not using MS dev tools to this end, but their Mono open-source equivalents :) The reason I’ve decided to learn C# at all is, I wanted to know about one of the two main managed languages of this day and age. The other being, of course, Java.

    Looking around at code samples, Java sounded like a language designed by crazy programming teachers who wanted to enforce “good” programming practices no matter what, to the point of making tasks which are simple in other languages ridiculously cumbersome. C#, on its side, sounded a lot cleaner, if a bit inconsistent in places*. Besides, I have experienced almost nothing but pain dealing with Java software and the runtimes they run on, so I’m not sure if I’d want to perpetuate such abominations, whereas the C#/.Net software I use has generally been undistinguishable from its native cousins.

    * Me learning C#: “WTF? Why are structs passed by values and objects passed by reference? And why are strings overriding this default behaviour to copy-on-write pass-by-value? Oh, God, what have I gotten myself into…”

    Can you produce a runtime error by trying to install and use two conflicting assemblies in the GAC? (Conflicting with each other, and not necessarily system assemblies).

    Adding the assemblies to the GAC works without a warning as long as they bear different names, but then they are not silently linked to the executables. As soon as I reference them through the command line, the compiler issues a warning about a class being defined multiple times.

    “> mono-csc /reference:/usr/lib/mono/Conflict1/Conflict1.dll,/usr/lib/mono/Conflict2/Conflict2.dll Main.cs
    Main.cs(27,9): error CS0433: The imported type `Conflict’ is defined multiple times
    /usr/lib/mono/gac/Conflict1/1.0.0.0__0686ae839ef13a8d/Conflict1.dll (Location of the symbol related to previous error)
    /usr/lib/mono/gac/Conflict2/1.0.0.0__0686ae839ef13a8d/Conflict2.dll (Location of the symbol related to previous error)
    Main.cs(24,16): warning CS0659: `Account’ overrides Object.Equals(object) but does not override Object.GetHashCode()
    Compilation failed: 1 error(s), 1 warnings”

  11. Alfman April 22, 2013 / 8:43 am

    Hadrien,

    Not sure I follow. Would you suggest using vendor-specific namespaces in the end, even if it would prevent the existence of multiple implementations of the same resource?

    I wasn’t clear, but here’s how it might work:

    1. Two or more modules have the same class name, and same developer URI. This implies that the conflict was caused by the developer (ignoring fraud). The most likely cause is that a new version was published to replace the old version, and so the OS should use the newest version.

    2. Two or more modules have the same class name, but different developer URIs. It’s impossible to know whether the developers were attempting to implement the same interface or just used the same class name (service name) by coincidence. I’m truly torn on how to solve this.

    A. Interface UUIDs like you mentioned.
    B. Generate an exception & let the user choose which developer URI is appropriate.
    C. Select one arbitrarily (which comes first in path, etc)
    D. Let the application explicitly default to a specific compile time URI, but allow it to use other URIs if the URI does not exist at run time. This way third parties could implement the same service, but you’d have to disable the default module first for that to work.

    A. is the Active-X way.
    VB handled these UUIDs automatically. This way seems ok until you realise that a project can be branched and a single interface UUID may or may not be appropriate as the interface is modified over time. VB handled this as well, saving the entire history of UUIDs corresponding to all versions of the interface a module is able to supersede. However a major source of grief on a legacy project I was working on was when multiple developers were trying to work on the same projects and inadvertently branched from the other developer instances, causing foreign UUIDs, which would cause inexplicable problems when running binaries linked on other machines. It didn’t help that active-x components usually just failed to load all together with no indication of any errors outside of the event log.

    B. seems to be the .net way
    It’s got simplicity in it’s favor, and I think there’s alot you could do in the UI to help the user resolve it quickly.

    C. is the Java & linux loader way
    They’re probably not even aware of a conflict because they stop looking after the first match. This is probably the least helpful for troubleshooting problems.

    D. This was one thought as to compromising between A and B.

    Well, know of a relational DB primer that covers SQL then? :)

    I’m sure all my college texts are out of print, I’ll email you a few real examples though.

    Actually, even if I’m coding in C#, I’m not using MS dev tools to this end, but their Mono open-source equivalents :)

    Oh ok, I’ve never used mono. I like .net alot, but I’m working mostly on linux these days and I’m really not sure if anyone runs production systems on mono? That’s probably not important to you.

  12. Hadrien April 23, 2013 / 8:56 am

    Regarding conflicting resources, I think the rules on what can generate a conflict and what the symptoms of said conflict are must be very clear and reproducible, otherwise this will result in random, hard-to-debug issues. This eliminates solution C.

    As for the other ones…

    A. Interface UUIDs like you mentioned.

    (…)

    A. is the Active-X way.
    VB handled these UUIDs automatically. This way seems ok until you realise that a project can be branched and a single interface UUID may or may not be appropriate as the interface is modified over time. VB handled this as well, saving the entire history of UUIDs corresponding to all versions of the interface a module is able to supersede. However a major source of grief on a legacy project I was working on was when multiple developers were trying to work on the same projects and inadvertently branched from the other developer instances, causing foreign UUIDs, which would cause inexplicable problems when running binaries linked on other machines. It didn’t help that active-x components usually just failed to load all together with no indication of any errors outside of the event log.

    If I get it, the issue is that sometimes, two different projects can implement their own version of an interface, which is initially common but gradually diverge from each other.

    A solution, which I have already studied for RPC, would be to make software specify not only the interface name, but parts of the interface itself (i.e. the signature of the methods which the software will need in order to work). This should disambiguate things more cleanly than an UUID, and avert the obfuscated ugliness of it altogether. Besides, it’s useful for forward and backward resource compatibility checks anyway.

    In that case, we could imagine going through the following variation of scenario D:

    1. Client software looks for the vendor and resource identifiers which it’s been deigned to use
    2. If a match is found, it is checked for interface compatibility, then accepted
    3. If no match is found, client software looks for the same resource name, but with different vendors
    4. If a match is found, it is checked for interface compatibility, then accepted
    5. If several matches are found, and all have a compatible interface, fallback to solution B
    6. If no match is found, client software gives up and crashes

    I’m not sure what should be done if an interface is found but turns out to be incompatible at step 2. In this case, we are facing a willful breach of compatibility by the original resource vendor. Should the search go on then, or should the software crash right away? In that case, letting the search continue has the advantage of allowing other vendors to propose alternate implementations of a deprecated library.

    Of course, having software specify all the methods which it relies on by hand is tedious. This should probably be automated. A way would be to specify the full resource interface at the time where the software was built. This effectively asks more than the client software needs, and thus artificially worsens compatibility, but it addresses some important use cases like code generation or library-based resource access. Besides, if the resource system is sufficiently fine-grained, not much is actually lost.

    B. Generate an exception & let the user choose which developer URI is appropriate.

    (…)

    B. seems to be the .net way
    It’s got simplicity in it’s favor, and I think there’s alot you could do in the UI to help the user resolve it quickly.

    When implementing this solution, one must be careful that the very action of generating an exception and asking the user to choose may reference the conflicting resource. This would be the case, as an example, if software reimplements resources used by the GUI subsystem.

    This is why a variant of scenario D sounds more appropriate to me, when possible: we can be sure that the system libraries will always be there, so the initial vendor check will never fail, and thus in the common case where resources are available and coming from the expected vendor, nothing will break.

    Oh ok, I’ve never used mono. I like .net alot, but I’m working mostly on linux these days and I’m really not sure if anyone runs production systems on mono? That’s probably not important to you.

    Actually, I’m not even sure I know what a “production system” is. For me, that brings image of giant automated factories building cars and filling up jars of jam. Which I’m almost sure is something you did not have in mind yourself. Hum.

    I do know of a well-known commercial product which uses Mono though, and that’s the Unity game engine. Basically, the engine core, which is written in C++, exposes a .Net interface which is used for scripting in C#, UnityScript (a variant of ECMAScript) or Boo (a Python-inspired language). That interface is built using Mono development tools and libraries, so as to allow for cross-platform game portability. It is also used by some of the tools which are provided to game developers.

  13. Alfman April 23, 2013 / 3:57 pm

    Hadrien,

    A solution, which I have already studied for RPC, would be to make software specify not only the interface name, but parts of the interface itself (i.e. the signature of the methods which the software will need in order to work).

    This is how Go does it, right?

    In that case, we could imagine going through the following variation of scenario D:

    Sounds like a winner.

    Of course, having software specify all the methods which it relies on by hand is tedious.

    Yes, it’s such a shame C++ doesn’t output reflection data, since it’s a notoriously difficult language to parse correctly. Doing it by hand is bad since there will be human mistakes that defeat the purpose of the signature checks.

    Maybe something like this could help.
    http://kifri.fri.uniza.sk/~chochlik/mirror-lib/html/

    Actually, I’m not even sure I know what a “production system” is. For me, that brings image of giant automated factories building cars and filling up jars of jam.

    It’s a widespread term here, it’s a system that is in active use for directly running a business. if a production server goes down, so does the business.

  14. Hadrien April 24, 2013 / 8:16 am
    A solution, which I have already studied for RPC, would be to make software specify not only the interface name, but parts of the interface itself (i.e. the signature of the methods which the software will need in order to work).

    This is how Go does it, right?

    Kind of. In Go, you can specify in code the part of the interface to an object which matters to you. This is used to implement a form of genericity, in which objects are solely considered as an implementation of an interface.

    As an example, if I declare the following Cat interface…

    “type Cat interface {
    ….Meow()
    ….Purr()
    }”

    …then a function taking a parameter of type Cat will accept as input any object which implements the Meow() and Purr() methods, without caring about whatever other methods or members which the object might have.

    My design for RPC, which I came up with before I knew about Go, worked on a single-method scale. Basically, a server process published the signature of the methods it implemented in the kernel database at initialization time. At binding time, a client process would publish the signature of the method which it expected to see on server side. If both signatures were incompatible, the kernel would return an error, otherwise it would create management structures for future RPC calls and return a unique connexion identifier to the client.

    In this process, the kernel would act as a glue between both, checking that both signatures are compatible with each other at binding time, and possibly performing some compatibility tweaks on RPC calls (such as adding default parameters to client method calls). It would also be the entity responsible of passing parameters from client code to server code and calling server code, managing threading or async queues as needed.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s