I. Enforcement
The concept of res communis when applied internationally (as opposed to internally, as when a state holds property within its jurisdiction and calls them community property) is incredibly difficult to enforce. If a group of citizens claiming no nationality or a new nationality tried to set up a new country in an area declared by treaty or convention as commonly held by the people of Earth (like the moon or international waters) then who's going to stop them? The UN doesn't really have the mandate for it, but neither does it have the credibility or police for it. Individual countries like the US might step up and do it, but that's prone to biased enforcement. More likely international courts would be brought in to deal with the issue, but how can international courts apply to people that don't recognize the court and aren't nationals of a signatory country? Naturally some courts (see: ICC) try to claim universal jurisdiction, but that's a problematic concept with very limited support.
Ultimately, if in the future somebody tried to set up a country in an area considered res communis then there wouldn't be a lot that countries would want to do to stop them.
It's the nature of owned property that the owner will protect the property. If international waters or the moon are owned in common by all of Earth, then all of Earth needs a property manager and rent-a-cops to protect the property. Otherwise it won't be protected. Somebody has to have the responsibility (and ideally the self-interest) to defend a property from being scavenged, polluted or squatted. By eliminating the privately-held and state-held options, only some IGO could be tasked with protecting these res communis areas, yet none satisfactorily exist. The closest thing is the International Seabed Authority, but it only relates to mineral-resource exploration and exploitation at the bottom of international waters; the moon, space, and of course the actual surface water of internationalw aters aren't really patrolled or policed.
II. Obstruction
Aside from the lack of any interested and empowered authorities to develop or protect the res communis areas, the practice also deters private or commercial progress toward them. It's entirely plausible that in fifty or a hundred years a company or group of people might want to establish a floating country, either moored or unmoored.
All it would really take is a group of super-sized cruise ships (connected or unconnected) and you could have a community of ten-thousand living more or less indefinitely on the water. They could get supplies by sending out boats or airplanes to the nearest land-based country and stocking up, and using the sea to make up the difference with fish or water. Energy could be supplied with nuclear reactors (like a battleship or a submarine) and you'd have a country. It would of course need financial support from tourism, or banking interests, or businesses seeking to avoid national regulation and intrusion or just a wacko trillionaire who likes the ocean; but just how it would operate is irrelevant. The point is that there's no real way to deal with people wanted to try this out, whether they were moored to the bottom of the ocean or sailing around the world.
With regard to space, it's obvious that eventually the technology of the space stations could be mimicked by companies or individuals either in orbit or on the Moon. The current treaties on the subject serve to inhibit development, especially on the Moon. Since nobody can own property on the moon, assuming it could somehow be profitable (financially or socially), there's no incentive to try - especially with the developed technology so far away.
The technology, though it's basically just science-fiction right now, will continue to stay far away and develop slowly if we don't allow for private initiative in this area. If there's a way to do it at an expected profit, it will eventually be found and done; all we have to do is find a way to allow for private property on the moon and other res communis areas.
If we want to encourage technological and scientific progress, especially towards the moon and space generally, we need to come up with an approach that is far less socialistic, far less hostile, and far less state-centered. Traditional and ancient international law may have been centered almost entirely on state and the domain of princes and governments, but today we must realize that individuals and enterprises can and do operate independently from states. It's only natural that laws reflect private endeavor.
III. Socialism and Power Politics
The idea of res communis sounds socialistic, and the rhetorical idealism behind it surely is. But the fact is that the law of the moon comes from two motivations: weak states and balancing states. The law of the moon is ideological realism dressed up by a little idealism.
The weak states, with little or no ability to send expeditions into space at all, let alone before the US, USSR and other, wanted to stop the big boys from making claims and keeping the weaker states out of the game. If you can't win, change the rules to cancel the game.
The balancing states didn't want to get into a fight over new territory on the Moon, a wonderful source of potential conflict, unrest and even violence or war. Rather than spending humongous cash reserves to colonize the Moon first or fight a war to stop the other from doing the same, they both agreed to keep it off-limits (see also: ABM).
Neither old-style socialism nor foreign policy realism offers the best alternative here (or most anywhere else). They lock out scientific and economic progress, not to mention a natural progression of humanity.
IV. Private Property of the Moon
My solution, naturally, is to find methods of allowing private property or even new countries to develop in these areas, just as happened during the Age of Exploration in the 1500s and 1600s. To figure out the law, some principles need to be drawn out. I'm limiting this part ot the moon, since open space, orbit and the international seas potentially involve moving territories (although I do think we ought to somehow accomodate moving territories somehow).
First, can sea, space or moon property be owned freehold or only by license? My undeniable preference is that we be talking about freehold, rather than licensing. That would be the only way to make a real country in these areas, since a country licensed by another country is more like a territory or protectorate. Governments already existing will want to make them licensed but otherwise controlled by said governments.
Second, how much property can be claimed at once and what has to be done to make it a valid claim? Obviously you can't just sit here on Earth, point to a map of the moon and make a credible claim. By the same token, you shouldn't be able to establish a claim to the entire moon based on having one outpost on it. For guidance, I'm turning to the US Homestead Acts, even though the moon isn't US property. The last version of the Homestead Act, from 1912 when it was mostly irrelevant, was that an area of 640 acres could be claimed either by purchasing it for $1.25 an acre after six months residence, or granted it after three years residence. Since it's hard to tell just what people would be doing on the moon (social versus financial reasons) it's hard to determine a size for a moon homesteading proposal. But I think it's a good starting point, in theory.
Third, do enterprises and individuals have the same options with regard to claiming land? It seems to me that they ought to be given equal weight, though of course any business enterprise could simply apply for adjoining homesteads, one for each employee, and get a larger area. But this trend would be sharply limited, since it would require actually living on the land and thus require a separate shelter for every homestead. The obstacle could be quartered by building a small community of shelters at the connecting point of four homesteads (assuming a grid formation) sort of like the borders of UT, Co, NM and AZ. All it would need is that at least one 'home' shelter be one each of the four claimed homestead areas.
Fourth, what power do governments have here? Do they grant the homesteads or does some international agencyhandle it? What powers of taxation are there? Who are the regulating bodies? These questions are hard to answer, and as a libertarian I'm not particularly thrilled at the idea that there has to be some form of government there at all, though in all practicality I realize it's nearly inevitable.
This subject is in a lot of ways fairly distant, but I think the issues involved ought to be handled sooner rather than later, especially if a change in law could encourage a change in technology.
Update: As a part of searching the Internet on this subject, I found an article from The Space Review on the same subject with the same position.


