I develop C++ computational library (adaptive mesh). Now I need to make it parallel (distributed-memory systems, MPI preferred).

My library is based on custom pointers, which sometimes point to "virtual" objects (objects which are results of computation, not real objects in memory).

So, I decided that all that I need is to upgrade these pointers for they to have ability to point to objects in the memory of another machine. The problem is that the object on another machine (dereferenced from such a pointer) must behave like real object!

OK, let us suppose that I added machine address to the pointer class.

1. All pointers will have this additional address. Even those which referring to the local objects. This is overhead. May be I should use 2 pointer classes: "local" and "remote"?
2. If program trying to dereference pointer, what should pointer do? Copy object from remote machine (I can do it using MPI-2 messaging system) and give a reference to it? But this copied object will not work, because it holds some regular pointers to its members. I need to make a copy of all its members, and members of their members,... the whole subtree need to be copied.
3. How the object should be modified? The local copy should be returned to the source machine and overwrite the original object?

So, may be I need another idea? I can, for example, put all interface of my objects (which can be pointed by remote pointers) into a single function with fixed parameters and fixed return value. Then instead of dereferencing a pointer I can only call this function from the pointer. The pointer will then transfer this call (function arguments) to the target machine, call function on this machine, and return result back to the caller.

Now the problem is that the caller must pack all the data into the fixed arguments of such function, even if object is local. May be it is possible in C++ to pass various parameters with various types into a function, which will transfer them directly to the target interface function if object is local, or pack them and transfer to another machine, if object is remote?

Any advice is appreciated.

Old way:

MyPointer p = this->GetNeighbor(nBottom);
p -> DoSomething(10,20,30);

New way:

MyPointer p = this->GetNeighbor(nBottom);
p.Call(DoSomething,10,20,30));

Inside the MyPointer class:

class MyPointer
{
private:
  MyClass *p;
  int machineAdr;

public:
inline bool Call(func, arguments,...) //return true if it is known that func modifies its arguments
{
  if(machineAdr == ThisMachineAdr())
  {
    p->func(arguments,...);
    return FuncModifiesData( func ); 
  } else {
    data = PackData( func, arguments,... );
    bool dataModified = SendMessageToRemoteMachine( machineAddr, data );
    if( dataModified ) UnPackdata( data, arguments, ... );
  }
};

1. Do I really need the RPC in my program?
2. If I need it, what will be the best way to implement it over pointers using MPI in my C++ program (to minimize overhead and code change)?

Instead of having pointers that know about "near" or "far" objects (on the same or another machine, respectively), have proxies that have the same interface as the objects you want to share between machines. So you can use normal pointers that point to an real object on the same machine or to a proxy that stands for an object on another machine. The proxy is the only one to know the handle for the corresponding object on the "other side", and if you call a method on the proxy it sends the correct message an waits for a return value, if necessary.
I think it would be a serious overhead to copy the complete object, especially if you only need one of its variables.
On the other hand, if in some cases you really need the whole data of the distant object, yout could do that on demand, i.e. only if the proxy really needs the data it requests the distant object to send the necessary data.
In addition, as you have multiple processes you would need some type of synchronization between the objects and their copies/proxies anyways.