Shared objects, a new approach to sharing data between processes

Originally published at: https://events.4d.com/summit2020/session/shared-objects-a-new-approach-to-sharing-data-between-processes/

Shared objects allow better data sharing between the different processes of 4D databases while ensuring simple management of concurrent access to these objects via mutexes.

1 Like

Omar, thanks for this session. Very informative. I do have one question. In the demonstration on Signal, you have a process that creates three other processes and then waits. Each of the three processes performs some work. At the end, each process checks to see if it is the last one to finish in order to call signal.trigger(). Each process knows if they are the last one by checking to see if there are 3 elements in the result collection. But I’m wondering if this is actually safe? What if two processes happened to read the length of the collection at exactly the same time. Wouldn’t both think that there were only 2 elements and then neither would call signal.wait()? And the original process would wait forever.

Please help me understand if I’m missing something. Thanks!

It really looks like that - and I myself repeatly fall in this trap and need to remind me how it works…

Signal is a shared object.
A shared object (or collection) is very special. It is a kind of protected thing. Whenever someone tries to read, there is a kind of microlock making sure only one can read. That makes this object a little bit slower to read compared to none shared objects.
And if somebody tries to write, it is denied, except USE is used to unlock it.

Because of the microlock, you are able to use a shared object with several processes. Else it would be not useable at all.
Imagine you store a long string in a shared object and one process starts to read while another one deletes it.
This shows that there must be a protection both for reading and writing.

In my experience, it is always best not to over-complicate the logic around signal. Most of the time, you just need it for a preemptive process to wait on its cooperative buddy, otherwise you have only yourself to blame for a deadlock situation.

As for shared objects, I always need to remind myself that I need to project the final structure in advance, before I start building a “multiple” type object.They are like lego except you do not work on modules to create a bigger object, you must keep adding new blocks directly to the main product.

I want to add something. More a warning…
Keep in mind, that you have to use Use also when reading if you have to make sure, that the reading values are in connection to each other. In other words:

$len:=sharedObj.coll.length
if($len>0)
$value:=sharedObj.coll[0] //may be invalid, because the collection may changed in meantime!
end if

To be safe: you should also use Use for reading:

Use(sharedObj.coll)
$len:=sharedObj.coll.length
if($len>0)
$value:=sharedObj.coll[0] //now it's safe to read
end if
End Use
1 Like

That makes more sense now. Thanks for everyone’s reply.

This visual explanation of shared groups (multiple objects) vs. single objects was the best! So must easier to understand with animation! Now I understand better the rules and logic behind negative and positive locker IDs.

The practical impact of the Storage object not sharing its locker ID with its dependents like a regular shared group was also made very clear, I feel really learned a lot.

I appreciate the need to watch out for logic that will create a dead lock only if the timing was unfortunate. Obvious dead locks are easily find and fix, as you only need to run the code once to see the error. But issues that are subject to timing are more elusive and easy to miss (as pointed out in the discussion above…).

The demo was a nice example of putting together all the features. Most developers know that preemptive mode helps speed up multiple concurrent tasks, but this example answers the question, “how can I use preemptive mode to improve the speed of a single task that is taking a long time to complete”. (Of course, we need to implement the exist code in the way that Christian suggested above).

The tip to create a “blank” shared object first and attach it to its eventual parent, before adding some children, is the most important lesson I unfortunate had to learn the hard way. It is counter-intuitive (you would normally construct an object in hierarchical order) but you are absolutely right, it must be done differently. The motto “push first, construct later” puts it so elegantly.

Last, but not least, we look forward to day when conversion from JSON to shared object becomes possible in one command, meanwhile the demo has a wrapper solution :slight_smile:

P.S.

Maybe not a big deal, depending on your network situation, but the demo is quite large (111 MB zip, 574 MB 4dbase, contains 1 million sample records), I started downloading, and finished watching the video before the transfer was complete. I understand, it was not the original plan to share the USB demo online.

For those interesting in running the demo, you might want to start downloading before you play the video.

Thank you Omar,
Learned a lot!