4D Server Caching and WAN

Since upgrading from v16 to v18.1, I have notice that most, but not every time, I log in from a computer located remotely on a WAN, 4D Remote reloads a new cache folder. This rapidly filling up the cache with a significant amount of files.
Note that

  1. the application on the Server has not been changed. (Other than some files in the resource folder which are often changed.)
  2. The Server has not been restarted.
  3. I am not running the legacy network.
  4. This does not happen when I open the application on my LAN

I understand that is the expected behaviour when I change the application, but do not understand why this should happen on the WAN.

Could someone please explain this. Is there anything I can do other than emptying out the cache folder every once and awhile.

Thanks

Mitch

Please check the names of your cache folders. I assume there are several of them, right?
They are supposed to be IP.Port.Hash, like
myapplication_192_168_0_100_19813_015

I expect that the name of the application, the IP address and the port does not change.
Is that correct?

4D creates a new hash if either the client application is moved to another folder/directory or if the previous one is already in use (running). In that case there is a text file inside the cache folder signalling that.

Does that fit to your folder?

Hi Thomas,

On my LAN I get the expected folders: i.e. myapplication_192_168_1_1_19813_056

On the WAN, I get myapplication_server.mydomain.com_123
In other words, no port number.

I have only one copy of 4D client on the machine. It resides in the Applications folder of my mac. It is never moved.

Thanks

Mitch

If you use a domain name instead of an IP address it uses the domain name in the folder name, correct.

But , I just test, it still includes the port AND it uses _ as separator and not “.” as in your example. The folder name is not supposed to use dots. So “server.mydomain.com” should be used as “xx_server_mydomain_com_19813_123”

I have no explanation, please contact your local tech support for further investigations.

Dear,
My customers noticed the same behavior of frequent reload of cache over WAN.
In my case I could trigger the “cache reload” by changing my network connection from Wifi to wired.
Changing network interface, and between the changes, make a new connection to my very same 4D server not restarted, results in new cache folders at each login.

TAOW says this is standard behavior. If remote IP changes -> new cache.

I have filed a feature request to improve this behavior to look deeper for actual changes and only download individual resource(s) in case of a change on server.

Hopefully this is the link:

Best Regards

Magnus

Thanks Magnus.
I ended up writing code to clear older copied from the cache on startup.

Mitch

Hello

we are experiencing the same issue using 18.2 build 18.252363 remote on Mac.

Everytime client connect, it is downloaded the full local resources.

We note on system cache folder the following folders, where XXX is the name of our custom application:
XXX_192_168_70_11_19833_231
XXX_192_168_70_11_19833_795
XXX_192_168_70_11_19833_954
XXX_192_168_70_11_19833_481
XXX_192_168_70_11_19833_844
XXX_192_168_70_11_19833_102
…

Everytime client connect, a new local folder is created, also after 4D server fresh restart.

We have noted this only in latest weeks.

If there is some suggestion for the solution, it will be appreciated.

The 3-digit suffix is supposed to be hashed from the application path.

Is it possible that your client app is being translocated from its origin?

Maybe I am dumb but what do you mean by translocated. Do you mean moved to quarantine. Otherwise , if I create a client, move it to folder A on hard drive and then move to folder B this would happen.

Regards

Chuck

“App translocation” also known as "“Gatekeeper Path Randomization”…there are a plenty of resources available online, but basically, the system copies an “untrusted” app to a sandboxed location.

The 4D version (v16, v17, v18) might not matter.

Apps that were installed on your computer prior to system update are all exempt from this new rule introduced in Sierra. The exact same app can be “untrusted” if copied to the system via network transfer after that.

You don’t have to do anything special.
Log in to your application from a WAN, check out some data.
Quit 4D and log in again. Do not restart your computer. Do nothing.
Just log back into the same program. Same structure. Same resources.
And most times a new cache folder is downloaded.

Do the same on the LAN and a new cache file is NOT created.

Note that the client has not actively changed anything on his / her computer. Now maybe something has changed in the way my internet provider connects me to the internet. But that is beyond my control and transparent to me.

From a conceptual perspective, the cache should only update when structure / plugins / resources in the server have changed.
It should not be updated when my network configuration changes.

I have no idea what 4D does behind the scene, but the result is that all of my computers get filled up with hundreds of megabytes of copies of cache files all which contain exactly the same data. Even though my structure has not changed at all.

Now 4D might call this a feature, but in the real world this is a bug.
It was not the behaviour prior to recent versions.

With all due respect, I don’t understand why 4D keeps explaining away something that is causing a problem for multiple users. This is not hard to reproduce.

FWIW, If anyone wants the code that I use to clear the old versions from my cache, just email me offline. If there are lots of requests, I will post it here.

Mitch

2 Likes

I am very sorry for that.

I totally agree that what you describe doesn’t sound like a useful feature or anything that can be “explained away” as nothing to be worried about…if it can be easily reproduced.

I hope I can be believed when I write that I don’t reproduce the issue, and the only way to replicate those folders is if I do some very specific manoeuvres that deliberately invoke the actual “feature” (hence the “explaining”). Unless I do that, WAN or LAN, I don’t reproduce those cache folders.

I am just a sample of one, but for our internal 4D client app (we connect from Japan to France every day) the cache folders are

  • {appName}_bugs_4d_com_19813_718
  • {appName}_bugs_4d_fr_19813_718

so there are only variations for the domain name, as explained.

also, I have

  • {anotherAppName}_192_168_10_52_19813_071
  • {anotherAppName}_192_168_10_52_19813_439

but that is because I simultaneously used 2 different builds for this project, each stored in a different folder, as explained.

Tthat does not seem to be the case. But again, I might be missing something.

We need to be asking questions like:

  • Does it happen on Mac only?
  • Is the client app signed and notarised with entitlements?
  • Does it happen with a regular 4D client and 4D Server that is not built?

to better understand what is going on.

In case of a crash, I think it might be useful to check the cache folder for

  • inuse.txt
  • uuid.txt

The former contains the process ID and full path of the app that is/was using this folder, to tell a second instance to go away and create its own.

Hello

thanks for your reply, we have not server crash case, as you wrote.
I was telling that we have this issue also after a clean 4D server restart.

I confirm that we have this issue on Mac.

In my machine it is replicated 100% of the time: everytime I connect, a new local resouurce foder is created, where XXX is the name of our custom application:

XXX_192_168_70_11_19833_231
XXX_192_168_70_11_19833_795
…

Using PC (we use remote session) on local resource there is only 1 folder, cannot replicate this issue in PC.

Details:

  • mac 10.15.5 (19F101)
  • 4D 18.2 build 18.252363
  • 4D database interpreted

For sure I have done a clean 4D install, but then I duplicated into the same 4D folder 4D app 2 times and renamed 4D app (added suffix “_production” and added suffix “_development”).
Don’t know if my 4D app rename action is the reason of this problem.

It is one issue so… I got at the moment 10 folders in 4D caches.

Dear,

I can only second that this is happening to me too.
As I wrote earlier it is easy to trigger for me by swopping
on and off the thunderbolt ethernet connection and jumping between
WiFi and cabled connection gives new caches at each connection.
It is Mac and Mojave. I Use 18.1 and start the connection with an
4dLink file and I connect to a dns name (not IP).
The WiFi and Cabled connection always have the same IP addresses
after reconnection (WiFi is enabled all the time)
NB Cabled have superior priority as network interface.
Wifi: 192.168.130.74
Cabled: 192:168:130:63

To provoke the behavior even further I tested following:

  • Have the ethernet cable connected
  • Connect 4D remote
  • Quit the application
  • Unplug the ethernet cable
  • Plug the cable again
  • Now I have the connection with same IP as before unplugging (…63)
  • Connect to 4D again
    … New cache folder is generated

I understand that something has been made in 4D to optimize
functionality when many Remote’s are run from the same computer
ala RDC or Citrix and the need to separate caches is important.
Maybe this is some kind of side-effect.

This behavior is definitely changed since v 17.

In my testing environment I do not see that I do anything different between
v17 and v18. However what has changed is:

v17.3 to v18.1
32 bit to 64 bit (Server and Remote)
Legacy network layer to New network layer.
Everything else that I can think of is the same. (hardware included)
I suppose that this is relevant for further investigation.

Best Regards

Magnus Torell

Hello

as new testing I have re-installed fresh on my mac 4D v18.2

I have not renamed 4D.app, as I did before.

Using clean 4D.app, I got in my local cache folder a new folder, most of the time (not always).

So my 4D app renaming was not the reason.

I can only add in the details that am using 4D remotely with VPN.

Re-confirm that actually on PC remote session I have only 1 local folder in cache resource, so on PC seems there is not this issue.

It is a problem… if you have some suggestion it will be appreciated.

Thanks !

I have some updates.

  • I can confirm that the problem is “easily reproducible”.

  • I can also say that WAN, LAN or VPN is not the issue.

Yesterday, I downloaded v18.2 (252958),

I created a new database with 4D Server, connect, disconnect, quit client, quit server.
I did it again, and voilĂ , I get a new client cache. But I swear that this was not the case before!

Some observations:

  • If I disconnect from the server, the “inuse.txt” file is removed.
  • If I reconnect without restarting the client, the same cache is used.
  • If I quit and relaunch the client, then connect to the same server, a new cache is created, with a new “inuse.txt” file and new “uuid.txt” file; this suggests that the app is indeed translocated.

To confirm, I look inside the “inuse.txt” file and I see the app path

/private/var/folders/19/45d7qmr50rsd5x0w5rrfzr540000gn/T/AppTranslocation/03299FEE-DEC9-43EE-A317-EF95E04B24F0/d/4D.app/Contents/MacOS/4D

I believe 4D is signed and notarised, but I suspect it is missing some entitlements, for example to access certain parts of the file system, or use the network. For my work, I actually sign and notarise 4D with my own Apple certificates and additional entitlements, for debugging plugins and doing other stuff. Today I am using an “official” copy so it does not have that advantage.

Perhaps we could discuss whether the official copy should be more entitled…

I went to system preferences and gave “Full Disk Access” to 4D. But that did not solve the translocation issue; a new cache is created yet again and the “inuse.txt” file says

/private/var/folders/19/45d7qmr50rsd5x0w5rrfzr540000gn/T/AppTranslocation/EC0472CF-FDD5-41A3-836F-71FF0967FEAC/d/4D.app/Contents/MacOS/4D

This is not cool…


By the way, this is how translocation looks like in Finder:

The app is sandboxed in a temporary volume.


Translocation is a major problem for 4D. The cache folder issue is just one manifestation, and it raises a red flag saying that something is not quite right, but the real issue is that the 4D is not trusted by the system and is being translocated.

1 Like

Follow-up:

I notarised 4D with my tool and I no longer see the cache issue. The file “inuse.txt” says:

/Applications/4D v18.2/252958/4D.app/Contents/MacOS/4D

which confirms that the app is no longer translocated.


Bug report reference:

ACI0100919: 4D downloaded and installed is translocated. Consequentially, the client cache folder is newly created every time the app is launched.


P.S.

Not sure if entitlements is the issue. Using codesign the only entitlement that the official 4D doesn’t have is com.apple.security.get-task-allow.

codesign -d --entitlements - /Applications/4D\ v18\ R3/4D.app
1 Like

Can you advise where I might get the tool

Thanks and regards

Chuck

I already shared a link.

Be glad to help, but may I ask which for part of the process you find to be unclear?