4D Server looses it's Lists in Structure

Product :4D - 4D Server
4D : v17.3
OS : Windows Server 2016

Hi to all,

we have a very strange and annoying problem with high inconvenience. Suddenly and without any notice 4D Server looses it’s lists in structure.

In the whole code, there is no single occurrence of
DELETE FROM LIST
SAVE LIST

only some
ARRAY TO LIST

BUT i don’t think, this is a problem created by user-code AS verifying the a corrupt structure results in errors like:

List ID=4101 exists but is not referenced by the list manager.
List ID=8286 exists but is not referenced by the list manager.
List ID=20358 exists but is not referenced by the list manager.
List ID=9197 exists but is not referenced by the list manager.
List ID=4109 exists but is not referenced by the list manager.
List ID=24686 exists but is not referenced by the list manager.
List ID=26555 exists but is not referenced by the list manager.
List ID=14616 exists but is not referenced by the list manager.

This behaviour startet in 16.4 (in my perception with the switch to the new network layer), went on in 16.5 (both on windows 2008). But our strong hopes in v17 together with an all new operating system (windows server 2016) also didn’t come true. We already posted this problem in the forum. But Thomas mentioned, that our configuration (v16 and windows server 2016) is outdated. This excuse does not longer exist.

As i mentioned: It started - in our perception - with the use of the new network layer. It seemed (very strongly) to happen more often, if the server is under high load (like copying 100 GByte over SMB). And it seemed to happen more often, if the users start the application (as this happens more often in the morning than in the afternoon)

With the update to v17 it seemed to be fine. We use v17 since 1 month. (This lost of lists happend only once, but were we are not 100% sure if this really happend as we didn’t conserve the damaged structure) BUT today this happend 3 times. The structure was running on the Server since monday continiously without such errors. Also this fact, that it runs without

We could switch back to the legacy layer and hope this will bring a relief to us but it won’t solve it. Are there any ideas to pin it down to a specific “action”. eg. do some logging, do some tests do something else

Thanks a lot

Christian

Hi,

We also encountered the exact same problem this week with 4D v17…

1 Like

We had/have this issue and for us, it started with 16.3. Below is our initial description for the bug opened with 4D.

Since converting from 14.6 to 16.3 HF2 and deploying our Production structure we have had three customer systems Lists go corrupt. All the lists except for less than 10 are lost completely. All attempts to restore the lists via manual, code, or using the MSC to repair that structure failed. The only solution that works is to shut the system down and replace the structure with the structure from our last production upgrade. This has caused us to have three unacceptable outages during critical usage periods. We need to determine the cause and how to prevent this from happening at any more sites.

We have multiple customers running the exact same structure and we had hundreds, if not thousands, of lists with many programmatic calls for ARRAY TO LIST. We did not have any Delete From List either.

We sent support multiple copies of the structure to 4D and the basic response was, you have too many lists and you’re updating them too often. Their response was that the List Editor is for more stagnant lists that do not change frequently. If you are frequently changing lists and values we should implement something using records in the database or some other solution.

I spent many, many hours cleaning out unused and duplicate lists (same content, different name) and then went and replaced all our calls for ARRAY TO LIST with code that compared the Array to the List and if they matched it didn’t update the code. This drastically cut the number of times the lists were updated. We had already moved to a [Lists] table for our web front end UI, but we still have some legacy 4D client UI access that heavily utilized lists.

Code is below. Hope this helps.

<code 4D>
If (Method called on error="") //added to check if there is an error call and if not call “ProcessError” - khollingsworth (7/26/2018) 13:05 -
ON ERR CALL(“ProcessError”)
End if

C_POINTER($1;$vpArray)
C_BOOLEAN($vbUpdate)
C_TEXT($vtMsg;$2;$vtListName)
//Init. Vars
$vpArray:=$1
$vtListName:=$2
ARRAY TEXT($atList;0) //added lines below minimize the writes to List manager - khollingsworth (7/12/2018) 16:18 - 15931 - JIRA CPG-76
LIST TO ARRAY($vtListName;$atList)
//changed below from > to # as it should update if the sizes don’t match regardless of which is larger - khollingsworth (9/7/2018) 09:30 - 16557
If (Size of array($atList)#Size of array($vpArray->)) //found that in testing the list could be bigger than the original which caused a syntax error - khollingsworth (7/26/2018) 13:02 -
$vbUpdate:=True
Else
For ($i;1;Size of array($atList))

	If ($atList{$i}#$vpArray->{$i})
		$i:=Size of array($atList)
		$vbUpdate:=True
	End if 
	
End for 

End if

If ($vbUpdate) //- khollingsworth (7/12/2018) 16:18 - 15931 - JIRA CPG-76

If (Test semaphore("WritingToList"))
	NewWindow (355;105;0;1)
	$vtMsg:=Char(13)+Char(13)+Char(13)+"List Management is locked. Please contact CardioPulse support immediately! : "+$vtListName+" ..."+Char(13)+Char(13)
	Repeat 
		MESSAGE($vtMsg)
		DELAY PROCESS(Current process;10)
	Until (Not(Test semaphore("WritingToList")))
	CLOSE WINDOW
End if 

If (Not(Semaphore("WritingToList")))
	ARRAY TO LIST($vpArray->;$vtListName)
	CLEAR SEMAPHORE("WritingToList")
End if 

End if
</code 4D>

Hello,
exact same problem here 17R4, only with the new network layer.

We eleminated all “Array To List” and replaced them with hierarchical List commands and we are using semaphores…

Nothing helped so far, except replacing structure and switching back to the legacy layer.

Very interesting that trying to repair lists at runtime seems impossible, so we will not even consider that…

We, respectively our customers, suffer that problem since v15.x.

Mostly on Mac, very rare on Win. Some can work for weeks or even month. Suddenly they ran into that ugly thing some days repeatedly.

Only solution: make a copy of the structure and replace it in case of a problem with the “old” version.

We have a check for missing lists with every call to open a window. Thus the customers are warned early.

Should I mention that we are unable to reproduce it in our office?

That’s discouraging to know that it is still happening in 17 R4, which probably means it’s still an issue in 18.

The impression that I received from the interactions on the ticket I opened with 4D was that structure lists were never intended to be repeatedly updated and are intended to be used in an almost write once, read many times way. I believe they expected us to create tables/record or outside data to use for any lists that would change frequently. If that was/is their intent, they could have saved all of us a lot of pain and better documented that.

We got the same issue yesterday , using a 17.3, for the second time in a few months.

No way to restate the lists from another structure as there is no tools provided by 4D for this…
We had to restore the structure.

Hope this is fixed in v18, this raises a huge mess in a production database.

I took over a project in v17.3 that showed the same issue. My hunch is it has something to do with writing/updating lists when running client server. Out of curiosity - how many instances of this problem happen when the server is uncompiled vs compiled?

Rather than screw around with trying to figure it out I simply moved the lists to a table. I recommend this approach if you make lists modifiable. I like the regular lists, but only for static purposes.

1 Like

Ours is uncompiled, so that might have something to do with it as well. We also moved all new development over to use a List table.

As another datapoint, I have not seen this happen with a structure that ran v16 for a year, and v17 for another year, on macOS Server with all macOS clients, always compiled mode.

This structure has a mechanism to modify some lists, which are also saved into the data file and overwritten every time the server launches.