Maintenance Security Centre gives inconsistent results on same file

I’ve got a reasonably large datafile (700KB). Maintenance Security Centre sometimes says it validates OK and sometimes gives errors. This is on the SAME DATAFILE - all I’ve done is copied it.

I took a clean copy that validated OK and put it in a different folder. Now I copy it back and it gives errors. (I’ve repeated this a couple of times to check)

The database that uses the file apparently works fine - it was just a routine check after a few days intensive editing: there wasn’t a problem that I was trying to solve.

The only thing I can think of is that my hard drive is a hybrid SSD (ie a hard disk with an SSD cache). I know 4D often goes quite deep into the system; could it be bypassing the cache?

Windows 10, 4D 17.1 build 17.232790 64-bit.

Malcom,

I’ve got a reasonably large datafile (700KB)
Do you mean 700MB? or 700GB?

700KB is like nothing.
My great-grand parents used to do 700KB databases before they had electricity.

You should try to drop the .4dindy file, I’ve often seen MSC giving
bad result due to this file corrupted.

: Malcolm STOREY

I know 4D often goes quite deep into the system; could it be
bypassing the cache?
I don’t think so. 4D cache, as far as I remember, relies on system cache. It’s not oracle :wink:

4D goes through the system API, it always uses system calls, it never communicates with hardware in low level mode.

That you have different results just by copying would ring all alert bells for me…

Google for free products creating hash codes for a file (something like https://support.microsoft.com/de-de/help/889768/how-to-compute-the-md5-or-sha-1-cryptographic-hash-values-for-a-file)

Duplicate your file several times, or copy it to other folders).
For each file create a hash code and compare them.
if they are not identical, stop everything asap, purchase a new disk and do a new setup of your computer. Don’t clone the old one. Install a fresh version of Windows and try to recover as much as possible from the old disk.

For system and data file it should not be a hybrid disk. Not because that would not work, but because SSD’s became so cheap, that this is a wrong place to try to save money. Use hyprid disks for your picture or music or video library, for stuff you don’t need every day and that’s too large to fit on an effordable SSD.

I do indeed mean 700mb. Sorry - it’d been a long day getting nowhere with corrupted files.

The copying was because I’d taken snapshots over the last few days and I was copying these into my work area to try to work out where things went wrong. The results were inconsistent, which could be a disk problem (or a memory problem or an OS problem or anything really).

Usually the validation would fail if I clicked the top button to validate records and indexes at once, but they would validate OK if I did them separately. The index fails were often not in my indexes but in some system stuff at the beginning.

Anyway, last night I reinstalled 4D and ran CCleaner on the registry.
This morning I deleted the index file (thanks Arnaud!), opened interpreted to rebuild indexes, checked it compiled, then ran Maintenance and everything has passed!

Validation produces a different log file and takes a lot longer to run when the file is corrupte. If it validates OK, it opens in IE as a short XML file. If it fails you get a (very) long formatted document with (in my case) 703 paragraphs.

I think (hope!) this has fixed it but I’ll wait a couple of days before closing this. Thanks very much everybody for your input.

: Malcolm STOREY

The copying was because I’d taken snapshots over the last few days
and I was copying these into my work area to try to work out where
things went wrong.
I’m not sur I understand all that, but if “taking snapshots” means “make copies of 4d files of a running 4d base”, you have chances that the copies are corrupted. The only clean copy of a running 4D file is the backup file.

By taking snapshots I meant close down 4D, copy the file(s), and then restart 4D. I’m aware of the potential problems of data in caches, transactions etc, (although 4D is remarkably good at coping with such foolish behaviour).

But thanks for the reminder.

Arnaud there is https://doc.4d.com/4Dv17/4D/17.1/Using-Volume-Shadow-Copy-Service-on-Windows.300-4203715.en.htmla new special option in v17 to manage correctly the snapshot> (only on PC). RTFM :razz:

: Malcolm STOREY

This morning I deleted the index file (thanks Arnaud!)
That’s very kind of you but I must unfortunately point out I wrote 4dindy, not 4dindx. :lol:

That said, in my practice the only files that really have to be “MSC clean” are 4db and 4dd, 4dindy and 4dindx are just results of these files.

  • I never hesitate to drop 4dindy
  • I rarely check the 4dindx file, more simple to drop it (even if I’d like it’s construction to be faster)

You did indeed say indy but I took it to mean index files in general and as it was the datafile that was giving trouble… But since even recovering from tags didn’t fix it, that does suggest the problem might be with indy.

I’ll drop them both again to make sure!

Thanks again

recovering from tags
I had a bad experience last year with that recovery. After MSC said everything was all right, the application began to work as if some relations were broken, or 4d unable to use them. I had to export/import all in a fresh new empty data to make it work correctly.
that does suggest the problem might be with indy.
I’ve seen more than once this file responsible of unexpected things.
An example i’ve seen more than once: launch MSC, run “verify records and index” -> it says 4db is correct, 4dindx is correct, but not both of them (the first button)! Delete 4dindy, run MSC again -> OK.
Another example: struggling to compile, always fails. Delete 4dindy, compile -> OK.

I spent yesterday doing manual editing and the datafile indexes are corrupt again. But I know what’s doing it. There’s been an index-corrupting bug in 4D for the 20+ years that I’ve been using it.

Take a table with a text field marked as “indexed, unique” and try to save a duplicate value, up pops the 4D error message. In my code at least, you then have to change the value before you can hit Cancel, or the message just pops up again (the value is checked and the error message appears when the entry field loses focus). Quit 4D and the index file is corrupted. I’ve seen this so many times, but not recently, so I’d forgotten about it until yesterday. (It presumably needs a sufficient number of records to populate the B-tree.)

Thanks, but I’m running single user 4D. It’s a database for personal use and I’m the only user. Every night after I’ve closed 4D I manually copy the files.

Now this is getting serious…

I’ve just deleted both indexes, opened 4D, recompiled, rebuilt, copied compiled to my work area, deleted the data index, and let it rebuild the indexes. It now reports that both the application and datafile are corrupt. :evil:

I’ve just repeated it but this time ran the validation without letting 4D rebuild the data indexes and it still says there are errors:

Line 114: Error:Problem on B-tree Index 0 on kind (TYPE only) index : Tag header of page address directory is invalid(28;108)
Line 124: Error:Problem on B-tree Index 1 on stamp (stamp) index : Tag header of page address directory is invalid(28;108)
Line 133: Error:Problem in NULLS cluster of B-tree Index 1 on stamp (stamp) index : Tag header is invalid(26;68)
Line 142: Error:Problem on B-tree Index 2 on kind , id (TYPE and ID) index : Tag header of page address directory is invalid(28;108)

:pray: :pray: :pray:

If you really think that this came from a bug you should contact the 4D support so they can see(reproduce) that and perhaps help you. They have some informations we don’t have here on the forum. If you are 4D Partner you should start to post an incident on https://taow.4d.comTAOW>.

Following the above, I opened 4D and let it build data indexes, then reran validation and everything has come up clean! (Just to be sure I quit and reran to check.)

My original title “Maintenance Security Centre gives inconsistent results on same file” was spot on!

I can only second what Manuel suggests, do open TAOW case. Also, the thing I would first ask you to do if I would still be at 4D is to provide me with your database so it could be tested on different computer/drive. You can try that yourself to see if you have hardware problem meaning it is local to your computer or you actually have problem with 4D.

I’ve opened a TAOW case.

The index has since become corrupted again. I hadn’t done very much. It never used to be so fragile.

It appears I need to delete the index file, let 4D rebuild it, then delete it again and rebuild it again to get a clean file.

I recently deleted a lot of records (c. 170K) from a table (let’s call it TABLEA) which is used to hold the raw data during a bulk import; it’s now empty. After the delete I ran compact and the file shrank from 975M to 770M.

<:?:> Validate displayed “checkling list of deleted records in TABLEA” for couple of seconds. I’ve run compression again and it still spends time checking that list of records that are no longer there. </:?:>

Hi Malcolm,
I’m thinking about the table property “Records definitively deleted”, see if it’s checked or not. I don’t use it so I’m not aware of consequences, but I suppose when it’s unchecked that place taken by records remains.

Did you try to:

  • “touch all” ? (force a re write of each records in the table of that so fragile index)
  • export/import all in an empty data?