Anyone using VERIFY DATA FILE or VERIFY CURRENT DATA FILE?

VERIFY DATA FILE and VERIFY CURRENT DATA FILE have been around since v14. Is anyone using these commands in their solutions? We’d like to automate some of our maintenance procedures and need some guidance. We’re curious about implementation. Is there an example of how we might use the Callback method? Call Form? for updates?

We started to experiment with this about a year or so ago and got stuck. Now with some slack time in our feature delivery, we are revisiting some old “nice to have” options.

Product :4D - 4D Server
OS : Mac OS X

Hi Ed,

I’ve been using this for several years now. Very nice to have. I don’t have the code in front of me, but basically each server runs VERIFY CURRENT DATA FILE every night. The callbacks are used to gather warnings and errors. When it is finished and if there are any issues, the server automatically notifies me so I can take action. It also sends the info from the callbacks so I have an immediate idea of the severity of the issue. Well worth doing, in my opinion!

HTH.

I’m not sure to understand:

<https://livedoc.4d.com/4D-Language-Reference-17-R4/4D-Environment/VERI
Y-DATA-FILE.301-4054754.en.html>https://livedoc.4d.com/4D-Language-Refe
ence-17-R4/4D-Environment/VERIFY-DATA-FILE.301-4054754.en.html>

To verify both the records and the indexes, pass the total of Verify
Records+Verify Indexes. The value 0 (zero) can also be used to obtain
the same result. The Verify All option carries out complete internal
verification. This verification is compatible with the creation of a
log.

Verify all = 16
Verify indexes = 8 + Verify records = 4 (it means all) = 12

Manual writes: The value 0 (zero) can also be used to obtain the same result.

I understand that to achieve the same job, we can pass 16 or 12 or 0!

Just remember that running a verify on an opened database is different than on a closed one.

It does not matter if you do that by code or by MSC, but usually people using code want to run it automatically every night, so they use the opened database.

A verify on closed is very detailed. The open one is less detailed.
You can see that if you run verify from admin window - or from MSC window.
Compare the needed time (often twice as fast) and check the log, you will see less entries.

Again, this is not about running by code, it is about “opened datafile”.

Similar if you want to check a disk drive, it can be done online, but usually you need to restart to verify the boot volume (to test it offline).

Cannon,

Thanks for the positive feedback. We were attempting to use the command on relaunching our server after an abnormal shutdown. It happens. The idea is to make it easier for the admin staff to relaunch the server if we are not available. Yes we have documents and a series of dialogs that guide them through the process, but we were looking to make things as automatic as possible. Maybe we don’t grasp the purpose of the callback. Our original thought was that we could use it like displaying a progress dialog. Sounds like you are writing out results and sending them automatically. More of a headless operation. Can you give us an example?

I don’t use the callback for a progress meter. It’s the way I get the result of the verification process. Here is some code:

<code 4D>
vlDataIntegrity_ErrorCount:=0
vlDataIntegrity_WarnCount:=0
vtDataIntegrity_AccumulateErr:=""
vtDataIntegrity_AccumulateWarn:=""

VERIFY CURRENT DATA FILE(Verify all;Do not create log file;“DataIntegrity_VerifyDataFile_CB”)
If (vlDataIntegrity_ErrorCount+vlDataIntegrity_WarnCount>0)
//Use the above process variables to send a message about the failed verification
End if

</code 4D>

The callback method looks like this:

<code 4D>
//See the language ref for VERIFY DATA FILE and VERIFY CURRENT DATA FILE. This method gets called while the
//verification happens. The Verification Finished (2) message can happen through out as it finishes verifying a part
//of the data file. End of execution (4) always happens at the end.

//I think we can just accumulate errors and warnings.

C_LONGINT($1;$lMessageType) //1 = Message; 2 = Verification Finished; 3 = Error; 4 = End of execution; 5 = Warning
C_LONGINT($2;$lObjectType)
C_TEXT($3;$tMessage)
C_LONGINT($4;$lTableNumber)
C_LONGINT($5;$lReserved)
C_LONGINT($0)

$lMessageType:=$1
$lObjectType:=$2
$tMessage:=$3
$lTableNumber:=$4
$lReserved:=$5

If ($lMessageType=3) //Found an error
vlDataIntegrity_ErrorCount:=vlDataIntegrity_ErrorCount+1
If (vtDataIntegrity_AccumulateErr#"")
vtDataIntegrity_AccumulateErr:=vtDataIntegrity_AccumulateErr+"\r\r"
End if
vtDataIntegrity_AccumulateErr:=vtDataIntegrity_AccumulateErr+$tMessage
End if

If ($lMessageType=5) //Found a warning
vlDataIntegrity_WarnCount:=vlDataIntegrity_WarnCount+1
If (vtDataIntegrity_AccumulateWarn#"")
vtDataIntegrity_AccumulateWarn:=vtDataIntegrity_AccumulateWarn+"\r\r"
End if
vtDataIntegrity_AccumulateWarn:=vtDataIntegrity_AccumulateWarn+$tMessage
End if

$0:=0 //We don’t want to stop anything

</code 4D>

: Ed HAMMOND

We were attempting to use the command on relaunching our server after
an abnormal shutdown.

Please think carefully about that idea.

If there was an abnormal shutdown and the risk is to have lost something, don’t trust verify. Don’t forget, the message is “seems to be ok”. It is not “it is ok”.

You will never know what you just lost. Nothing, something, a lot, you don’t know.

4D offers since 1992 a save way. You know 100% what happend. Just use it.

When 4D starts to write in the data file, it sets a flag. Then it writes new/modified/deleted content and update the index. Then it writes the last operation number in the data file, finally it removes the flag.

Then 4D starts and there is no “dirty” flag, there was no open read operation. No need to verify, that is a fact. Then 4D compares the operation counter with the operation counter in the journal. If both are identical, there was no unsaved data in the cache. No question, it is a fact.

If the operations counter is different AND you followed best practice and both “automatic restore” check boxes are set, 4D automatically integrates the missing operations from the journal in data file and update index. Data is correct, no need to verify, all is fine.

If dirty flag is set, 4D restores last full backup and integrates log file. All data here, all data clean, no need to verify, all is fine.

This are facts.

Verify/Repair is to be used if you have lost your backup disk and your data disk became corrupt. This is a disaster operations, to be used after a fire or massive hardware fault. This is not for daily usage.
Verify cannot find missing or lost or never saved data.

Edgar,

VERIFY DATA FILE
Yes; I have all my systems, once a week after a backup: restore the backup and verify the datafile from the backup, and email me if there is damage.
Out of 300+ databases, I have about one/month that comes up with something.

VERIFY CURRENT DATA FILE
no: I tried it for a while, and abandoned that idea, for several reasons.

Tony,

I’m curious, are the system you do this on running 24/7?

Maybe this is an ignorant statement, but I’m not sure how you restore and validate if the system is still being used?

Our customers run around the clock and the fact that 4D is a blocking backup is a significant issue for us, so as much as I like the idea of what your doing, I don’t know if we could suffer even more downtime during a backup.

Best,

Steve

Hey Steve,

are the system you do this on running 24/7?
Absolutely.

I have a separate agent running on each server machine (a 4D app that I wrote) that watches for backups, restores the backup to a separate directory, and verifies it.

So, the live server doesn’t come down: That’s the whole point: a separate app runs the verify on a restored backup.

Tony,

Excellent…should have thought of that option.

Steve

Hi Tony,

I like this idea. I’m planning on doing something similar, and was wondering what’s the largest data file this setup of yours deals with, and how long does the whole restore and verify process take on that one?

My largest installation has a 160GB+ data file, and ideally I’d like this whole process to kick off and complete outside work hours.

Thanks

Hi Peter,

My largest datafile that this runs on is about 130GB.
It’s all running on RAID SSD’s, but not the absolute fastest. This particular backup is NOT compressed (because running a compressed backup of that side would have the client unusable for too long)

The auto-restore takes about 10 to 15 minutes.
The verify takes around 4 hours I think.
Because my system is on SSDs and RAID, I don’t think there’s any disk contention: it does run on the same machine, but the machine’s well resourced.

Thanks Tony,

My setup is RAIDed SSD on a very well speced server also, so it should be fine. 4-6 hours overnight on a Friday should not cause any issues with availability.