[tip]removing tokens from .4dm files

the tokens make it really hard to use an external editor.
Thankfully, there’s a regex for that.
(In Atom)

  1. Open your .4dm file
  2. Open find and replace
  3. You’ll see a button labeled .* like this. Click on it:
  4. Enter the following regex:
    :C\d{1,}
  5. Make sure the Replace With area is empty
  6. Hit Replace All
    Add a thumb up to this post.
1 Like

I see I forgot constants.
That regex is
:K\d{1,}:\d{1,}

Just have to watch for false positives when you are using replace all :wink:

VP SET FORMULA(VP Cell("ViewProArea";5;2);"SUM(A1:C10)")
Begin SQL
  SELECT C1, C2, C3
  FROM Data
  INTO :C1, :C2, :C3
End SQL
$Text:=Document to text(":docs:C1Vertebrae.csv")
1 Like

ooh. good catches. three of them. I forgot about sql parameters and spreadsheet formulas. So I now have
(?<=[A-Za-z])((:C\d{1,})|(:K\d{1,}:\d{1,}))

Unfortunately it will still catch :C1Vertebrae in the last line, but Atom only supports fixed lookbehind in regex, so I’m not sure how to overcome that.

And now I see I have also neglected table, field and plugin tokens.
Adding plugins would make it
(?<=[A-Za-z])((:C\d{1,})|(:(K|P)\d{1,}:\d{1,}))
I’ll have to add table and field tokens, unless someone beats me to it.

Hi Mickey,
I use these in a 4d method (*) to remove tokens I know:
//command 4D - CommandName:C123
$rx_t:="[\w ]{1,31}(:C\d{1,4})"
//constant - ConstantName:K123:456
$rx_t:="[A-Za-z][\w ]{1,31}(:K\d{1,4}:\d{1,4})"
//[TABLE]field - [TableName:123]field:456
$rx_t:="[\w{1,31}(:\d{1,4})]\w{1,31}(:\d{1,4})"

[edit]
(*) from here

that’s going to match both the token and the identifier, but now you’re going to make me sit through and figure it out.

Remove what you don’t need, I meant.
It’s from a method that parses a line of code and removes optionally:
a/ only the tokens
b/ the tokens AND the tokenised texts
See the (capture group) in these regex.

If (someParameter)
	$remove_l:=0  //remove token and tokenised text
Else 
	$remove_l:=1  //remove token only
End if 
//...
$rx_t:="[\\w ]{1,31}(:C\\d{1,4})"
$start_l:=1
While (Match regex($rx_t;$out_t;$start_l;$pos_al;$len_al))
	$out_t:=Delete string($out_t;$pos_al{$remove_l};$len_al{$remove_l})
	$start_l:=$pos_al{0}  //+$len_al
End while 

I don’t think you need it to be 100% perfect, awareness that it might happen is probably enough.

Besides, I don’t think you will be able to prevent all false positives with regex. For perfect matching you would need to parse the code into a syntax tree so that you know for certain whether you are looking at a command or constant or whatever.

Another (silly) example with XML:

$ref:=DOM Create XML Ref("ALERT:C41";"http://www.mydomain/ALERT")
// produces: <ALERT:C41 xmlns:ALERT="http://www.mydomain/ALERT"/>

right. there were two instances that needed to be ignored: inside of quotes and in a comment.
the inside of quotes could be easy enough. basically you count the number of quotes between the current position and the end of the line. if it’s even, you’re outside of quotes, if it’s odd, you’re inside of quotes.
of course, 4d could always store the .4dm files without the tokens…