Modify .4dm format for easier editing in text editors

I opened a .4dm file in ST a few minutes ago and was disappointed to discover that it isn’t going to be as straightforward as I was expecting.

What did you expect?

I’m confused by the question. Have you looked at a .4dm file in a text editor? Have you noticed that the syntax?

Here’s a simple one


//%attributes = {}

If (False:C215)

$startingLocker:=Num:C11(Request:C163("What is the first locker number?"))

If (ok=0)

ABORT:C156

End if

$lastLocker:=Num:C11(Request:C163("What is the last locker number?"))

If (ok=0)

ABORT:C156

End if

$location:=Request:C163("What is the location?")

If (ok=0)

ABORT:C156

End if

For ($i;$startingLocker;$lastLocker)

CREATE RECORD:C68([Locker:7])

[Locker:7]Locker Number:1:=$i

[Locker:7]Building:2:="North"

[Locker:7]Location:3:=$location

[Locker:7]In Service:5:=True:C214

SAVE RECORD:C53([Locker:7])

End for

End if   //false

No, I haven’t seen that, we are not yet working productively with v18, but I had a look at it recently. It didn’t look that bad.
Yes, you have these token numbers as suffixes to all commands. But that’s more of a feature. Consult the documentation (I don’t know where at the moment).
But what I saw was properly indented 4D code that is easy to read.
Like I said, not so bad.
Your example seems to be rather a problem with some settings of your editor, there is at least an issue with the line endings. And indenting is wrong too.

The main advantage of editing these files outside of 4D is the use of sophisticated refactory options with regex and the like, provided by some editors and IDEs, for example. At least in my opinion.
And yes, 4D’s editor features are not as good as e.g. in IntelliJ, but on the other hand, these IDEs can’t use their full functionality in conjunction with the 4D syntax. So there will always be a switch between the two tools. IMO this is not really a problem, since v18 imports changes from outside immediately.

Don’t look at the indenting. I copied/pasted.

The tokens are not helpful when one is trying to use the editor to write new code or edit existing code. It certainly makes it harder to read (otherwise, we’d all be clamoring for it in the method editor). I want to be able to use the editor to actually work on code in some weird variant of 4D that is mainly 4D except it’s some other 4D with suffixes on the keywords. It’s not SAVE RECORD, it’s SAVE RECORD:C53. No thanks.

There are so many things that dedicated text editors can do that script editors do not do. Take indenting, since you brought it up - you can write custom indenting rules. With indenting rules, you don’t have to wait for 4DSA to add Code Blocks, maybe, someday. Example: I want to look for problems in my employee locker data. The code is several hundred lines long, with lots of control structures, but it boils down to several main tasks: Find missing data, Find cross-linked data, Find instances where a record has in incorrect state. Using code blocks, I can mark those exact sections in my code


//<find missing data>

...

//<find cross-linked data>

...

//<find incorrect state>

4D won’t recognize those those comments as meaning anything, but a custom indent rule in a text editor can, and then indent the code in those sections to make the method easier to read. Even better - now that the blocks are indented, a text editor will gladly fold them for me.

Theming - is there a dark mode in 4D? What about a theme that’s inspired by Material Design? Halloween? Experiment 626?

Last one, because I just ran into this one today, too and I was reminded about it above - more thoughtful shortcuts. I was trying to fold different parts of a method today inside of the method editor. There is no shortcut. It’s actually kind-of a pain to fold and unfold in the method editor. Open Sublime Text or Atom, though, and there is a whole series of shortcuts for folding at different levels, different points, etc. Fold all the control structures in my method in Sublime? Cmd-K-1. Unfold? Cmd-K-0. Oh, yes, you can Cmd-K-2, and so on, too.

shameless plug:

you can use the Atom editor.

(you don’t need to install from this page, you can add packages directly in Atom)

language-4dm use the “tree-sitter” parser.

Go to preferences > shortcut, and you can add your custom shortcuts for folding, unfolding, etc.

wait - never mind, I never added a feature request for the shortcut.
AFA the shortcuts, thanks for the tip. It would be nice to be able to specify the level of folding the way you can in atom or ST, too, especially once classes become more of a thing in 4D.

When there are only single methods, it’s not as much of a deal, but once you throw the entire class definition in one file, the hairball will get much larger, so being able to fold at level 1, 2, etc. using or ignoring the selection is a great feature.

OK, NOW it’s a feature request.

and whoops, never mind, i did find it in the atom package manager. I was in the wrong tab.

The token suffixes are going to be a problem, so I think removing them should be considered.

Problem how?

The tokens are the commands, constants, tables and fields. Removing them means you just have comments in your code.

Of course, technically it is possible to register all commands, constants, tables and fields, like a dictionary. But that would give you a false sense of security because the language is never static.

I do hard-code control flow and compiler directives, because I know they are basic building blocks and more than regular commands. However, I purposefully avoided doing the same for all commands, constants, tables and fields, because they are supposed to be dynamic in nature.

Only the runtime, or the compiler, with the help of an abstract syntax tree can know for certain what a token represents. In fact they are always correct, because they are the ones who make the final call. An external editor can make an educated guess, with some knowledge of the structure, the version, the symbol list, the language environment, but sometimes it is impossible to tell just by looking at code if a token is a variable, constant or method in a 4D application (calling fields without the table has been deprecated in project mode, so we can forget about those).

For me, external code editors are support tools, not Method Editor replacements.

I think this implementation for VSCode takes a more preemptive approach:

It looks like Sublime Text does not have a 4dm extension yet, maybe you can be the first to publish :slight_smile:

1 Like

I suppose one may remove these tokens easily with no side effect. But:
• if your code is opened by someone (still) using french langage, it will not work (I suppose you don’t mind about this one :smile:)
• if you open some code written in another version in which commands or constants names have been changed, you will mind…

I agree tokens are ugly, but the feature request cannot be “remove tokens and that’s all”, but “stop changing names first, then remove tokens”.

And even if they’re ugly, I often use tokens in my formulas:
$formula:="[myTable]myField" //formula will become false if i change table or field name
$formula:="[myTable:5]myField:11" //formula insensitive to renaming

1 Like

@keisuke_miyako
this is in regard to being able to write and edit code in an external editor. If you have to know and append the token suffix to every token, that is not going to work very well.

@de_montard.arnaud

we’re not planning on supporting code in other languages, but i understand the issue, since 4d is trying to allow developers to work in a variety of tongues.

i understand what you are saying about the token names changing in the future, and that is something that will have to be discussed and debated. is the priority going to be to withstand those sorts of issues, have an update migrate code forward when token names change, have the project be text files, but when they load update the binary database project, and then regenerate the text files, etc.? i get it. the goal of this feature request is to let me stop using the method editor and instead use whichever text editor i choose. i contend that the token suffix is an impediment to that.

Language Server Protocol is a new and growing method for editors to support a variety of programming languages in a standardized way. I have not looked at the spec in detail, but perhaps there is a way for the LSP server to remove the tokens for editing and add them back on save just like 4D does it.

4D really needs something like LSP to help properly identify language elements. Regex/grammar based approaches can only go so far.

https://langserver.org

https://nshipster.com/language-server-protocol/

I’m afraid, it’s a matter of performance.
4D permanently needs to re-parse those files, and the token numbers help to accelerate.
So 4D cannot do without these token suffixes.

If I’m wrong, then we should get a preference “4DM without tokens”.
So the developer can decide whether he prefers language and update safety, or readability.

Tokens in text are optional. If you are using an external editor, you can type code without.
Their purpose is only to make your code resilient to the renaming of commands, constants, tables or fields.

About “Tokens” there is very much to say and discuss.
I had no solution for all what can be needed and a bridge between all.
Tokens sometimes good, Tokens sometimes bad, Tokens better done by meta dictionary append to all stuffs…and so on …it belongs to different use-cases.
No we can use all this extern stuffs thats good,
extern stuffs missing a large range of features what can be done internaly in 4D-MethodEditor or internaly 4D-StructSearch or internaly 4D-Syntax-Check.
And extern stuffs have features what are missing 4D internally(built-in).
So i decide i use both externStuffs and 4DinternStuffs, to take the good one of both :wink:

This code is only one small jigsaw piece (as workaround) of 1000 pieces we need :wink:

  // Just my workaround (i had no better idea) to remove or to add tokens on the fly
  // because there is no 4D-command to add/remove tokens for whole method-text
  // imported as text from any project-source-folder.
  // This ist only for use inner 4D and belongs to 4DVersion and own DB(table,field,plugin,...)
C_COLLECTION($colLines)
C_TEXT($txtLine;$txtIndent)
C_LONGINT($pos;$length)
C_BOOLEAN($hasIndent)
$colLines:=New collection
$colLines:=Split string(Form.soCurrEl.content;Char(13))  // create a code-lines-collection from my method-text
Form.soCurrEl.content:=""  // ...my method-text (with or without tokens) set first to empty
  // ...now fill my method-text again (with or without tokens) from splitted lines in $colLines
For each ($txtLine;$colLines)
	$hasIndent:=Match regex("^\\t+";$txtLine;1;$pos;$length)
	If ($hasIndent)
		$txtIndent:="\t"*$length
	Else 
		$txtIndent:=""
	End if 
	If (Form.toggleViewTokens=1)
		  // remove all tokens
		  // - CommentParts in code are correctly ignored: // some comment :C000 is here my pattern...
		  // - StringExpressions in code are correctly ignored: $x:=Replace string(":C000";"")
		Form.soCurrEl.content:=Form.soCurrEl.content+$txtIndent+Parse formula($txtLine)+Char(13)
	Else 
		  // create/add tokens
		  // by current 4DVersion(commands, constants, ...)
		  // and by currentStuffsFromOwnDB (table, fields, plugin-functions ...)
		  // - CommentParts in code are correctly ignored: // some comment :C000 is here my pattern...
		  // - StringExpressions in code are correctly ignored: $x:=Replace string(":C000";"")
		Form.soCurrEl.content:=Form.soCurrEl.content+$txtIndent+Parse formula($txtLine;Formula out with tokens)+Char(13)
	End if 
End for each

Hi Laurent. Interesting. How do we turn off the “Include Tokens” option.

Excuse me the interjection.

Laurent wrote that you can “type” without tokens in an external editor.

You seem to be looking for an option to stop 4D from adding tokens to the backend files that are automatically managed in project mode. Those are two different conversations.

2 Likes

interesting. i did not know they were optional. that makes it simpler, now doesn’t it? i mean, it will still make the first pass a bit of a mess unless you go through with some regex and remove them all, but still…