[AUCTeX] fontification question

[François Patte]

Bonjour,

I added to my .Xresources file:
"Emacs.font-latex-sedate-face.attributeForeground: indianred2" in order
to have all LaTeX commands in indianred.

But when I use a command like \- for discretionnary hyphen, the whole
word after the \- command is turned into indian red, ie. the
fontification is not applied only to the command (which is actually in
red).

For instance: anti\-constitutionnel: \- is in red and constitutionnel
in indian red.

Is there a way to avoid this?

Thank you.

[Tassilo Horn]

François Patte <***@mi.parisdescartes.fr> writes:

Bonjour François,

Post by François Patte
"Emacs.font-latex-sedate-face.attributeForeground: indianred2" in
order to have all LaTeX commands in indianred.
But when I use a command like \- for discretionnary hyphen, the whole
word after the \- command is turned into indian red, ie. the
fontification is not applied only to the command (which is actually in
red).
For instance: anti\-constitutionnel: \- is in red and constitutionnel
in indian red.
Is there a way to avoid this?

That's a bug. I just taught font-latex that \-something is not a
command and should not be highlighted as such.

I also made a new ELPA release but sadly just *before* I read your mail
and fixed the bug, so it might take a while until the fix becomes
available (through ELPA at least).

Merçi,
Tassilo

[Mosè Giordano]

Hi Tassilo,

I added also "," to the characters to be ignored, because with the patch

foo\,bar

had "\,bar" highlighted.

Bye,
Mosè

Bonjour François,

Post by François Patte
"Emacs.font-latex-sedate-face.attributeForeground: indianred2" in
order to have all LaTeX commands in indianred.
But when I use a command like \- for discretionnary hyphen, the whole
word after the \- command is turned into indian red, ie. the
fontification is not applied only to the command (which is actually in
red).
For instance: anti\-constitutionnel: \- is in red and constitutionnel
in indian red.
Is there a way to avoid this?

That's a bug. I just taught font-latex that \-something is not a
command and should not be highlighted as such.
I also made a new ELPA release but sadly just *before* I read your mail
and fixed the bug, so it might take a while until the fix becomes
available (through ELPA at least).
Merçi,
Tassilo
_______________________________________________
auctex mailing list
https://lists.gnu.org/mailman/listinfo/auctex

[Tassilo Horn]

Post by Mosè Giordano
I added also "," to the characters to be ignored, because with the patch
foo\,bar
had "\,bar" highlighted.

Right, I've forgotten that.

Thanks,
Tassilo

[Arash Esbati]

Post by Tassilo Horn

Post by Mosè Giordano
I added also "," to the characters to be ignored, because with the patch
foo\,bar
had "\,bar" highlighted.

Right, I've forgotten that.

Hi all,

as for the next iteration, I also added " " (space) to the ignored
characters list. Otherwise,

\foo\ bar

had "bar" highlighted.

Best, Arash

[Arash Esbati]

Post by Arash Esbati
as for the next iteration, I also added " " (space) to the ignored
characters list. Otherwise,
\foo\ bar
had "bar" highlighted.

Following up myself, the patch was not good enough. Here a test case of
fontification right now:

[Tassilo Horn]

Post by Arash Esbati

Post by Arash Esbati
as for the next iteration, I also added " " (space) to the ignored
characters list. Otherwise,
\foo\ bar
had "bar" highlighted.

Following up myself, the patch was not good enough.

Oh, yeah, I introduced the bug. My change said that the first char
after the \ must not be a - or a ,. However, then it did not need to be
a symbol or word character, too! ;-)

Ok, I fixed it now but that "\ followed by word and symbol chars but not
starting with - or ," couldn't be captured in a regex anymore (tell me
if I'm wrong).

Bye,
Tassilo

[Arash Esbati]

Post by Tassilo Horn

Post by Arash Esbati
Following up myself, the patch was not good enough.

Oh, yeah, I introduced the bug.

To avoid any misunderstandings: I meant that my last patch was not good
enough.

Post by Tassilo Horn
My change said that the first char after the \ must not be a - or a ,.
However, then it did not need to be a symbol or word character, too!
;-)
Ok, I fixed it now but that "\ followed by word and symbol chars but not
starting with - or ," couldn't be captured in a regex anymore (tell me
if I'm wrong).

Thanks for your patch, it looks much better. I think the only thing
missing is when you have a character with punctuation syntax after \
like \"o, \^o or \~o. What do you think about a variation like this:

--8<---------------cut here---------------start------------->8---
diff --git a/font-latex.el b/font-latex.el
index 746193bc..d6bf8d37 100644
--- a/font-latex.el
+++ b/font-latex.el
@@ -1786,10 +1786,10 @@ marks boundaries for searching for group ends."
;; foo\,bar) or a - (hyphenation: foo\-bar).
(let* ((search (lambda ()
(TeX-re-search-forward-unescaped
- "\\\\\\(\\s_\\|\\sw\\)\\(?:\\s_\\|\\sw\\)*" limit t)))
+ "\\\\\\(\\s.\\|\\s_\\|\\sw\\)\\(?:\\s_\\|\\sw\\)*" limit t)))
(pos (funcall search)))
(while (and pos
- (member (match-string 1) '("-" ",")))
+ (member (match-string 1) '("-" "," "/" "&" "#" "_")))
(setq pos (funcall search)))
pos))
--8<---------------cut here---------------end--------------->8---

We can extend the exclude list if other characters appear.

Best, Arash

[Tassilo Horn]

Post by Arash Esbati

Post by Tassilo Horn
Oh, yeah, I introduced the bug.

To avoid any misunderstandings: I meant that my last patch was not
good enough.

Well, you did not succeed in fixing the bug I've introduced. ;-)

Post by Arash Esbati

Post by Tassilo Horn
My change said that the first char after the \ must not be a - or a
,. However, then it did not need to be a symbol or word character,
too! ;-)

Thanks for your patch, it looks much better. I think the only thing
missing is when you have a character with punctuation syntax after \
like \"o, \^o or \~o.

Are those valid commands? When I try to define them using

\newcommand{\"o}{tüddelchen o}
\newcommand{\^o}{dach o}
\newcommand{\~o}{tilde o}

I get the error

ERROR: LaTeX Error: Command \~ already defined.
Or name \end... illegal, see p.192 of the manual.

Bye,
Tassilo

[Arash Esbati]

Post by Tassilo Horn
Well, you did not succeed in fixing the bug I've introduced. ;-)

I'm not that much on top of the game ;-)

Post by Tassilo Horn
Are those valid commands? When I try to define them using
\newcommand{\"o}{tüddelchen o}
\newcommand{\^o}{dach o}
\newcommand{\~o}{tilde o}

Yes, they are. Probably not interesting in UTF era any more, but valid
code:

\documentclass{article}
\begin{document}
\`o \'o \^o \"o \~o \=o \.o \t oo \o\ \oe\ \~o
\S \P
\end{document}

Best, Arash

[Tassilo Horn]

Ah, right. Well, then I guess your patch makes sense. Feel free to commit it.

Bye,
Tassilo

Post by Arash Esbati

Post by Tassilo Horn
Well, you did not succeed in fixing the bug I've introduced. ;-)

I'm not that much on top of the game ;-)

Post by Tassilo Horn
Are those valid commands? When I try to define them using
\newcommand{\"o}{tüddelchen o}
\newcommand{\^o}{dach o}
\newcommand{\~o}{tilde o}

Yes, they are. Probably not interesting in UTF era any more, but valid
\documentclass{article}
\begin{document}
\`o \'o \^o \"o \~o \=o \.o \t oo \o\ \oe\ \~o
\S \P
\end{document}
Best, Arash

[Arash Esbati]

Post by Tassilo Horn
Ah, right. Well, then I guess your patch makes sense. Feel free to commit it.

Done. Best, Arash

[Arash Esbati]

Post by Arash Esbati

Post by Tassilo Horn
Ah, right. Well, then I guess your patch makes sense. Feel free to commit it.

Done. Best, Arash

Well, I have found one use-case where the patch still doesn't fit: expl3
syntax. Please consider this snippet:

\cs_new:Npn \foo_bar:Nn #1#2
{
\cs_if_exist:NTF #1
{ \__foo_bar:n {#2} }
{ \__foo_bar:nn {#2} { literal } }
}

Current code doesn't work on \__foo_bar:n. My plan is to install this
patch:

--8<---------------cut here---------------start------------->8---
diff --git a/font-latex.el b/font-latex.el
index ea3e829a..2f662965 100644
--- a/font-latex.el
+++ b/font-latex.el
@@ -1782,6 +1782,11 @@ marks boundaries for searching for group ends."
(throw 'extend group-start)))))
nil)))

+(defvar font-latex-match-simple-exclude-list
+ '("-" "," "/" "&" "#" "_")
+ "List of characters directly after \"\\\" excluded from fontification.
+Each character is a string.")
+
(defun font-latex-match-simple-command (limit)
"Search for command like \\foo before LIMIT."
;; \s_ matches chars with symbol syntax, \sw chars with word syntax, \s. chars
@@ -1793,7 +1798,10 @@ marks boundaries for searching for group ends."
"\\\\\\(\\s_\\|\\sw\\|\\s.\\)\\(?:\\s_\\|\\sw\\)*" limit t)))
(pos (funcall search)))
(while (and pos
- (member (match-string 1) '("-" "," "/" "&" "#" "_")))
+ (member (match-string 1)
+ (if (eq major-mode 'doctex-mode)
+ (remove "_" font-latex-match-simple-exclude-list)
+ font-latex-match-simple-exclude-list)))
(setq pos (funcall search)))
pos))
--8<---------------cut here---------------end--------------->8---

Then it works at least in doctex-mode. I'm reluctant to include
something like this in expl3.el:

(set (make-local-variable 'font-latex-match-simple-exclude-list)
(remove "_" font-latex-match-simple-exclude-list))

It breaks too much for little benefit. I will also contact the author
of expl3.el; he tends to have good ideas :-)

Best, Arash

[Mosè Giordano]

I think that start writing tests for fontification may be a good idea
at some point ;-) Fontification tends to be often broken and it's
hard to understand what was actually broken and what's the status-quo
that we want to preserve.

Bye,
Mosè

Post by Arash Esbati

Post by Arash Esbati

Post by Tassilo Horn
Ah, right. Well, then I guess your patch makes sense. Feel free to commit it.

Done. Best, Arash

Well, I have found one use-case where the patch still doesn't fit: expl3
\cs_new:Npn \foo_bar:Nn #1#2
{
\cs_if_exist:NTF #1
{ \__foo_bar:n {#2} }
{ \__foo_bar:nn {#2} { literal } }
}
Current code doesn't work on \__foo_bar:n. My plan is to install this
--8<---------------cut here---------------start------------->8---
diff --git a/font-latex.el b/font-latex.el
index ea3e829a..2f662965 100644
--- a/font-latex.el
+++ b/font-latex.el
@@ -1782,6 +1782,11 @@ marks boundaries for searching for group ends."
(throw 'extend group-start)))))
nil)))
+(defvar font-latex-match-simple-exclude-list
+ '("-" "," "/" "&" "#" "_")
+ "List of characters directly after \"\\\" excluded from fontification.
+Each character is a string.")
+
(defun font-latex-match-simple-command (limit)
"Search for command like \\foo before LIMIT."
;; \s_ matches chars with symbol syntax, \sw chars with word syntax, \s. chars
@@ -1793,7 +1798,10 @@ marks boundaries for searching for group ends."
"\\\\\\(\\s_\\|\\sw\\|\\s.\\)\\(?:\\s_\\|\\sw\\)*" limit t)))
(pos (funcall search)))
(while (and pos
- (member (match-string 1) '("-" "," "/" "&" "#" "_")))
+ (member (match-string 1)
+ (if (eq major-mode 'doctex-mode)
+ (remove "_" font-latex-match-simple-exclude-list)
+ font-latex-match-simple-exclude-list)))
(setq pos (funcall search)))
pos))
--8<---------------cut here---------------end--------------->8---
Then it works at least in doctex-mode. I'm reluctant to include
(set (make-local-variable 'font-latex-match-simple-exclude-list)
(remove "_" font-latex-match-simple-exclude-list))
It breaks too much for little benefit. I will also contact the author
of expl3.el; he tends to have good ideas :-)
Best, Arash

[Arash Esbati]

Post by Mosè Giordano
I think that start writing tests for fontification may be a good idea
at some point ;-)

True, I started with it sometimes ago as I was playing with
fontification of verb macros:

http://git.savannah.gnu.org/cgit/auctex.git/tree/tests/latex/fontification-verb-macros.tex

Post by Mosè Giordano
Fontification tends to be often broken and it's hard to understand
what was actually broken and what's the status-quo that we want to
preserve.

You're right about "how do we want to have it in the first place?" I
put my idea behind my patch in the document attached.

Comments welcome.

Best, Arash

[François Patte]

Post by Tassilo Horn
Bonjour François,

Post by François Patte
"Emacs.font-latex-sedate-face.attributeForeground: indianred2" in
order to have all LaTeX commands in indianred.
But when I use a command like \- for discretionnary hyphen, the whole
word after the \- command is turned into indian red, ie. the
fontification is not applied only to the command (which is actually in
red).
For instance: anti\-constitutionnel: \- is in red and constitutionnel
in indian red.
Is there a way to avoid this?

That's a bug. I just taught font-latex that \-something is not a
command and should not be highlighted as such.
I also made a new ELPA release but sadly just *before* I read your mail
and fixed the bug, so it might take a while until the fix becomes
available (through ELPA at least).

Thank you for the corrections in the last Auctex version, but openning
an old file, when I typed diacritical mark using TeX command for
accented characters, I can see that the problem remains for all these
commands; examples:

m\^eme, \'etant, r\'eit\'er\'ee, S\=uryad\=asa and a lot of others...

So, nowadays, using xelatex and unicode we can avoid this... But for
people using a basic 7bits keyboard it could be nice to correct also
these commands...

Regards.

[Arash Esbati]

Post by François Patte
Thank you for the corrections in the last Auctex version, but openning
an old file, when I typed diacritical mark using TeX command for
accented characters, I can see that the problem remains for all these
m\^eme, \'etant, r\'eit\'er\'ee, S\=uryad\=asa and a lot of others...

Hi François,

looking at lshort.pdf, the list of characters to exclude would be like
this:

\begin{tabular}{@{}*{4}{cl}@{}}
\hline
\`o & \verb|\`o| & \'o & \verb|\'o|
& \^o & \verb|\^o| & \~o & \verb|\~o| \\
\=o & \verb|\=o| & \.o & \verb|\.o|
& \"o & \verb|\"o| \\
\hline
\end{tabular}

I can fix this with a patch I proposed here:

http://lists.gnu.org/archive/html/auctex/2017-04/msg00047.html

Applied locally, it looks like this: