Enhance isatty() for GCC for use with GNU make?

[Paul Smith]

Hi all. In GNU make 4.0+ a new capability was introduced which allows
GNU make to separate output for individual jobs when run in parallel, to
avoid corrupted output with multiple commands writing to stdout at the
same time.

In order to make this work, of course, GNU make must capture the output
of the commands it invokes then dump them out after ensuring it has sole
write access to the terminal. This means that when this feature is
enabled the isatty() function will always return false, _even if_ the
output will really be sent to a TTY eventually. This causes all
utilities that make invokes that check for terminal settings to behave
incorrectly (as if they were not printing to the terminal).

GNU make 4.1+ will set an environment variable MAKE_TERMOUT to a
non-empty value if it (make) believes that the output it finds will
eventually be sent to a terminal (that is, if the top-most make
invocation was printing to a terminal on stdout). Ditto for
MAKE_TERMERR and stderr. This environment variable will be not present
or have an empty value if either (a) this new feature is not used or (b)
make thinks that stdout/stderr is not a tty.

For tools, like gcc, which are often invoked by make I wonder if it
would be worthwhile to provide a special variation of isatty() which
checked this env.var. first before calling isatty(). Maybe something
like:

int
is_a_terminal (int fd)
{
char const *v = NULL;
if (fd == STDOUT_FILENO)
v = "MAKE_TERMOUT";
else if (fd == STDERR_FILENO)
v = "MAKE_TERMERR";

if (v)
{
v = getenv (v);
if (v && v[0] == '\0')
return 1;
}

return isatty (fd);
}

(untested). Then replace calls to isatty() with calls to this function
instead.

Of course, people can always change their makefiles to add something
like this to their GCC invocations:

$(if $(MAKE_TERMERR),--fdiagnostics-color=always)

but this is actually not a very robust solution (how does a user turn it
off?), and it doesn't change things like terminal width (number of
columns) computation, etc.

It would be nice if this were handled automagically, although I agree it
introduces a bit of (unseemly?) chumminess between GCC and GNU make.

[Ian Lance Taylor]

Post by Paul Smith
For tools, like gcc, which are often invoked by make I wonder if it
would be worthwhile to provide a special variation of isatty() which
checked this env.var. first before calling isatty().

Sounds reasonable to me.

Ian

[Ángel González]

Post by Paul Smith
Hi all. In GNU make 4.0+ a new capability was introduced which allows
GNU make to separate output for individual jobs when run in parallel, to
avoid corrupted output with multiple commands writing to stdout at the
same time.
In order to make this work, of course, GNU make must capture the output
of the commands it invokes then dump them out after ensuring it has sole
write access to the terminal. This means that when this feature is
enabled the isatty() function will always return false, _even if_ the
output will really be sent to a TTY eventually. This causes all
utilities that make invokes that check for terminal settings to behave
incorrectly (as if they were not printing to the terminal).
GNU make 4.1+ will set an environment variable MAKE_TERMOUT to a
non-empty value if it (make) believes that the output it finds will
eventually be sent to a terminal
(...)
but this is actually not a very robust solution (how does a user turn it
off?), and it doesn't change things like terminal width (number of
columns) computation, etc.
It would be nice if this were handled automagically, although I agree it
introduces a bit of (unseemly?) chumminess between GCC and GNU make.

Wouldn't be preferible for make to preload such fake isatty() to those
programs?
Or even better, create a pseudoterminal in which to run the child
processes. You
would only need as many pts as jobs, which seems reasonable.

[Paul Smith]

Post by Ángel González

Post by Paul Smith
It would be nice if this were handled automagically, although I agree it
introduces a bit of (unseemly?) chumminess between GCC and GNU make.

Wouldn't be preferible for make to preload such fake isatty() to those
programs? Or even better, create a pseudoterminal in which to run the
child processes. You would only need as many pts as jobs, which seems
reasonable.

I do agree that it would be nice to have a single solution and avoid
adding special-case code to a wide variety of user programs.
Unfortunately, it's not so simple for GNU make. First recall GNU make
is highly portable: all different manner of UNIX-based systems including
pretty ancient ones, DOS, Windows, VMS, even AmigaOS still works. Of
course not all those support parallel builds so not all are relevant
here. Still, enough to be problematic.

Preload involves a significant increase in GNU make deployment
complexity (switching from a single executable "make" that can be copied
anywhere and run without any problems, to multiple files which need to
be copied together and where GNU make needs to be able to locate the
preload shared library). And building a shared library adds a lot of
complexity to configuring/building make itself; currently I have no need
of libtool for example. I'm not excited by the prospect :-).

For pseudo-terminals it's actually as many pts as jobs _PER MAKE
INVOCATION_, which if you use recursive make can be as many as one make
invocation per job, so for jobs = N it could be (N-1)^2 pts. I suppose
it might be possible to share the pt slave FDs through the jobserver
pipe, as long as we were sure no FD was >255, rather than creating new
ones for every instance of make. It would be tricky to ensure that all
the other pt FDs were closed before exec'ing the child (if the child is
not a recursive make).

On the other hand it's not unheard of on the GNU make mailing lists to
hear from people using it on systems with many hundreds of cores and
very large -j. And, it so happens that those massively parallel systems
are exactly the ones where the new ordered output feature is most useful
(although I admit it is probably less likely they would use color output
since they're very likely writing to log files anyway).

And we haven't even touched on folks using "-j" alone with no limit...

[Ángel González]

Post by Paul Smith

Post by Ángel González

Post by Paul Smith
It would be nice if this were handled automagically, although I agree it
introduces a bit of (unseemly?) chumminess between GCC and GNU make.

Wouldn't be preferible for make to preload such fake isatty() to those
programs? Or even better, create a pseudoterminal in which to run the
child processes. You would only need as many pts as jobs, which seems
reasonable.

I do agree that it would be nice to have a single solution and avoid
adding special-case code to a wide variety of user programs.
Unfortunately, it's not so simple for GNU make. First recall GNU make
is highly portable: all different manner of UNIX-based systems including
pretty ancient ones, DOS, Windows, VMS, even AmigaOS still works. Of
course not all those support parallel builds so not all are relevant
here. Still, enough to be problematic.
Preload involves a significant increase in GNU make deployment
complexity (switching from a single executable "make" that can be copied
anywhere and run without any problems, to multiple files which need to
be copied together and where GNU make needs to be able to locate the
preload shared library).

Note it would be a weak dependency. It could even be distributed as a
separate
package named eg. libfaketty
If libfaketty is not , (or the host doesn't support library preloading),
the programs
would simply behave as if not using a tty.

Post by Paul Smith
And building a shared library adds a lot of
complexity to configuring/building make itself; currently I have no need
of libtool for example. I'm not excited by the prospect :-).

It wouldn't need things like fallback to use a static library, but I
don't know if there
would be room to simplify the build procedure. I'm not experienced in
libtool, either.

Post by Paul Smith
For pseudo-terminals it's actually as many pts as jobs _PER MAKE
INVOCATION_, which if you use recursive make can be as many as one make
invocation per job, so for jobs = N it could be (N-1)^2 pts.

Wouldn't that mean that you can end up with (N-1)^2 make instances?
That seems contradictory with the -j N goal (I am probably missing
something).

Post by Paul Smith
I suppose
it might be possible to share the pt slave FDs through the jobserver
pipe, as long as we were sure no FD was>255, rather than creating new
ones for every instance of make. It would be tricky to ensure that all
the other pt FDs were closed before exec'ing the child (if the child is
not a recursive make).

Looks funny. Surely would be hard to do right.

Post by Paul Smith
On the other hand it's not unheard of on the GNU make mailing lists to
hear from people using it on systems with many hundreds of cores and
very large -j. And, it so happens that those massively parallel systems
are exactly the ones where the new ordered output feature is most useful
(although I admit it is probably less likely they would use color output
since they're very likely writing to log files anyway).
And we haven't even touched on folks using "-j" alone with no limit...

Maybe a mixed approach of using a limited number of pts is warranted.
It seemed much easier on the surface. :)