Processing backspace control character (^H) in real time while logging sdout to file

Question

I am working on a script to test new-to-me hard drives in the background (so I can close the terminal window) and log the outputs. My problem is in getting badblocks to print stdout to the log file so I can monitor its multi-day progress and create properly formatted update emails.

I have been able to print stdout to a log file with the following: (flags are r/w, % monitor, verbose)
sudo badblocks -b 4096 -wsv /dev/sdx 2>&1 | tee sdx.log

Normally the output would look like:
Testing with pattern 0xaa: 2.23% done, 7:00 elapsed. (0/0/0 errors)

No new-line character is used, the ^H control command backs up the cursor, and then the new updated status overwrites the previous status.

Unfortunately, the control character is not processed but saved as a character in the file, producing the above output followed by 43 copies of ^H, the new updated stats, 43 copies of ^H, etc.

Since the output is updated at least once per second, this produces a much larger file than necessary, and makes it difficult to retrieve the current status.

While working in terminal, the solution cat sdx.log && echo"" prints the expected/wanted results by parsing the control characters (and then inserting a carriage return so it is not immediately printed over by the next terminal line), but using cat sdx.log > some.file or cat sdx.log | mail both still include all of the extra characters (though in email they are interpreted as spaces). This solution (or ones like it which decode or remove the control character at the time of access still produce a huge, unnecessary output file.

I have worked my way through the following similar questions, but none have produced (at least that I can figure out) a solution which works in real time with the output to update the file, instead requiring that the saved log file be processed separately after the task has finished writing, or that the log file not be written until the process is done, both of which defeat the stated goal of monitoring progress.

Bash - process backspace control character when redirecting output to file

How to "apply" backspace characters within a text file (ideally in vim)

Thank you!

Answer 1

The main place I've run into this in real life is trying to process man pages. In the past, I've always used a simple script that post processes by stripping out the backspace appropriately. One could probably do this sort of thing in 80 character of perl, but here's an approach that handles backspace and cr/nl fairly well. I've not tested extensively, but it produces good output for simple cases. eg:

$ printf 'xxx\rabclx\bo\rhel\nworld\n' | ./a.out output
hello
world
$ cat output
hello
world
$ xxd output
00000000: 6865 6c6c 6f0a 776f 726c 640a            hello.world.

If your output starts to have a lot of csi sequences, this approach just isn't worth the trouble. cat will produce nice human consumable output for those cases.

#include <assert.h>
#include <ctype.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>

FILE * xfopen(const char *path, const char *mode);
off_t xftello(FILE *stream, const char *name);
void xfseeko(FILE *stream, off_t offset, int whence, const char *name);

int
main(int argc, char **argv)
{
        const char *mode = "w";
        char *name = strchr(argv[0], '/');
        off_t last = 0, max = 0, curr = 0;
        name = name ? name + 1 : argv[0];
        if( argc > 1 && ! strcmp(argv[1], "-a")) {
                argv += 1;
                argc -= 1;
                mode = "a";
        }
        if( argc > 1 && ! strcmp(argv[1], "-h")) {
                printf("usage: %s [-a] [-h] file [ file ...]\n", name);
                return EXIT_SUCCESS;
        }
        if( argc < 2 ) {
                fprintf(stderr, "Missing output file.  -h for usage\n");
                return EXIT_FAILURE;
        }
        assert( argc > 1 );
        argc -= 1;
        argv += 1;

        FILE *ofp[argc];
        for( int i = 0; i < argc; i++ ) {
                ofp[i] = xfopen(argv[i], mode);
        }
        int c;
        while( ( c = fgetc(stdin) ) != EOF ) {
                fputc(c, stdout);
                for( int i = 0; i < argc; i++ ) {
                        if( c == '\b' ) {
                                xfseeko(ofp[i], -1, SEEK_CUR, argv[i]);
                        } else if( isprint(c) ) {
                                fputc(c, ofp[i]);
                        } else if( c == '\n' ) {
                                xfseeko(ofp[i], max, SEEK_SET, argv[i]);
                                fputc(c, ofp[i]);
                                last = curr + 1;
                        } else if( c == '\r' ) {
                                xfseeko(ofp[i], last, SEEK_SET, argv[i]);
                        }
                }
                curr = xftello(ofp[0], argv[0]);
                if( curr > max ) {
                        max = curr;
                }
        }
        return 0;
}

off_t
xftello(FILE *stream, const char *name)
{
        off_t r = ftello(stream);
        if( r == -1 ) {
                perror(name);
                exit(EXIT_FAILURE);
        }
        return r;
}

void
xfseeko(FILE *stream, off_t offset, int whence, const char *name)
{
        if( fseeko(stream, offset, whence) ) {
                perror(name);
                exit(EXIT_FAILURE);
        }
}

FILE *
xfopen(const char *path, const char *mode)
{
        FILE *fp = fopen(path, mode);
        if( fp == NULL ) {
                perror(path);
                exit(EXIT_FAILURE);
        }
        return fp;
}

Answer 2

You can delete the ^H

sudo badblocks -b 4096 -wsv /dev/sdx 2>&1 | tr -d '\b' | tee sdx.log

Answer 3

I have found col -b and colcrt usefull, but none worked perfect for me. These will apply control characters, not just drop them:

sudo badblocks -b 4096 -wsv /dev/sdx 2>&1 | col -b | tee sdx.log