Subtract smoothed data from original

Question

I wonder whether there is a way to subtract smoothed data from original ones when doing things of the kind:

plot ["17.12.2020 08:00:00":"18.12.2020 20:00:00"] 'data3-17-28.csv1' using 4:5 title 'Sensor 3' with lines, \
    '' using 4:5 smooth acsplines

Alternatively I would need to do it externally, of course.

Suntory · Answer 1 · 2020-12-30T14:09:15.760

If I well understand you would like this :

First write your smooth's data in out.csv file

set table "out.csv" separator comma
    plot 'file' u 4:5 smooth acsplines
unset table

Then this line will paste 'out.csv' to file as an appended column.You will maybe need to delete first lines using sed command (sed '1,4d' out.csv)

stats 'file' matrix

Thanks to stats we automatically get the number of column in your original data (STATS_size_x).

plot "< paste -d' ' file out.csv" u 4:($5-$(STATS_size_x+2)) w l

Could you please try this small code on your data.

theozh · Answer 2 · 2021-01-01T05:42:00.503

As @Suntory already suggested you can plot smoothed data into a table. However, keep in mind, the number of datapoints will be determined by set samples, default setting is 100 and the smoothed datapoints will be equidistant. So, if you set samples to the number of your datapoints and your data is equidistant as well, then all should be fine.

Concatenating data line by line is not straightforward in gnuplot, since gnuplot is not intended to do such operations.

The following gnuplot-only solution assumes that you have your data in a datablock $Data without headers and empty lines. If not, you could either plot it with table from file into a table named $Data or use the following approach in the accepted answer of this question: gnuplot: load datafile 1:1 into datablock

If you don't have equidistant data, you need to interpolate data, which is also not straightforward in gnuplot, see: Resampling data with gnuplot

It's up to you: either you use external tools (which might not be platform-independent) or you apply a somewhat cumbersome platform independent gnuplot-only solution.

Code:

### plot difference of data to smoothed data
reset session

$Data <<EOD
1   0
2   13
3   16
4   17
5   11
6    8
7    0
EOD

stats $Data u 0 nooutput   # get number of rows or datapoints
set samples STATS_records
set table $Smoothed
    plot $Data u 1:2 smooth acsplines
unset table

# put both datablock into one
set print $Difference
    do for [i=1:|$Data|] {
        print sprintf('%s %s',$Data[i],$Smoothed[i+4])
    }
set print
     
plot $Data u 1:2 w lp pt 7, \
     $Smoothed u 1:2 w lp pt 6, \
     $Difference u 1:($2-$4) w lp pt 4 lc "red"
### end of code

Result:

Yeah, this will work. Thanks. I finally decided to make calculations outside of Gnuplot and did them in using Numpy with Python... — Sergei, Dec 31 '20 at 15:20
@Sergei ok, if it works for you using Python. By the way, besides smoothing... maybe subtracting from the running average would be another option https://stackoverflow.com/a/56813651/7295599 — theozh, Jan 01 '21 at 05:36

Subtract smoothed data from original

2 Answers2