Npgsql Exception while reading from stream, Postgres

Question

I am getting this error intermittently in my code. Sometimes it happens time-after-time-after-time. Sometimes it happens 1-out-of-10 times. I am not doing anything unique or special in my SQL unlike the other poster on StackOverflow who was doing a COPY command. All I am doing is SELECTs.

Here is the stack trace:

Exception while reading from stream
at Npgsql.ReadBuffer.Ensure(Int32 count, Boolean dontBreakOnTimeouts)
at Npgsql.NpgsqlConnector.DoReadMessage(DataRowLoadingMode  dataRowLoadingMode, Boolean isPrependedMessage)
at Npgsql.NpgsqlConnector.ReadMessageWithPrepended(DataRowLoadingMode dataRow LoadingMode) 
at Npgsql.NpgsqlConnector.ReadMessage(DataRowLoadingMode dataRowLoadingMode) 
at Npgsql.NpgsqlConnector.ReadExpecting[T]() 
at Npgsql.NpgsqlDataReader.NextResultInternal() 
at Npgsql.NpgsqlDataReader.NextResult() 
at Npgsql.NpgsqlCommand.Execute(CommandBehavior behavior) 
at Npgsql.NpgsqlCommand.ExecuteDbDataReaderInternal(CommandBehavior behavior) 
at Npgsql.NpgsqlCommand.ExecuteDbDataReader(CommandBehavior behavior) 
at System.Data.Common.DbCommand.ExecuteReader() 
at Npgsql.NpgsqlCommand.ExecuteReader() 
at JBetaFinder.Program.portfolioSimulation(String beginResult, String endResult) in c:\Users\j\Documents\Visual Studio 2013\Projects\JBetaFinder\JBetaFinder\Program.cs:line 571

Any suggestions on how to avoid this error? Is this a problem with Npgsql and postgres?

Here is my SQL Statement that seems to be the most problematic:

select leg1.trade_date, sum(p.qty) as totalqty, max(gas.net_change)*10000 as avggaschange,  
            sum(((leg1.settlement_price - leg2.settlement_price) - (leg3.settlement_price - leg4.settlement_price))*qty*1000000) as spread_value_weight
            from quant_portfolio p
            inner join (select distinct trade_date, hub, product, strip, settlement_price, net_change
                            from public.icecleared_gas where contract = 'H') leg1
                            on p.leg1 = leg1.strip
            inner join (select distinct trade_date, hub, product, strip, settlement_price, net_change
                            from public.icecleared_gas where contract = 'H') leg2
                            on p.leg2 = leg2.strip and leg1.trade_date = leg2.trade_date                
            inner join (select distinct trade_date, hub, product, strip, settlement_price, net_change
                            from public.icecleared_gas where contract = 'H') leg3
                            on p.leg3 = leg3.strip and leg1.trade_date = leg3.trade_date                
            inner join (select distinct trade_date, hub, product, strip, settlement_price, net_change
                            from public.icecleared_gas where contract = 'H') leg4
                            on p.leg4 = leg4.strip and leg1.trade_date = leg4.trade_date  
            inner join (select distinct trade_date, hub, product, strip, contract, settlement_price, net_change
                            from public.icecleared_gas where contract = 'H') gas
                            on gas.strip = (select min(leg1) from quant_portfolio where commodity = 'NG') and gas.trade_date = leg1.trade_date                         
            where p.commodity = 'NG'
            AND (leg1.trade_date>='xxx' and leg1.trade_date<='yyy')
            group by leg1.trade_date
            order by leg1.trade_date

I tried re-arranging the SQL to take out the sub-SELECTS and make them all joins; didn't help, same error.

Here is the C# code calling Npgsql:

query = new NpgsqlCommand(getFullQuantPortBeta.ToString().Replace("xxx", beginResult.ToString()).Replace("yyy", endResult.ToString()), conn);
            dr = query.ExecuteReader();//code does not get past this line!
            beta = 0;
            while (dr.Read())
            {
                baselineData.Add(double.Parse(dr[2].ToString()));
                responseData.Add(double.Parse(dr[3].ToString()));
                if (baselineData.Count > 3)
                {
                    Tuple<double, double> result = MathNet.Numerics.LinearRegression.SimpleRegression.Fit(baselineData.ToArray(), responseData.ToArray());
                    beta = result.Item2 * BETA_MULT;
                    Console.WriteLine("WEIGHT BETA = " + beta);
                }
            }
            dr.Close();

Welcome to Stack Overflow, please can you post the C# where you're calling Npgsql? — Dr Rob Lang, Nov 01 '16 at 16:46
Your exception report is incomplete: you don't post the type of the exception, nor any inner exceptions (would probably contain the real problem). Please add those to your question. — Shay Rojansky, Nov 01 '16 at 17:05

score 39 · Answer 1 · edited Jan 02 '20 at 17:47

39

conn = new NpgsqlConnection("Server=myserver;
User Id=postgres;
Password=somepw;
Database=somedb;
Pooling=false;
Timeout=300;
CommandTimeout=300");

I added the CommandTimeout property to my connection string and it seems to be working now. Weird exception for a timeout error...

edited Jan 02 '20 at 17:47

Broken_Window

2,037
3
21
47

answered Nov 01 '16 at 17:17

RDS_JAF

911
1
8
10

"Weird exception for a timeout error" - the timeout error appears in the Inner Exception, probably "System.TimeoutException: Timeout during reading attempt" – Click Ok Jul 01 '21 at 17:18

score 10 · Answer 2 · edited Jun 12 '19 at 13:14

10

Please try the setting

KeepAlive = 300

The number of seconds of connection inactivity before Npgsql sends a keepalive query. Set to 0 (the default) to disable.

See the documentation.

edited Jun 12 '19 at 13:14

Laurenz Albe

209,280
17
206
263

answered Jun 12 '19 at 12:49

RafaelSantos

101
1
2

score 1 · Answer 3 · answered Mar 27 '23 at 12:41

1

Check if you forgot to use await before calling method. In my case it was

advisorRepository.UpdateAsync(entity, cancellationToken);

without await

answered Mar 27 '23 at 12:41

Akhmed

1,141
4
21
49

Npgsql Exception while reading from stream, Postgres

3 Answers3