0

I have the following demo code written in Sequelize. I'd like to pull a list of Endpoint entities get a list of EndpointCall entities grouped by 1 minute time buckets. I can't seem to be able to solve this is Sequelize with a findAll with a nested table:

const Sequelize = require('sequelize');
const sequelize = new Sequelize(process.env.MYSQL_DATABASE, process.env.MYSQL_USER, process.env.MYSQL_PASSWORD, {
  host: process.env.MYSQL_HOST,
  dialect: 'mysql',
  logging: false,
  benchmark: false,
});

const model = {
    Endpoint: sequelize.define('endpoint', {
        id:                 { type: Sequelize.INTEGER,          autoIncrement: true,  primaryKey: true },
        name:               { type: Sequelize.STRING('256'),    allowNull: false },
    }),

    EndpointCall: sequelize.define('endpoint_call', {
        endpoint_id:        { type: Sequelize.INTEGER,          allowNull: false },
        date:               { type: Sequelize.DATE,             allowNull: false },
        response_time:      { type: Sequelize.DOUBLE,           allowNull: true },
    }),
    
    init: function () {
        this.Endpoint.hasMany(this.EndpointCall, { foreignKey: 'endpoint_id' });
        return sequelize.sync();
    },
};

model.init()
.then(async () => {
    var result = await model.Endpoint.findAll({
        include: [{
            model: model.EndpointCall,
            required: false,
            attributes: [
                [Sequelize.literal(`FROM_UNIXTIME(ROUND(UNIX_TIMESTAMP(date)/60)*60)`), 'time_bucket'],
                [Sequelize.literal(`AVG(response_time)`), 'avg_response_time']
            ],
            group: 'time_bucket'
        }],
    });
    console.log(result);
});

This is the closest I got to but this still throws the following error:

In aggregated query without GROUP BY, expression #1 of SELECT list contains nonaggregated column 'database.endpoint.id'; this is incompatible with sql_mode=only_full_group_by

I don't necessarily want to configure the MySQL server a non-standard way, therefore I was trying to play with the group clause but with no luck:

group: [ 'id', 'name', 'time_bucket' ]

Moving the group clause to the top level table also didn't help.

UPDATE:

  1. Tried group: [ 'endpoint_call.time_bucket' ] on the top level

Result:

SequelizeDatabaseError: Unknown column 'endpoint_call.time_bucket' in 'group statement'
  1. Tried group: [ 'endpoint_calls.time_bucket' ] on the top level (with the -s at the end)

Result:

SequelizeDatabaseError: Unknown column 'endpoint_calls.time_bucket' in 'group statement'

WORKAROUND:

Thanks to Emma in the comments I found a workaround. I still thing that Sequelize should solve this internally, I'm publishing it here in case someone else gets stuck.

The point is removing sql_mode=only_full_group_by in a local session by executing this before the aggregated query:

SET sql_mode=(SELECT REPLACE(@@sql_mode,'ONLY_FULL_GROUP_BY',''));

The full code is here:

model.init()
.then(async () => {
    await sequelize.query(`SET sql_mode=(SELECT REPLACE(@@sql_mode,'ONLY_FULL_GROUP_BY',''));`);
    var result = await model.Endpoint.findAll({
        include: [{
            model: model.EndpointCall,
            required: false,
            separate: true,
            attributes: [
                [Sequelize.literal(`FROM_UNIXTIME(ROUND(UNIX_TIMESTAMP(date)/60)*60)`), 'time_bucket'],
                [Sequelize.literal(`AVG(response_time)`), 'avg_response_time']
            ],
            group: [ 'endpoint_id', 'time_bucket' ],
        }],
    });
    console.log(result);
});
adamsfamily
  • 1,746
  • 19
  • 37
  • how about `group: ['endpoint_call.time_bucket']` at top level? – Emma Jun 09 '21 at 01:30
  • @Emma Tried it, got another error, I will update my original post for better code formatting. – adamsfamily Jun 09 '21 at 08:11
  • Sorry for the wrong advice in the previous one. I think the original code you have is fine, however, the issue is related to mysql setting. Here is the link to the issue. https://stackoverflow.com/q/41887460/2956135 – Emma Jun 09 '21 at 18:38
  • The solution worked for me is https://stackoverflow.com/a/66776167/2956135 – Emma Jun 09 '21 at 18:39
  • Got it! My only question is, what are the negative impacts of modifying this configuration variable? Do we know why MySQL has decided to change it to OFF by default? – adamsfamily Jun 10 '21 at 08:03
  • This explains the effect of the config in detail. https://dev.mysql.com/doc/refman/5.7/en/group-by-handling.html And if you would like to keep default config which I think totally reasonable, you can try `any_value` query or locally enable the config. For `any_value` query, check the link and look for `any_value`. For temporary enable the config, I haven't tried but `SET LOCAL sql_mode` or `SET SESSION sql_mode` looks promising. ref: https://dev.mysql.com/doc/refman/5.7/en/set-variable.html – Emma Jun 10 '21 at 16:22
  • 1
    @Emma Did some research. The `any_value` approach looks clean enough for me, sadly the list of columns in the query are constructed by Sequelize and I'd need to manually override with the list of fields from the model. The `SET LOCAL` approach seems less invasive to me and it works! I'm going to put it into my original question, feel free to suggest that as an answer if you are happy with that. IMHO Sequelize should do a much better job by handling this out-of-box (it should simply group by all entities in the model). – adamsfamily Jun 11 '21 at 05:32
  • Just curious, your `sql_mode` value back to original after the current session (ie in the next request)? I agree with you on wish Sequelize can have ability to handle this groupby case. – Emma Jun 11 '21 at 05:51
  • @Emma (Sorry for the late answer) Yes, I just tried it in a for cycle and I confirm that it's enough to set the `sql_mode` once per session. However, bearing in mind that the connection to MySQL server might be lost and Sequelize might be configured with a reconnect policy (in my case it is) I wouldn't bet on this to keep working. I rather run the SET sql_mode command before each query that requires it. – adamsfamily Jun 15 '21 at 16:33

0 Answers0