The main risk is that the session identifier effectively becomes the password if it never expires. Anyone with access to the session identifier could record this offline, and then use this at a later time to login to the application. For example, somebody could copy the session token from the cookie with brief access to a user's machine.
A mitigation for this is to routinely rotate session identifiers. Maybe you could have an AJAX request that fires off every 10 minutes and gets a new token for the current session - even with standard session expiration times (e.g. 10-20 minutes), this would be enough to keep the session alive so that it does not time-out before the form is submitted.
Brute forcing is not an issue: As long as the session identifier has enough entropy, then there is very little risk of this being brute forced. OWASP guidance here on selecting a strong method for session identifier generation.
More on the performance side than security side is that if you're storing objects in memory for each session, then eventually memory will fill as the number of sessions increase.
Another risk with long sessions is that any CSRF or XSS vulnerabilities have a long exposure time for exploitation. A short session timeout would mitigate any attack if the user visited a malicious site targeting your app, because the user would not be authenticated. Even with persistent login, this would be mitigated if you had a long term "refresh token" with a short term (i.e. session) "access token" if your site was sufficiently locked down (e.g. it only allows a request with CSRF protection itself to exchange a refresh token for an access token).
For example if there was an CSRF vulnerability:
[User] --> [Attacker's Site] --> [Your site]
Browser --> Malicious Page builds form for your site --> Submits form via AJAX
With an endless session timeout, this attack will succeed if the user has ever visited your site with their browser.
However if you had two session tokens: A refresh token and an access token and you required an access token to submit the form, this would prevent the attack. As the access token can only be retrieved by a same-site request, making sure that the handler for this request is sufficiently protected against CSRF would mitigate other vulnerabilities that may be present on your site.
Therefore if you must make the session infinite, use a different token that must be exchanged in order to authenticate with your website.
See this post for how to implement "remember me" functionality (aka our refresh token). The drawback is that you must implement the refresh token to access token logic yourself and require that the user must re-send any requests again with the access token (which can be implemented using client-side logic with JavaScript).