Toit Getting "Out of memory" error while posting to HTTP API

Question

I am tryng to scan BLE devices and once I find some particular vendor, post some data to HTTP server. For this purpose I'm using two different ESP32 devkits and Toit platform.

My code is very simple:

import ble
import http
import net
import device

main:
  bd := ble.Device.default

  bd.scan: | bleDevice/ble.RemoteDevice |
    print "Device detected " + bleDevice.address.stringify;
    if bleDevice.data.manufacturer_data.size == 13 and bleDevice.data.manufacturer_data[0] == 0x9a and bleDevice.data.manufacturer_data[1] == 0x08 :
      print "Teltonika sensor"
      post bleDevice
      
post sensor:
  network := net.open
  client := http.Client network
  json_payload := {
    "t": (sensor.data.manufacturer_data[4].to_float * 256 + sensor.data.manufacturer_data[5].to_float) / 100,
    "h": sensor.data.manufacturer_data[6].to_float,
    "tower" : device.name, // ESP32 Tower name 
    "mac" : sensor.address.stringify,
    "sensor" : sensor.data.name
  }

  client.post_json --host="ingest.XXXXXXXXX.eu" --path="/" json_payload

However, I often see exception:

    EXCEPTION error. 
    Out of memory
    0: tcp_connect_              system/modules/tcp.toit:176:3
    1: TcpSocket.connect         system/modules/tcp.toit:136:12
    2: TcpSocket.connect         system/modules/tcp.toit:132:12
    3: WifiNetworkInterface.tcp_connect system/components/wifi.toit:98:7
    4: NetworkInterfaceResource.tcp_connect system/components/network.toit:91:26
    5: register_network.<lambda> system/components/network.toit:26:23
    6: RpcBroker.get_handler_.<block>.<lambda> tools/rpc.toit:81:17
    7: RpcBroker.process_handler.<block>.<block> tools/rpc.toit:122:27
    8: RpcBroker.process_handler.<block> tools/rpc.toit:120:43
    9: catch.<block>             <sdk>/core/exceptions.toit:114:10
    10: catch                     <sdk>/core/exceptions.toit:112:1
    11: catch                     <sdk>/core/exceptions.toit:87:10
    12: RpcBroker.process_handler tools/rpc.toit:120:5
    13: RpcBroker.listen_.<lambda> tools/rpc.toit:117:25
    14: TaskCache_.broker_task_   <sdk>/monitor.toit:434:14
    15: TaskCache_.run.<block>.<lambda> <sdk>/monitor.toit:430:42

also:

EXCEPTION error. 
FAILED TO CONNECT
  0: Network.connect           system/kernel/network.toit:101:5
  1: NetworkInterfaceResource.open system/components/network.toit:72:34
  2: register_network.<lambda> system/components/network.toit:16:29
  3: RpcBroker.process_handler.<block>.<block> tools/rpc.toit:122:27
  4: RpcBroker.process_handler.<block> tools/rpc.toit:120:43
  5: catch.<block>             <sdk>/core/exceptions.toit:114:10
  6: catch                     <sdk>/core/exceptions.toit:112:1
  7: catch                     <sdk>/core/exceptions.toit:87:10
  8: RpcBroker.process_handler tools/rpc.toit:120:5
  9: RpcBroker.listen_.<lambda> tools/rpc.toit:117:25
 10: TaskCache_.broker_task_   <sdk>/monitor.toit:434:14
 11: TaskCache_.run.<block>.<lambda> <sdk>/monitor.toit:430:42

What is the problem here? Should it not use GC to control memory? Can I explicitly delete variables? Do I have to try/catch something here?

Florian Loitsch · Answer 1 · 2022-12-20T13:47:50.053

0

TSL connections (especially the handshake) are very resource intensive.

Unfortunately, they sometimes push Toit programs over the limit.

As a user you should not need to do anything differently (unless you have big data structures or buffers lying around that aren't necessary).

The v1.6 release (released early February 2022) has lots of improvements with respect to memory, so hopefully this error should become much less frequent.

There is also work started on a better garbage collector, which will also improve the memory situation.

Update 2022-12-20: Toit v2 (the open-source version) now has a significantly better garbage collector.

Also, I missed the fact that the client.post_json didn't do anything with the response. Ideally, the response should be drained to close the connection and release the associated resources/memory. The garbage collector will do this eventually by itself, but it might take longer than needed.

With the http-package v1.* you could do the following:

  response := client.post_json --host="ingest.XXXXXXXXX.eu" --path="/" json_payload
  while response.body.read: /* do nothing */

edited Dec 20 '22 at 13:47

answered Feb 06 '22 at 13:47

Florian Loitsch

7,698
25
30

Thanks for comment, Florian. Howerver im only using HTTP (not TLS). – Sakvojage Feb 07 '22 at 13:10
That uses less memory and shouldn't make problems. However, v1.6 has some fixes for that one as well. If you still see issues, then please file an issue (or use one of the other ways of contacting Toit). – Florian Loitsch Feb 08 '22 at 10:32
Thank You Florian, I will test it with 1.6. So far I really like ToiT - the concept is great, just a bit of a work to figure another programming language:) It would be great if there could be a simple way to change WIFI parameters thru serial, so I could do website for customers and they could launch ESP32 nodes remotely. That would help to monetize my projects big time. – Sakvojage Feb 08 '22 at 14:26
Best is to hop onto https://github.com/toitlang/toit/discussions and present your use-case there. There might be different ways to achieve your goal. – Florian Loitsch Feb 08 '22 at 15:16

Toit Getting "Out of memory" error while posting to HTTP API

1 Answers1