2

I am running Geomesa-Hbase on an EMR cluster, set up as described here. I'm able to ssh into the Master and ingest / export from there. How would I ingest / export the data remotely from for example a lambda function (preferably a python solution). Right now for the ingest part I'm running a lambda function that just sends a shell command via SSH:

c = paramiko.SSHClient()
c.connect(hostname = host, username = "ec2-user", pkey = k )
c.exec_command("geomesa-hbase ingest <file_to_ingest_on_S3> ...")

But I imagine I should be able to ingest / export remotely without using ssh. I've been looking for days for a solution but no luck so far.

Rob Wanders
  • 25
  • 2

1 Answers1

2

You can ingest or export remotely just by running GeoMesa code on a remote box. This could mean installing the command-line tools, or using the GeoTools API in a processing framework of your choice. GeoServer is typically used for interactive (not bulk) querying.

There isn't any out-of-the-box solution for ingest/export via AWS lambdas, but you could create a docker image with the GeoMesa command-line tools and invoke that.

Also note that the command-line tools support ingest and export via map/reduce job, which allows you to run a distributed process using your local install.

Emilio Lahr-Vivaz
  • 1,439
  • 6
  • 5
  • Could you some detail on how to install the command-line tools? I followed [this](https://www.geomesa.org/documentation/stable/user/hbase/install.html#setting-up-the-hbase-command-line-tools) guide and copied the hbase-site.xml from my cluster to my local geomesa-hbase_2.11-$VERSION/conf/ folder. But according to the guide, I also need to edit geomesa-hbase_2.11-$VERSION/conf/dependencies.sh, but this file I cannot find. – Rob Wanders Jan 24 '21 at 10:19
  • what version of geomesa are you using? that file should exist in newer versions of geomesa, but there are archived versions of the docs available for older versions. You can see them on the main geomesa.org page by clicking on 'documentation' – Emilio Lahr-Vivaz Jan 25 '21 at 12:54
  • Sorry I was messing around a bit and downloading different distributions of Geomesa, I am working with GeoMesa 2.4.0 right now. I have followed your advice and installed GeoServer. GeoServer is running but I have to configure it still. I installed it on the same machine where GeoMesa is running aswell. From the documentation I understand that I should provide Geoserver with the Hbase, Hadoop & Zookeeper (and some additional) JARs corresponding to my Geomesa installation by placing them into GeoServer's WEB-INF/lib directory. What would be the method to collect these JARS? – Rob Wanders Jan 25 '21 at 17:08
  • Could you post a new question, or ask in the GeoMesa [gitter](https://gitter.im/locationtech/geomesa) channel? I believe stack overflow discourages asking and answering new questions in the comments. thanks! – Emilio Lahr-Vivaz Jan 25 '21 at 21:09