Python - Connecting to AWS Neptune

Question

I have created a neptune instance in aws. How can I connect to it now?

I tried the the example given in the documentation locally from my laptop.

from gremlin_python.structure.graph import Graph
from gremlin_python.driver.driver_remote_connection import DriverRemoteConnection

graph = Graph()

g = graph.traversal().withRemote(DriverRemoteConnection('ws://my_endpoint:8182/gremlin','g'))

print(g.V().limit(2).toList())

But I get Timeout exception with the following stacktrace

File "/home/cegprakash/.virtualenvs/cegprakash-6Wq6Rd61/lib/python3.5/site-packages/gremlin_python/driver/driver_remote_connection.py", line 45, in __init__
    password=password)
  File "/home/cegprakash/.virtualenvs/cegprakash-6Wq6Rd61/lib/python3.5/site-packages/gremlin_python/driver/client.py", line 76, in __init__
    self._fill_pool()
  File "/home/cegprakash/.virtualenvs/cegprakash-6Wq6Rd61/lib/python3.5/site-packages/gremlin_python/driver/client.py", line 88, in _fill_pool
    conn = self._get_connection()
  File "/home/cegprakash/.virtualenvs/cegprakash-6Wq6Rd61/lib/python3.5/site-packages/gremlin_python/driver/client.py", line 101, in _get_connection
    self._transport_factory, self._executor, self._pool)
  File "/home/cegprakash/.virtualenvs/cegprakash-6Wq6Rd61/lib/python3.5/site-packages/gremlin_python/driver/connection.py", line 40, in __init__
    self.connect()
  File "/home/cegprakash/.virtualenvs/cegprakash-6Wq6Rd61/lib/python3.5/site-packages/gremlin_python/driver/connection.py", line 46, in connect
    self._transport.connect(self._url)
  File "/home/cegprakash/.virtualenvs/cegprakash-6Wq6Rd61/lib/python3.5/site-packages/gremlin_python/driver/tornado/transport.py", line 33, in connect
    lambda: websocket.websocket_connect(url))
  File "/home/cegprakash/.virtualenvs/cegprakash-6Wq6Rd61/lib/python3.5/site-packages/tornado/ioloop.py", line 458, in run_sync
    return future_cell[0].result()
  File "/home/cegprakash/.virtualenvs/cegprakash-6Wq6Rd61/lib/python3.5/site-packages/tornado/concurrent.py", line 238, in result
    raise_exc_info(self._exc_info)
  File "<string>", line 4, in raise_exc_info
  File "/home/cegprakash/.virtualenvs/cegprakash-6Wq6Rd61/lib/python3.5/site-packages/tornado/stack_context.py", line 316, in wrapped
    ret = fn(*args, **kwargs)
  File "/home/cegprakash/.virtualenvs/cegprakash-6Wq6Rd61/lib/python3.5/site-packages/tornado/simple_httpclient.py", line 307, in _on_timeout
    raise HTTPError(599, error_message)
tornado.httpclient.HTTPError: HTTP 599: Timeout while connecting

Is there any authentication that I'm missing for the DB to get connected?

Timeout while connecting --> Seems port is not listening (or) closed. Where is this client running? on EC2? — INVOKE Cloud, Aug 14 '18 at 15:59
I tried both local machine and on EC2. I get the same issue. — cegprakash, Aug 14 '18 at 16:48
Did you try this discussion? https://github.com/tornadoweb/tornado/issues/1400 — INVOKE Cloud, Aug 14 '18 at 17:06
The link you shared may be specific to tornado. But I do not use tornado setup anywhere. Definitely I've missed something with private network or something. I'm not sure what it is. — cegprakash, Aug 14 '18 at 17:10
You might try the connection _without_ tornado to start with. The [AWS outline](https://docs.aws.amazon.com/neptune/latest/userguide/access-graph-gremlin-python.html) should connect an EC2 with neptune without any issues. Also make sure your neptune server is actually running. — technomage, Aug 15 '18 at 19:15
Also ensure that your neptune instance shares a VPC with your EC2. — technomage, Aug 15 '18 at 19:16
Can you confirm that you have setup EC2 security group correctly? Make sure you allow http traffic at port 8182. You can simply check connectivity via http curl. See public doc https://docs.aws.amazon.com/neptune/latest/userguide/get-started-prerequisites.html#get-started-vpc-security-group, https://docs.aws.amazon.com/neptune/latest/userguide/access-graph-gremlin-rest.html — Ankit Gupta, Sep 07 '18 at 04:15
@cegprakash Do you need any other help with regards to this issue? — The-Big-K, Oct 24 '18 at 06:31
for anyone looking for a sample code, this might help: https://github.com/hardikvasa/database-journal/blob/master/code-samples/aws_neptune.py — hnvasa, Jun 16 '19 at 03:16

score 4 · Answer 1 · answered Oct 17 '18 at 05:25

Connectivity problems are generally attributed to some issue with your security group settings. This was already answered in another question [1]. Posting the response here, in case it helps.

If you are seeing timeouts while connecting to the database, the first step would be to check if you have network connectivity to the endpoint.

Try: telnet endpoint port

If you have connectivity, you would see something like this:

Trying 172.217.5.110...
Connected to endpoint (172.217.5.110).
Escape character is '^]'

If this does work, then any HTTP client should be able to connect to your database. (CURL, POSTMAN etc)

If telnet does not work, then it is almost certain that you have not configured your EC2 Security Groups correctly. The gist of what you need to do is:

Create a security Group (say 'ec2') and attach that to your EC2 client instance. By default, this security group should allow outbound connections to all IPs. If that is not the case, add it.
Create a security Group (say 'db'). In Inbound rules, add a rule that allows inbound TCP connections to your database port, and source as the security group created in #1.
Now modify your Neptune Cluster, and attach 'db' to it.
Security Group changes propagate pretty fast, so you should be able to test this using telnet.

You may find other answers that say that you need the database and the EC2 instance to be in the same security group. That is not entirely true, it is just a special case of the steps mentioned above where instead of creating 2 security groups, you can use a single security group for both - db and the client instance. From a security and design perspective, its best if you have separate security groups for your DB and your client instances.

Hope this helps.

[1] https://stackoverflow.com/a/51940587/3069919

Joshua Wolff · Answer 2 · 2019-06-30T10:07:27.147

2

AWS Neptune is only accessible from an EC2 instance (or something similar) running in the VPC in which you set up your cluster.

https://docs.aws.amazon.com/neptune/latest/userguide/security-vpc.html

If this proves to be a barrier, you can rapidly prototype using AWS Lambda, which allows access to Neptune via this tutorial.

https://docs.aws.amazon.com/neptune/latest/userguide/get-started-cfn-lambda.html

edited Jun 30 '19 at 10:07

answered Jun 30 '19 at 09:58

Joshua Wolff

2,687
1
25
42

score 1 · Answer 3 · edited May 04 '20 at 07:33

1

Make sure your EC2 and Neptune are in the same VPC.

In the Security Group allow TCP connections for port 8182.

Try changing the URL from ('ws://my_endpoint:8182/gremlin','g') to ('wss://my_endpoint:8182/gremlin','g').

It worked for me.

edited May 04 '20 at 07:33

char

2,063
3
15
26

answered May 04 '20 at 05:36

AbhiK

247
3
19

Python - Connecting to AWS Neptune

3 Answers3