0

Use case: There is a certain file located on a remote server. Instead of hard coding the path to that file in my program I would like to be prompted by the program to let me specify the path to that directory on the remote server. This will help in program portability. I am able to connect to remote server in pyCharm 4.5 professional edition. Using SFTP to connect to remote server in pyCharm. Password or keyfile is not an issue to be concerned with at least for now.

Question: the function raw_input() works for local interpreter. But what method do I use to prompt the user to enter the directory path to a file located in a remote server?

For now I am hard-coding the file path in the program like 

input_file="/home/ashish/PyCharm_proj/raw_data/all_user_and_tweets_only_raw.csv"

I also tried the following code which off course does not work when executed on the remote server

import os,sys
user_input = raw_input("Enter the path of your file (use \: ")
assert os.path.exists(user_input), "I did not find the file at,  "+str(user_input)
input_file = open(user_input,'r+')
print("Hooray we found your file!")

Similar questions are 1,2,3,4 but I could not find anything relevant that satisfies my use case. Any suggestions to solve this?

Community
  • 1
  • 1
mnm
  • 1,962
  • 4
  • 19
  • 46
  • Is this on Linux, Windows, or something else? – Paul Aug 07 '15 at 03:10
  • Here's an actually related question - http://stackoverflow.com/questions/14392432/checking-a-file-existence-on-a-remote-ssh-server-using-python – selllikesybok Aug 07 '15 at 03:12
  • @Paul the remote server is on CentOS and the host machine is on Windows 7 Professional – mnm Aug 07 '15 at 03:18
  • The name of a remote computer and a filename is insufficient to open the file. There is more complexity involved. For instance, a URL `http://some.host.com/path/to/file.html` only allows access to files accessible to the remote web server, and not arbitrary files. Linux's `open` only opens files locally, which can include NFS or SAMBA remote filesystems `mount` ed by the system administrator, but once again, not arbitrary files on arbitrary systems. `open()` doesn't open URLs, nor FTP sites, not SCP, or other ways to get to remote files. – Paul Aug 07 '15 at 03:21
  • @Paul, Thanks for your response. All I want is to be prompted for file location by the remote server. Is it seriously that difficult to implement? How about any predefined packages? Or are you suggesting that hard-coding the path to the file located on the remote server in the program is the only solution? – mnm Aug 07 '15 at 03:25
  • @Ashish All I am saying is you are missing a fileserver, or other layer, that will allow those files to be accessed remotely. On Linux you can install `samba` which would allow Windows to mount files. A Windows shared file used to look like `\\machine\some\file`, I am not familiar with Python on Windows, maybe that is acceptable? But the sharing has to be set up in advance. In a work environment sometimes this is possible. But not arbitrarily. If you need to be able to grab files from anywhere, you need to consider carefully how that happens, e.g. perhaps via the web. – Paul Aug 07 '15 at 03:29
  • @Ashish There are python modules to access remote files in various ways, e.g. ssh, ftp, scp, http (web). But they each have their own open-a-file procedure and quirks. – Paul Aug 07 '15 at 03:31
  • @selllikesybok thanks for your response but No that is not 'an actual related question'! This question in context discusses about checking the file existence on the remote server whereas in my question, I already know that the file exists on the remote server. Moreover, I am not asking to check the file existence! I am asking how to prompt for the file path where the file is located on the remote server. – mnm Aug 07 '15 at 03:31
  • @Ashish - What Paul is saying is similar to my question. What kind of protocol do you have to access this file on the remote server? That was the missing detail from your question. In my answer below, I was assuming/guessing you are trying to access a file on UNIX system via SSH, hence paramiko would work. – Maelstrom Aug 07 '15 at 03:33
  • @Maelstrom thank you for your response. The protocol is SSH. In pyCharm I have configured the remote server and remote interpreter to use SFTP. I think SFTP uses SSH. – mnm Aug 07 '15 at 03:36
  • @Ashish Well, there are python modules to open SFTP, but it raises another issue: passwords/keyfiles. Are you going to ask the human his password so you have a way to login as him to get his files? Or are you going to create a new userid on the remote machine in advance, so that your program can use a keyfile to get in, and remind the human to set permissions properly with `chmod` so that the fetcher userid can actually read the files... – Paul Aug 07 '15 at 03:40
  • @Ashish At some point I begin to wonder, wouldn't it be easier to set up filesharing, or a shared hard drive, or ask the user to copy the file manually, that have to code up all this stuff? – Paul Aug 07 '15 at 03:41
  • @Paul for now lets forget about user passwords/key files issue because the user is me and my colleague. Filesharing or a shared hard drive is not relevant in this use case context either because the file to be read are server log files to which I will be doing further processing. Therefore, the question that I asked still remains to be solved. There is an answer to it that I have to check – mnm Aug 07 '15 at 03:47
  • @Ashish - raw_input is sufficient for you to provide the (known) path to your script. For your script to do anything with that information is another matter. Perhaps your question is not clear, to any of us? – selllikesybok Aug 07 '15 at 03:48
  • @selllikesybok Thanks for your response. I have updated the question so as to make it more readable. If anything is still missing do let me know, I will rectify it. Already tried raw_input but it does not work. When executed the server does not prompt me for the path to enter. – mnm Aug 07 '15 at 03:54
  • @Ashish - See the example below. That should work for your case. Yes, SFTP and SSH shares the same authentication protocol, not the same transport though. – Maelstrom Aug 07 '15 at 04:00
  • @selllikesybok, my aplogies. I was executing the wrong program. Yes, raw_input does work just like I wanted. Thank you. But I get the following error Traceback (most recent call last): File "/home/ashish/PyCharm_proj/data_cleaning/tweet_analysis.py", line 21, in with open(input_file,'r') as myfile: TypeError: coercing to Unicode: need string or buffer, file found – mnm Aug 07 '15 at 04:03
  • @Ashish Looks like you used the same variable name for the file, and the file name. You should not be open()ing the already open input_file object. – selllikesybok Aug 07 '15 at 04:06
  • @selllikesybok, thank you for the explanation. It resolved the problem. Perhaps you can post your comment as an answer to which I will mark it as solved. Thank you once again and silly me for hunting for a package for something so trivial and making it so overtly complicated. – mnm Aug 07 '15 at 05:42

2 Answers2

2

To validate the file exists on remote host/server you need a way to connect to that host. There are many different ways depending on your situation.

  • From your description above it looks like you have ssh access to the remote system? In this case Paramiko can be used to connect via ssh and execute remote command
  • If the remote host files are published on a website via HTTP you can verify it via URL, but beware that this may not reflect the remote host full path since it depends on HTTP root.

So it really depends on your case.

If using paramiko, one of the easiest way is to use SFTP class. Something like this:

ssh_client = paramiko.SSHClient()
ssh_client.connect(host, port, user, password)
sftp = ssh_client.open_sftp()
file_stat = sftp.stat(path)           # can use stat
try:
    sftp_obj1 = sftp.file(path)   # or use file
    sftp_obj2 = sftp.open(path)   # or use open
except:
    print "Error! File not found!"
Maelstrom
  • 443
  • 2
  • 9
  • thank you for your answer. Let me check out Paramiko and I will revert back if it works for me. – mnm Aug 07 '15 at 04:06
  • for now I will stick to the suggestion given earlier to use raw_input because I do not want to complicate it. The reason is that I would have to install this library on the server and that is a treacherous task which i would rather not do. I would like to thank you for taking the pains to post your answer in such an elaborate fashion. – mnm Aug 07 '15 at 05:47
  • This was a great answer to the question we all thought was being asked. Upvoted, and encourage others to do so. – selllikesybok Aug 07 '15 at 05:55
  • Thanks guys. Yes @Ashish. I saw you did.. It was selllikesybok who was encouraging another upvote. :-) Thanks selllikesybok. – Maelstrom Aug 09 '15 at 07:16
1

So, the question was a bit unclear at first. OP was ultimately looking for a way to give input to his script about the location (known) of a file on a remote server.

OP tried raw_input but got an unspecified error.

Further discussion in comments reveals the original approach was correct for this use case. The error was in an unrelated section of code, where OP used the same identifier for the input string and the file object, resulting in trying to open the file object.

selllikesybok
  • 1,250
  • 11
  • 17