23

Is there an HTTP client like wget/lynx/GET that is distributed by default in POSIX or *nix operating systems that could be used for maximum portability?

I know most systems have wget or lynx installed, but I seem to remember installing some Ubuntu server systems using default settings and they had neither wget or lynx installed in the base package.

I am writing a shell script for Linux (and probably Mac) to install a piece of software onto the computer. To prevent having to distribute a couple of large files, I would like to fetch these files from the internet instead of packaging in with the installer. Currently, the install script is to be distributed in a single file created from Makeself.

I'd like to avoid having the install script be over 100 MB which it would be if the files were included, and also they may not be required if the person is upgrading or re-installing the software. Maybe the most portable thing to do is include the files in the pacakage.

Right now I am just thinking of having the script check for wget, lynx, and GET, in that order and it will use whichever one it can for downloading, but I could avoid this altogether if there was a way I could download the files that would work on all systems.

EDIT:

Does anyone know much about lwp-request (GET) and its availability? This seems to be readily available on several of the systems I have checked so far, and I remember this always being around 10+ years ago going back to RedHat.

drew010
  • 68,777
  • 11
  • 134
  • 162
  • 1
    You could also use `curl` or (extreme example) `netcat` (often aliased as `nc`). However, I think `wget` should be the most universally available. If this isn't available, chances are that all the others won't be as well – Niklas B. Feb 28 '12 at 22:52
  • True `curl` is a possibility too, I figured that one would be less common than `wget` though. I didn't think about `nc`, so that could be a possibility too, I know its been around a long time. – drew010 Feb 28 '12 at 22:58
  • 4
    By the way, you can also [use bash itself](http://thesmithfam.org/blog/2006/05/23/bash-socket-programming-with-devtcp-2/) to make the HTTP request. I guess bash is even more commonly available than wget (although that particular feature might not so commonly be enabled). – Niklas B. Feb 28 '12 at 23:01
  • Cool find on the Bash `/dev/tcp` thing. If you wanna add an answer about `nc` and that bash trick, I will probably accept that. I'm starting to think that maybe I am over-analyzing this particular piece and could always just tell them to install one of those packages in the worst case. – drew010 Feb 28 '12 at 23:44
  • I've done that, but sarnold's answer is even a bit more informative :) – Niklas B. Feb 28 '12 at 23:55
  • 3
    Since you mentioned Mac: OS X comes with curl (and ftp, sftp, nc, and of course bash) but not wget. – Gordon Davisson Feb 29 '12 at 05:46

6 Answers6

17

Edit in 2019-11-04: I'm rewriting my answer to reflect the importance of ensuring that a transfer isn't tampered with while in flight. I'll leave my original answer below the rule.

I suggest using rsync over ssh to transfer your files. rsync's interface may look overwhelming, but most users may be able to pick rsync -avzP, and if you need more flexibility, rsync can adapt. Using ssh will provide integrity, authenticity, and privacy to your connection.

curl is the de facto standard for http transfers; if plain http or https are preferred, curl or tools based on curl are probably a good choice.

In my experience, tools are available about in this order:

  • wget
  • curl
  • sftp
  • ftp
  • GET (I use HEAD all the time and often forget it is just one tool in the suite)
  • tftp
  • nc (not as common as I wish)
  • socat (even less common)

The bash /dev/tcp tool is available on most systems I've used (some used dash or pdksh instead), but using echo with bash, nc, or socat is going the long-way-around for HTTP access -- you'll have to handle headers somehow, which reduces its elegance.

sarnold
  • 102,305
  • 22
  • 181
  • 238
  • Many `ftp` clients, `ncftp`, `lftp`, etc., may or may not install `ftp` symlinks; Debian-derived systems will at least populate `/etc/alternatives/ftp` with the system-preferred `ftp` client, and `/usr/bin/ftp` will point to that. – sarnold Feb 28 '12 at 23:56
  • 1
    I've found wget is available on most systems I've used however curl has generally always been something I've compiled by hand. By contrast however because SFTP is based on having SSH access, I've found that it's generally also always available. So I'd rank that higher – Mark D Apr 01 '13 at 01:29
  • @sarnold it might be helpful if you give some kind of clue about what kind of OS's your experience samples. I imagine it might be mostly unix and linux systems? But I just noticed that, on my macbook, curl is installed but not wget, which might indicate that's typical for macos users, which would mean your list would look quite different if it included the mac world (which is way bigger than the linux world). Just guessing on details for the most part here, of course. Also I just noticed your answer is from 2012 so I shouldn't hold you to anything you've said here anyway :-) – Don Hatch Oct 31 '19 at 04:06
  • @DonHatch, thanks for the ping. I'll update this answer a little. – sarnold Nov 04 '19 at 18:32
11

Official list of POSIX 7 utilities

http://pubs.opengroup.org/onlinepubs/9699919799/utilities/contents.html

The following are not present in the list:

  • wget
  • curl
  • ftp

The same goes for the LSB which essentially only guarantees the POSIX utilities.

But I do think that the POSIX C is enough to implement most of netcat functionality, so it is really a missed opportunity. E.g.: How to make an HTTP get request in C without libcurl?

Likely it is because network protocols like HTTP were deemed too specific/didn't exist at the time POSIX was still evolving, and POSIX basically froze forever. Notably, HTTPS encryption is likely not trivial to implement.

Ciro Santilli OurBigBook.com
  • 347,512
  • 102
  • 1,199
  • 985
3

gnu awk (gawk) has built in TCP handeling even on non-linux systems http://www.gnu.org/software/gawk/manual/gawkinet/html_node/

Fenn
  • 31
  • 1
3

Curl is probably even more common than wget. At least in my experience merely because more other tools depend on it. But both curl and wget are a super simple install and will be available on any system.

Manfred Moser
  • 29,539
  • 13
  • 92
  • 123
  • I agree, but when it comes to being pre-installed on a system, I don't think it is commonly installed "out of the box". Especially when we are talking about 5+ year old linux distros. – drew010 Feb 28 '12 at 23:45
  • You're right, I don't know what I'm thinking. I don't think it will be going on anything that old. Thanks for the comments – drew010 Feb 29 '12 at 03:57
  • 1
    @ManfredMoser to give an example: Debian (Stretch) comes with wget pre-installed but not with curl (and I doubt that it was any different in 2012). In my experience, wget is more often installed than curl (I wish it would be different). – JepZ Nov 10 '18 at 15:26
  • 1
    @JepZ I'm not sure "In my experience, wget is more often installed than curl" is that meaningful of a statement, other than as a statement of what subculture you happen to hang out in :-) I say that because I see that, although wget (and not curl) is installed on my offshoot-of-ubuntu linux machine, curl (and not wget) is installed on my macbook. I'm guessing the latter implies that most people in the mac world (which is way bigger than the linux world) would say "In my experience, curl is more often installed than wget". – Don Hatch Oct 31 '19 at 03:57
  • @DonHatch you are probably correct and it depends on the subculture. A few days after I wrote that comment, I saw the situation on macOS too and was quite surprised. And it is true, lately, I didn't switch a lot between distributions, so even if curl was kinda rare (in default installations) 15 years ago that situation might have changed too. – JepZ Oct 31 '19 at 13:52
1

Ready-made (auto-detect) script

I wrote a Posix-compliant Shell function, which can also be used as a command:

http_get

#!/bin/sh
## Usage: http_get <url> [filepath]
## Copyright (C) 2022 AJ ONeal <aj@therootcompany.com>
## Permission to copy and modify is granted under the CC0-1.0 license
set -e
set -u

http_get() { (
    http_get_url="${1}"
    http_get_filepath="${2:-"$(basename "${http_get_url}")"}"
    if [ -z "${http_get_url}" ]; then
        echo >&2 "Usage: http_get <url> [filepath]"
        return 1
    fi

    if command -v wget > /dev/null; then
        # wget supports resumable downloads
        if ! wget -q -c "${http_get_url}" -O "${http_get_filepath}.part"; then
            echo >&2 "failed to download ${http_get_url} to ${http_get_filepath}.part"
            return 1
        fi
    elif command -v curl > /dev/null; then
        if ! curl -fsSL "${http_get_url}" -o "${http_get_filepath}.part"; then
            echo >&2 "failed to download ${http_get_url} to ${http_get_filepath}.part"
            return 1
        fi
    fi

    # move downloaded file into file location
    mv "${http_get_filepath}.part" "${http_get_filepath}"
); }

http_get "${1:-}" "${2:-}"

Benefits

  • Safe for command or sourced function use (variables in a subshell, and namespaced)
  • Runs in "strict mode" (clean exit on error)
  • shellcheck-linted (Posix-compliant, no bashisms)
  • Safe for batch mode (silences progress bar)
  • Safe for partial & resumable downloads (prefers wget when available)

Extra features

There's an extended version that also detects if the shell is interactive (to enable the progress bar) at https://therootcompany.com/blog//posix-shell-http-request-curl-vs-wget/

coolaj86
  • 74,004
  • 20
  • 105
  • 125
1

I suggest using ftp or wget, as they are the most common in Linux distributions. The best practice might be to have your script look to see if a command is available, if not go to the next command.

Ben Ashton
  • 1,385
  • 10
  • 15