How do I append to an existing string variable inside a loop in bash?

Question

I have a simple bash script that downloads stock prices and appends them to a variable, which is then outputted:

#!/bin/bash

output=" "
for stock in GOOG AAPL
do
  price=$(curl -s "http://download.finance.yahoo.com/d/quotes.csv?s=$stock&f=l1")
  output+="$stock: $price "
done

echo "$output"

This script only displays AAPL: 524.896, the last piece of data fetched. According to whatswrongwithmyscript, there isn't anything wrong with the script, and I thought I was following this answer properly. This answer discussed a similar problem (appending to a string variable inside a loop) and suggested a different method which I used like this:

#!/bin/bash

output=" "
for stock in GOOG AAPL
do
  price=$(curl -s "http://download.finance.yahoo.com/d/quotes.csv?s=$stock&f=l1")
  output="$output$stock: $price "
done

echo "$output"

The output is still the same. I'm using bash 4.2.45 on debian jessie x64.

More info

I echoed the result in a loop to debug, and from the first script, this is what I get:

 GOOG: 1030.42
 AAPL: 524.896
 AAPL: 524.896

And the second script gives the same thing:

 GOOG: 1030.42
 AAPL: 524.896
 AAPL: 524.896

What you're seeing, I think, is `curl` not getting a stock price for Google. Have you echoed stock and price in the loop as a debugging measure? If the second technique is not working correctly, then the problem is in the data. The first should also work, but... — Jonathan Leffler, Oct 31 '13 at 00:09
@JonathanLeffler It looks like `curl` is fetching the data. See the edit; I echoed stock and price, and the data is being fetched correctly. — Michael A, Oct 31 '13 at 00:14

score 5 · Answer 1 · answered Oct 31 '13 at 00:20

I'm pretty sure that the problem is that curl is returning a carriage return and this is messing with the printing of both values. If you redirect the output of the curl command to a file and view it in vi, you'll see it's created a DOS file.

This seems to work:

#!/bin/bash

output=""
for stock in GOOG AAPL
do
  price=$(curl -s "http://download.finance.yahoo.com/d/quotes.csv?s=$stock&f=l1" | tr -d '\r')
  output+="$stock $price\n"
done

echo -e "$output"

score 5 · Accepted Answer · answered Oct 31 '13 at 00:25

When I run your script and pipe the output to od -c, the result is illuminating:

0000000       G   O   O   G   :       1   0   3   0   .   4   2  \r    
0000020   A   A   P   L   :       5   2   4   .   8   9   6  \r      \n
0000040

So you can see that it IS in fact getting all the entries and concatenating them, but its ALSO getting CR characters (the \r in the od output), which causes them to print over the top of each other when you print the string.

You can pipe the output of curl to tr -d '\r' to strip off the problematic CRs:

price=$(curl -s "...." | tr -d '\r')

I wasn't aware of the `od` command, and that's super helpful! Thanks! — Michael A, Oct 31 '13 at 03:10

How do I append to an existing string variable inside a loop in bash?

More info

2 Answers2