169

How can I execute an SQL command through a shell script so that I can make it automated?

I want to restore data I have collected in a SQL file using a shell script. I want to connect to a server and restore data. The command works when executed separately via SSH command line.

This is the command I use:

mysql -h "server-name" -u root "password" "database-name" < "filename.sql"

This is the shell script code that creates the file ds_fbids.sql and pipes it into mysql.

perl fb_apps_frm_fb.pl
perl fb_new_spider.pl ds_fbids.txt ds_fbids.sql
mysql -h dbservername -u username -ppassword dbname < ds_fbids.sql

What is the correct way to do this?

Eric Leschinski
  • 146,994
  • 96
  • 417
  • 335
MUFC
  • 1,943
  • 3
  • 14
  • 13

16 Answers16

206

You need to use the -p flag to send a password. And it's tricky because you must have no space between -p and the password.

$ mysql -h "server-name" -u "root" "-pXXXXXXXX" "database-name" < "filename.sql"

If you use a space after -p it makes the mysql client prompt you interactively for the password, and then it interprets the next command argument as a database-name:

$ mysql -h "server-name" -u "root" -p "XXXXXXXX" "database-name" < "filename.sql"
Enter password: <you type it in here>
ERROR 1049 (42000): Unknown database 'XXXXXXXX'

Actually, I prefer to store the user and password in ~/.my.cnf so I don't have to put it on the command-line at all:

[client]
user = root
password = XXXXXXXX

Then:

$ mysql -h "server-name" "database-name" < "filename.sql"

Re your comment:

I run batch-mode mysql commands like the above on the command line and in shell scripts all the time. It's hard to diagnose what's wrong with your shell script, because you haven't shared the exact script or any error output. I suggest you edit your original question above and provide examples of what goes wrong.

Also when I'm troubleshooting a shell script I use the -x flag so I can see how it's executing each command:

$ bash -x myscript.sh
Bill Karwin
  • 538,548
  • 86
  • 673
  • 828
  • Thanks for a quick response. I tired by putting the password in the command line itself. The real problem is I am putting this command in .sh file and then executing this shell script. The command in the file is not executed on command line, but the same command works perfectly fine when I execute only the command on command line. – MUFC Nov 08 '11 at 19:33
  • `+ mysql -h dbservername -u user-name -ppassword dbname : No such file or directoryids.sql + $'\r' : command not found2:` This is the error message I got – MUFC Nov 08 '11 at 21:25
  • Ok, then I would infer that your current working directory is not where the ids.sql file is located. Also you may have embedded newlines in your script. – Bill Karwin Nov 08 '11 at 21:29
  • I do have new lines embedded in my shell script after every command. All that my shell script contain are 3 command line command which I dont want to run separately so i created a shell script to make them run without my intervention and I put newline afetr every commmand. Is that is causing problem ? – MUFC Nov 08 '11 at 21:43
  • best to avoid `-p` if the password is null or empty string, perhaps you can update your post? :) – James Oravec Jan 06 '15 at 23:44
  • @VenomFangs, I frequently do use `-p` even if I have a blank password. It prompts you for a password, then you just press return. – Bill Karwin Jan 07 '15 at 00:09
  • 1
    @racl101, if you want it to be more clear in your scripts, use `--password=xxxxxx` – Bill Karwin Jan 18 '15 at 16:51
  • Sorry for come so late, but I tried the suggestion to put user and password in ~/.my.cnf and the script doesn't get data from there. Any other step missing? https://gist.github.com/shadyueh/789654ba216e25f71ad8eb07de5e4315 – Shad Apr 05 '16 at 03:42
  • How to add Port number ?? – RU Ahmed Jun 15 '20 at 08:45
  • @RUAhmed, use the `port` option. You may be interested in reviewing the documentation on MySQL client options: https://dev.mysql.com/doc/refman/8.0/en/mysql-command-options.html#option_mysql_port – Bill Karwin Jun 15 '20 at 14:49
143

Use this syntax:

mysql -u $user -p$passsword -Bse "command1;command2;....;commandn"
Chris Gerken
  • 16,221
  • 6
  • 44
  • 59
Kshitij Sood
  • 1,439
  • 1
  • 9
  • 2
  • 28
    Some more details about the options from the manual: *-B* is for batch, print results using tab as the column separator, with each row on a new line. With this option, mysql does not use the history file. Batch mode results in nontabular output format and escaping of special characters. *-s* is silent mode. Produce less output. *-e* is to execute the statement and quit – wranvaud Sep 14 '16 at 11:30
  • Could it run with a heredoc? – zx1986 Aug 10 '18 at 12:04
  • 1
    @zx1986 Yes and No, to HEREDOC. Depends on how you mean to use it. Using it to replace the `"command1;command2;....;commandn"` part of this answer will not work. Using it to replace the use of the redirected file in the OP's syntax _can_ work. I've addressed that issue in my [answer](https://stackoverflow.com/a/54264982/7412956) to this question. – Chindraba Jan 19 '19 at 07:30
  • `-s` is redundant if `-B` is used – Tim Feb 05 '19 at 21:13
59

All of the previous answers are great. If it is a simple, one line sql command you wish to run, you could also use the -e option.

mysql -h <host> -u<user> -p<password> database -e \
  "SELECT * FROM blah WHERE foo='bar';"
Jericon
  • 4,992
  • 3
  • 20
  • 22
22

How to execute an SQL script, use this syntax:

mysql --host= localhost --user=root --password=xxxxxx  -e "source dbscript.sql"

If you use host as localhost you don't need to mention it. You can use this:

mysql --user=root --password=xxxxxx  -e "source dbscript.sql"

This should work for Windows and Linux.

If the password content contains a ! (Exclamation mark) you should add a \ (backslash) in front of it.

Eric Leschinski
  • 146,994
  • 96
  • 417
  • 335
Milinda Bandara
  • 542
  • 8
  • 23
10

The core of the question has been answered several times already, I just thought I'd add that backticks (`s) have beaning in both shell scripting and SQL. If you need to use them in SQL for specifying a table or database name you'll need to escape them in the shell script like so:

mysql -p=password -u "root" -Bse "CREATE DATABASE \`${1}_database\`;
CREATE USER '$1'@'%' IDENTIFIED BY '$2';
GRANT ALL PRIVILEGES ON `${1}_database`.* TO '$1'@'%' WITH GRANT OPTION;"

Of course, generating SQL through concatenated user input (passed arguments) shouldn't be done unless you trust the user input.It'd be a lot more secure to put it in another scripting language with support for parameters / correctly escaping strings for insertion into MySQL.

Li1t
  • 622
  • 6
  • 16
7

You forgot -p or --password= (the latter is better readable):

mysql -h "$server_name" "--user=$user" "--password=$password" "--database=$database_name" < "filename.sql"

(The quotes are unnecessary if you are sure that your credentials/names do not contain space or shell-special characters.)

Note that the manpage, too, says that providing the credentials on the command line is insecure. So follow Bill's advice about my.cnf.

PointedEars
  • 14,752
  • 4
  • 34
  • 33
5
mysql -h "hostname" -u usr_name -pPASSWD "db_name" < sql_script_file

(use full path for sql_script_file if needed)

If you want to redirect the out put to a file

mysql -h "hostname" -u usr_name -pPASSWD "db_name" < sql_script_file > out_file
Gus
  • 6,719
  • 6
  • 37
  • 58
vine
  • 51
  • 1
  • 1
  • @Gus, First of all thanks for the valuable comments. It worked like a charm for me. I want the output to be an excel or .csv file. how can I achieve that. Thanks in Advance. – Ash_and_Perl Nov 20 '14 at 16:37
  • @Ash_and_Perl I only edited this answer, thank vine not me, it's his answer. If you have a question of your own, *and you have tried to find a solution on your own already*, I suggest you create a question. That way you can detail what you tried, how it failed, and people can give you a full, complete answer (and get points for it!). – Gus Nov 20 '14 at 22:25
5

As stated before you can use -p to pass the password to the server.

But I recommend this:

mysql -h "hostaddress" -u "username" -p "database-name" < "sqlfile.sql"

Notice the password is not there. It would then prompt your for the password. I would THEN type it in. So that your password doesn't get logged into the servers command line history.

This is a basic security measure.

If security is not a concern, I would just temporarily remove the password from the database user. Then after the import - re-add it.

This way any other accounts you may have that share the same password would not be compromised.

It also appears that in your shell script you are not waiting/checking to see if the file you are trying to import actually exists. The perl script may not be finished yet.

  • 2
    You missed the "automated" part of the question, and temporarily removing the password is a Really Bad Idea. – PointedEars Nov 08 '11 at 19:15
  • I read it as "restore" and "automated" which means "automated but not forever". But like I said "if security is not a concern". I agree - it's a REALLY bad idea. – Sterling Hamilton Nov 08 '11 at 19:22
  • I am sorry if I have created confusion. What I meant by Automated is, I have two perl scripts which are used to generate the .sql file, but the command to dump that file in to DB is not run by the shell script, but it works absolutely file if I run that command on a command line. I want to eleminate the effort of running that command on command line and run it through the shell script itself. – MUFC Nov 08 '11 at 21:15
  • 1
    Vaibav: if you could put the actual shell script inside your question, I may be able to help further. – Sterling Hamilton Nov 08 '11 at 21:24
  • `perl fb_apps_frm_fb.pl` perl fb_new_spider.pl ds_fbids.txt ds_fbids.sql` `mysql -h dbservername -u username -ppassword dbname < ds_fbids.sql` – MUFC Nov 08 '11 at 21:30
  • Vaibav: Could you take what you just posted here AND what is contained in fb_apps_frm_fb.pl and put it in the Question's description at the top? This way others can see it. – Sterling Hamilton Nov 08 '11 at 21:34
  • Vaibav: It appears that you are writing a shell scripts that invokes a few perl scripts that generate some files. Then you are trying to use the generated file and import that into a database. Is that correct? – Sterling Hamilton Nov 08 '11 at 21:48
  • I am not getting any errors but the script just stops after executing teh first command and does not execute anything after that. – MUFC Nov 09 '11 at 18:17
  • Your (ba)sh code, if executed verbatim, would mean to build a new command from the **output** of each of the three commands, then execute the new command. Put each command on its own line and remove the backticks unless you mean to reuse the output. Also, if you have newline in your output/variable values, you need to double-quote the backticks or variable reference to preserve the newline (else it is collapsed to a single space). RTFM, really. – PointedEars Nov 10 '11 at 02:13
3

Use

echo "your sql script;" | mysql -u -p -h db_name
piet.t
  • 11,718
  • 21
  • 43
  • 52
senninha
  • 96
  • 3
3

To "automate" the process of importing the generated .sql file, while avoiding all the traps that can be hidden in trying to pass files through stdin and stdout, just tell MySQL to execute the generated .sql file using the SOURCE command in MySQL.

The syntax in the short, but excellent, answer, from Kshitij Sood, gives the best starting point. In short, modify the OP's command according to Kshitij Sood's syntax and replace the commands in that with the SOURCE command:

#!/bin/bash
mysql -u$user -p$password $dbname -Bse "SOURCE ds_fbids.sql
SOURCE ds_fbidx.sql"

If the database name is included in the generated .sql file, it can be dropped from the command.

The presumption here is that the generated file is valid as an .sql file on its own. By not having the file redirected, piped, or in any other manner handled by the shell, there is no issue with needing to escape any of the characters in the generated output because of the shell. The rules with respect to what needs to be escaped in an .sql file, of course, still apply.

How to deal with the security issues around the password on the command line, or in a my.cnf file, etc., has been well addressed in other answers, with some excellent suggestions. My favorite answer, from Danny, covers that, including how to handle the issue when dealing with cron jobs, or anything else.


To address a comment (question?) on the short answer I mentioned: No, it cannot be used with a HEREDOC syntax, as that shell command is given. HEREDOC can be used in the redirection version syntax, (without the -Bse option), since I/O redirection is what HEREDOC is built around. If you need the functionality of HEREDOC, it would be better to use it in the creation of a .sql file, even if it's a temporary one, and use that file as the "command" to execute with the MySQL batch line.

#!/bin/bash
cat >temp.sql <<SQL_STATEMENTS
...
SELECT \`column_name\` FROM \`table_name\` WHERE \`column_name\`='$shell_variable';
...
SQL_STATEMENTS
mysql -u $user -p$password $db_name -Be "SOURCE temp.sql"
rm -f temp.sql

Bear in mind that because of shell expansion you can use shell and environment variables within the HEREDOC. The down-side is that you must escape each and every backtick. MySQL uses them as the delimiters for identifiers but the shell, which gets the string first, uses them as executable command delimiters. Miss the escape on a single backtick of the MySQL commands, and the whole thing explodes with errors. The whole issue can be solved by using a quoted LimitString for the HEREDOC:

#!/bin/bash
cat >temp.sql <<'SQL_STATEMENTS'
...
SELECT `column_name` FROM `table_name` WHERE `column_name`='constant_value';
...
SQL_STATEMENTS
mysql -u $user -p$password $db_name -Be "SOURCE temp.sql"
rm -f temp.sql

Removing shell expansion that way eliminates the need to escape the backticks, and other shell-special characters. It also removes the ability to use shell and environment variables within it. That pretty much removes the benefits of using a HEREDOC inside the shell script to begin with.

The other option is to use the multi-line quoted strings allowed in Bash with the batch syntax version (with the -Bse). I don't know other shells, so I cannot say if they work therein as well. You would need to use this for executing more than one .sql file with the SOURCE command anyway, since that is not terminated by a ; as other MySQL commands are, and only one is allowed per line. The multi-line string can be either single or double quoted, with the normal effects on shell expansion. It also has the same caveats as using the HEREDOC syntax does for backticks, etc.

A potentially better solution would be to use a scripting language, Perl, Python, etc., to create the .sql file, as the OP did, and SOURCE that file using the simple command syntax at the top. The scripting languages are much better at string manipulation than the shell is, and most have in-built procedures to handle the quoting and escaping needed when dealing with MySQL.

Chindraba
  • 820
  • 1
  • 18
  • 19
3

I have written a shell script which will read data from properties file and then run mysql script on shell script. sharing this may help to others.

#!/bin/bash
    PROPERTY_FILE=filename.properties

    function getProperty {
       PROP_KEY=$1
       PROP_VALUE=`cat $PROPERTY_FILE | grep "$PROP_KEY" | cut -d'=' -f2`
       echo $PROP_VALUE
    }

    echo "# Reading property from $PROPERTY_FILE"
    DB_USER=$(getProperty "db.username")
    DB_PASS=$(getProperty "db.password")
    ROOT_LOC=$(getProperty "root.location")
    echo $DB_USER
    echo $DB_PASS
    echo $ROOT_LOC
    echo "Writing on DB ... "
    mysql -u$DB_USER -p$DB_PASS dbname<<EOFMYSQL

    update tablename set tablename.value_ = "$ROOT_LOC" where tablename.name_="Root directory location";
    EOFMYSQL
    echo "Writing root location($ROOT_LOC) is done ... "
    counter=`mysql -u${DB_USER} -p${DB_PASS} dbname -e "select count(*) from tablename where tablename.name_='Root directory location' and tablename.value_ = '$ROOT_LOC';" | grep -v "count"`;

    if [ "$counter" = "1" ]
    then
    echo "ROOT location updated"
    fi
flopcoder
  • 1,195
  • 12
  • 25
  • If you are a root user then using "sudo mysql" plainly will also work. auth_socket comes enabled by default for the root user. Was looking if there's any way of interpolating SQL queries with bash variables. And this answered it. – Blaze Nov 24 '20 at 10:04
2

An important consideration for accessing mysql from a shell script used in cron, is that mysql looks at the logged in user to determine a .my.cnf to load.

That does not work with cron. It can also get confusing if you are using su/sudo as the logged in user might not be the user you are running as.

I use something like:

mysql --defaults-extra-file=/path/to/specific/.my.cnf -e 'SELECT something FROM sometable'

Just make sure that user and group ownership and permissions are set appropriately and tightly on the .my.cnf file.

Danny
  • 56
  • 5
2

To take input from terminal using script shell and send it to mysql, write this command in shell script or bash script.

echo "Creating MySQL user and database"

echo "What is your DB_NAME?"

read db_name

echo "What is your DB_PASS?"

read db_pass

echo "What is your DB_HOST?"

read db_host

echo "What is your USER_NAME?"

read user_name

echo "What is your USER_PASS?"

read user_pass

mysql -u$db_name -p$db_pass -h$db_host -e "CREATE USER '$user_name'@'%' IDENTIFIED BY '$user_pass';GRANT SELECT ON *.* TO '$user_name'@'%';"

echo "YOU HAVE SUCCESSFULLY CREATED A USER $user_name WITH PASSWORD $user_pass"
buddemat
  • 4,552
  • 14
  • 29
  • 49
1
mysql_config_editor set --login-path=storedPasswordKey --host=localhost --user=root --password

How do I execute a command line with a secure password?? use the config editor!!!

As of mysql 5.6.6 you can store the password in a config file and then execute cli commands like this....

mysql --login-path=storedPasswordKey ....

--login-path replaces variables... host, user AND password. excellent right!

Artistan
  • 1,982
  • 22
  • 33
1
#!/bin/sh
#Procedures = update
#Scheduled at : Every 00.05 

v_path=/etc/database_jobs
v_cnt=0

MAILTO="indd@abc.in joanson@abc.in sturt@abc.in"
touch "$v_path/db_db_log.log"

#test
mysql -uusername -ppassword -h111.111.111.111 db_name -e "CALL functionName()" > $v_path/db_db_log.log 2>&1
if [ "$?" -eq 0 ]
  then
   v_cnt=`expr $v_cnt + 1`
  mail -s "db Attendance Update has been run successfully" $MAILTO < $v_path/db_db_log.log
 else
   mail -s "Alert : db Attendance Update has been failed" $MAILTO < $v_path/db_db_log.log
   exit
fi
kartavya soni
  • 91
  • 1
  • 8
0

automate code mysql in shell script

this code connection to mysql by shell

#!/bin/bash
sudo mysql -u root -p "--password=pass"

OR

In addition connect , show value one tabel

        #!/bin/bash
        sudo mysql -u root -p "--password=4310260985" "database" 
-e "select * from table order by id desc;"

Hint: -e =======> You can get this -Bse

Netwons
  • 1,170
  • 11
  • 14