Is there a simple way to export the data from a meteor deployed app?

Question

So, for example, if I had deployed an app named test.meteor.com...

How could I easily download the data that has been collected by that app - so that I could run it locally with data from the deployed app?

Access meteor DB directly.. see this http://stackoverflow.com/a/31639624/2837412 — Nishchit, Jul 26 '15 at 17:04

Scott Anderson · Answer 1 · 2012-10-08T13:03:06.213

90

To get the URL for your deployed site at meteor.com use the command (you may need to provide your site password if you password protected it):

meteor mongo --url YOURSITE.meteor.com

Which will return something like :

mongodb://client:PASSWORD@sky.member1.mongolayer.com:27017/YOURSITE_meteor_com

Which you can give to a program like mongodump

mongodump -u client -h sky.member1.mongolayer.com:27017 -d YOURSITE_meteor_com\
          -p PASSWORD

The password is only good for one minute. For usage:

$ meteor --help mongo

edited Oct 08 '12 at 13:03

answered Sep 16 '12 at 14:31

Scott Anderson

1,666
12
12

1

Thank you! This was very helpful, except I would add that the -d should be yoursite_meteor_com. – Erick Oct 04 '12 at 21:37
Thanks, I was really looking for this. Just as a side note, the hostname for my meteor project was skybreak.member1.mongolayer.com instead of sky.member1.mongolayer.com – Dec 19 '12 at 10:41
2

Yes, important to note here that the \ on the end of the database name is meant to let you continue the command on the next line in your terminal. It is not a part of the database name and you will see an error if you write out your command all on one line. – Pathogen Feb 21 '15 at 23:26
Not working for me. After I enter `meteor mongo --url YOURSITE.meteor.com`, It asked me `username & password`. I try the username & password which I use to login to my account in meteor but no luck. – Stiger Sep 17 '15 at 04:44

score 38 · Answer 2 · answered May 05 '13 at 03:02

38

And here's how to do the opposite: (uploading your local monogo db to meteor)

https://gist.github.com/IslamMagdy/5519514

# How to upload local db to meteor:

# -h = host, -d = database name, -o = dump folder name
mongodump -h 127.0.0.1:3002 -d meteor -o meteor

# get meteor db url, username, and password
meteor mongo --url myapp.meteor.com

# -h = host, -d = database name (app domain), -p = password, folder = the path to the dumped db
mongorestore -u client -h c0.meteor.m0.mongolayer.com:27017 -d myapp_meteor_com -p 'password' folder/

answered May 05 '13 at 03:02

iMagdy

511
4
5

I'm getting the following errors any ideas ERROR: ERROR: root directory must be a dump of a single database ERROR: when specifying a db name with --db – Almog Aug 17 '13 at 19:47
@almog, be sure your `mongorestore` folder points to where the `.bson` files are. For example, with `mongodump -h 127.0.0.1:3002 -d meteor`, the path to the dumped db for `mongorestore` (executed from the same directory as the `mongodump`) is `dump/meteor/` – Donny Winston Jan 06 '14 at 18:15

jerico · Answer 3 · 2013-01-23T21:18:33.137

Based on Kasper Souren's solution I created an updated script that works with current versions of Meteor and also works when you protect your remote Meteor app with a password.

Please create the following script parse-mongo-url.coffee:

spawn = require('child_process').spawn
mongo = spawn 'meteor', ['mongo', '--url', 'YOURPROJECT.meteor.com'], stdio: [process.stdin, 'pipe', process.stderr]

mongo.stdout.on 'data', (data) ->
    data = data.toString()
    m = data.match /mongodb:\/\/([^:]+):([^@]+)@([^:]+):27017\/([^\/]+)/
    if m?
        process.stdout.write "-u #{m[1]} -p #{m[2]} -h #{m[3]} -d #{m[4]}"
    else
        if data == 'Password: '
            process.stderr.write data

Then execute it like this in a *nix shell:

mongodump `coffee parse-mongo-url.coffee`

This is incredibly useful. Fumbling around with the username and password inside of the one minute timebox where the credentials are valid to put together a correct mongodump or mongorestore command gets old very quickly. — Pathogen, Feb 21 '15 at 23:24

score 6 · Answer 4 · answered Apr 11 '15 at 20:14

I have created a tool, mmongo, that wraps all the Mongo DB client shell commands for convenient use on a Meteor database. If you use npm (Node Package Manager), you can install it with:

npm install -g mmongo

Otherwise, see README.

To back up your Meteor database, you can now do:

mmongo test.meteor.com dump

To upload it to your local development meteor would be:

mmongo restore dump/test_meteor_com

And if you accidentally delete your production database:

mmongo test.meteor.com --eval 'db.dropDatabase()'   # whoops!

You can easily restore it:

mmongo test.meteor.com restore dump/test_meteor_com

If you'd rather export a collection (say tasks) to something readable:

mmongo test.meteor.com export -c tasks -o tasks.json

Then you can open up tasks.json in your text editor, do some changes and insert the changes with:

mmongo test.meteor.com import tasks.json -c tasks --upsert

Github, NPM

I just wanted to add that in order to avoid [“system.users: not authorized” error](http://docs.mlab.com/backups/) you can also do this: `mmongo test.meteor.com dump --excludeCollection=system.users` — Giwan, Mar 06 '16 at 15:56

mna · Answer 5 · 2012-06-14T01:54:09.237

5

I suppose your data is in a mongodb database, so if that's the case, the question is more mongo-related than meteor. You may take a look at mongoexport and mongoimport command line tools.

Edit (for example):

mongoexport -h flame.mongohq.com:12345 -u my_user -p my_pwd -d my_db -c my_coll

You need to install mongodb on your computer to have this command line tool, and obviously you need your mongodb informations. In the above example, I connect to MongoHQ (flame.mongohq.com is the host, '12345' is the port of your mongo server), but I don't know which Mongo host is actually used by the meteor hosting. If you tried the Meteor examples (TODOs, Leaderboard, etc.) locally, chances are you already installed Mongo, since it uses a local server by default.

edited Jun 14 '12 at 01:54

answered Jun 14 '12 at 00:15

mna

22,989
6
46
49

Not sure if I can run mongo command line on the meteor server. Does anyone know? – Greg Mercer Jun 14 '12 at 01:41
In fact, you run it from your computer, but you specify the cloud-hosted mongo information (host, port, probably user & pwd). I'll try to give you an example in a few minutes. – mna Jun 14 '12 at 01:45
I edited the answer, hopefully this helps you. The command is pretty straightforward, I showed the most useful/common arguments, by default it outputs to stdout, you can redirect it to a file or specify it directly as an argument. mongoexport --help is well detailed. – mna Jun 14 '12 at 01:55

score 3 · Answer 6 · answered Feb 11 '14 at 01:34

Here is another solution in bash

#! /bin/bash
# inspired by http://stackoverflow.com/questions/11353547/bash-string-extraction-manipulation

# http://www.davidpashley.com/articles/writing-robust-shell-scripts/
set -o nounset
set -o errexit
set -o pipefail
set -x

# stackoverflow.com/questions/7216358/date-command-on-os-x-doesnt-have-iso-8601-i-option
function nowString {
    date -u +"%Y-%m-%dT%H:%M:%SZ"
}

NOW=$(nowString)

# prod_url="mongodb://...:...@...:.../..."
prod_pattern="mongodb://([^:]+):([^@]+)@([^:]+):([^/]+)/(.*)"
prod_url=$(meteor mongo katapoolt --url | tr -d '\n')
[[ ${prod_url} =~ ${prod_pattern} ]]
PROD_USER="${BASH_REMATCH[1]}"
PROD_PASSWORD="${BASH_REMATCH[2]}"
PROD_HOST="${BASH_REMATCH[3]}"
PROD_PORT="${BASH_REMATCH[4]}"
PROD_DB="${BASH_REMATCH[5]}"
PROD_DUMP_DIR=dumps/${NOW}
mkdir -p dumps

# local_url="mongodb://...:.../..."
local_pattern="mongodb://([^:]+):([^/]+)/(.*)"
local_url=$(meteor mongo --url | tr -d '\n')
[[ ${local_url} =~ ${local_pattern} ]]
LOCAL_HOST="${BASH_REMATCH[1]}"
LOCAL_PORT="${BASH_REMATCH[2]}"
LOCAL_DB="${BASH_REMATCH[3]}"

mongodump --host ${PROD_HOST} --port ${PROD_PORT} --username ${PROD_USER} --password ${PROD_PASSWORD} --db ${PROD_DB} --out ${PROD_DUMP_DIR}
mongorestore --port ${LOCAL_PORT} --host ${LOCAL_HOST} --db ${LOCAL_DB} ${PROD_DUMP_DIR}/${PROD_DB}

Would be killer if remote url was an argv of the bash script! Thanks a lot. — gabrielhpugliese, Feb 16 '14 at 15:24
Does not work under Git Shell, as it seems the libregex shipped with msysgit does not support the =~ operator. — steph643, May 26 '15 at 11:30

the · Answer 7 · 2015-03-05T18:03:01.113

meteor-backup is by far the easiest way to do this.

sudo npm install -g meteor-db-utils
meteor-backup [domain] [collection...]

As of March 2015 you still need to specify all collections you want to fetch though (until this issue is resolved).

Stuff from the past below

I'm doing

mongodump $(meteor mongo -U example.meteor.com | coffee url2args.cfee)

together with this little coffeescript, with a mangled extension in order not to confuse Meteor, url2args.cfee:

stdin = process.openStdin()
stdin.setEncoding 'utf8'
stdin.on 'data', (input) ->
  m = input.match /mongodb:\/\/(\w+):((\w+-)+\w+)@((\w+\.)+\w+):27017\/(\w+)/
  console.log "-u #{m[1]} -h #{m[4]} -p #{m[2]} -d #{m[6]}"

(it would be nicer if meteor mongo -U --mongodumpoptions would give these options, or if mongodump would accept the mongo:// URL)

score 2 · Answer 8 · answered Apr 22 '15 at 11:14

# How to upload local db to meteor:

# -h = host, -d = database name, -o = dump folder name
mongodump -h 127.0.0.1:3001 -d meteor -o meteor

# get meteor db url, username, and password
meteor mongo --url myapp.meteor.com

# -h = host, -d = database name (app domain), -p = password, folder = the path to the dumped db
mongorestore -u client -h http://production-db-a2.meteor.io:27017 -d myapp_meteor_com -p 'password' folder/

While uploading local db to remote db, having an assertion Exception

shubham@shubham-PC:$ mongorestore -u client -h http://production-db-a2.meteor.io:27017 -d myapp_meteor_com -p my_password local/
2015-04-22T16:37:38.504+0530 Assertion failure _setName.size() src/mongo/client/dbclientinterface.h 219
2015-04-22T16:37:38.506+0530 0xdcc299 0xd6c7c8 0xd4bfd2 0x663468 0x65d82e 0x605f98 0x606442 0x7f5d102f8ec5 0x60af41 
 mongorestore(_ZN5mongo15printStackTraceERSo+0x39) [0xdcc299]
 mongorestore(_ZN5mongo10logContextEPKc+0x198) [0xd6c7c8]
 mongorestore(_ZN5mongo12verifyFailedEPKcS1_j+0x102) [0xd4bfd2]
 mongorestore(_ZN5mongo16ConnectionStringC2ENS0_14ConnectionTypeERKSsS3_+0x1c8) [0x663468]
 mongorestore(_ZN5mongo16ConnectionString5parseERKSsRSs+0x1ce) [0x65d82e]
 mongorestore(_ZN5mongo4Tool4mainEiPPcS2_+0x2c8) [0x605f98]
 mongorestore(main+0x42) [0x606442]
 /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5) [0x7f5d102f8ec5]
 mongorestore() [0x60af41]
terminate called after throwing an instance of 'mongo::AssertionException'
  what():  assertion src/mongo/client/dbclientinterface.h:219
Aborted (core dumped)

score 1 · Answer 9 · answered Apr 07 '13 at 21:23

I made this simple Rakefile to copy the live db to local.

To restore the live db to my local machine I just do...

rake copy_live_db

Replace myapp with the name of your meteor.com - e.g myapp.meteor.com.

require 'rubygems'
require 'open-uri'

desc "Backup the live db to local ./dump folder"
task :backup_live_db do
  uri = `meteor mongo myapp --url`
  pass = uri.match(/client:([^@]+)@/)[1]
  puts "Using live db password: #{pass}"
  `mongodump -h meteor.m0.mongolayer.com:27017 -d myapp_meteor_com -u client -p #{pass}`
end


desc "Copy live database to local"
task :copy_live_db => :backup_live_db do
  server =  `meteor mongo --url`
  uri = URI.parse(server)
  `mongorestore --host #{uri.host} --port #{uri.port} --db meteor --drop dump/myapp_meteor_com/`
end

desc "Restore last backup"
task :restore do
  server =  `meteor mongo --url`
  uri = URI.parse(server)
  `mongorestore --host #{uri.host} --port #{uri.port} --db meteor --drop dump/myapp_meteor_com/`
end

score 0 · Answer 10 · edited May 23 '17 at 12:08

To use an existing local mongodb database on your meteor deploy myAppName site, you need to dump, then restore the mongodb.

Follow the instructions above to mongodump (remember the path) and then run the following to generate your 'mongorestore' (replaces the second step and copy/pasting):

CMD=meteor mongo -U myAppName.meteor.com | tail -1 | sed 's_mongodb://\([a-z0-9\-]*\):\([a-f0-9\-]*\)@\(.*\)/\(.*\)_mongorestore -u \1 -p \2 -h \3 -d \4_'

then

$CMD /path/to/dump

From Can mongorestore take a single url argument instead of separate arguments?

score 0 · Answer 11 · answered Jun 08 '15 at 11:05

I think you can use a remotely mounted file system via sshfs and then rsync to synchronize the mongodb's folder itself or your entire Meteor folder I believe as well. This is like doing an incremental backup and potentially more efficient. It's possible to use the same solution for sending changes of your code, etc. so why not get you database changes back at the same time too?! (killing 2 birds with 1 stone)

score 0 · Answer 12 · answered Aug 27 '15 at 08:28

Here is a simple bash script that lets you dump your database from meteor.com hosted sites.

#!/bin/bash

site="rankz.meteor.com"
name="$(meteor mongo --url $site)"
echo $name

IFS='@' read -a mongoString <<< "$name"

echo "HEAD: ${mongoString[0]}"
echo "TAIL: ${mongoString[1]}"

IFS=':' read -a pwd <<< "${mongoString[0]}"

echo "${pwd[1]}"
echo "${pwd[1]:2}"
echo "${pwd[2]}"


IFS='/' read -a site <<< "${mongoString[1]}"

echo "${site[0]}"
echo "${site[1]}"


mongodump -u ${pwd[1]:2} -h ${site[0]} -d ${site[1]}\
          -p ${pwd[2]}

Is there a simple way to export the data from a meteor deployed app?

12 Answers12

Linked