Write results into a file using CasperJS

Question

How do I create a file in the file system and place the contents of this.getPageContent() inside it?

score 70 · Accepted Answer · answered Oct 11 '12 at 18:28

70

var fs = require('fs');
fs.write(myfile, myData, 'w');

for saving daily scrapes I do:

var currentTime = new Date();
var month = currentTime.getMonth() + 1;
var day = currentTime.getDate();
var year = currentTime.getFullYear();
var myfile = "data-"+year + "-" + month + "-" + day+".html";

answered Oct 11 '12 at 18:28

Henry

1,374
2
14
24

4

huh? and since when can you use node modules inside casperjs? – abbood Apr 08 '13 at 05:37
36

@abbood `fs` is a [PhantomJS module](https://github.com/ariya/phantomjs/wiki/API-Reference-FileSystem), not a NodeJS module. – Chaser324 May 10 '13 at 19:46
3

Just wanted to comment that I import Node modules in to Casper scripts by creating a blank module file and module.exports the require call to the Node script. I can import these proxy modules in to a casper script and use node modules without any problems. Even in caspers test environment. – Kylee Jan 27 '14 at 17:50
1

The title of this question needs to be reworded. You're dumping page HTML into a file not writing results. – Darwin Allen Nov 12 '15 at 20:48
You could also put `console.log(this.getPageContent());` in your code and redirect the output to a file. But if you do that it has to only console.log the page content or you will get all the messages you logged to standard out in the file along with the content. – Dec 03 '15 at 10:03

iChux · Answer 2 · 2014-11-10T07:39:20.360

You could also append to a text file using the method below

    var casper = require('casper').create();
    var fs = require('fs');
    var fname = new Date().getTime() + '.txt';
    var save = fs.pathJoin(fs.workingDirectory, 'nwaomachux', fname);

    casper.start('http://www.po3w.com/', function() {
        fs.write(save, this.getTitle() + '\n', 'w');
    });

    casper.thenOpen('http://phantomjs.org', function() {
        fs.write(save, this.getTitle(), 'a');
    });

    casper.run();

If the folder, nwaomachux, doesn't yet exist, it'll be automatically created for you.
If you saved the file as save.js, run it from PhantomJS with the following command on a Linux PC

./phantom casperjs/bin/bootstrap.js --casper-path=casperjs --cli save.js

score 6 · Answer 3 · edited Aug 26 '14 at 07:05

Here is a helper function that you can use to add this functionality to the casper object.

/**
 * Save page markup to a file. Respect an existing savePageContent function, if
 * casper.js core introduces one.
 * 
 * @param String targetFile
 *   A target filename.
 * @return Casper
 */
casper.savePageContent = casper.savePageContent || function(targetFile) {
  var fs = require('fs');
  var f  = require('utils').format;

  // Get the absolute path.
  targetFile = fs.absolute(targetFile);
  // Let other code modify the path.
  targetFile = this.filter('page.target_filename', targetFile) || targetFile;
  this.log(f("Saving page html to %s", targetFile), "debug");
  // Try saving the file.
  try {
    fs.write(targetFile, this.getPageContent(), 'w');
  } catch(err) {
    this.log(f("Failed to save page html to %s; please check permissions", targetFile), "error");
    this.log(err, "debug");
    return this;
  }

  this.log(f("Page html saved to %s", targetFile), "info");
  // Trigger the page.saved event.
  this.emit('page.saved', targetFile);

  return this;
};

It is helpful to note that 'fs' in this case is not the Node JS FileSystem object, but rather a PhantomJS module.

score 5 · Answer 4 · edited Dec 28 '13 at 14:35

A full 'then' function, that scrap data from a site, returns a json and store it to a file"myFile" should looks like this:

casper.then(function paso2() {
var jsonStr = this.evaluate(function(){
    var puntos = {};
    puntos.alafecha = document.querySelector('div.cont_item_productos_puntos > p.txt_negro').textContent;
    puntos.totales = document.querySelector('ul.lista_prod_puntos > li.ppuntos_1 > span.ppuntos_2').textContent;
    return JSON.stringify(puntos);
});
console.log("this is a response in json format: "+json);
fs.write('myFile.json', jsonStr, 'w');
});

Write results into a file using CasperJS

4 Answers4

Linked