C# WebClient - Getting an HTML from URI instead of CSV from OBIEE

Question

Following this post:

C# WebClient - Getting an HTML from URI instead of CSV_

The problem is that the csv was compiled with javascript in the browser. In the same post, recommendation is use Casperjs but there are options using c# or how could use casperjs to get a csv from OBIEE?

Someone have an example of that?

Additional information:

I found a webpage with this links: xls, xml, csv

When i clic each link a get the .csv file for example, but if i try to use WebClient c# a got HTML with javascript functions and i understood that the page is something like a template with following functions:

an array filled with data that i need:

var datos = [];

function to fill that array:

function llenaArreglo(){ 
 var serie = new Array("12/01/1984", "18/01/1984", "DTF", "36,45%");
    datos.push(serie);

... and it repeat the same code for each data tuple (it's funny), filling the data array.

function that choose the type of download from request:

function downloadByType(){
    var type = gup("download_type", location.href);
    
    if(type=="csv"){
        downloadCsv()
    }
    if(type=="xml"){
        downloadXml()
    }
}

function to download, it seem that re-write content of the current page with the data in the choosen format:

function download(data, filename, type) {
    var a = document.createElement("a"),
    //textEncoder = new TextEncoder('utf-16');
    //var contentEncoded = textEncoder.encode([data]);
    file = new Blob([data], {type: type});
    if (window.navigator.msSaveOrOpenBlob) // IE10+
        window.navigator.msSaveOrOpenBlob(file, filename);
    else { // Others
        var url = URL.createObjectURL(file);
        a.href = url;
        a.download = filename;
        document.body.appendChild(a);
        a.click();
        setTimeout(function() {
            document.body.removeChild(a);
            window.URL.revokeObjectURL(url);  
        }, 0); 
    }
}

And the function for csv format:

function downloadCsv(){
    var dataCsv = "\ufeff"+"fecha_inicio;fecha_final;tasa_interes;valor_tasa\n";
    for(var i=0; i<datos.length;i++){
        serie = datos[i];
        var fecha_inicio = serie[0];
        var fecha_final = serie[1];
        var tasa_interes = serie[2];
        var valor_tasa = validarNulo(serie[3]);
                                            
        dataCsv += fecha_inicio+";"+fecha_final+
        ";"+tasa_interes+
        ";"+valor_tasa+"\n";
    }
    download(dataCsv, "Tasas_captacion_semanales_DTF_CDT_TCC.csv", "data:text/txt;charset=windows-1252;");
}

If i use the WebClient c# option, i only get this HTML with javascript code... i tried to use "CasperJS" but i get same result and i don't know how to call "download" function or may be get "datos" variable.

I will try in CasperJS the following code:

casper.start('URL').thenEvaluate(
function(){
    data= datos;
});

casper.run(function() {
    this.echo('my data:'    + data);
    this.exit();        });

But i thing i am wrong and i need to know how to get the last result of the webpate.

You are not getting CSV from a WebClient. You are getting an HTML Response where the body of the response contains the csv. So you are already getting HTML. — jdweng, Jun 21 '20 at 15:49
hi, i got a HTML code but it have javascript functions that generate output file (csv, xml) and i found an Array with values that use those convert functions and i could scraping it but i read in the other post that is possible use "casperjs" to pre-process (i susspose that) the URL and get directly CSV file... but i need some help to how to use "casperjs" to do that :) — Eider Falla, Jun 22 '20 at 16:01
If you are pre-processing than how are you getting data directly? You are processing the data and will not get directly. Pre-Processing and Directly as OPPOSITE. You can't be doing both. — jdweng, Jun 22 '20 at 16:05

score 0 · Answer 1 · edited Mar 13 '21 at 21:38

I resolve my trouble with CasperJS, not is the ideal solution but it's a solution. I scraped the code to get the variable "datos" (array with data):

var casper = require('casper').create();
casper.start('uri', function() {
    //this code copied from other post about casperjs
    var myVarInCasper = this.evaluate(function() {
        var myVarInBrowser;
        myVarInBrowser = datos;
        return myVarInBrowser;
    });
    var dataCsv = "\ufeff"+"col001;col002;col003;col003\n";
    //the same code, copied from HTML 
    for(var i=0; i<myVarInCasper.length;i++){
        serie = myVarInCasper[i];
        var col1 = serie[0];
        var col2 = serie[1];
        var col4 = serie[2];
        var col5 = serie[3];
        dataCsv += col1+";"+col2+";"+col3+";"+col4+"\n";
    }
    this.echo(dataCsv);
});
casper.run();

with the above code i get CSV in console and i re-direct ouput to file in Windows and create a BATCH file with command line. Something like:

casperjs.exe sample.js > result.csv

C# WebClient - Getting an HTML from URI instead of CSV from OBIEE

1 Answers1