DivideOverflow

Evaluating Remote Javascript with CasperJs

Posted on

I needed to retrieve evaluated Javascript object from a remote page. The code on the page looked like this:

<script language="Javascript" type="text/javascript">

        var XXXGen = function(options){
                !options && (options = {});

                this.url = '';
                this.options = options;
                ....
        };

        ...

        var gen = new XXXGen({
                trackuri: 'some.uri',
                campaignId: 000000,
                programs: [{"key":"123","value":"345"}, {.....}]
        });

I was trying to scrap this webpage and retrieve 'gen' object as it was after the Javascript gets executed, with a subsequent task to enumerate the list of all 'programs' in the object.

Using CasperJs, it should have been as simple as:

var casper = require('casper').create();

casper.start('https://some.website.com/login');

casper.thenEvaluate(function() {
        .. sign in code here ..
});

casper.thenOpen('https://some.website.com/target-page', function() {
                var x = this.getGlobal('gen.options.programs');
                this.echo(x);
});

casper.run();

However, it failed with a type error:

TypeError: JSON.stringify cannot serialize cyclic structures.

From this post on StackOverflow.com, I learned that: "In PhantomJS (and thus also CasperJS), evaluate runs in a jailed environment. Only primitive objects, something you can serialize via JSON.stringify and JSON.parse is accepted."

I struggled to find a work-around until I thought about using underlying PhantomJs API which Casperjs exposes via .page class.

Bingo! From within casper.thenOpen call, using this.page.evaluate I was able to access 'gen' object as a property of 'window' object and return it back to my casperjs code:

                var programs = this.page.evaluate(function() {
                        return window.gen.options.programs;
                });

                for(var i = 0; i < programs.length; i++) {
                        console.log(programs[i].key, programs[i].value);
                }

Neat :)