Jump to content
Sign in to follow this  
fh13

Simulations using ripple-lib and historical data

Recommended Posts

12 hours ago, JoelKatz said:

The "replay" option exists to do exactly this. If you do "--a -load --ledger <hash> --replay" rippled will load the specified ledger and its predecessor, then extract the state of the predecessor and the transactions from that ledger, it will wait for a "ledger_accept" command, and then replay the close of the specified ledger;. You can use "ledger_request" in a prior run to ensure both ledgers are fully resident to ensure the replay will succeed.

This is a great way to understand precisely why and how a particular transaction had a particular result. You can replay the close of the appropriate ledger with logging at trace level and even try modifying the code to get more information or see the effects of code changes on particular transactions.

Thank you for this detailed answer. Unfortunately I couldn't find anything about this option on any official ripple web page.

And still this isn't what I am looking for. I was looking for a way to replay a whole sequence of ledgers in order to test my trading scripts with real world data. Since this is common pratice for testing automated trading software, I think there would be a real demand for this kind of option.

Yet the option you describe can be really helpful, however it doesnt really fit this requirement.

Share this post


Link to post
Share on other sites

https://wiki.ripple.com/Rippled

You can get the transactions from subsequent ledgers, resubmit them to your server and just replay them as well (submit the actual transactions and run ledger_accept, then submit the ones from the next ledger etc.)

If you submit your own orders in between, you'll change the future state, so some transactions might become invalid (imagine you buy an offer that was later actually taken by someone else). If you don't submit your transactions to the simulation and just want to watch, you might also just get all historic ledgers and analyze them as much as you want. The dataset is relatively serious in size though - depending on what you are used to work with - e.g. recent ledgers in binary (ASCII) format have about 220 MB of state.

Share this post


Link to post
Share on other sites
16 minutes ago, Sukrim said:

https://wiki.ripple.com/Rippled

You can get the transactions from subsequent ledgers, resubmit them to your server and just replay them as well (submit the actual transactions and run ledger_accept, then submit the ones from the next ledger etc.)

If you submit your own orders in between, you'll change the future state, so some transactions might become invalid (imagine you buy an offer that was later actually taken by someone else). If you don't submit your transactions to the simulation and just want to watch, you might also just get all historic ledgers and analyze them as much as you want. The dataset is relatively serious in size though - depending on what you are used to work with - e.g. recent ledgers in binary (ASCII) format have about 220 MB of state.

Thanks Sukrim.

For now I just want to "watch" these past ledgers indeed. So yeah, thats what I had in mind doing now:
running rippled and download ledgers -> export transactions and headers to some format -> load first ledger in standalone mode -> replay them from file and check hashes for integrity

Another option would be running 2 rippled instances, one loading the transactions and one applying them. Both would be pretty hardware intense so they are far from ideal...
 

Share this post


Link to post
Share on other sites
20 minutes ago, fh13 said:

Thanks Sukrim.

For now I just want to "watch" these past ledgers indeed. So yeah, thats what I had in mind doing now:
running rippled and download ledgers -> export transactions and headers to some format -> load first ledger in standalone mode -> replay them from file and check hashes for integrity

Another option would be running 2 rippled instances, one loading the transactions and one applying them. Both would be pretty hardware intense so they are far from ideal...
 

If you just watch them, then you can just request the real ledger from the rippled. There's no need to replay the transactions because they were already applied to a real ledger.

I don't know what you're trying to test, but something doesn't seem right. If you're just backtesting bot strategy, then it is probably easier/faster to mock components. You can generate test fixtures from real ripple ledgers. If you're trying to write integration tests that test "real time" behaviour of your bot, then you probably need more focused/different tests.

 

Edited by T8493

Share this post


Link to post
Share on other sites
6 minutes ago, T8493 said:

If you just watch them, then you can just request the real ledger from the rippled. There's no need to replay the transactions because they were already applied to a real ledger.

My script uses the ripple-lib-orderbook extension, which, at least from what I know, doesn't have the option to be manually updated from requested ledgers, but updates automatically on ledger closed / transction event from the ripple server. 

Thats why I hoped to find a way to replay real txn on a real rippled, because that would be the easiest way to test the ingetration of this script under normal conditions.

Maybe there is a way to request the ledgers from my Node.js code and then emitting these "update" events by myself. But I guess setting up a mock object that requests these ledgers from a rippled server and which integrates right with this orderbook extension would be quite a task.

Share this post


Link to post
Share on other sites
1 minute ago, fh13 said:

My script uses the ripple-lib-orderbook extension, which, at least from what I know, doesn't have the option to be manually updated from requested ledgers, but updates automatically on ledger closed / transction event from the ripple server. 

Thats why I hoped to find a way to replay real txn on a real rippled, because that would be the easiest way to test the ingetration of this script under normal conditions.

Maybe there is a way to request the ledgers from my Node.js code and then emitting these "update" events by myself. But I guess setting up a mock object that requests these ledgers from a rippled server and which integrates right with this orderbook extension would be quite a task.

I will have to take a look what this orderbook extension actually does.

However, you can always replace ripple-lib-orderbook with mock object. I wouldn't even try to simulate environment in which ripple-lib-orderbook operates. If you manually generate and advance rippled ledgers, the events may differ from events that were emitted in live environment.

But your real problem may be the replication of ripple-lib-orderbook results (generating test fixtures).

 

Share this post


Link to post
Share on other sites

I just had a deeper dive into the ripple-lib API as well as the rippled API and I came across the   book_offers method. I knew this method exists but I didnt realize I could request the orderbook for a specific ledger.

Now since the ripple-lib API doesnt seem to support this method directly I had to format my request by myself, but no big deal.

So for a quick test I came up with this code:

var book = rippleOrderbook.OrderBook.createOrderBook(api, {
    currency_gets: 'XRP',
    issuer_pays: 'rvYAfWj5gh67oV6fW32ZzP3Aw4Eubs59B',
    currency_pays: 'USD'
});

book.on('model', function(offers) {

    api.connection.request({
        "id": 4,
        "command": "book_offers",
        "taker_gets": {
            "currency": "XRP"
        },
        "taker_pays": {
            "currency": "USD",
            "issuer": "rvYAfWj5gh67oV6fW32ZzP3Aw4Eubs59B"
        },
        "limit": 200
    }).then(function(response){
        var offreq = response.offers;

        console.log("========================================================");
        var map = {};
        var cummulative =0;
        offers.forEach(function (off) {
            cummulative += off.taker_gets_funded/10E5;
            map[off.quality*10E5]=cummulative;
        });

        for (var key in map) {
            if (map.hasOwnProperty(key)) {
                console.log(key + ";" + map[key]);
            }
        }

        console.log("########################################################");
        map={};
        cummulative =0;
        offreq.forEach(function (off) {
            cummulative += (!!off.taker_gets_funded ? off.taker_gets_funded : off.TakerGets)/10E5;
            map[off.quality*10E5]=cummulative;
        });

        for (var key in map) {
            if (map.hasOwnProperty(key)) {
                console.log(key + ";" + map[key]);
            }
        }
    });
});

(I just realize that code formation in this forum itegrates perfect with WebStorm - cool ! )

So this codes replicates one table side of an orderbook with the cumulative size of the orders (like on ripplecharts). I compared the response of the book_offers method to the data of the orderbook extension manually and noticed it is exactly the same.

Who thought it'd be that easy :beach:  Anyways thank you everybody.

Share this post


Link to post
Share on other sites
4 hours ago, Sukrim said:

If you clean it up a bit, maybe you can upstream it to ripple-lib? Otoh it seems simple enough that it might be just useful as an example too.

Yeah its kinda messy... but if you want I can clean it up and add some comments maybe

Share this post


Link to post
Share on other sites

You seem to have garnered quite the interesting conversation here; I knew nothing about the --reply command.  Thanks for posing your questions and thanks for your contributions, I'm sure that other community developers will appreciate it @fh13, we all start from somewhere ;)

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
Sign in to follow this  

×
×
  • Create New...