Runescape Data Mining

As I try to learn various languages I will think of projects to work on. One of those languages that I thought of was Google code project. Most of them are limited javascript projects integrated in with Google's Docs. I first starting thinking about doing something in docs when I played Runescape for a monetary gain. I still get on it from time to time to see if I can improve what I have coded.

I have created a document that allows me to see what current prices are for what I am interested in at the time. For this document I have various pages that are aimed for a specific skill. Each document has algorithms for dictating what is the best way to earn money. Basically you put in how much money you are willing to risk and it will say - based on current prices - you will make this much money.

Now that may not seem to complicated, and it probably isn't "that" complicated. But at the time I had thought about what kind of information I would have liked to have seen. One of the things that bugged me the most about Runescape was that they didn't give enough detail about the grand exchange. (The grand exchange is the place where all members try to buy or sell items across all worlds.) A couple of iterations later I tried to store the information that I had gathered from Runescape.com. I had to drop the idea not because I was doing it wrong, but because the browser couldn't support the memory it took to load the data.

I am currently tracking 91 items in Runescape. They are the items I think I can make the most money on.

The google docs api requires you to create something that is more API oriented. So I created something that would be used over and over. It could also be used by multiple people by using Google's general caching system. The way it works is, it will go out and find the item's price data. Once it does, it saves it to Google's public cache. (The public cache is shared between anyone trying to use Google Doc's Spreadsheet. This means that if I am the first one to use the api, then everyone else will see the data I see for at least a specified time. I had to do this because Runescape has a limit on their api. You can only make so many requests per second to their system before it stops responding to any of them. I was first thinking that maybe I could start implementing a HEAD request system to find out when they last updated their response. I have only recently found out that their "last updated response" variable says epoch time. Which means their entire api system has been consistently crap. I have ran into various problems with their api that I have needed to sort out. It has been rather a big growing pain; but I have been able to manage it.)

One of the various problems with their api is the fact they don't return raw data types. Instead they seem to like the idea of strings. Strings aren't all bad until you want to do mathematical calculations on them. So I created a parser that would take that data and use REGEX to parse the price data out of items and set them back to the original objects' price. Some of the things they would return is 2.5k, 5.3m and 10.4b.

I don't mean to boast and place myself on a pedestal but I am proud of what I have accomplish. I am proud of the fact that I saw a problem and found a solution through code. I really like coding and think it can be very useful but sometimes it isn't the answer to everything. Doing it brute force can be the better solution.

Anyways back to the main topic... So yes, I have a Google Doc's Api built on Runescape's Api. What I also have gained from it is a doc that can calculate how much money you would make per hour based on what item you are training on/merchanting/using. So far it isn't used by anyone but my closest friend. When I took a javascript programming course at Neumont. I tried to expand on it by creating a Chrome extension that used the same Runescape Api. Basically what it would do is let you search for an item via a category; once found you could see it's historical data and then you could make a judgement call on whether or not you should buy or sell the item. It didn't update nearly as frequently as the Google Doc's Spreadsheet.

It was fun to do and it was definitely a challenge to get all the features I wanted done.

So maybe at this point you might be asking, where is the sheet. I don't play Runescape, but I want to see this piece. At this point I would like to say that I am in the process of moving it over to the new Google Spreadsheets. I have found that the preprocessing is 10x faster than the older version on the newer Google Spreadsheets. So for right now you can see the old version. Docs [Clicky]

Anyways, this is one of the many things that drive me nuts during the day. I constantly think about these things to try to improve them.

Note: Not all the code in this article is a perfect representation of what is working in production code. Most of this code has been modified for presentation.