NodeJs Runescape API
It's been 4 years that this project has been around, and it still finds ways to make my mind spin. The project was originally written just in excel using data queries. I moved it to Google spreadsheets using similar commands. After I found that Google cached the results I decided to make my own functions to get the data. Finally after reaching the limitations of a Google app script only being able to run for 5 minutes, I have ported the project over to NodeJS using mongodb while still using google spreadsheets as the view.
Colten Rouska
Status
Well at this very moment I have become a little frustrated with the project. In the past I was frustrated with not understanding the language. Also not understanding what Google did to wrap Javascript to make Google App Script. At this point I feel like I am proficient enough to be blocked by it. Nope, this time I am completely and utterly at the whim of the Runescape Api. Here is the deal.
The app I was trying to make was for my own interest. I wanted a better way of tracking items and their prices. I wanted a good way to show that if Iron Ore and Coal was going up in price that a Steel bar would also go up. This would also affect Cannonballs because every single one of these items are related. So if I am a mass producer in the game and I want to make an item that would give me the most amount of profit without spending a lot of cash, I could look at these recipes and find out what is the best profit margin.
This kind of pattern also happens in the real world stock market. Sometimes you can find a stock that starts to rise in price. Then another company that also benefits from the stock of another company will also rise. I have always been interested in the stock market and I feel like Runescape is a really good free platform for practicing my analytical skills.
Problem:
The Runescape API is unuseable past simple queries. At this current moment I am tracking 122 items of interest. The problem comes in when the Runescape api starts to block requests that come in too fast and for too long. I have read online that if you want to query a lot of items, a good 5 second delay and groups of 50-60 itemsets seems to work. So I set a query to get all items from the db starting at 3 second delay to see how far it gets. I got about 300 items before I was blocked. I tried 4 seconds; still blocked only a few more items in. 5 seconds still produces the same result.
Previously I was using another api that would let you pass in multiple item id's to query, then you would get all the items back at once. However that api endpoint went down, as I had feared it might one day.
Solutions:
- I may be able to contact Jagex to see if they would take off the limit for my specific circumstance. The chances of that happening are slim but there has been some light on this in the past. Reddit link to analytics opportunity from Jagex
- I can figure out a way to make the requests distribute across multiple IP Addresses. This would cost, the first thought is to make a Raspberry Pi that I can put at my parents to send requests from. Haha But that only gives me two IP Addresses.
- Tracking only most active items. I had this thought the other day where I can mark an Item to track for a certain period of time. This would allow me to focus on a subgroup of items and not need to track all of them. This isn't as efficient as I want to detect items that may be affecting other item prices.
Well I guess this can be saved for a later date. I can't say if this project is going to die or not, but I can say that it has taught me a lot. But until that happens, expect more info to appear.