2018-04-03MALstats died about 7 months ago when the measly amount of traffic it got became too much (Barely a thousand in a day). Instead of fixing the issues it had, I'd rather just start over again since its foundation was pretty shoddy to begin with, being my first project really in webdev. Now after doing a fair bit of webdev at work, I'll be taking another crack at it.
I had set up the database schema a month or two ago and started work on the queuing shortly after, but motivation is in rare quantities when you get home from coding to do additional coding. I suppose with the winter season ending, I didn't have really any shows left to watch as I was just doing seasonals, so I was able to open the flood gates and start work on Saturday, working pretty much nonstop, besides to eat, work, and sleep up until now. I started using two online APIs (In case one fails) instead of my own scraping to get show information and the bonus is that the APIs export to json, which consumes a lot memory, allowing me to get the full casting and staff on mammoth shows like One Piece that has 611 character roles, and that's only including the Japanese ones. Incidentally, they're named Jikan, and Mikan. From my testing, it's still pretty slow to get staffing data because I grabbed all the production staff, but I'm still hoping to use it in filters and graphs. While I was populating the database using people that have large lists, I figured that other people in the queue with a smaller amount of shows probably already have most of their shows in the database already, and since at this time I have not yet gotten into getting episode history, I figured it wouldn't hurt to let people get a sneak peak (There's a notice on your page if your list is not complete yet). I'm also going to try downloading the user's xml, so I shouldn't have to worry about wasting bandwidth grabbing several of them every minute (They're surprisingly hefty, mine was about 600kB).
For now the only content ready is the list and it's pretty much the same as before. I'm keeping the filters design from before, it seems pretty logical to me and I was able to make the queries powering it much more efficient than before (I don't have any numbers backing this up), I have removed a few though for the time being until I have them tracked, like rewatches and popularity. In contrast the horizontal tabs of the previous version, I've decided on a vertical side bar that'll house the username, profile picture, tabs, and an area for statistics based on the currently selected tab. I think this works pretty well for the list as I can display info like the rating distribution for the filtered list or it's percentage of the total list in a compact area. I'm not sure exactly what use it'll have for the other tabs but I'm sure that'll come when I get to designing those the pages. I previously had a graph on version one where you could select the studio or year or what not and it'd show the rating distribution, but now that's built in to the list.
The next step will be one of three:
Optimizing the VA/Staff database enough to not crash under usage by hundreds
Getting episode history without user credentials, or if not, a way that doesn't involve massive scraping