Moved all statistics to global. User pages currently only display their name and avatar; queue list also removed.
Global displays tables for two DBs, Anime and User.
- Type (TV, movie, etc)
- Standardized avg rating
- # people watch, finished, dropped, and holding
- # watch, finished, etc
- Hipster %
- Std Dev
Most of these are raw values, but some, like Std Avg gotta be computed first. Thanks to the datatables jQuery plugin, it loads the whole DB and includes sorts and searches by default. Std Avg is calculated by first standardizing everyone's ratings, finding the lowest and highest of those scores, and then shifting those standardized scores into a 1-10 scale. A show then has an average of those ratings. To make sure shows finished and rated highly by one person don't overtake a show rated slightly less high by a big group of people, the average is weighted by # people seen
Average * (1.25^(#Seen/Total#) - 0.25)
It was originally 0.3 but that was too strong and resulted in the top show only being an 8.2 or something.
This time around, I used 4 workers instead of 2, which I assumed was fine, but because I forgot to put a unique id in the people db, it didn't need to scrape that many entries. After fixing that, I found out that 4 workers were working too fast and MAL denied them for DDoS protection, so it is back to 2. I realized also that the 4th worker may not necessarily finish last, so I set each worker to set their queue to FINISHED when done and when all workers are finished, the last one done will reset the queue Interestingly, I have over twice as many odd numbered anime as even numbered, database id wise.
Everything had to be truncated again because I added some new columns, so will have to be repopulated. With 46 users, 3200 anime, and at least 14000 entries, it took about 10-15 minutes to update the database, probably from the standardizing. So I'll have to either increase update period or optimize it somehow. As for the peopleDB. I wrote an SQL query that displays top 25 people by show occurance, but in a DB with just me, it took 2 seconds for the query, which is too long. Also for some reason, I can get that query working in PHPMyAdmin, but not in PHP code. Plans to fix that soon.