Some of you might have experienced problems to reach the CSI:FingerID web service on Thursday, April 12, 2018. The reason is: After a “relatively quiet” Wednesday with “only” 120k compound queries (irony warning), the CSI:FingerID web service had to handle a real-world stress test on Thursday, with 260k compound queries submitted on a single day. That is 3 compound queries per second on average.
This brought our job database into some trouble: The database runs on a redundant twin server and is regularly mirrored between the two servers, in case one of them is hosed. Unfortunately, the log files used for mirroring jobs became too big, stalling the web server.
The problem resided until Friday 13th noon. Bad luck.
We are sorry if this caused any inconvenience. All systems should be up and running again.
As a courtesy towards other users: In case you want to submit complete databases with hundreds of runs, it would be great if you could distribute your computations over the course of several days. Also, you might want to contact us in advance, so that we do not accidentally block you.
By the way: We have obviously passed 1.5 million compound queries after these two days.