Commit Graph

31 Commits (cca06d252121bd3233f06c17b8d9f19aaa65f6c5)

Author SHA1 Message Date
ansuz d4e6d86961 enforce asynchrony in new database offset methods
ansuz 466072d03b read files starting from the oldest known point of relevance when computing indices
ansuz d72e053560 make a note to improve stream timeout error handling
ansuz 67430de7ff Make efforts to avoid closing streams mid-read
1. Close streams when you're done with them
2. Close streams if they seem to have been abandoned
ansuz 6e57366b7f address a file descriptor leak
ansuz 46ebd7b40b remove unsupported storage configuration parameters
ansuz 5358fab615 more WIP inactive account eviction
ansuz d794e0b48f include inactive accounts in the archival script
ansuz b04133e6c3 remove some dead code
ansuz fd169ff39c close streams after 2 minutes instead of 5
Hopefully this helps to even out our memory and CPU usage.
There's also some improved labels so that IF streams time out
we will know the exact origin of the offending call.
ansuz 7fac997e93 increase some file storage timeouts related to streams
ansuz d802173325 remove an invalid line
ansuz adfd1eaaa2 avoid a silly typeError in the event of an error
ansuz bc13a21796 give fileStreams a little bit more time before closing them
ansuz c2fcba31e3 remove an XXX
ansuz e128683ffb call back with errors instead of throwing within the Storage module
yflory dc0b26da0b Merge branch 'soon' into staging
ansuz ed63d98ab0 Merge branch 'scaling-index' into soon
ansuz 590e249500 resolve merge conflict
ansuz 31c7cecaf9 add a few checks for safety
ansuz d386e223e4 simplify open/close of writeStreams
ansuz 32cd0f3c4d increase timeout value from 15s to 45. ought to be enough for anybody
ansuz 05a4e86cdb close streams whenever we finish using them. time out if necessary
ansuz 4522ffa18a compute indexes in child processes
ansuz f345998956 fix undefined reference
ansuz 30f17040ac close streams when possible, simplify some code, enforce asynchrony
ansuz e1a0daac9c possible server fixes
ansuz 32d769447a begin standardizing our method of streaming lines from files
ansuz 35eca2c5d2 pull file streaming out into its own file, leave a few notes
ansuz ff73e96cb8 reimplement the trim history fix from staging
ansuz 725d10fc60 nest storage directory inside './lib'