8bee95d13d
improve error reporting in GET_HISTORY
5 years ago
e05891184e
fix regression in maxWorkers
5 years ago
b56c73be6e
delegate more work from getOlderHistory to the worker
5 years ago
fd169ff39c
close streams after 2 minutes instead of 5
...
Hopefully this helps to even out our memory and CPU usage.
There's also some improved labels so that IF streams time out
we will know the exact origin of the offending call.
5 years ago
bac8ef3c6e
lint compliance
5 years ago
4e57e390da
write tasks inside of workers
5 years ago
ba6faca02e
make the number of workers configurable
5 years ago
0465f31a45
add a FIXME note
5 years ago
04ab7f538a
recover from worker faults and unify worker types
5 years ago
e0a6852b79
correctly serialize an error log
5 years ago
9ed82640bf
Merge branch 'worker-queue' into staging
5 years ago
e8b1fcf710
solve some cases where crashing workers could result in an invalid state for the queue
5 years ago
9ed26cfeb0
lint compliance
5 years ago
dc0460780f
Merge branch 'soon' into staging
5 years ago
74567e0cf4
return process memory usage in GET CACHE STATS RPC
5 years ago
e3f42cbc20
Allow config.defaultStorageLimit to be zero bytes
5 years ago
89262cd29e
resolve silly conflict and merge staging
5 years ago
e527867e2e
queue blob and channel deletions per-user
5 years ago
bd3e18d1a8
only cache a user's pinned channels if there are no errors when loading them
5 years ago
bef18a9320
hotfix
5 years ago
7fac997e93
increase some file storage timeouts related to streams
5 years ago
95965c1dee
keep a parallel implementation of the pin loader to validate the new one
5 years ago
a57bf94058
remove XXX note
5 years ago
c53675c9d5
WIP worker rpc call queue
5 years ago
d802173325
remove an invalid line
5 years ago
2b3f6e3464
Merge branch 'soon' into staging
5 years ago
adfd1eaaa2
avoid a silly typeError in the event of an error
5 years ago
32406a5995
Merge branch 'master' into staging
5 years ago
834c96a4fc
update comments and remove some notes
5 years ago
ed5d97f899
WIP history-keeper fixes
5 years ago
eddbe80eef
Merge branch 'soon' into staging
5 years ago
3d8e78a268
fix overly broad config interpretation for daily check
5 years ago
e6ec891d9a
improve admin rpc for diagnosing memory used by the cache
5 years ago
77961e3954
trim lookup tables for channels without checkpoints
5 years ago
db9eaa8d4f
Merge branch 'soon' into staging
5 years ago
6d0dee979a
allow admins to inspect index/metadata caches
5 years ago
cbd3547814
remove log statements
5 years ago
9058a59555
reassign db tasks if the responsible worker fails
5 years ago
172823c954
lint compliance
5 years ago
b5649707d1
export 'tryParse' command
5 years ago
64b0879984
improve logging for parse errors
5 years ago
3f86b6141e
rename worker processes
5 years ago
d8a88cb4ca
run expiration tasks in a worker instead of the main process
5 years ago
b0179eaad9
drop XXX note
5 years ago
9dbd32758a
improve worker logging and move blob deletion to worker processes as well
5 years ago
5f69fc18d0
suppress some noisy errors
5 years ago
5f2d7c8dcf
increase worker rpc wait time before timeout
5 years ago
33e8e65507
handle errors in the server's workers
5 years ago
a4c8039cc7
improve error handling with rpc response API
5 years ago
cb53bd1c15
lint compliance
5 years ago
e15196bd53
Merge branch 'reconnect' into staging
5 years ago
14feef1757
calculate pin list checksums in a worker
5 years ago
70a0d4efb4
move more database reads into the database worker
5 years ago
bc13a21796
give fileStreams a little bit more time before closing them
5 years ago
4ba36a9173
load user pins in the database worker
5 years ago
50e8893b24
move the 'getOlderHistory' call into the database worker
5 years ago
471e374533
compute metadata in the same child process that builds indexes
5 years ago
479b76f848
lint compliance
5 years ago
c2fcba31e3
remove an XXX
5 years ago
967ca6afa9
suppress ENOENT errors when checking empty chat channels
5 years ago
40251948d4
check authenticated rpc signatures in separate threads
5 years ago
0e09c73a60
Merge branch 'master' into soon
5 years ago
a2facc4a0d
Merge pull request #513 from 7adietri/patch-1
...
Add shutdown() to non-disk logger
5 years ago
e128683ffb
call back with errors instead of throwing within the Storage module
5 years ago
1240f4a2a5
Merge branch 'master' into soon
5 years ago
90899aa2f4
start reviewing and removing XXX notes
5 years ago
fe14399e67
Merge branch 'whiteboard' into soon
5 years ago
2ef0af29e5
guard against index worker rpc responses for a txid with no supplied callback
5 years ago
ee8d5c9c36
fix the premiumUploadSize validation to check the correct attribute
5 years ago
dc0b26da0b
Merge branch 'soon' into staging
5 years ago
a190af079d
Add shutdown() to non-disk logger
...
The script `evict-inactive.js` calls `Log.shutdown()` at the end, which currently fails if logging to disk has been disabled via `logPath`.
5 years ago
ed63d98ab0
Merge branch 'scaling-index' into soon
5 years ago
590e249500
resolve merge conflict
5 years ago
31c7cecaf9
add a few checks for safety
5 years ago
cea9705bbe
Merge branch 'aggressive-stream-closing' into staging
5 years ago
faa133aab8
remove an XXX
5 years ago
d386e223e4
simplify open/close of writeStreams
5 years ago
32cd0f3c4d
increase timeout value from 15s to 45. ought to be enough for anybody
5 years ago
05a4e86cdb
close streams whenever we finish using them. time out if necessary
5 years ago
4522ffa18a
compute indexes in child processes
5 years ago
f8ad649b45
[style] bail out early to avoid nesting
5 years ago
019e5e708b
wrap workers in a function scope and add a validateMessage method to HK's Env
5 years ago
5467e1ffac
replace ad-hoc response handler with Util.response
5 years ago
9e85a1411e
abstract the logic around worker choice out of message handling
5 years ago
631ea54b49
lint compliance
5 years ago
67dd4a608c
Merge remote-tracking branch 'origin/scaling' into scaling2
5 years ago
b2fcde87d8
implement an admin command to update the default storage limit without a restart
5 years ago
923616aef0
Use each process in order instead of using a random one
5 years ago
f345998956
fix undefined reference
5 years ago
fb0eb1b20c
Use more subprocesses
5 years ago
0d636dabc9
Check signature for history keeper in a different process
5 years ago
30f17040ac
close streams when possible, simplify some code, enforce asynchrony
5 years ago
2a9780abcf
lint compliance
5 years ago
578042154e
Better reconnect after invalid lastKnownHash
5 years ago
2d47e7e2cf
implement an admin rpc to fetch ulimit data
5 years ago
81c7416095
lint compliance
5 years ago
f644c24362
add an admin-rpc to count the number of open files
5 years ago
1230057970
disable IS_PAD_PINNED
5 years ago
e1a0daac9c
possible server fixes
5 years ago
bc034d95a3
enforce asynchrony in 'gethistoryOffset'
5 years ago
ansuz