ansuz
fbfb25bf29
lint compliance
4 years ago
ansuz
100b417646
guard against several serverside typeErrors
...
and warn in cases where they would have occurred
4 years ago
ansuz
e8428a2a73
prevent a case of multiple callbacks
4 years ago
ansuz
fbc9edd795
rename latest admin commands and implement metadata getter/remover
4 years ago
ansuz
488ec93ece
allow expert admins to get and clear cached channel indices
4 years ago
ansuz
faa7ebf399
Fix some bugs with integrated eviction
...
1. implement SET_LAST_EVICTION as an admin command, not a decree
2. expect a return value from Env.evictInactive and expose it via Env.evictionReport
4 years ago
ansuz
ac322c8e82
make the eviction script call back with accumulated statistics
4 years ago
ansuz
a29c5641b8
introduce some artificial delays in the eviction script
...
...and reduce the margin of error in its bloom filters
4 years ago
ansuz
90f046f896
prevent long-running worker tasks from timing out
4 years ago
ansuz
6e57366b7f
address a file descriptor leak
4 years ago
ansuz
a2c0d2165b
implement a few more admin decrees
4 years ago
ansuz
abd84665ae
lint compliance
4 years ago
ansuz
6ec5171518
add support for changing a few more Env parameters at runtime
4 years ago
ansuz
adb988058d
stop logging dev/fresh mode
4 years ago
ansuz
f2ec9cbe33
load premium and customLimits to avoid evicting them even if inactive
4 years ago
ansuz
45b063e378
suppress irrelevant errors from Decrees.load
4 years ago
ansuz
854e4c06ff
report lastEviction in an admin INSTANCE_STATUS
4 years ago
ansuz
471a9a33ac
take note of a minor error with pin RPCs
4 years ago
ansuz
f86409e1ba
take note of which Environment variables should be made mutable via decree
4 years ago
ansuz
6d13a785c6
downgrade a non-critical XXX to a FIXME
4 years ago
ansuz
46ebd7b40b
remove unsupported storage configuration parameters
4 years ago
ansuz
369c92c01d
initialize Env from server and deduplicate several attributes
4 years ago
ansuz
b5d3a10dc2
guard against possible typeErrors from unvalidated config input
4 years ago
ansuz
964aa2bb79
validate custom limits provided via config at launch
4 years ago
ansuz
f4f803ccd9
reuse Keys.canonicalize to accept any pubkey format
4 years ago
ansuz
914139dbfa
lint compliance
4 years ago
yflory
51e1b9b043
Fix custom limit not applied when using a raw edPublic
4 years ago
ansuz
145386e944
suppress an inconsequential error
4 years ago
ansuz
aa35e092f7
update an inaccurate comment
4 years ago
ansuz
928b7e3583
implement GET_LIMITS admin rpc
...
...and label stored usage limits according to their source
4 years ago
ansuz
5029281a53
implement RM_QUOTA decree
4 years ago
ansuz
af47b2cab1
implement ADD_QUOTA admin decree
4 years ago
ansuz
998f2267c4
ensure the path to the decree file exists before attempting to write to it
4 years ago
ansuz
c2918ac888
fix inverted conditional
4 years ago
ansuz
b62d4387a6
drop support for SET_DEFAULT_STORAGE_LIMIT
...
it's redundant now that we support persistent admin decrees
and it was never exposed via the UI anyway
4 years ago
ansuz
0c5db31fbc
refactor quota logic for external usage
4 years ago
ansuz
bd6234c5bc
lint compliance and some notes
4 years ago
ansuz
1781ee2585
implement 'UPDATE_DEFAULT_STORAGE' as a decree
4 years ago
ansuz
7aa7d5978f
implement persistence of config changes from admin panel
4 years ago
ansuz
fcfcb3dca6
fix history trim for channels with access lists
4 years ago
ansuz
d509fd8b8f
downgrade severity of several XXX notes
4 years ago
ansuz
1ba080739b
Merge branch 'X' into staging
4 years ago
ansuz
cfcfe2f65f
enable APIs for XLSX export in firefox
4 years ago
yflory
e698241ee9
Merge branch 'historyOO' into staging
4 years ago
ansuz
04d1bc9cd7
archive inactive data in one of the server's workers
4 years ago
ansuz
6ada5fd751
always consider premium accounts to be active
4 years ago
ansuz
5358fab615
more WIP inactive account eviction
4 years ago
ansuz
a9b95865ba
Merge branch 'staging' into inactive-accounts
4 years ago
ansuz
d9219f4ecc
add a note for later
4 years ago
ansuz
5749b6fd7b
add an admin RPC to archive documents
4 years ago
ansuz
d794e0b48f
include inactive accounts in the archival script
4 years ago
yflory
25606b92bf
lint compliance
4 years ago
yflory
4f147d4fd2
Add support for onlyoffice history
4 years ago
yflory
b31707098a
Improve getHistoryRange to support 'toHash'
4 years ago
ansuz
95778cc7c9
respond with ENOENT instead of E_NO_OWNERS when archiving a file that does not exist
4 years ago
ansuz
b04133e6c3
remove some dead code
4 years ago
ansuz
a21c3372c0
avoid a potential type error
4 years ago
yflory
04af753a81
lint compliance
5 years ago
ansuz
03be102ce4
deduplicate some code by reusing the client definition on the server
5 years ago
ansuz
ebcc9a069b
add server-side support for the new format of public signing keys
5 years ago
ansuz
48deced859
fix serverside index corruption
5 years ago
ansuz
5b2929a6fc
suppress some useless info from some commonly logged errors
5 years ago
ansuz
9f1f01f3b4
disable worker task queueing and address a probable memory leak
5 years ago
ansuz
b0af6b5488
fix an incorrect condition when checking for available server workers
...
Also, log when the RPC queue has a backlog and when it is drained
5 years ago
ansuz
4037270a23
add a FIXME note for a sketchy implementation in history keeper
5 years ago
ansuz
02347d134a
queue signature validation per-channel to ensure correct ordering
5 years ago
ansuz
9975c27e84
increase timeout from one minute to three and decrease the number of parallel jobs per worker
5 years ago
ansuz
488f876ad6
lint compliance
5 years ago
ansuz
d35b015c3f
log some info on successful file uploads
5 years ago
ansuz
37d3995ac1
unify format of console output with stored logs
5 years ago
ansuz
8a5d012edf
add special error handling for a few special serverside errors
5 years ago
ansuz
8bee95d13d
improve error reporting in GET_HISTORY
5 years ago
ansuz
e05891184e
fix regression in maxWorkers
5 years ago
ansuz
b56c73be6e
delegate more work from getOlderHistory to the worker
5 years ago
ansuz
fd169ff39c
close streams after 2 minutes instead of 5
...
Hopefully this helps to even out our memory and CPU usage.
There's also some improved labels so that IF streams time out
we will know the exact origin of the offending call.
5 years ago
ansuz
bac8ef3c6e
lint compliance
5 years ago
ansuz
4e57e390da
write tasks inside of workers
5 years ago
ansuz
ba6faca02e
make the number of workers configurable
5 years ago
ansuz
0465f31a45
add a FIXME note
5 years ago
ansuz
04ab7f538a
recover from worker faults and unify worker types
5 years ago
ansuz
e0a6852b79
correctly serialize an error log
5 years ago
ansuz
9ed82640bf
Merge branch 'worker-queue' into staging
5 years ago
ansuz
e8b1fcf710
solve some cases where crashing workers could result in an invalid state for the queue
5 years ago
ansuz
9ed26cfeb0
lint compliance
5 years ago
ansuz
dc0460780f
Merge branch 'soon' into staging
5 years ago
ansuz
74567e0cf4
return process memory usage in GET CACHE STATS RPC
5 years ago
Martin Alexander Neumann
e3f42cbc20
Allow config.defaultStorageLimit to be zero bytes
5 years ago
ansuz
89262cd29e
resolve silly conflict and merge staging
5 years ago
ansuz
e527867e2e
queue blob and channel deletions per-user
5 years ago
ansuz
bd3e18d1a8
only cache a user's pinned channels if there are no errors when loading them
5 years ago
ansuz
bef18a9320
hotfix
5 years ago
ansuz
7fac997e93
increase some file storage timeouts related to streams
5 years ago
ansuz
95965c1dee
keep a parallel implementation of the pin loader to validate the new one
5 years ago
ansuz
a57bf94058
remove XXX note
5 years ago
ansuz
c53675c9d5
WIP worker rpc call queue
5 years ago
ansuz
d802173325
remove an invalid line
5 years ago
ansuz
2b3f6e3464
Merge branch 'soon' into staging
5 years ago
ansuz
adfd1eaaa2
avoid a silly typeError in the event of an error
5 years ago
yflory
32406a5995
Merge branch 'master' into staging
5 years ago
ansuz
834c96a4fc
update comments and remove some notes
5 years ago
ansuz
ed5d97f899
WIP history-keeper fixes
5 years ago
ansuz
eddbe80eef
Merge branch 'soon' into staging
5 years ago
ansuz
3d8e78a268
fix overly broad config interpretation for daily check
5 years ago
ansuz
e6ec891d9a
improve admin rpc for diagnosing memory used by the cache
5 years ago
ansuz
77961e3954
trim lookup tables for channels without checkpoints
5 years ago
ansuz
db9eaa8d4f
Merge branch 'soon' into staging
5 years ago
ansuz
6d0dee979a
allow admins to inspect index/metadata caches
5 years ago
ansuz
cbd3547814
remove log statements
5 years ago
ansuz
9058a59555
reassign db tasks if the responsible worker fails
5 years ago
ansuz
172823c954
lint compliance
5 years ago
ansuz
b5649707d1
export 'tryParse' command
5 years ago
ansuz
64b0879984
improve logging for parse errors
5 years ago
ansuz
3f86b6141e
rename worker processes
5 years ago
ansuz
d8a88cb4ca
run expiration tasks in a worker instead of the main process
5 years ago
ansuz
b0179eaad9
drop XXX note
5 years ago
ansuz
9dbd32758a
improve worker logging and move blob deletion to worker processes as well
5 years ago
ansuz
5f69fc18d0
suppress some noisy errors
5 years ago
ansuz
5f2d7c8dcf
increase worker rpc wait time before timeout
5 years ago
ansuz
33e8e65507
handle errors in the server's workers
5 years ago
ansuz
a4c8039cc7
improve error handling with rpc response API
5 years ago
ansuz
cb53bd1c15
lint compliance
5 years ago
yflory
e15196bd53
Merge branch 'reconnect' into staging
5 years ago
ansuz
14feef1757
calculate pin list checksums in a worker
5 years ago
ansuz
70a0d4efb4
move more database reads into the database worker
5 years ago
ansuz
bc13a21796
give fileStreams a little bit more time before closing them
5 years ago
ansuz
4ba36a9173
load user pins in the database worker
5 years ago
ansuz
50e8893b24
move the 'getOlderHistory' call into the database worker
5 years ago
ansuz
471e374533
compute metadata in the same child process that builds indexes
5 years ago
ansuz
479b76f848
lint compliance
5 years ago
ansuz
c2fcba31e3
remove an XXX
5 years ago
ansuz
967ca6afa9
suppress ENOENT errors when checking empty chat channels
5 years ago
ansuz
40251948d4
check authenticated rpc signatures in separate threads
5 years ago
ansuz
0e09c73a60
Merge branch 'master' into soon
5 years ago
ansuz
a2facc4a0d
Merge pull request #513 from 7adietri/patch-1
...
Add shutdown() to non-disk logger
5 years ago
ansuz
e128683ffb
call back with errors instead of throwing within the Storage module
5 years ago
ansuz
1240f4a2a5
Merge branch 'master' into soon
5 years ago
ansuz
90899aa2f4
start reviewing and removing XXX notes
5 years ago
yflory
fe14399e67
Merge branch 'whiteboard' into soon
5 years ago
ansuz
2ef0af29e5
guard against index worker rpc responses for a txid with no supplied callback
5 years ago
ansuz
ee8d5c9c36
fix the premiumUploadSize validation to check the correct attribute
5 years ago
yflory
dc0b26da0b
Merge branch 'soon' into staging
5 years ago
Alexander
a190af079d
Add shutdown() to non-disk logger
...
The script `evict-inactive.js` calls `Log.shutdown()` at the end, which currently fails if logging to disk has been disabled via `logPath`.
5 years ago
ansuz
ed63d98ab0
Merge branch 'scaling-index' into soon
5 years ago
ansuz
590e249500
resolve merge conflict
5 years ago
ansuz
31c7cecaf9
add a few checks for safety
5 years ago
ansuz
cea9705bbe
Merge branch 'aggressive-stream-closing' into staging
5 years ago
ansuz
faa133aab8
remove an XXX
5 years ago
ansuz
d386e223e4
simplify open/close of writeStreams
5 years ago
ansuz
32cd0f3c4d
increase timeout value from 15s to 45. ought to be enough for anybody
5 years ago
ansuz
05a4e86cdb
close streams whenever we finish using them. time out if necessary
5 years ago
ansuz
4522ffa18a
compute indexes in child processes
5 years ago
ansuz
f8ad649b45
[style] bail out early to avoid nesting
5 years ago
ansuz
019e5e708b
wrap workers in a function scope and add a validateMessage method to HK's Env
5 years ago
ansuz
5467e1ffac
replace ad-hoc response handler with Util.response
5 years ago
ansuz
9e85a1411e
abstract the logic around worker choice out of message handling
5 years ago
ansuz
631ea54b49
lint compliance
5 years ago
ansuz
67dd4a608c
Merge remote-tracking branch 'origin/scaling' into scaling2
5 years ago
ansuz
b2fcde87d8
implement an admin command to update the default storage limit without a restart
5 years ago
yflory
923616aef0
Use each process in order instead of using a random one
5 years ago
ansuz
f345998956
fix undefined reference
5 years ago
yflory
fb0eb1b20c
Use more subprocesses
5 years ago
yflory
0d636dabc9
Check signature for history keeper in a different process
5 years ago
ansuz
30f17040ac
close streams when possible, simplify some code, enforce asynchrony
5 years ago
ansuz
2a9780abcf
lint compliance
5 years ago
yflory
578042154e
Better reconnect after invalid lastKnownHash
5 years ago
ansuz
2d47e7e2cf
implement an admin rpc to fetch ulimit data
5 years ago
ansuz
81c7416095
lint compliance
5 years ago
ansuz
f644c24362
add an admin-rpc to count the number of open files
5 years ago
ansuz
1230057970
disable IS_PAD_PINNED
5 years ago
ansuz
e1a0daac9c
possible server fixes
5 years ago
ansuz
bc034d95a3
enforce asynchrony in 'gethistoryOffset'
5 years ago
ansuz
d4038344ba
stop iterating over tasks if you can't read the root directory
5 years ago
ansuz
4e8ebac3c1
guard against the possible non-existence of metadata for a channel
5 years ago
ansuz
b32f3e34af
fix local worker debugging on firefox
5 years ago
ansuz
6d8cdca7a3
add 'rejected' field to metadata when getMetadata requests are blocked
5 years ago
ansuz
9d8bb43d03
finally untangle metadata and index caches
5 years ago
ansuz
32d769447a
begin standardizing our method of streaming lines from files
5 years ago
ansuz
35eca2c5d2
pull file streaming out into its own file, leave a few notes
5 years ago
ansuz
ce6cf6f727
revert some changes that couldn't have possibly worked
5 years ago
ansuz
b126e4456b
simplify quota logic
5 years ago
ansuz
53ed247bc2
expose maxUploadSize and premiumUploadSize via /api/config
5 years ago
ansuz
170aa6d47e
clean up a few tasks related to allow lists
5 years ago
ansuz
2f00ff3278
simplify check-in deactivation
5 years ago
ansuz
03ff9bd0d7
start deprecating usage of myDomain
5 years ago
ansuz
de6b9e2228
apply some defaults when loading the config
5 years ago
ansuz
94d8e7f2c2
provide a consistent type for 'customLimits'
5 years ago
ansuz
3cf09924ae
fix default CSP headers
5 years ago
ansuz
8d509fd6d5
Merge branch 'staging' into small-config
5 years ago
ansuz
c26560e3f9
commit stashed admin-rpc refactor
5 years ago
ansuz
02fc343727
Merge branch 'staging' into small-config
5 years ago
ansuz
3df47a1415
continued WIP config reorganization
5 years ago
ansuz
294a444603
WIP removing defaults from the example config file
5 years ago
ansuz
59ad80d7f1
support larger upload sizes for users with premium accounts
5 years ago
ansuz
3f2ebb773b
merge master back to staging
5 years ago
ansuz
542150b775
merge server components from staging
5 years ago
ansuz
f951951077
merge communities-allow-list and lint compliance
5 years ago
ansuz
f0e5fc7614
merge some missing code from communities-allow-list
5 years ago
ansuz
ad08fe0c08
silence a few more routine websocket errors
5 years ago
ansuz
021a6b361f
remove a bogus XXX note
5 years ago
ansuz
033bd361f9
add backwards compatibility with older servers
5 years ago