48deced859
fix serverside index corruption
5 years ago
5b2929a6fc
suppress some useless info from some commonly logged errors
5 years ago
9f1f01f3b4
disable worker task queueing and address a probable memory leak
5 years ago
b0af6b5488
fix an incorrect condition when checking for available server workers
...
Also, log when the RPC queue has a backlog and when it is drained
5 years ago
4037270a23
add a FIXME note for a sketchy implementation in history keeper
5 years ago
02347d134a
queue signature validation per-channel to ensure correct ordering
5 years ago
9975c27e84
increase timeout from one minute to three and decrease the number of parallel jobs per worker
5 years ago
488f876ad6
lint compliance
5 years ago
d35b015c3f
log some info on successful file uploads
5 years ago
37d3995ac1
unify format of console output with stored logs
5 years ago
8a5d012edf
add special error handling for a few special serverside errors
5 years ago
8bee95d13d
improve error reporting in GET_HISTORY
5 years ago
e05891184e
fix regression in maxWorkers
5 years ago
b56c73be6e
delegate more work from getOlderHistory to the worker
5 years ago
fd169ff39c
close streams after 2 minutes instead of 5
...
Hopefully this helps to even out our memory and CPU usage.
There's also some improved labels so that IF streams time out
we will know the exact origin of the offending call.
5 years ago
bac8ef3c6e
lint compliance
5 years ago
4e57e390da
write tasks inside of workers
5 years ago
ba6faca02e
make the number of workers configurable
5 years ago
0465f31a45
add a FIXME note
5 years ago
04ab7f538a
recover from worker faults and unify worker types
5 years ago
e0a6852b79
correctly serialize an error log
5 years ago
9ed82640bf
Merge branch 'worker-queue' into staging
5 years ago
e8b1fcf710
solve some cases where crashing workers could result in an invalid state for the queue
5 years ago
9ed26cfeb0
lint compliance
5 years ago
dc0460780f
Merge branch 'soon' into staging
5 years ago
74567e0cf4
return process memory usage in GET CACHE STATS RPC
5 years ago
e3f42cbc20
Allow config.defaultStorageLimit to be zero bytes
5 years ago
89262cd29e
resolve silly conflict and merge staging
5 years ago
e527867e2e
queue blob and channel deletions per-user
5 years ago
bd3e18d1a8
only cache a user's pinned channels if there are no errors when loading them
5 years ago
bef18a9320
hotfix
5 years ago
7fac997e93
increase some file storage timeouts related to streams
5 years ago
95965c1dee
keep a parallel implementation of the pin loader to validate the new one
5 years ago
a57bf94058
remove XXX note
5 years ago
c53675c9d5
WIP worker rpc call queue
5 years ago
d802173325
remove an invalid line
5 years ago
2b3f6e3464
Merge branch 'soon' into staging
5 years ago
adfd1eaaa2
avoid a silly typeError in the event of an error
5 years ago
32406a5995
Merge branch 'master' into staging
5 years ago
834c96a4fc
update comments and remove some notes
5 years ago
ed5d97f899
WIP history-keeper fixes
5 years ago
eddbe80eef
Merge branch 'soon' into staging
5 years ago
3d8e78a268
fix overly broad config interpretation for daily check
5 years ago
e6ec891d9a
improve admin rpc for diagnosing memory used by the cache
5 years ago
77961e3954
trim lookup tables for channels without checkpoints
5 years ago
db9eaa8d4f
Merge branch 'soon' into staging
5 years ago
6d0dee979a
allow admins to inspect index/metadata caches
5 years ago
cbd3547814
remove log statements
5 years ago
9058a59555
reassign db tasks if the responsible worker fails
5 years ago
172823c954
lint compliance
5 years ago
b5649707d1
export 'tryParse' command
5 years ago
64b0879984
improve logging for parse errors
5 years ago
3f86b6141e
rename worker processes
5 years ago
d8a88cb4ca
run expiration tasks in a worker instead of the main process
5 years ago
b0179eaad9
drop XXX note
5 years ago
9dbd32758a
improve worker logging and move blob deletion to worker processes as well
5 years ago
5f69fc18d0
suppress some noisy errors
5 years ago
5f2d7c8dcf
increase worker rpc wait time before timeout
5 years ago
33e8e65507
handle errors in the server's workers
5 years ago
a4c8039cc7
improve error handling with rpc response API
5 years ago
cb53bd1c15
lint compliance
5 years ago
e15196bd53
Merge branch 'reconnect' into staging
5 years ago
14feef1757
calculate pin list checksums in a worker
5 years ago
70a0d4efb4
move more database reads into the database worker
5 years ago
bc13a21796
give fileStreams a little bit more time before closing them
5 years ago
4ba36a9173
load user pins in the database worker
5 years ago
50e8893b24
move the 'getOlderHistory' call into the database worker
5 years ago
471e374533
compute metadata in the same child process that builds indexes
5 years ago
479b76f848
lint compliance
5 years ago
c2fcba31e3
remove an XXX
5 years ago
967ca6afa9
suppress ENOENT errors when checking empty chat channels
5 years ago
40251948d4
check authenticated rpc signatures in separate threads
5 years ago
0e09c73a60
Merge branch 'master' into soon
5 years ago
a2facc4a0d
Merge pull request #513 from 7adietri/patch-1
...
Add shutdown() to non-disk logger
5 years ago
e128683ffb
call back with errors instead of throwing within the Storage module
5 years ago
1240f4a2a5
Merge branch 'master' into soon
5 years ago
90899aa2f4
start reviewing and removing XXX notes
5 years ago
fe14399e67
Merge branch 'whiteboard' into soon
5 years ago
2ef0af29e5
guard against index worker rpc responses for a txid with no supplied callback
5 years ago
ee8d5c9c36
fix the premiumUploadSize validation to check the correct attribute
5 years ago
dc0b26da0b
Merge branch 'soon' into staging
5 years ago
a190af079d
Add shutdown() to non-disk logger
...
The script `evict-inactive.js` calls `Log.shutdown()` at the end, which currently fails if logging to disk has been disabled via `logPath`.
5 years ago
ed63d98ab0
Merge branch 'scaling-index' into soon
5 years ago
590e249500
resolve merge conflict
5 years ago
31c7cecaf9
add a few checks for safety
5 years ago
cea9705bbe
Merge branch 'aggressive-stream-closing' into staging
5 years ago
faa133aab8
remove an XXX
5 years ago
d386e223e4
simplify open/close of writeStreams
5 years ago
32cd0f3c4d
increase timeout value from 15s to 45. ought to be enough for anybody
5 years ago
05a4e86cdb
close streams whenever we finish using them. time out if necessary
5 years ago
4522ffa18a
compute indexes in child processes
5 years ago
f8ad649b45
[style] bail out early to avoid nesting
5 years ago
019e5e708b
wrap workers in a function scope and add a validateMessage method to HK's Env
5 years ago
5467e1ffac
replace ad-hoc response handler with Util.response
5 years ago
9e85a1411e
abstract the logic around worker choice out of message handling
5 years ago
631ea54b49
lint compliance
5 years ago
67dd4a608c
Merge remote-tracking branch 'origin/scaling' into scaling2
5 years ago
b2fcde87d8
implement an admin command to update the default storage limit without a restart
5 years ago
923616aef0
Use each process in order instead of using a random one
5 years ago
f345998956
fix undefined reference
5 years ago
fb0eb1b20c
Use more subprocesses
5 years ago
0d636dabc9
Check signature for history keeper in a different process
5 years ago
30f17040ac
close streams when possible, simplify some code, enforce asynchrony
5 years ago
2a9780abcf
lint compliance
5 years ago
578042154e
Better reconnect after invalid lastKnownHash
5 years ago
2d47e7e2cf
implement an admin rpc to fetch ulimit data
5 years ago
81c7416095
lint compliance
5 years ago
f644c24362
add an admin-rpc to count the number of open files
5 years ago
1230057970
disable IS_PAD_PINNED
5 years ago
e1a0daac9c
possible server fixes
5 years ago
bc034d95a3
enforce asynchrony in 'gethistoryOffset'
5 years ago
d4038344ba
stop iterating over tasks if you can't read the root directory
5 years ago
4e8ebac3c1
guard against the possible non-existence of metadata for a channel
5 years ago
b32f3e34af
fix local worker debugging on firefox
5 years ago
6d8cdca7a3
add 'rejected' field to metadata when getMetadata requests are blocked
5 years ago
9d8bb43d03
finally untangle metadata and index caches
5 years ago
32d769447a
begin standardizing our method of streaming lines from files
5 years ago
35eca2c5d2
pull file streaming out into its own file, leave a few notes
5 years ago
ce6cf6f727
revert some changes that couldn't have possibly worked
5 years ago
b126e4456b
simplify quota logic
5 years ago
53ed247bc2
expose maxUploadSize and premiumUploadSize via /api/config
5 years ago
170aa6d47e
clean up a few tasks related to allow lists
5 years ago
2f00ff3278
simplify check-in deactivation
5 years ago
03ff9bd0d7
start deprecating usage of myDomain
5 years ago
de6b9e2228
apply some defaults when loading the config
5 years ago
94d8e7f2c2
provide a consistent type for 'customLimits'
5 years ago
3cf09924ae
fix default CSP headers
5 years ago
8d509fd6d5
Merge branch 'staging' into small-config
5 years ago
c26560e3f9
commit stashed admin-rpc refactor
5 years ago
02fc343727
Merge branch 'staging' into small-config
5 years ago
3df47a1415
continued WIP config reorganization
5 years ago
294a444603
WIP removing defaults from the example config file
5 years ago
59ad80d7f1
support larger upload sizes for users with premium accounts
5 years ago
3f2ebb773b
merge master back to staging
5 years ago
542150b775
merge server components from staging
5 years ago
f951951077
merge communities-allow-list and lint compliance
5 years ago
f0e5fc7614
merge some missing code from communities-allow-list
5 years ago
ad08fe0c08
silence a few more routine websocket errors
5 years ago
021a6b361f
remove a bogus XXX note
5 years ago
033bd361f9
add backwards compatibility with older servers
5 years ago
cac114bb52
merge staging into pending allow-list work
5 years ago
386add7c58
fix an invalid use of dropChannel
5 years ago
33c12c27a1
ignore successive metadata errors with the same message
5 years ago
7ee8778210
kick unauthorized users from restricted channels
...
and update dependencies
5 years ago
fe0e02a421
restrict access to metadata
5 years ago
791aad53f2
WIP allow list changes
5 years ago
597f417ad6
WIP block history requests that bypass JOIN restriction (where appropriate)
5 years ago
79bc8830ef
disable access restriction if all owners are removed
5 years ago
f579c9b059
semi-functional allow-list implementation in historyKeeper
5 years ago
0edcdcb1b8
oups! fix a missing callback that's been here for a while...
5 years ago
27e57e7af0
oups! fix a missing callback that's been here for a while...
5 years ago
eac3e4cbcd
sketch out some historyKeeper functionality related to access lists
5 years ago
156c37942d
implement new metadata commands related to allow lists and mailboxes
5 years ago
a280cc85d2
Merge branch 'staging' into communities-allow-list
5 years ago
626a031115
resolve conflict with master/staging
5 years ago
ff73e96cb8
reimplement the trim history fix from staging
5 years ago
a4be6185de
merge staging and do a little lint compliance
5 years ago
f478ae725d
Merge branch 'staging' into communities-allow-list
5 years ago
79f1280cf3
Merge branch 'soon' into staging
5 years ago
b56367414b
don't overwrite cached indices when new users join a channel
5 years ago
253ea0d336
Merge branch 'staging' into communities-allow-list
5 years ago
e8949168ec
lint compliance
5 years ago
5dff6535ed
add a simple guard against unparsed messages when trimming history
5 years ago
8694c17023
sketch out metadata commands for allow lists
5 years ago
f86196e40a
implement shared environment between historyKeeper and RPC
5 years ago
3f606d8c75
remove some duplicated code
5 years ago
725d10fc60
nest storage directory inside './lib'
5 years ago
38c1700173
Respond to pinning RPCs as soon as possible
...
(instead of waiting until you've read an unbounded number
of pin logs while queries back up in memory)
Also replace instances of 'publicKey' with 'safeKey' or 'unsafeKey'
to clearly and correctly indicate their format.
5 years ago
cded52f83f
replicate existing pinned.load API correctly
5 years ago
65f88617cf
add support for an optional handler for each pin log
...
great for analyzing which files are most in need of optimization
5 years ago
3dc789cbca
replace pinned.js with low-profile streaming implementation
5 years ago
609eddc9e1
reimplement the the Pinned.load API
...
* recurse into directories on demand
* optimized for lower memory consumption
* less parallelism than before
5 years ago
2345323f0d
lint compliance
5 years ago
2d6626234b
yet another async scheduler, this time supporting flexible runtime control flow based on declarative priority levels
5 years ago
0f697ac865
Merge branch 'staging' into communities-trim
5 years ago
1231cacef1
remove XXX
5 years ago
57be324850
remove some XXX notes
5 years ago
68a77885d4
suppress some routine server logs
5 years ago
7f22da0f57
fix use of incorrect key format in admin-rpc
5 years ago
7000be0e70
clean up historykeeper
5 years ago
802034616c
centralize historykeeper-rpc interaction in rpc methods
5 years ago
fd095644a2
Merge branch 'staging' into communities-trim
5 years ago
64c0f3a90d
Add missing RPC command
5 years ago
9a857ea058
Merge branch 'staging' into communities-trim
5 years ago
b4172676f3
lint compliance
5 years ago
f17d14fd99
simpler limit updates
5 years ago
bde17a62a1
continue refactoring rpc
5 years ago
fa525817ba
fix broken removePins RPC
5 years ago
a00a9fd018
don't hang when reading unparseable lines from history
5 years ago
1d56c08700
Merge branch 'staging' into communities-trim
5 years ago
3741cbcbc0
rewrite trimmed history WITH newlines
5 years ago
4079f1bfdb
Fix RPC command
5 years ago
6655f493e0
Merge branch 'staging' into communities-trim
5 years ago
5808c534b5
fix function signature for trimHistory RPC
5 years ago
c9676e1c91
Merge branch 'staging' into communities-trim
5 years ago
9c28941f6c
lint compliance
5 years ago
5cb266838a
Fix syntax error
5 years ago
0d69620687
tweak the metadata line handler to handle an edge case in trim history
5 years ago
f8f3a48e8b
use latest chainpad-server
5 years ago
46dfa026f0
fix an API change that caused a typeError
5 years ago
88be40ede3
standardize some function signatures and factor out a lot of boilerplate
5 years ago
65ba85d97b
clear historyKeeper cache when we trim a channel
5 years ago
43307ffb1a
define all server intervals in a map so we can easily clear them all
5 years ago
6523974ca2
fix a WRITE_PRIVATE_MESSAGE rpc regression
5 years ago
779e817443
stop relying on netflux-server internals
...
* create RPC module from inside historyKeeper
* stop passing around netflux-server context
* update to use newer netflux-server's formal APIs
* manage your own cache of indexes instead of storing things in the netflux context
5 years ago
06c29ef1d1
latest api changes to match the netflux-server refactor
5 years ago
b922860339
drop usage of historyKeeper.setConfig
5 years ago
80c012f34d
prepare to merge history keeper and rpc
5 years ago
b093d3f0d2
WIP massive rpc refactor
5 years ago
ceb351326c
split out some more rpc functionality and fix broken module paths
5 years ago
c1f222dd6c
move metadata commands from rpc to their own module
5 years ago
bb7e8e4512
move login block functionality into its own rpc module
5 years ago
c765362744
move more rpc functionality into modules
5 years ago
c93b39c094
separate more rpc functionality into pinning and core submodules
5 years ago
ba6e3f33bd
move admin commands into their own module
5 years ago
39b0785406
apply custom limits immediately at startup
5 years ago
9cdf54aff2
untested implementation of trimHistory
5 years ago
c388641479
drop support for 'retainData' configuration
5 years ago
f45de2b52f
move some server deps from repo root to lib/
5 years ago
b585dd998d
throw in a little asynchrony
5 years ago
7072fe4fa4
implement and test a complex constrained scheduler
5 years ago
a714e41d49
Log errors in faulty custom config file
...
CryptPAD now terminates and does not fallback to default config if the custom config is faulty.
The error in the custom config is logged to the terminal now.
Signed-off-by: Adrian Nöthlich <git@promasu.tech>
5 years ago
1da662687f
guarantee asynchronous execution for batch reads and write queues
5 years ago
9b74136b66
Merge branch 'invitations' into staging
5 years ago
69e61ec2f2
ignore metadata commands which add invalid owners
5 years ago
069776e804
use fast implementations of tweetnacl serverside
5 years ago
cb6b4a8eb3
always supply a callback when writing messages
5 years ago
ab868237a0
replace ad hoc line stream with readline module
5 years ago
73854a8b6a
implement Util.both
5 years ago
a3eff2728e
update lib/once.js to match clientside definition
5 years ago
29813e485d
add a little debugging info to batch-read
5 years ago
fe884278f4
ignore non-interesting errors when processing metadata
5 years ago
dc078c1ca1
test mailboxes and RPCs from the command line, fix some minor errors
5 years ago
d81ff791ad
WIP cli tests for rpc
5 years ago
9ce7cea9cc
abstract historyKeeper's batched reads implementation into a library
5 years ago
4902554a61
implement per-channel fifo queues for metadata and channel writes
5 years ago
70161dbd73
Merge branch 'staging' of github.com:xwiki-labs/cryptpad into teams-server-features
5 years ago
ba18d5c79d
WIP CLI client
5 years ago
4027672678
Store the mailbox of all the owners
5 years ago
d52e449f87
Fix setMetadata RPC saferphore
5 years ago
ed5671a548
Don't write metadata updates to the log if it doesn't change the value
5 years ago
738030e8c0
Add pending_owners commands
5 years ago
e12e5e1f25
Allow pending_owners to claim ownership
5 years ago
28b2341f2c
Serverside metadata update
5 years ago
c5ca82f0e1
throw errors if you try to update expiration times, as this isn't implemented yet
5 years ago
0a0c64d240
only increment the metadata line index if you can parse the line
5 years ago
1c37ec7aee
prevent duplicated owners in metadata amendments
5 years ago
ce847700c4
expose the index of the metadata log processor
5 years ago
b36f50f2c9
update metadata queries to include edits
6 years ago
efd0efede4
implement RESET_OWNERS command for editable metadata
6 years ago
ansuz