a190af079d
Add shutdown() to non-disk logger
...
The script `evict-inactive.js` calls `Log.shutdown()` at the end, which currently fails if logging to disk has been disabled via `logPath`.
2020-03-21 14:43:25 +01:00
ed63d98ab0
Merge branch 'scaling-index' into soon
2020-03-20 18:57:51 -04:00
590e249500
resolve merge conflict
2020-03-20 13:07:35 -04:00
31c7cecaf9
add a few checks for safety
2020-03-20 10:54:01 -04:00
cea9705bbe
Merge branch 'aggressive-stream-closing' into staging
2020-03-20 10:50:17 -04:00
faa133aab8
remove an XXX
2020-03-20 10:22:48 -04:00
d386e223e4
simplify open/close of writeStreams
2020-03-19 17:33:22 -04:00
32cd0f3c4d
increase timeout value from 15s to 45. ought to be enough for anybody
2020-03-19 16:13:24 -04:00
05a4e86cdb
close streams whenever we finish using them. time out if necessary
2020-03-19 16:11:24 -04:00
4522ffa18a
compute indexes in child processes
2020-03-19 10:46:18 -04:00
f8ad649b45
[style] bail out early to avoid nesting
2020-03-18 10:30:42 -04:00
019e5e708b
wrap workers in a function scope and add a validateMessage method to HK's Env
2020-03-17 17:23:35 -04:00
5467e1ffac
replace ad-hoc response handler with Util.response
2020-03-17 16:52:41 -04:00
9e85a1411e
abstract the logic around worker choice out of message handling
2020-03-17 16:06:01 -04:00
631ea54b49
lint compliance
2020-03-17 15:40:06 -04:00
67dd4a608c
Merge remote-tracking branch 'origin/scaling' into scaling2
2020-03-17 15:36:58 -04:00
b2fcde87d8
implement an admin command to update the default storage limit without a restart
2020-03-17 10:11:52 -04:00
923616aef0
Use each process in order instead of using a random one
2020-03-17 15:10:23 +01:00
f345998956
fix undefined reference
2020-03-17 10:10:20 -04:00
fb0eb1b20c
Use more subprocesses
2020-03-17 14:18:41 +01:00
0d636dabc9
Check signature for history keeper in a different process
2020-03-17 13:29:53 +01:00
30f17040ac
close streams when possible, simplify some code, enforce asynchrony
2020-03-16 16:13:38 -04:00
2a9780abcf
lint compliance
2020-03-16 12:29:26 -04:00
578042154e
Better reconnect after invalid lastKnownHash
2020-03-13 17:39:16 +01:00
2d47e7e2cf
implement an admin rpc to fetch ulimit data
2020-03-12 12:09:44 -04:00
81c7416095
lint compliance
2020-03-12 11:40:46 -04:00
f644c24362
add an admin-rpc to count the number of open files
2020-03-12 11:35:16 -04:00
1230057970
disable IS_PAD_PINNED
2020-03-12 11:31:08 -04:00
e1a0daac9c
possible server fixes
2020-03-12 11:23:43 -04:00
bc034d95a3
enforce asynchrony in 'gethistoryOffset'
2020-03-12 11:13:23 -04:00
d4038344ba
stop iterating over tasks if you can't read the root directory
2020-03-11 16:34:00 -04:00
4e8ebac3c1
guard against the possible non-existence of metadata for a channel
2020-03-09 17:42:10 -04:00
b32f3e34af
fix local worker debugging on firefox
2020-03-06 11:43:34 -05:00
6d8cdca7a3
add 'rejected' field to metadata when getMetadata requests are blocked
2020-03-05 12:09:18 -05:00
9d8bb43d03
finally untangle metadata and index caches
2020-03-04 12:56:33 -05:00
32d769447a
begin standardizing our method of streaming lines from files
2020-03-04 11:38:14 -05:00
35eca2c5d2
pull file streaming out into its own file, leave a few notes
2020-03-04 11:38:14 -05:00
ce6cf6f727
revert some changes that couldn't have possibly worked
2020-03-04 09:59:17 -05:00
b126e4456b
simplify quota logic
2020-03-03 16:55:40 -05:00
53ed247bc2
expose maxUploadSize and premiumUploadSize via /api/config
2020-03-03 16:47:44 -05:00
170aa6d47e
clean up a few tasks related to allow lists
2020-03-03 15:52:49 -05:00
2f00ff3278
simplify check-in deactivation
2020-02-28 14:24:53 -05:00
03ff9bd0d7
start deprecating usage of myDomain
2020-02-28 11:45:41 -05:00
de6b9e2228
apply some defaults when loading the config
2020-02-28 10:50:30 -05:00
94d8e7f2c2
provide a consistent type for 'customLimits'
2020-02-28 10:49:20 -05:00
3cf09924ae
fix default CSP headers
2020-02-28 10:46:44 -05:00
8d509fd6d5
Merge branch 'staging' into small-config
2020-02-27 15:25:39 -05:00
c26560e3f9
commit stashed admin-rpc refactor
2020-02-27 15:00:31 -05:00
02fc343727
Merge branch 'staging' into small-config
2020-02-27 13:54:15 -05:00
3df47a1415
continued WIP config reorganization
2020-02-27 13:24:19 -05:00
294a444603
WIP removing defaults from the example config file
2020-02-27 13:09:12 -05:00
59ad80d7f1
support larger upload sizes for users with premium accounts
2020-02-26 13:09:11 -05:00
3f2ebb773b
merge master back to staging
2020-02-25 11:20:20 -05:00
542150b775
merge server components from staging
2020-02-25 10:09:35 -05:00
f951951077
merge communities-allow-list and lint compliance
2020-02-25 09:48:41 -05:00
f0e5fc7614
merge some missing code from communities-allow-list
2020-02-25 09:09:16 -05:00
ad08fe0c08
silence a few more routine websocket errors
2020-02-25 08:01:25 -05:00
021a6b361f
remove a bogus XXX note
2020-02-21 14:33:28 -05:00
033bd361f9
add backwards compatibility with older servers
2020-02-21 13:58:21 -05:00
cac114bb52
merge staging into pending allow-list work
2020-02-21 13:52:10 -05:00
386add7c58
fix an invalid use of dropChannel
2020-02-21 13:46:11 -05:00
33c12c27a1
ignore successive metadata errors with the same message
2020-02-21 13:38:56 -05:00
7ee8778210
kick unauthorized users from restricted channels
...
and update dependencies
2020-02-21 13:05:24 -05:00
fe0e02a421
restrict access to metadata
2020-02-21 10:31:10 -05:00
791aad53f2
WIP allow list changes
2020-02-21 08:18:05 -05:00
597f417ad6
WIP block history requests that bypass JOIN restriction (where appropriate)
2020-02-19 15:32:15 -05:00
79bc8830ef
disable access restriction if all owners are removed
2020-02-19 14:39:28 -05:00
f579c9b059
semi-functional allow-list implementation in historyKeeper
2020-02-19 14:22:12 -05:00
0edcdcb1b8
oups! fix a missing callback that's been here for a while...
2020-02-18 16:15:05 -05:00
27e57e7af0
oups! fix a missing callback that's been here for a while...
2020-02-18 16:07:14 -05:00
eac3e4cbcd
sketch out some historyKeeper functionality related to access lists
2020-02-18 16:05:12 -05:00
156c37942d
implement new metadata commands related to allow lists and mailboxes
2020-02-18 14:16:07 -05:00
a280cc85d2
Merge branch 'staging' into communities-allow-list
2020-02-17 12:04:41 -05:00
626a031115
resolve conflict with master/staging
2020-02-17 12:04:01 -05:00
ff73e96cb8
reimplement the trim history fix from staging
2020-02-17 12:02:49 -05:00
a4be6185de
merge staging and do a little lint compliance
2020-02-17 11:54:27 -05:00
f478ae725d
Merge branch 'staging' into communities-allow-list
2020-02-17 11:48:56 -05:00
79f1280cf3
Merge branch 'soon' into staging
2020-02-17 11:42:08 -05:00
b56367414b
don't overwrite cached indices when new users join a channel
2020-02-17 11:31:00 -05:00
253ea0d336
Merge branch 'staging' into communities-allow-list
2020-02-17 10:38:40 -05:00
e8949168ec
lint compliance
2020-02-17 10:30:44 -05:00
5dff6535ed
add a simple guard against unparsed messages when trimming history
2020-02-17 10:19:51 -05:00
8694c17023
sketch out metadata commands for allow lists
2020-02-17 09:26:36 -05:00
f86196e40a
implement shared environment between historyKeeper and RPC
2020-02-17 09:01:10 -05:00
3f606d8c75
remove some duplicated code
2020-02-14 16:45:53 -05:00
725d10fc60
nest storage directory inside './lib'
2020-02-14 16:29:30 -05:00
38c1700173
Respond to pinning RPCs as soon as possible
...
(instead of waiting until you've read an unbounded number
of pin logs while queries back up in memory)
Also replace instances of 'publicKey' with 'safeKey' or 'unsafeKey'
to clearly and correctly indicate their format.
2020-02-14 13:46:40 -05:00
cded52f83f
replicate existing pinned.load API correctly
2020-02-14 11:45:51 -05:00
65f88617cf
add support for an optional handler for each pin log
...
great for analyzing which files are most in need of optimization
2020-02-13 18:16:32 -05:00
3dc789cbca
replace pinned.js with low-profile streaming implementation
2020-02-13 17:46:10 -05:00
609eddc9e1
reimplement the the Pinned.load API
...
* recurse into directories on demand
* optimized for lower memory consumption
* less parallelism than before
2020-02-13 17:41:36 -05:00
2345323f0d
lint compliance
2020-02-13 17:40:55 -05:00
2d6626234b
yet another async scheduler, this time supporting flexible runtime control flow based on declarative priority levels
2020-02-13 16:44:11 -05:00
0f697ac865
Merge branch 'staging' into communities-trim
2020-02-11 11:10:12 +01:00
1231cacef1
remove XXX
2020-02-10 11:21:04 -05:00
57be324850
remove some XXX notes
2020-02-10 10:01:02 -05:00
68a77885d4
suppress some routine server logs
2020-02-10 09:03:16 -05:00
7f22da0f57
fix use of incorrect key format in admin-rpc
2020-02-10 08:07:29 -05:00
7000be0e70
clean up historykeeper
2020-02-07 18:58:57 -05:00
802034616c
centralize historykeeper-rpc interaction in rpc methods
2020-02-07 12:53:12 -05:00
fd095644a2
Merge branch 'staging' into communities-trim
2020-02-06 11:55:07 +01:00
64c0f3a90d
Add missing RPC command
2020-02-06 11:54:46 +01:00
9a857ea058
Merge branch 'staging' into communities-trim
2020-02-06 10:55:12 +01:00
b4172676f3
lint compliance
2020-02-05 18:12:09 -05:00
f17d14fd99
simpler limit updates
2020-02-05 18:01:48 -05:00
bde17a62a1
continue refactoring rpc
2020-02-05 17:31:44 -05:00
fa525817ba
fix broken removePins RPC
2020-02-05 13:05:06 -05:00
a00a9fd018
don't hang when reading unparseable lines from history
2020-02-05 13:05:05 -05:00
1d56c08700
Merge branch 'staging' into communities-trim
2020-02-05 18:23:10 +01:00
3741cbcbc0
rewrite trimmed history WITH newlines
2020-02-05 12:22:53 -05:00
4079f1bfdb
Fix RPC command
2020-02-05 18:12:10 +01:00
6655f493e0
Merge branch 'staging' into communities-trim
2020-02-05 17:56:14 +01:00
5808c534b5
fix function signature for trimHistory RPC
2020-02-05 11:53:47 -05:00
c9676e1c91
Merge branch 'staging' into communities-trim
2020-02-05 17:05:45 +01:00
9c28941f6c
lint compliance
2020-02-05 10:47:43 -05:00
5cb266838a
Fix syntax error
2020-02-05 11:28:27 +01:00
0d69620687
tweak the metadata line handler to handle an edge case in trim history
2020-02-04 17:33:51 -05:00
f8f3a48e8b
use latest chainpad-server
2020-02-04 11:15:44 -05:00
46dfa026f0
fix an API change that caused a typeError
2020-02-03 18:47:18 -05:00
88be40ede3
standardize some function signatures and factor out a lot of boilerplate
2020-02-03 18:32:21 -05:00
65ba85d97b
clear historyKeeper cache when we trim a channel
2020-02-03 17:14:42 -05:00
43307ffb1a
define all server intervals in a map so we can easily clear them all
2020-02-03 17:14:23 -05:00
6523974ca2
fix a WRITE_PRIVATE_MESSAGE rpc regression
2020-02-03 15:47:41 -05:00
779e817443
stop relying on netflux-server internals
...
* create RPC module from inside historyKeeper
* stop passing around netflux-server context
* update to use newer netflux-server's formal APIs
* manage your own cache of indexes instead of storing things in the netflux context
2020-02-03 14:20:05 -05:00
06c29ef1d1
latest api changes to match the netflux-server refactor
2020-02-03 10:03:43 -05:00
b922860339
drop usage of historyKeeper.setConfig
2020-01-27 18:54:16 -05:00
80c012f34d
prepare to merge history keeper and rpc
2020-01-27 17:57:39 -05:00
b093d3f0d2
WIP massive rpc refactor
2020-01-24 14:45:53 -05:00
ceb351326c
split out some more rpc functionality and fix broken module paths
2020-01-24 13:36:14 -05:00
c1f222dd6c
move metadata commands from rpc to their own module
2020-01-24 13:19:40 -05:00
bb7e8e4512
move login block functionality into its own rpc module
2020-01-24 13:14:26 -05:00
c765362744
move more rpc functionality into modules
2020-01-24 13:06:46 -05:00
c93b39c094
separate more rpc functionality into pinning and core submodules
2020-01-24 12:43:11 -05:00
ba6e3f33bd
move admin commands into their own module
2020-01-24 11:25:48 -05:00
39b0785406
apply custom limits immediately at startup
2020-01-24 09:57:12 -05:00
9cdf54aff2
untested implementation of trimHistory
2020-01-23 17:58:24 -05:00
c388641479
drop support for 'retainData' configuration
2020-01-23 15:13:19 -05:00
f45de2b52f
move some server deps from repo root to lib/
2020-01-23 15:01:56 -05:00
b585dd998d
throw in a little asynchrony
2020-01-23 13:27:12 -05:00
7072fe4fa4
implement and test a complex constrained scheduler
2020-01-15 13:32:52 -05:00
a714e41d49
Log errors in faulty custom config file
...
CryptPAD now terminates and does not fallback to default config if the custom config is faulty.
The error in the custom config is logged to the terminal now.
Signed-off-by: Adrian Nöthlich <git@promasu.tech>
2020-01-04 23:58:01 +01:00
1da662687f
guarantee asynchronous execution for batch reads and write queues
2019-12-05 13:35:26 -05:00
9b74136b66
Merge branch 'invitations' into staging
2019-09-30 10:53:10 +02:00
69e61ec2f2
ignore metadata commands which add invalid owners
2019-09-27 14:28:59 +02:00
069776e804
use fast implementations of tweetnacl serverside
2019-09-27 13:30:17 +02:00
cb6b4a8eb3
always supply a callback when writing messages
2019-09-12 18:04:24 +02:00
ab868237a0
replace ad hoc line stream with readline module
2019-09-11 11:01:41 +02:00
73854a8b6a
implement Util.both
2019-09-11 10:54:44 +02:00
a3eff2728e
update lib/once.js to match clientside definition
2019-09-09 16:39:54 +02:00
29813e485d
add a little debugging info to batch-read
2019-09-06 15:03:42 +02:00
fe884278f4
ignore non-interesting errors when processing metadata
2019-09-06 13:36:06 +02:00
dc078c1ca1
test mailboxes and RPCs from the command line, fix some minor errors
2019-09-06 13:35:36 +02:00
d81ff791ad
WIP cli tests for rpc
2019-09-05 18:01:21 +02:00
9ce7cea9cc
abstract historyKeeper's batched reads implementation into a library
2019-09-04 17:22:06 +02:00
4902554a61
implement per-channel fifo queues for metadata and channel writes
2019-09-04 11:51:33 +02:00
70161dbd73
Merge branch 'staging' of github.com:xwiki-labs/cryptpad into teams-server-features
2019-09-03 17:08:16 +02:00
ba18d5c79d
WIP CLI client
2019-09-03 16:36:27 +02:00
4027672678
Store the mailbox of all the owners
2019-09-03 15:11:23 +02:00
d52e449f87
Fix setMetadata RPC saferphore
2019-09-03 13:29:57 +02:00
ed5671a548
Don't write metadata updates to the log if it doesn't change the value
2019-09-02 17:16:14 +02:00
738030e8c0
Add pending_owners commands
2019-08-30 12:07:03 +02:00
e12e5e1f25
Allow pending_owners to claim ownership
2019-08-30 11:49:15 +02:00
28b2341f2c
Serverside metadata update
2019-08-28 12:40:35 +02:00
c5ca82f0e1
throw errors if you try to update expiration times, as this isn't implemented yet
2019-08-21 12:28:19 +02:00
0a0c64d240
only increment the metadata line index if you can parse the line
2019-08-21 12:26:46 +02:00
1c37ec7aee
prevent duplicated owners in metadata amendments
2019-08-21 12:25:17 +02:00
ce847700c4
expose the index of the metadata log processor
2019-08-12 15:26:23 +02:00
b36f50f2c9
update metadata queries to include edits
2019-06-27 18:21:12 +02:00
efd0efede4
implement RESET_OWNERS command for editable metadata
2019-06-27 12:13:29 +02:00
7b18bf91ce
implement functions for iterating over metadata logs
2019-06-27 11:11:22 +02:00
35dd3abbb5
make the config path configurable
2019-05-15 14:56:07 +02:00
ecd9647a2e
add shutdown command for the logstore
2019-04-23 13:25:03 +02:00
cd793b295d
provide a standard api for calculating pin data for a given public key
2019-04-17 16:30:45 +02:00
5c9cd5ffc6
standardize pin calculation routine
2019-04-17 14:46:46 +02:00
d5a9eacef0
standardize loading of server configuration
2019-04-17 13:48:39 +02:00
dde28dfa02
reorder logLevel priority
2019-04-11 11:23:22 +02:00
f9eb1fde67
oops
2019-04-11 10:54:00 +02:00
0bd14bfcbe
fix logging to stdout and document logging configuration
2019-04-09 14:29:02 +02:00
b3e0f9e886
fix use before definition
2019-04-09 11:57:33 +02:00
4e9273f35c
improved logger behaviour and updated default configuration
2019-04-09 11:40:51 +02:00
e1e6d51038
new logging api
2019-04-08 18:11:36 +02:00
Alexander