Commit Graph

265 Commits (27b4f175c0c1e8f8ce49dfc1324f0a9b8bd42316)

Author SHA1 Message Date
ansuz cbd3547814 remove log statements 2020-03-27 17:20:57 -04:00
ansuz 9058a59555 reassign db tasks if the responsible worker fails 2020-03-27 17:17:42 -04:00
ansuz 172823c954 lint compliance 2020-03-27 16:59:41 -04:00
ansuz b5649707d1 export 'tryParse' command 2020-03-27 15:36:34 -04:00
ansuz 64b0879984 improve logging for parse errors 2020-03-27 15:15:54 -04:00
ansuz 3f86b6141e rename worker processes 2020-03-27 15:14:45 -04:00
ansuz d8a88cb4ca run expiration tasks in a worker instead of the main process 2020-03-27 14:38:58 -04:00
ansuz b0179eaad9 drop XXX note 2020-03-27 14:25:07 -04:00
ansuz 9dbd32758a improve worker logging and move blob deletion to worker processes as well 2020-03-27 13:38:27 -04:00
ansuz 5f69fc18d0 suppress some noisy errors 2020-03-26 17:11:43 -04:00
ansuz 5f2d7c8dcf increase worker rpc wait time before timeout 2020-03-26 15:53:00 -04:00
ansuz 33e8e65507 handle errors in the server's workers 2020-03-26 15:13:20 -04:00
ansuz a4c8039cc7 improve error handling with rpc response API 2020-03-26 14:45:24 -04:00
ansuz cb53bd1c15 lint compliance 2020-03-26 14:44:37 -04:00
yflory e15196bd53 Merge branch 'reconnect' into staging 2020-03-26 15:24:38 +01:00
ansuz 14feef1757 calculate pin list checksums in a worker 2020-03-25 18:37:42 -04:00
ansuz 70a0d4efb4 move more database reads into the database worker 2020-03-25 17:43:57 -04:00
ansuz bc13a21796 give fileStreams a little bit more time before closing them 2020-03-25 11:50:45 -04:00
ansuz 4ba36a9173 load user pins in the database worker 2020-03-25 11:39:14 -04:00
ansuz 50e8893b24 move the 'getOlderHistory' call into the database worker 2020-03-25 10:54:29 -04:00
ansuz 471e374533 compute metadata in the same child process that builds indexes 2020-03-24 17:43:15 -04:00
ansuz 479b76f848 lint compliance 2020-03-24 17:40:07 -04:00
ansuz c2fcba31e3 remove an XXX 2020-03-24 15:02:46 -04:00
ansuz 967ca6afa9 suppress ENOENT errors when checking empty chat channels 2020-03-24 15:02:26 -04:00
ansuz 40251948d4 check authenticated rpc signatures in separate threads 2020-03-24 14:31:40 -04:00
ansuz 0e09c73a60 Merge branch 'master' into soon 2020-03-24 13:32:42 -04:00
ansuz a2facc4a0d
Merge pull request #513 from 7adietri/patch-1
Add shutdown() to non-disk logger
2020-03-24 13:32:13 -04:00
ansuz e128683ffb call back with errors instead of throwing within the Storage module 2020-03-24 13:31:03 -04:00
ansuz 1240f4a2a5 Merge branch 'master' into soon 2020-03-24 13:04:25 -04:00
ansuz 90899aa2f4 start reviewing and removing XXX notes 2020-03-24 13:03:49 -04:00
yflory fe14399e67 Merge branch 'whiteboard' into soon 2020-03-24 16:15:13 +01:00
ansuz 2ef0af29e5 guard against index worker rpc responses for a txid with no supplied callback 2020-03-23 22:03:59 -04:00
ansuz ee8d5c9c36 fix the premiumUploadSize validation to check the correct attribute 2020-03-23 16:13:48 -04:00
yflory dc0b26da0b Merge branch 'soon' into staging 2020-03-23 12:33:17 +01:00
Alexander a190af079d
Add shutdown() to non-disk logger
The script `evict-inactive.js` calls `Log.shutdown()` at the end, which currently fails if logging to disk has been disabled via `logPath`.
2020-03-21 14:43:25 +01:00
ansuz ed63d98ab0 Merge branch 'scaling-index' into soon 2020-03-20 18:57:51 -04:00
ansuz 590e249500 resolve merge conflict 2020-03-20 13:07:35 -04:00
ansuz 31c7cecaf9 add a few checks for safety 2020-03-20 10:54:01 -04:00
ansuz cea9705bbe Merge branch 'aggressive-stream-closing' into staging 2020-03-20 10:50:17 -04:00
ansuz faa133aab8 remove an XXX 2020-03-20 10:22:48 -04:00
ansuz d386e223e4 simplify open/close of writeStreams 2020-03-19 17:33:22 -04:00
ansuz 32cd0f3c4d increase timeout value from 15s to 45. ought to be enough for anybody 2020-03-19 16:13:24 -04:00
ansuz 05a4e86cdb close streams whenever we finish using them. time out if necessary 2020-03-19 16:11:24 -04:00
ansuz 4522ffa18a compute indexes in child processes 2020-03-19 10:46:18 -04:00
ansuz f8ad649b45 [style] bail out early to avoid nesting 2020-03-18 10:30:42 -04:00
ansuz 019e5e708b wrap workers in a function scope and add a validateMessage method to HK's Env 2020-03-17 17:23:35 -04:00
ansuz 5467e1ffac replace ad-hoc response handler with Util.response 2020-03-17 16:52:41 -04:00
ansuz 9e85a1411e abstract the logic around worker choice out of message handling 2020-03-17 16:06:01 -04:00
ansuz 631ea54b49 lint compliance 2020-03-17 15:40:06 -04:00
ansuz 67dd4a608c Merge remote-tracking branch 'origin/scaling' into scaling2 2020-03-17 15:36:58 -04:00
ansuz b2fcde87d8 implement an admin command to update the default storage limit without a restart 2020-03-17 10:11:52 -04:00
yflory 923616aef0 Use each process in order instead of using a random one 2020-03-17 15:10:23 +01:00
ansuz f345998956 fix undefined reference 2020-03-17 10:10:20 -04:00
yflory fb0eb1b20c Use more subprocesses 2020-03-17 14:18:41 +01:00
yflory 0d636dabc9 Check signature for history keeper in a different process 2020-03-17 13:29:53 +01:00
ansuz 30f17040ac close streams when possible, simplify some code, enforce asynchrony 2020-03-16 16:13:38 -04:00
ansuz 2a9780abcf lint compliance 2020-03-16 12:29:26 -04:00
yflory 578042154e Better reconnect after invalid lastKnownHash 2020-03-13 17:39:16 +01:00
ansuz 2d47e7e2cf implement an admin rpc to fetch ulimit data 2020-03-12 12:09:44 -04:00
ansuz 81c7416095 lint compliance 2020-03-12 11:40:46 -04:00
ansuz f644c24362 add an admin-rpc to count the number of open files 2020-03-12 11:35:16 -04:00
ansuz 1230057970 disable IS_PAD_PINNED 2020-03-12 11:31:08 -04:00
ansuz e1a0daac9c possible server fixes 2020-03-12 11:23:43 -04:00
ansuz bc034d95a3 enforce asynchrony in 'gethistoryOffset' 2020-03-12 11:13:23 -04:00
ansuz d4038344ba stop iterating over tasks if you can't read the root directory 2020-03-11 16:34:00 -04:00
ansuz 4e8ebac3c1 guard against the possible non-existence of metadata for a channel 2020-03-09 17:42:10 -04:00
ansuz b32f3e34af fix local worker debugging on firefox 2020-03-06 11:43:34 -05:00
ansuz 6d8cdca7a3 add 'rejected' field to metadata when getMetadata requests are blocked 2020-03-05 12:09:18 -05:00
ansuz 9d8bb43d03 finally untangle metadata and index caches 2020-03-04 12:56:33 -05:00
ansuz 32d769447a begin standardizing our method of streaming lines from files 2020-03-04 11:38:14 -05:00
ansuz 35eca2c5d2 pull file streaming out into its own file, leave a few notes 2020-03-04 11:38:14 -05:00
ansuz ce6cf6f727 revert some changes that couldn't have possibly worked 2020-03-04 09:59:17 -05:00
ansuz b126e4456b simplify quota logic 2020-03-03 16:55:40 -05:00
ansuz 53ed247bc2 expose maxUploadSize and premiumUploadSize via /api/config 2020-03-03 16:47:44 -05:00
ansuz 170aa6d47e clean up a few tasks related to allow lists 2020-03-03 15:52:49 -05:00
ansuz 2f00ff3278 simplify check-in deactivation 2020-02-28 14:24:53 -05:00
ansuz 03ff9bd0d7 start deprecating usage of myDomain 2020-02-28 11:45:41 -05:00
ansuz de6b9e2228 apply some defaults when loading the config 2020-02-28 10:50:30 -05:00
ansuz 94d8e7f2c2 provide a consistent type for 'customLimits' 2020-02-28 10:49:20 -05:00
ansuz 3cf09924ae fix default CSP headers 2020-02-28 10:46:44 -05:00
ansuz 8d509fd6d5 Merge branch 'staging' into small-config 2020-02-27 15:25:39 -05:00
ansuz c26560e3f9 commit stashed admin-rpc refactor 2020-02-27 15:00:31 -05:00
ansuz 02fc343727 Merge branch 'staging' into small-config 2020-02-27 13:54:15 -05:00
ansuz 3df47a1415 continued WIP config reorganization 2020-02-27 13:24:19 -05:00
ansuz 294a444603 WIP removing defaults from the example config file 2020-02-27 13:09:12 -05:00
ansuz 59ad80d7f1 support larger upload sizes for users with premium accounts 2020-02-26 13:09:11 -05:00
ansuz 3f2ebb773b merge master back to staging 2020-02-25 11:20:20 -05:00
ansuz 542150b775 merge server components from staging 2020-02-25 10:09:35 -05:00
ansuz f951951077 merge communities-allow-list and lint compliance 2020-02-25 09:48:41 -05:00
ansuz f0e5fc7614 merge some missing code from communities-allow-list 2020-02-25 09:09:16 -05:00
ansuz ad08fe0c08 silence a few more routine websocket errors 2020-02-25 08:01:25 -05:00
ansuz 021a6b361f remove a bogus XXX note 2020-02-21 14:33:28 -05:00
ansuz 033bd361f9 add backwards compatibility with older servers 2020-02-21 13:58:21 -05:00
ansuz cac114bb52 merge staging into pending allow-list work 2020-02-21 13:52:10 -05:00
ansuz 386add7c58 fix an invalid use of dropChannel 2020-02-21 13:46:11 -05:00
ansuz 33c12c27a1 ignore successive metadata errors with the same message 2020-02-21 13:38:56 -05:00
ansuz 7ee8778210 kick unauthorized users from restricted channels
and update dependencies
2020-02-21 13:05:24 -05:00
ansuz fe0e02a421 restrict access to metadata 2020-02-21 10:31:10 -05:00
ansuz 791aad53f2 WIP allow list changes 2020-02-21 08:18:05 -05:00
ansuz 597f417ad6 WIP block history requests that bypass JOIN restriction (where appropriate) 2020-02-19 15:32:15 -05:00