diff --git a/CHANGELOG.md b/CHANGELOG.md index 0c80b2b8f..566758ad4 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,3 +1,50 @@ +# PigFootedBandicoot release (3.15.0) + +## Goals + +Our plan for this release was to allow our server's code to stabilize after a prologued period of major changes. The massive surge of new users on cryptpad.fr forced us to change our plans and focus instead on increasing performance and scalability of our serverside code and its supporting infrastructure. Most of this release's changes have been thoroughly tested as they've been deployed to our instance on an ongoing basis, however, we're still looking forward to stabilizing as planned. + +We also ended up making significant improvements to our clientside code, since the increased load on the server seemed to exacerbate a few race conditions which occurred less frequently under the previous circumstances. + +## Update notes + +Updating from version 3.14.0 should follow the usual process: + +1. stop your server +2. fetch the latest code with git +3. install clientside dependencies with `bower update` +4. install serverside dependencies with `npm i` +5. start your server + +You may notice that the server now launches a number of child processes named `crypto-worker.js` and `db-worker.js`. These worker processes make use of however many cores your server has available to perform more CPU-intensive tasks in parallel. + +## Features + +* As noted above, the server uses an multi-process architecture and parallelizes more routines. This improvement will be the most noticeable when the server is run on ARM processors which validate cryptographic signatures particularly slowly. +* The admin panel available to instance administrators now displays a list of "Open files". We added this to help us diagnose a "file descriptor leak" which will be described in the _Bug fixes_ section. +* We received a large number of contributions from translators via our [weblate instance](https://weblate.cryptpad.fr/projects/cryptpad/app/). Most notably, Italian is the fourth language to be fully translated with Finnish and Spanish seemingly in line to take the fifth and sixth spots. +* We've addressed some usability issues in our whiteboard app in response to increased interest. Its canvas now automatically resizes according to the size of your screen and the content you've drawn. Unfortunately, we noticed that the "embed image" functionality was imposing some additional strain on our server, so we decided to implement an admittedly arbitrary limit of 1MB on the size of images embedded in whiteboards. We'll consider removing this restriction when we have time to design a more efficient embedding system. +* We've removed the per-user setting which previously allowed registered users to skip the "pad creation screen" which is displayed before creating a document. This setting has not been the default for some time and was not actively tested, so this "feature" is our way of guaranteeing no future regressions in its behaviour. +* As a part of our effort to improve the server's scalability we evaluated which clientside requests could be sent less often. One such request came from the "usage bar" found in users' drives, teams, and settings pages. Previously it would update every 30 seconds no matter what. Now it only updates if that tab is focused. +* Most actions that an administrator can take with regard to a user's account require the "public key" which is used to identify their account. This key is available on the user's settings page, but many users share their profile URL instead. We've added a button to profile pages which copies the user's public key to the clipboard, so now either page will be sufficient. +* We've updated our [mermaidjs](https://mermaid-js.github.io/mermaid/#/) dependency. For those that don't know, Mermaid is a powerful markup syntax for producing a variety of charts. It's integrated into our code editor. This updated version supports GANTT chart tasks with multiple dependencies, pie charts, and a variety of other useful formats. +* We found that in practice our mermaid charts and other embedded media were sufficiently detailed that they became difficult to read on some screens. In response we've added the ability to view these elements in a "lightbox UI" which is nearly full-screen. This interface is can be used to view media contained in the "preview pane" of the code editor as well as within user and team drives, as well as a few other places where Markdown is used. + +## Bug fixes + +This release contains fixes for a lot of bugs. We'll provide a brief overview, but in the interest of putting more time towards development I'll just put my strong recommendation that you update. + +* The server process didn't always close file descriptors that it opened, resulting in an EMFILE error when the system ran out of available file descriptors. Now it closes them. +* The server also kept an unbounded amount of data in an in-memory cache under certain circumstances. Now it doesn't. +* A simple check to ignore the `premiumUploadSize` config value if it was less than `maxUploadSize` incorrectly compared against `defaultStorageLimit`. Premium upload sizes were disabled on our instance when we increased the default storage limit to 1GB. It's fixed now. +* We accepted a [PR](https://github.com/xwiki-labs/cryptpad/pull/513) to prevent a typeError when logging to disk was entirely disabled. +* We identified and fixed the cause of [This issue](https://github.com/xwiki-labs/cryptpad/issues/518) which caused spreadsheets not to load. +* Emojis at the start of users display names were not displayed correctly in the Kanban's "cursor" +* We (once again) believe we've fixed the [duplicated text bug](https://github.com/xwiki-labs/cryptpad/issues/352). Time will tell. +* Our existing Mermaidjs integration supported the special syntax to make elements clickable, but the resulting links don't work within CryptPad. We now remove them. +* Rather than having messages time out if they are not received by the server within a certain timeframe we now wait until the client reconnects, at which point we can check whether those messages exist in the document's history. On a related note we now detect when the realtime system is in a bad state and recreate it. +* Finally, we've fixed a variety of errors in spreadsheets. + # OrienteCaveRat release (3.14.0) ## Goals diff --git a/customize.dist/pages.js b/customize.dist/pages.js index d782e4a16..63cbbf154 100644 --- a/customize.dist/pages.js +++ b/customize.dist/pages.js @@ -107,7 +107,7 @@ define([ ])*/ ]) ]), - h('div.cp-version-footer', "CryptPad v3.14.0 (OrienteCaveRat)") + h('div.cp-version-footer', "CryptPad v3.15.0 (PigFootedBandicoot)") ]); }; diff --git a/customize.dist/src/less2/include/modal.less b/customize.dist/src/less2/include/modal.less index 635a65b71..12d3c908c 100644 --- a/customize.dist/src/less2/include/modal.less +++ b/customize.dist/src/less2/include/modal.less @@ -1,5 +1,8 @@ @import (reference) "./colortheme-all.less"; @import (reference) "./variables.less"; +@import (reference) './buttons.less'; + + .modal_base() { font-family: @colortheme_font; @@ -36,6 +39,8 @@ background-color: @colortheme_modal-dim; .cp-modal { + .buttons_main(); + background-color: @colortheme_modal-bg; color: @colortheme_modal-fg; box-shadow: @variables_shadow; @@ -75,6 +80,7 @@ background-color: @colortheme_modal-input-fg; color: @cryptpad_text_col; border: 1px solid @colortheme_modal-input; + width: auto; } } diff --git a/customize.dist/src/less2/include/modals-ui-elements.less b/customize.dist/src/less2/include/modals-ui-elements.less index 62b175f0e..0cc580ec0 100644 --- a/customize.dist/src/less2/include/modals-ui-elements.less +++ b/customize.dist/src/less2/include/modals-ui-elements.less @@ -125,6 +125,9 @@ video, iframe { margin-bottom: -5px; } + button { + line-height: 1.5; + } & > iframe { width: 100%; height: 100%; diff --git a/docs/example.nginx.conf b/docs/example.nginx.conf index ea8224c14..f6e163910 100644 --- a/docs/example.nginx.conf +++ b/docs/example.nginx.conf @@ -54,6 +54,7 @@ server { add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always; add_header X-XSS-Protection "1; mode=block"; add_header X-Content-Type-Options nosniff; + add_header Access-Control-Allow-Origin "*"; # add_header X-Frame-Options "SAMEORIGIN"; # Insert the path to your CryptPad repository root here diff --git a/lib/commands/channel.js b/lib/commands/channel.js index d35e15241..fce5b048c 100644 --- a/lib/commands/channel.js +++ b/lib/commands/channel.js @@ -54,16 +54,8 @@ Channel.clearOwnedChannel = function (Env, safeKey, channelId, cb, Server) { }); }; -Channel.removeOwnedChannel = function (Env, safeKey, channelId, cb, Server) { - if (typeof(channelId) !== 'string' || !Core.isValidId(channelId)) { - return cb('INVALID_ARGUMENTS'); - } +var archiveOwnedChannel = function (Env, safeKey, channelId, cb, Server) { var unsafeKey = Util.unescapeKeyCharacters(safeKey); - - if (Env.blobStore.isFileId(channelId)) { - return void Env.removeOwnedBlob(channelId, safeKey, cb); - } - Metadata.getMetadata(Env, channelId, function (err, metadata) { if (err) { return void cb(err); } if (!Core.hasOwners(metadata)) { return void cb('E_NO_OWNERS'); } @@ -124,6 +116,24 @@ Channel.removeOwnedChannel = function (Env, safeKey, channelId, cb, Server) { }); }; +Channel.removeOwnedChannel = function (Env, safeKey, channelId, __cb, Server) { + var _cb = Util.once(Util.mkAsync(__cb)); + + if (typeof(channelId) !== 'string' || !Core.isValidId(channelId)) { + return _cb('INVALID_ARGUMENTS'); + } + + // archiving large channels or files can be expensive, so do it one at a time + // for any given user to ensure that nobody can use too much of the server's resources + Env.queueDeletes(safeKey, function (next) { + var cb = Util.both(_cb, next); + if (Env.blobStore.isFileId(channelId)) { + return void Env.removeOwnedBlob(channelId, safeKey, cb); + } + archiveOwnedChannel(Env, safeKey, channelId, cb, Server); + }); +}; + Channel.trimHistory = function (Env, safeKey, data, cb) { if (!(data && typeof(data.channel) === 'string' && typeof(data.hash) === 'string' && data.hash.length === 64)) { return void cb('INVALID_ARGS'); diff --git a/lib/commands/pin-rpc.js b/lib/commands/pin-rpc.js index f3ade0489..a6f1642c6 100644 --- a/lib/commands/pin-rpc.js +++ b/lib/commands/pin-rpc.js @@ -49,16 +49,19 @@ var loadUserPins = function (Env, safeKey, cb) { // only put this into the cache if it completes session.channels = value; } - session.channels = value; done(value); }); }); }; var truthyKeys = function (O) { - return Object.keys(O).filter(function (k) { - return O[k]; - }); + try { + return Object.keys(O).filter(function (k) { + return O[k]; + }); + } catch (err) { + return []; + } }; var getChannelList = Pinning.getChannelList = function (Env, safeKey, _cb) { diff --git a/lib/historyKeeper.js b/lib/historyKeeper.js index f70dba006..2b5ec8770 100644 --- a/lib/historyKeeper.js +++ b/lib/historyKeeper.js @@ -38,6 +38,7 @@ module.exports.create = function (config, cb) { metadata_cache: {}, channel_cache: {}, queueStorage: WriteQueue(), + queueDeletes: WriteQueue(), batchIndexReads: BatchRead("HK_GET_INDEX"), batchMetadata: BatchRead('GET_METADATA'), diff --git a/lib/pins.js b/lib/pins.js index 41e871446..d840991a3 100644 --- a/lib/pins.js +++ b/lib/pins.js @@ -7,6 +7,9 @@ const Path = require("path"); const Util = require("./common-util"); const Plan = require("./plan"); +const Semaphore = require('saferphore'); +const nThen = require('nthen'); + /* Accepts a reference to an object, and... either a string describing which log is being processed (backwards compatibility), or a function which will log the error with all relevant data @@ -194,3 +197,63 @@ Pins.list = function (_done, config) { }).start(); }); }; + +Pins.load = function (cb, config) { + const sema = Semaphore.create(config.workers || 5); + + let dirList; + const fileList = []; + const pinned = {}; + + var pinPath = config.pinPath || './pins'; + var done = Util.once(cb); + + nThen((waitFor) => { + // recurse over the configured pinPath, or the default + Fs.readdir(pinPath, waitFor((err, list) => { + if (err) { + if (err.code === 'ENOENT') { + dirList = []; + return; // this ends up calling back with an empty object + } + waitFor.abort(); + return void done(err); + } + dirList = list; + })); + }).nThen((waitFor) => { + dirList.forEach((f) => { + sema.take((returnAfter) => { + // iterate over all the subdirectories in the pin store + Fs.readdir(Path.join(pinPath, f), waitFor(returnAfter((err, list2) => { + if (err) { + waitFor.abort(); + return void done(err); + } + list2.forEach((ff) => { + if (config && config.exclude && config.exclude.indexOf(ff) > -1) { return; } + fileList.push(Path.join(pinPath, f, ff)); + }); + }))); + }); + }); + }).nThen((waitFor) => { + fileList.forEach((f) => { + sema.take((returnAfter) => { + Fs.readFile(f, waitFor(returnAfter((err, content) => { + if (err) { + waitFor.abort(); + return void done(err); + } + const hashes = Pins.calculateFromLog(content.toString('utf8'), f); + hashes.forEach((x) => { + (pinned[x] = pinned[x] || {})[f.replace(/.*\/([^/]*).ndjson$/, (x, y)=>y)] = 1; + }); + }))); + }); + }); + }).nThen(() => { + done(void 0, pinned); + }); +}; + diff --git a/lib/storage/file.js b/lib/storage/file.js index 2d27ce185..00222c3f6 100644 --- a/lib/storage/file.js +++ b/lib/storage/file.js @@ -14,6 +14,28 @@ const readFileBin = require("../stream-file").readFileBin; const BatchRead = require("../batch-read"); const Schedule = require("../schedule"); + +/* Each time you write to a channel it will either use an open file descriptor + for that channel or open a new descriptor if one is not available. These are + automatically closed after this window to prevent a file descriptor leak, so + writes that take longer than this time may be dropped! */ +const CHANNEL_WRITE_WINDOW = 300000; + +/* Each time you read a channel it will have this many milliseconds to complete + otherwise it will be closed to prevent a file descriptor leak. The server will + lock up if it uses all available file descriptors, so it's important to close + them. The tradeoff with this timeout is that some functions, the stream, and + and the timeout itself are stored in memory. A longer timeout uses more memory + and running out of memory will also kill the server. */ +const STREAM_CLOSE_TIMEOUT = 300000; + +/* The above timeout closes the stream, but apparently that doesn't always work. + We set yet another timeout to allow the runtime to gracefully close the stream + (flushing all pending writes/reads and doing who knows what else). After this timeout + it will be MERCILESSLY DESTROYED. This isn't graceful, but again, file descriptor + leaks are bad. */ +const STREAM_DESTROY_TIMEOUT = 30000; + const isValidChannelId = function (id) { return typeof(id) === 'string' && id.length >= 32 && id.length < 50 && @@ -64,7 +86,7 @@ const destroyStream = function (stream) { try { stream.close(); } catch (err) { console.error(err); } setTimeout(function () { try { stream.destroy(); } catch (err) { console.error(err); } - }, 15000); + }, STREAM_DESTROY_TIMEOUT); }; const ensureStreamCloses = function (stream, id, ms) { @@ -74,7 +96,7 @@ const ensureStreamCloses = function (stream, id, ms) { // this can only be a timeout error... console.log("stream close error:", err, id); } - }), ms || 45000), []); + }), ms || STREAM_CLOSE_TIMEOUT), []); }; // readMessagesBin asynchronously iterates over the messages in a channel log @@ -729,7 +751,7 @@ var getChannel = function (env, id, _callback) { delete env.channels[id]; destroyStream(channel.writeStream, path); //console.log("closing writestream"); - }, 120000); + }, CHANNEL_WRITE_WINDOW); channel.delayClose(); env.channels[id] = channel; done(void 0, channel); diff --git a/package-lock.json b/package-lock.json index 8d0cebc6b..cefaca30d 100644 --- a/package-lock.json +++ b/package-lock.json @@ -1,6 +1,6 @@ { "name": "cryptpad", - "version": "3.14.0", + "version": "3.15.0", "lockfileVersion": 1, "requires": true, "dependencies": { diff --git a/package.json b/package.json index 668eb53a0..cd684660f 100644 --- a/package.json +++ b/package.json @@ -1,7 +1,7 @@ { "name": "cryptpad", "description": "realtime collaborative visual editor with zero knowlege server", - "version": "3.14.0", + "version": "3.15.0", "license": "AGPL-3.0+", "repository": { "type": "git", diff --git a/scripts/compare-pin-methods.js b/scripts/compare-pin-methods.js new file mode 100644 index 000000000..de7ef114d --- /dev/null +++ b/scripts/compare-pin-methods.js @@ -0,0 +1,42 @@ +/* jshint esversion: 6, node: true */ +const nThen = require("nthen"); +const Pins = require("../lib/pins"); +const Assert = require("assert"); + +const config = require("../lib/load-config"); + +var compare = function () { + console.log(config); + var conf = { + pinPath: config.pinPath, + }; + + var list, load; + + nThen(function (w) { + Pins.list(w(function (err, p) { + if (err) { throw err; } + list = p; + console.log(p); + console.log(list); + console.log(); + }), conf); + }).nThen(function (w) { + Pins.load(w(function (err, p) { + if (err) { throw err; } + load = p; + console.log(load); + console.log(); + }), conf); + }).nThen(function () { + console.log({ + listLength: Object.keys(list).length, + loadLength: Object.keys(load).length, + }); + + Assert.deepEqual(list, load); + console.log("methods are equivalent"); + }); +}; + +compare(); diff --git a/scripts/evict-inactive.js b/scripts/evict-inactive.js index a3a595ca4..1d7b87e91 100644 --- a/scripts/evict-inactive.js +++ b/scripts/evict-inactive.js @@ -42,7 +42,7 @@ nThen(function (w) { store = _; })); // load the list of pinned files so you know which files // should not be archived or deleted - Pins.list(w(function (err, _) { + Pins.load(w(function (err, _) { if (err) { w.abort(); return void console.error(err); diff --git a/www/auth/main.js b/www/auth/main.js index 84a17d922..fcbeaf3af 100644 --- a/www/auth/main.js +++ b/www/auth/main.js @@ -71,7 +71,7 @@ define([ // Get contacts and extract their avatar channel and key var getData = function (obj, href) { var parsed = Hash.parsePadUrl(href); - if (parsed.type !== "file") { return; } + if (!parsed || parsed.type !== "file") { return; } var secret = Hash.getSecrets('file', parsed.hash); if (!secret.keys || !secret.channel) { return; } obj.avatarKey = Hash.encodeBase64(secret.keys && secret.keys.cryptKey); @@ -81,6 +81,7 @@ define([ contacts.friends = proxy.friends || {}; Object.keys(contacts.friends).map(function (key) { var friend = contacts.friends[key]; + if (!friend) { return; } var ret = { edPublic: friend.edPublic, name: friend.displayName, @@ -90,6 +91,7 @@ define([ }); Object.keys(contacts.teams).map(function (key) { var team = contacts.teams[key]; + if (!team) { return; } var avatar = team.metadata && team.metadata.avatar; var ret = { edPublic: team.keys && team.keys.drive && team.keys.drive.edPublic, diff --git a/www/common/common-ui-elements.js b/www/common/common-ui-elements.js index aae6b09b9..a9c682afc 100644 --- a/www/common/common-ui-elements.js +++ b/www/common/common-ui-elements.js @@ -2557,18 +2557,6 @@ define([ var $advanced; - var $advancedContainer = $('