Merge branch 'staging' into toolbarUI

CHANGELOG.md

@@ -1,3 +1,65 @@
+# RedGazelle's revenge release (3.17.1)
+
+In recent months a growing amount of our time has been going towards answering support tickets, emails, and GitHub issues. This has made it a little more difficult to also maintain a bi-weekly release schedule, since there's some overhead involved in deploying our latest code and producing release notes.
+
+To ease our workload, we've decided to switch to producing a full release every three weeks, with an optional patch release at some point in the middle. Patch releases may fix major issues that can't wait three weeks or may simply consist of a few minor fixes that are trivial to deploy.
+
+This release fixes a few spreadsheet issues and introduces a more responsive layout for user drives in list mode.
+
+Updating to 3.17.1 from 3.17.0 is pretty standard:
+
+1. Stop your server
+2. Get the latest code with git
+3. Restart your server 
+
+# RedGazelle release (3.17.0)
+
+## Goals
+
+Our goal for this release was to introduce a first version of comments and mentions in our rich text editor as a part of a second R&D project funded by [NLnet](https://nlnet.nl/). We also received the results of an "accessibility audit" that was conducted as a part of our first NLnet PET project and so we've begun to integrate the auditor's feedback into the platform.
+
+Otherwise we've continued with our major goal of continuing to support a growing number of users on our instance via server improvements (without introducing any regressions).
+
+## Update notes
+
+The most drastic change in this release is that we've removed all docker-related files from the platform's repository. These files were all added via community contributions. Having them in the main repo gave the impression that we support installation via docker (which we do not).
+
+Docker-related files can now be found in the community-support [cryptpad-docker](https://github.com/xwiki-labs/cryptpad-docker/) repository.
+If you have an existing instance that you've installed using docker and you'd like to update, you may review the [migration guide](https://github.com/xwiki-labs/cryptpad-docker/blob/master/MIGRATION.md). If you encounter any problems in the process we advise that you create an issue in the repository's issue-tracker.
+
+Once again, this repository is **community-maintained**. If you are using this repository then _you are a part of the community_! Bug reports are useful, but fixes are even better!
+
+Otherwise, this is a fairly standard release. We've updated two of our client-side dependencies:
+
+1. ChainPad features a memory management optimization which is particularly relevant to editing very large documents or loading a drive with a large number of files. In one test we were able to reduce memory consumption in Chrome from 1.7GB to 20MB.
+2. CKEditor (the third-party library we use for our rich-text editor) has been updated so that we could make use of some more recent APIs for the _comments_ feature.
+
+To update from **3.16.0** to **3.17.0**:
+
+1. Stop your server
+2. Fetch the latest source with git
+3. Install the latest client-side dependencies with `bower update`
+4. Restart your server
+
+## Features
+
+* As noted above, this release introduces a first version of [comments at the right of the screen](https://github.com/xwiki-labs/cryptpad/issues/143) in our rich text editor. We're aware of a few usability issues under heavy concurrent usage, and we have some more improvements planned, but we figured that these issues were minor enough that people would be happy to use them in the meantime. The comments system integrates with the rest of our social functionality, so you'll have the ability to mention other users with the `@` symbol when typing within a comment.
+* We've made some minor changes to the server's logging system to suppress some uninformative log statements and to include some useful information in logs to improve our ability to debug some serverside performance issues. This probably won't affect you directly, but indirectly you'll benefit from some bug fixes and performance tweaks as we get a better understanding of what the server does at runtime.
+* We've received an _enormous_ amount of support tickets on CryptPad.fr (enough that if we answered them all we'd have very little time left for development). In response, we've updated the support ticket inbox available to administrators to highlight unanswered messages from non-paying users in yellow while support tickets from _premium users_ are highlighted in red. Administrators on other instances will notice that users of their instance with quotas increased via the server's `customLimits` config block will be counted as _premium_ as well.
+* Finally, we've continued to receive translations in a number of languages via our [Weblate instance](https://weblate.cryptpad.fr/projects/cryptpad/app/).
+
+## Bug fixes
+
+* We've fixed a minor bug in our code editor in which hiding _author colors_ while they were still enabled for the document caused a tooltip containing `undefined` to be displayed when hovering over the text.
+* A race condition in our server which was introduced when we started validating cryptographic signatures in child processes made it such that incoming messages could be written to the database in a different order than they were received. We implemented a per-channel queue which should now guarantee their ordering.
+* It used to be that an error in the process of creating a thumbnail for an encrypted file upload would prevent the file upload from completing (and prevent future uploads in that session). We've added some guards to catch these errors and handle them appropriately, closing [#540](https://github.com/xwiki-labs/cryptpad/issues/540).
+* CryptPad builds some CSS on the client because the source files (written in LESS) are smaller than the produced CSS. This results in faster load times for users with slow network connections. We identified and fixed bug in the loader which caused some files to be included in the compiled output multiple times, resulting in faster load times.
+* We addressed a minor bug in the drive's item sorting logic which was triggered when displaying inverse sortings.
+* Our last release introduced a set of custom styles for the mermaidjs integration in our code editor and featured one style which was not applied consistently across the wide variety of elements that could appear in mermaid graphs. As such, we've reverted the style (a color change in mermaid `graph` charts).
+* In the process of implementing comments in our rich text editor we realized that there were some bugs in our cursor recovery code (used to maintain your cursor position when multiple people are typing in the same document). We made some small patches to address a few very specific edge cases, but it's possible the improvements will have a broader effect with cursors in other situations.
+* We caught (and fixed) a few regressions in the _access_ and _properties_ modals that were introduced in the previous release.
+* It came to our attention that the script `cryptpad/scripts/evict-inactive.js` was removing inactive blobs after a shorter amount of time than intended. After investigating we found that it was using `retentionTime` instead of `inactiveTime` (both of which are from the server's config file. As such, some files were being archived after 15 days of inactivity instead of 90 (in cases where the files were not stored in anyone's drive). This script must be run manually (or periodically via a `cron`), so unless you've configured your instance to do so this will not have affected you.
+
 # Quagga release (3.16.0)
 
 ## Goals

customize.dist/pages.js

@@ -107,7 +107,7 @@ define([
                     ])*/
                 ])
             ]),
-            h('div.cp-version-footer', "CryptPad v3.17.0 (RedGazelle)")
+            h('div.cp-version-footer', "CryptPad v3.17.1 (RedGazelle's revenge)")
         ]);
     };
 

customize.dist/src/less2/include/drive.less

@@ -107,6 +107,7 @@
         display: flex;
         flex-flow: row;
         min-height: 0;
+        min-width: 0;
         @media screen and (max-width: @browser_media-medium-screen) {
             display: block;
             overflow-y: auto;

lib/api.js

@@ -14,6 +14,7 @@ module.exports.create = function (config) {
         var special_errors = {};
         ['EPIPE', 'ECONNRESET'].forEach(function (k) { special_errors[k] = noop; });
         special_errors.NF_ENOENT = function (error, label, info) {
+            delete info.stack;
             log.error(label, {
                 info: info,
             });
@@ -27,10 +28,11 @@ module.exports.create = function (config) {
             .on('sessionClose', historyKeeper.sessionClose)
             .on('error', function (error, label, info) {
                 if (!error) { return; }
-                if (error && error.code) {
+                var code = error && (error.code || error.message);
+                if (code) {
                     /*  EPIPE,ECONNERESET, NF_ENOENT */
-                    if (typeof(special_errors[error.code]) === 'function') {
+                    if (typeof(special_errors[code]) === 'function') {
-                        return void special_errors[error.code](error, label, info);
+                        return void special_errors[code](error, label, info);
                     }
                 }
 

package-lock.json

@@ -1,6 +1,6 @@
 {
   "name": "cryptpad",
-  "version": "3.17.0",
+  "version": "3.17.1",
   "lockfileVersion": 1,
   "requires": true,
   "dependencies": {

package.json

@@ -1,7 +1,7 @@
 {
   "name": "cryptpad",
   "description": "realtime collaborative visual editor with zero knowlege server",
-  "version": "3.17.0",
+  "version": "3.17.1",
   "license": "AGPL-3.0+",
   "repository": {
     "type": "git",

www/common/common-interface.js

@@ -1044,6 +1044,7 @@ define([
         var MutationObserver = window.MutationObserver;
         var addTippy = function (i, el) {
             if (el._tippy) { return; }
+            if (!el.getAttribute('title')) { return; }
             if (el.nodeName === 'IFRAME') { return; }
             var opts = {
                 distance: 15

www/common/drive-ui.js

@@ -2746,7 +2746,6 @@ define([
                 $list.find('.cp-app-drive-sort-foldername').addClass('cp-app-drive-sort-active').prepend($icon);
             }
         };
-        Messages.fm_sort = "Sort"; // XXX
         var getSortDropdown = function () {
             var $fhSort = $(h('span.cp-dropdown-container.cp-app-drive-element-sort.cp-app-drive-sort-clickable'));
             var options = [{

www/common/onlyoffice/inner.js

@@ -52,7 +52,7 @@ define([
         $: $
     };
 
-    var CHECKPOINT_INTERVAL = 50;
+    var CHECKPOINT_INTERVAL = 100;
     var DISPLAY_RESTORE_BUTTON = false;
     var NEW_VERSION = 2;
     var PENDING_TIMEOUT = 30000;
@@ -76,6 +76,7 @@ define([
         var privateData = metadataMgr.getPrivateData();
         var readOnly = false;
         var offline = false;
+        var ooLoaded = false;
         var pendingChanges = {};
         var config = {};
         var content = {
@@ -95,6 +96,8 @@ define([
         // This structure is used for caching media data and blob urls for each media cryptpad url
         var mediasData = {};
 
+        var startOO = function () {};
+
         var getMediasSources = APP.getMediasSources =  function() {
             content.mediasSources = content.mediasSources || {};
             return content.mediasSources;
@@ -221,10 +224,12 @@ define([
 
         var now = function () { return +new Date(); };
 
-        var getLastCp = function (old) {
+        var getLastCp = function (old, i) {
             var hashes = old ? oldHashes : content.hashes;
             if (!hashes || !Object.keys(hashes).length) { return {}; }
-            var lastIndex = Math.max.apply(null, Object.keys(hashes).map(Number));
+            i = i || 0;
+            var idx = Object.keys(hashes).map(Number).sort();
+            var lastIndex = idx[idx.length - 1 - i];
             var last = JSON.parse(JSON.stringify(hashes[lastIndex]));
             return last;
         };
@@ -262,11 +267,22 @@ define([
             }
         };
 
+        var checkDrawings = function () {
+            var editor = getEditor();
+            if (!editor) { return false; }
+            var s = editor.GetSheets();
+            return s.some(function (obj) {
+                return obj.worksheet.Drawings.length;
+            });
+        };
+
         // Loading a checkpoint reorder the sheet starting from ID "5".
         // We have to reorder it manually when a checkpoint is created
         // so that the messages we send to the realtime channel are
         // loadable by users joining after the checkpoint
         var fixSheets = function () {
+            var hasDrawings = checkDrawings();
+            if (hasDrawings) { return; }
             try {
                 var editor = getEditor();
                 // if we are not in the sheet app
@@ -291,13 +307,24 @@ define([
                 console.error(err);
                 return void UI.alert(Messages.oo_saveError);
             }
-            var i = Math.floor(ev.index / CHECKPOINT_INTERVAL);
+            // Get the last cp idx
+            var all = Object.keys(content.hashes || {}).map(Number).sort();
+            var current = all[all.length - 1] || 0;
+            // Get the expected cp idx
+            var _i = Math.floor(ev.index / CHECKPOINT_INTERVAL);
+            // Take the max of both
+            var i = Math.max(_i, current);
             content.hashes[i] = {
                 file: data.url,
                 hash: ev.hash,
                 index: ev.index
             };
             oldHashes = JSON.parse(JSON.stringify(content.hashes));
+            var hasDrawings = checkDrawings();
+            if (hasDrawings) {
+                content.locks = {};
+                content.ids = {};
+            }
             // If this is a migration, set the new version
             if (APP.migrate) {
                 delete content.migration;
@@ -344,6 +371,31 @@ define([
         };
         APP.FM = common.createFileManager(fmConfig);
 
+        // Add a lock
+        var isLockedModal = {
+            content: UI.dialog.customModal(h('div.cp-oo-x2tXls', [
+                h('span.fa.fa-spin.fa-spinner'),
+                h('span', Messages.oo_isLocked)
+            ]))
+        };
+
+        var resetData = function (blob, type) {
+            if (!isLockedModal.modal) {
+                isLockedModal.modal = UI.openCustomModal(isLockedModal.content);
+            }
+            myUniqueOOId = undefined;
+            setMyId();
+            APP.docEditor.destroyEditor(); // Kill the old editor
+            $('iframe[name="frameEditor"]').after(h('div#cp-app-oo-placeholder')).remove();
+            ooLoaded = false;
+            oldLocks = {};
+            Object.keys(pendingChanges).forEach(function (key) {
+                clearTimeout(pendingChanges[key]);
+                delete pendingChanges[key];
+            });
+            startOO(blob, type, true);
+        };
+
         var saveToServer = function () {
             var text = getContent();
             var blob = new Blob([text], {type: 'plain/text'});
@@ -354,6 +406,16 @@ define([
                 index: ooChannel.cpIndex
             };
             fixSheets();
+
+            var hasDrawings = checkDrawings();
+            if (hasDrawings) {
+                ooChannel.ready = false;
+                ooChannel.queue = [];
+                data.callback = function () {
+                    resetData(blob, file);
+                };
+            }
+
             APP.FM.handleFile(blob, data);
         };
 
@@ -619,13 +681,6 @@ define([
             });
         };
 
-        // Add a lock
-        var isLockedModal = {
-            content: UI.dialog.customModal(h('div.cp-oo-x2tXls', [
-                h('span.fa.fa-spin.fa-spinner'),
-                h('span', Messages.oo_isLocked)
-            ]))
-        };
         var handleLock = function (obj, send) {
             if (content.saveLock) {
                 if (!isLockedModal.modal) {
@@ -838,9 +893,8 @@ define([
             });
         };
 
-        var ooLoaded = false;
+        startOO = function (blob, file, force) {
-        var startOO = function (blob, file) {
+            if (APP.ooconfig && !force) { return void console.error('already started'); }
-            if (APP.ooconfig) { return void console.error('already started'); }
             var url = URL.createObjectURL(blob);
             var lock = readOnly || APP.migrate;
 
@@ -868,7 +922,7 @@ define([
                         "id": String(myOOId), //"c0c3bf82-20d7-4663-bf6d-7fa39c598b1d",
                         "firstname": metadataMgr.getUserData().name || Messages.anonymous,
                     },
-                    "mode": lock ? "view" : "edit",
+                    "mode": "edit",
                     "lang": (navigator.language || navigator.userLanguage || '').slice(0,2)
                 },
                 "events": {
@@ -910,7 +964,6 @@ define([
                         }
                     },
                     "onDocumentReady": function () {
-
                         // The doc is ready, fix the worksheets IDs and push the queue
                         fixSheets();
                         // Push changes since last cp
@@ -926,7 +979,21 @@ define([
                         APP.onLocal();
                         handleNewLocks(oldLocks, content.locks || {});
                         // Allow edition
+
+                        if (lock) {
+                            setTimeout(function () {
                                 setEditable(true);
+                                getEditor().setViewModeDisconnect();
+                            }, 5000);
+                        } else {
+                            setEditable(true);
+                        }
+
+                        if (isLockedModal.modal && force) {
+                            isLockedModal.modal.closeModal();
+                            delete isLockedModal.modal;
+                            $('#cp-app-oo-editor > iframe')[0].contentWindow.focus();
+                        }
 
                         if (APP.migrate && !readOnly) {
                             var div = h('div.cp-oo-x2tXls', [
@@ -1365,9 +1432,7 @@ define([
             }, 100);
         };
 
-        var loadLastDocument = function () {
+        var loadLastDocument = function (lastCp, onCpError, cb) {
-            var lastCp = getLastCp();
-            if (!lastCp) { return; }
             ooChannel.cpIndex = lastCp.index || 0;
             var parsed = Hash.parsePadUrl(lastCp.file);
             var secret = Hash.getSecrets('file', parsed.hash);
@@ -1381,6 +1446,7 @@ define([
             xhr.responseType = 'arraybuffer';
             xhr.onload = function () {
                 if (/^4/.test('' + this.status)) {
+                    onCpError();
                     return void console.error('XHR error', this.status);
                 }
                 var arrayBuffer = xhr.response;
@@ -1389,19 +1455,32 @@ define([
                     FileCrypto.decrypt(u8, key, function (err, decrypted) {
                         if (err) { return void console.error(err); }
                         var blob = new Blob([decrypted.content], {type: 'plain/text'});
+                        if (cb) {
+                            return cb(blob, getFileType());
+                        }
                         startOO(blob, getFileType());
                     });
                 }
             };
+            xhr.onerror = function () {
+                onCpError();
+            };
             xhr.send(null);
         };
-        var loadDocument = function (noCp, useNewDefault) {
+        var loadDocument = function (noCp, useNewDefault, i) {
             if (ooLoaded) { return; }
             var type = common.getMetadataMgr().getPrivateData().ooType;
             var file = getFileType();
             if (!noCp) {
+                var lastCp = getLastCp(false, i);
+                // If the last checkpoint is empty, load the "initial" doc instead
+                if (!lastCp || !lastCp.file) { return void loadDocument(true, useNewDefault); }
                 // Load latest checkpoint
-                return void loadLastDocument();
+                return void loadLastDocument(lastCp, function () {
+                    // Checkpoint error: load the previous one
+                    i = i || 0;
+                    loadDocument(noCp, useNewDefault, ++i);
+                });
             }
             var newText;
             switch (type) {
@@ -1646,6 +1725,18 @@ define([
 
         var reloadPopup = false;
 
+        var checkNewCheckpoint = function () {
+            var hasDrawings = checkDrawings();
+            if (hasDrawings) {
+                var lastCp = getLastCp();
+                loadLastDocument(lastCp, function () {
+                    // On error, do nothing
+                }, function (blob, type) {
+                    resetData(blob, type);
+                });
+            }
+        };
+
         config.onRemote = function () {
             if (initializing) { return; }
             var userDoc = APP.realtime.getUserDoc();
@@ -1671,10 +1762,18 @@ define([
                 var latest = getLastCp(true);
                 var newLatest = getLastCp();
                 if (newLatest.index > latest.index) {
+                    var hasDrawings = checkDrawings();
+                    if (hasDrawings) {
+                        ooChannel.ready = false;
+                        ooChannel.queue = [];
+                    }
+                    // New checkpoint
                     sframeChan.query('Q_OO_SAVE', {
                         hash: newLatest.hash,
                         url: newLatest.file
-                    }, function () { });
+                    }, function () {
+                        checkNewCheckpoint();
+                    });
                 }
                 oldHashes = JSON.parse(JSON.stringify(content.hashes));
             }

www/common/sframe-common-codemirror.js

@@ -523,7 +523,9 @@ define([
                     : 'background-color: rgba(255,0,0,0.2)';
                 marks[id] = editor.markText(pos1, pos2, {
                     css: css,
+                    attributes: {
                         'data-cptippy-html': true,
+                    },
                     title: makeTippy(cursor),
                     className: 'cp-tippy-html'
                 });

www/common/sframe-common-outer.js

@@ -490,6 +490,13 @@ define([
 
             // Put in the following function the RPC queries that should also work in filepicker
             var addCommonRpc = function (sframeChan, safe) {
+                Cryptpad.universal.onEvent.reg(function (data) {
+                    sframeChan.event('EV_UNIVERSAL_EVENT', data);
+                });
+                sframeChan.on('Q_UNIVERSAL_COMMAND', function (data, cb) {
+                    Cryptpad.universal.execCommand(data, cb);
+                });
+
                 sframeChan.on('Q_ANON_RPC_MESSAGE', function (data, cb) {
                     Cryptpad.anonRpcMsg(data.msg, data.content, function (err, response) {
                         cb({error: err, response: response});
@@ -1374,13 +1381,6 @@ define([
                 Cryptpad.cursor.execCommand(data, cb);
             });
 
-            Cryptpad.universal.onEvent.reg(function (data) {
-                sframeChan.event('EV_UNIVERSAL_EVENT', data);
-            });
-            sframeChan.on('Q_UNIVERSAL_COMMAND', function (data, cb) {
-                Cryptpad.universal.execCommand(data, cb);
-            });
-
             Cryptpad.onTimeoutEvent.reg(function () {
                 sframeChan.event('EV_WORKER_TIMEOUT');
             });

www/common/translations/messages.fr.json

@@ -686,7 +686,7 @@
     "features_f_subscribe": "S'abonner à un compte premium",
     "features_f_subscribe_note": "Vous devez d'abord vous connecter à un compte CryptPad",
     "faq_link": "FAQ",
-    "faq_title": "Foire Aux Questions",
+    "faq_title": "Foire aux questions",
     "faq_whatis": "Qu'est-ce que <span class='cp-brand-font'>CryptPad</span> ?",
     "faq": {
         "keywords": {
@@ -1366,5 +1366,6 @@
     "mentions_notification": "{0} vous a mentionné dans <b>{1}</b>",
     "unknownPad": "Pad inconnu",
     "comments_notification": "Réponses à votre commentaire \"{0}\" sur <b>{1}</b>",
-    "comments_error": "Impossible d'ajouter un commentaire ici"
+    "comments_error": "Impossible d'ajouter un commentaire ici",
+    "fm_sort": "Trier"
 }

www/common/translations/messages.json

@@ -1366,5 +1366,6 @@
     "settings_padNotifTitle": "Comment notifications",
     "settings_padNotifHint": "Ignore notifications when someone replies to one of your comments",
     "settings_padNotifCheckbox": "Disable comment notifications",
-    "comments_error": "Can't add a comment here"
+    "comments_error": "Can't add a comment here",
+    "fm_sort": "Sort"
 }

www/settings/inner.js

@@ -942,8 +942,7 @@ define([
     create['drive-import-local'] = function() {
         if (!common.isLoggedIn()) { return; }
         var $div = $('<div>', { 'class': 'cp-settings-drive-import-local cp-sidebarlayout-element' });
-        $('<label>', { 'for': 'cp-settings-import-local-pads' })
+        $('<label>').text(Messages.settings_import).appendTo($div);
-            .text(Messages.settings_import).appendTo($div);
         $('<span>', { 'class': 'cp-sidebarlayout-description' })
             .text(Messages.settings_importTitle).appendTo($div);
         var $button = $('<button>', {