Merge branch 'staging' of github.com:xwiki-labs/cryptpad into staging

CHANGELOG.md

@@ -1,3 +1,48 @@
+# Upupa release (v2.20.0)
+
+## Goals
+
+After all the features we've added over time, the root of the CryptPad repository had gotten to be something of a mess. We decided to spend a lot of this release period cleaning things up. We also prioritized some other features which make it easier to manage a CryptPad instance.
+
+## Update notes
+
+This release makes a number of serverside changes. Read the following notes carefully before updating from an earlier version of CryptPad!
+
+* We realized that docker images persisted `config.js` by copying it into the `customize` volume. Since customize is exposed by the webserver, this meant that potentially private information in the configuration file would be accessible over the web. We've moved `config.js` to a `cryptpad/config/`, along with `config.example.js` and modified the docker setup so that nothing in this folder will be exposed to the web.
+  * Consequently, you'll need to move your own `config.js` to the new location in order for your server to read it when you restart.
+* We also noticed that the configuration values for alternate paths to various were not universally supported, and that they couldn't be deeper than one directory, in any case. We've reviewed the server's source and introduced support for arbitrary filepaths to each of the directories.
+  * In the near future we plan to simplify server maintenance by moving all user data into a new `data` directory. This will make docker setups easier to maintain, as well as simplifying the task of migrating or backing up your database.
+* CryptPad now features a rudimentary administration panel, accessible at the /admin/ URL. Server operators can add their **Public signing key** (found on their settings page) to their config file in the `adminKeys` array. See config.example.js for more info.
+* We've also moved all our scripts out of the repository root and into a dedicated `scripts` directory. We recommend reviewing any crontabs or other scripts that might be calling them.
+* After receiving a number of support requests for third-party instances due to our email being displayed on the contact page, we've decided to display the `adminEmail` from `config.js` to users.
+  * If you leave the default `i.did.not.read.my.config@cryptpad.fr`, nothing will be displayed. We'd appreciate it if you did leave your own contact information, as time we spend trying to help users on your instance is time we spend _not developing new features_.
+* We've introduced a basic logging API which standardizes how various messages are printed, as well as logging them to the disk.
+  * If you do not specify `logPath` in your config file, it will not log to the disk.
+  * Unless `logToStdout` is true, it will not print to the console either.
+  * You can configure the degree of logging by setting `logLevel` to one of the supported settings. If no level is set, it will use the default `info` setting, which includes _warnings_ and _errors_. See the example config for more information.
+* We've dropped support for number of configuration points:
+  * `enableUploads` no longer has any effect, as the clientside code assumed the server supported uploads. This value was added when file uploads were still considered experimental, but they have been a core part of the platform for some time.
+  * `restrictUploads` no longer has any effect either, for the same reason.
+* We've made some small updates to `example.nginx.conf` to expose `/datastore/` over the web, as there are some scripts which depend on expect the log files to be exposed.
+* Depending on when you last updated, you may need to update your clientside dependencies. Run `bower update` to get the latest code.
+* Finally, we've introduced a server-side dependency (get-folder-size) and updated one of our own libraries (chainpad-server). Run `npm install` to get the updated versions. The server won't work without them.
+
+## Features
+
+* Our rich text editor is now configured to support the insertion of LaTeX equations via CKEditor's  _mathjax_ plugin.
+* The contact page now lists our Mastodon account, which is quickly catching up to our twitter account's number of followers.
+  * If configured correctly, instances will also display the contact email for the instance administrator.
+* We've reorganized the home page a little bit, making more of our applications visible at a glance. It also features changes to the header and footer.
+* The chat box and help text are no longer shown by default, making the interface much cleaner for new users.
+* Pads which were created with an expiration date are now displayed with a clock icon in users' Drives.
+* The settings page now remembers which tab you'd selected, in the event of a page reload.
+* We received contributions to our [German](https://weblate.cryptpad.fr/projects/cryptpad/app/de/#history) and [Russian](https://weblate.cryptpad.fr/projects/cryptpad/app/ru/#history) translations.
+* Our code and slide editors now features a first version of support for rendering [Mermaid rendering](https://mermaidjs.github.io/).
+
+## Bug fixes
+
+* fixed the corner dialog's z-index for the slide app
+
 # Tapir release (v2.19.0)
 
 ## Goals

config/config.example.js

@@ -279,7 +279,7 @@ module.exports = {
     blobStagingPath: './blobstage',
 
     /* CryptPad supports logging events directly to the disk in a 'logs' directory
-     * Set its location here, or set it to false if you'd rather not log
+     * Set its location here, or set it to false (or nothing) if you'd rather not log
      */
     logPath: './data/logs',
 
@@ -295,7 +295,7 @@ module.exports = {
     /* CryptPad can be configured to log more or less
      * the various settings are listed below by order of importance
      *
-     * silly, debug, verbose, feedback, info, warn, error
+     * silly, verbose, debug, feedback, info, warn, error
      *
      * Choose the least important level of logging you wish to see.
      * For example, a 'silly' logLevel will display everything,

customize.dist/pages/contact.js

@@ -13,7 +13,7 @@ define([
                 )
             ]),
             h('div.container.cp-container', [
-                Config.adminEmail !== 'i.did.not.read.my.config@cryptpad.fr' ? h('div.row.cp-iconCont.align-items-center', [
+                Config.adminEmail && Config.adminEmail !== 'i.did.not.read.my.config@cryptpad.fr' ? h('div.row.cp-iconCont.align-items-center', [
                     h('div.col-12',
                         Pages.setHTML(h('h4.text-center'), Msg.contact_admin),
                         h('p', Msg.contact_adminHint)

docs/example.nginx.conf

@@ -67,6 +67,25 @@ server {
         proxy_set_header Connection upgrade;
     }
 
+    location ^~ /datastore/ {
+        alias /home/cryptpad/office.cryptpad/data/datastore;
+        if ($request_method = 'OPTIONS') {
+            add_header 'Access-Control-Allow-Origin' '*';
+            add_header 'Access-Control-Allow-Methods' 'GET, POST, OPTIONS';
+            add_header 'Access-Control-Allow-Headers' 'DNT,X-CustomHeader,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Content-Range,Range';
+            add_header 'Access-Control-Max-Age' 0;
+            add_header 'Content-Type' 'application/octet-stream; charset=utf-8';
+            add_header 'Content-Length' 0;
+            return 204;
+        }
+        add_header Cache-Control max-age=0;
+        add_header 'Access-Control-Allow-Origin' '*';
+        add_header 'Access-Control-Allow-Methods' 'GET, POST, OPTIONS';
+        add_header 'Access-Control-Allow-Headers' 'DNT,X-CustomHeader,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Content-Range,Range';
+        add_header 'Access-Control-Expose-Headers' 'DNT,X-CustomHeader,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Content-Range,Range';
+        try_files $uri =404;
+    }
+
     location ^~ /customize.dist/ {
         # This is needed in order to prevent infinite recursion between /customize/ and the root
     }

lib/log.js

@@ -19,7 +19,7 @@ var write = function (ctx, content) {
 };
 
 // various degrees of logging
-const logLevels = ['silly', 'debug', 'verbose', 'feedback', 'info', 'warn', 'error'];
+const logLevels = ['silly', 'verbose', 'debug', 'feedback', 'info', 'warn', 'error'];
 
 var handlers = {
     silly: function (ctx, time, tag, info) {

lib/once.js

@@ -0,0 +1,8 @@
+module.exports = function (f) {
+    var called;
+    return function () {
+        if (called) { return; }
+        called = true;
+        f.apply(this, Array.prototype.slice.call(arguments));
+    };
+};

package.json

@@ -8,7 +8,7 @@
     "url": "git://github.com/xwiki-labs/cryptpad.git"
   },
   "dependencies": {
-    "chainpad-server": "~2.1.0",
+    "chainpad-server": "~3.0.0",
     "express": "~4.16.0",
     "fs-extra": "^7.0.0",
     "nthen": "~0.1.0",

rpc.js

@@ -245,6 +245,7 @@ var loadUserPins = function (Env, publicKey, cb) {
     };
 
     var unpin = function (channel) {
+        // TODO delete?
         pins[channel] = false;
     };
 
@@ -263,6 +264,7 @@ var loadUserPins = function (Env, publicKey, cb) {
                     parsed[1].forEach(unpin);
                     break;
                 case 'RESET':
+                    // TODO just wipe out the object?
                     Object.keys(pins).forEach(unpin);
 
                     if (parsed[1] && parsed[1].length) {
@@ -600,9 +602,21 @@ var sumChannelSizes = function (sizes) {
 
 // inform that the
 var loadChannelPins = function (Env) {
-    Pinned.load(function (data) {
+    Pinned.load(function (err, data) {
+        if (err) {
+            Log.error("LOAD_CHANNEL_PINS", err);
+
+            // FIXME not sure what should be done here instead
+            Env.pinnedPads = {};
+            Env.evPinnedPadsReady.fire();
+            return;
+        }
+
+
         Env.pinnedPads = data;
         Env.evPinnedPadsReady.fire();
+    }, {
+        pinPath: Env.paths.pin,
     });
 };
 var addPinned = function (

scripts/check-account-deletion.js

@@ -3,6 +3,8 @@ const Fs = require('fs');
 const nThen = require('nthen');
 const Pinned = require('./pinned');
 const Nacl = require('tweetnacl');
+const Path = require('path');
+const Config = require('./load-config');
 
 const hashesFromPinFile = (pinFile, fileName) => {
     var pins = {};
@@ -54,7 +56,9 @@ let data = [];
 let pinned = [];
 
 nThen((waitFor) => {
-    let f = '../pins/' + edPublic.slice(0, 2) + '/' + edPublic + '.ndjson';
+    var pinPath = Config.pinPath || './pins';
+
+    let f = Path.join(pinPath, edPublic.slice(0, 2), edPublic + '.ndjson');
     Fs.readFile(f, waitFor((err, content) => {
         if (err) { throw err; }
         pinned = hashesFromPinFile(content.toString('utf8'), f);

scripts/check-accounts.js

@@ -5,6 +5,7 @@ var Package = require("../package.json");
 
 var body = JSON.stringify({
     domain: Config.myDomain,
+    subdomain: Config.mySubdomain || null,
     adminEmail: Config.adminEmail,
     version: Package.version,
 });

scripts/delete-inactive.js

@@ -5,9 +5,9 @@ const Saferphore = require("saferphore");
 const PinnedData = require('./pinneddata');
 let config;
 try {
-    config = require('../config/config');
+    config = require('./config/config');
 } catch (e) {
-    config = require('../config/config.example');
+    config = require('./config/config.example');
 }
 
 if (!config.inactiveTime || typeof(config.inactiveTime) !== "number") { return; }
@@ -16,7 +16,11 @@ let inactiveTime = +new Date() - (config.inactiveTime * 24 * 3600 * 1000);
 let inactiveConfig = {
     unpinned: true,
     olderthan: inactiveTime,
-    blobsolderthan: inactiveTime
+    blobsolderthan: inactiveTime,
+
+    filePath: config.filePath,
+    blobPath: config.blobPath,
+    pinPath: config.pinPath,
 };
 let toDelete;
 nThen(function (waitFor) {

scripts/expire-channels.js

@@ -3,12 +3,7 @@ var Path = require("path");
 
 var nThen = require("nthen");
 
-var config;
+var config = require("./load-config");
-try {
-    config = require('../config/config');
-} catch (e) {
-    config = require('../config/config.example');
-}
 
 var FileStorage = require('../' + config.storage || './storage/file');
 var root = Path.resolve('../' + config.taskPath || './tasks');
@@ -52,10 +47,12 @@ var handleTask = function (str, path, cb) {
     nThen(function (waitFor) {
         switch (command) {
             case 'EXPIRE':
+                // FIXME noisy!
                 console.log("expiring: %s", args[0]);
                 store.removeChannel(args[0], waitFor());
                 break;
             default:
+                // FIXME noisy
                 console.log("unknown command", command);
         }
     }).nThen(function () {
@@ -83,6 +80,7 @@ nt = nThen(function (w) {
 }).nThen(function () {
     dirs.forEach(function (dir, dIdx) {
         queue(function (w) {
+            // FIXME noisy!
             console.log('recursing into %s', dir);
             Fs.readdir(Path.join(root, dir), w(function (e, list) {
                 list.forEach(function (fn) {
@@ -90,6 +88,7 @@ nt = nThen(function (w) {
                         var filePath = Path.join(root, dir, fn);
                         var cb = w();
 
+                        // FIXME noisy!
                         console.log("processing file at %s", filePath);
                         Fs.readFile(filePath, 'utf8', function (e, str) {
                             if (e) {

scripts/load-config.js

@@ -0,0 +1,7 @@
+var config;
+try {
+    config = require("../config/config");
+} catch (e) {
+    config = require("../config/config.example");
+}
+module.exports = config;

scripts/pinned.js

@@ -1,6 +1,8 @@
 /* jshint esversion: 6, node: true */
 const Fs = require('fs');
+const Path = require("path");
 const Semaphore = require('saferphore');
+const Once = require("../lib/once");
 const nThen = require('nthen');
 
 const sema = Semaphore.create(20);
@@ -9,7 +11,9 @@ let dirList;
 const fileList = [];
 const pinned = {};
 
-const hashesFromPinFile = (pinFile, fileName) => {
+// FIXME this seems to be duplicated in a few places.
+// make it a library and put it in ./lib/
+const checkPinStatus = (pinFile, fileName) => {
     var pins = {};
     pinFile.split('\n').filter((x)=>(x)).map((l) => JSON.parse(l)).forEach((l) => {
         switch (l[0]) {
@@ -27,32 +31,47 @@ const hashesFromPinFile = (pinFile, fileName) => {
                 l[1].forEach((x) => { delete pins[x]; });
                 break;
             }
-            default: throw new Error(JSON.stringify(l) + '  ' + fileName);
+            default:
+                // TODO write to the error log
+                /* Log.error('CORRUPTED_PIN_LOG', {
+                    line: JSON.stringify(l),
+                    fileName: fileName,
+                }); */
+                console.error(new Error (JSON.stringify(l) + '  ' + fileName));
         }
     });
     return Object.keys(pins);
 };
 
 module.exports.load = function (cb, config) {
+    var pinPath = config.pinPath || './pins';
+    var done = Once(cb);
+
     nThen((waitFor) => {
-        Fs.readdir('../pins', waitFor((err, list) => {
+        // recurse over the configured pinPath, or the default
+        Fs.readdir(pinPath, waitFor((err, list) => {
             if (err) {
                 if (err.code === 'ENOENT') {
                     dirList = [];
-                    return;
+                    return; // this ends up calling back with an empty object
                 }
-                throw err;
+                waitFor.abort();
+                return void done(err);
             }
             dirList = list;
         }));
     }).nThen((waitFor) => {
         dirList.forEach((f) => {
             sema.take((returnAfter) => {
-                Fs.readdir('../pins/' + f, waitFor(returnAfter((err, list2) => {
+                // iterate over all the subdirectories in the pin store
-                    if (err) { throw err; }
+                Fs.readdir(Path.join(pinPath, f), waitFor(returnAfter((err, list2) => {
+                    if (err) {
+                        waitFor.abort();
+                        return void done(err);
+                    }
                     list2.forEach((ff) => {
                         if (config && config.exclude && config.exclude.indexOf(ff) > -1) { return; }
-                        fileList.push('../pins/' + f + '/' + ff);
+                        fileList.push(Path.join(pinPath, f, ff));
                     });
                 })));
             });
@@ -61,8 +80,11 @@ module.exports.load = function (cb, config) {
         fileList.forEach((f) => {
             sema.take((returnAfter) => {
                 Fs.readFile(f, waitFor(returnAfter((err, content) => {
-                    if (err) { throw err; }
+                    if (err) {
-                    const hashes = hashesFromPinFile(content.toString('utf8'), f);
+                        waitFor.abort();
+                        return void done(err);
+                    }
+                    const hashes = checkPinStatus(content.toString('utf8'), f);
                     hashes.forEach((x) => {
                         (pinned[x] = pinned[x] || {})[f.replace(/.*\/([^/]*).ndjson$/, (x, y)=>y)] = 1;
                     });
@@ -70,14 +92,20 @@ module.exports.load = function (cb, config) {
             });
         });
     }).nThen(() => {
-        cb(pinned);
+        done(void 0, pinned);
     });
 };
 
 if (!module.parent) {
-    module.exports.load(function (data) {
+    module.exports.load(function (err, data) {
+        if (err) {
+            return void console.error(err);
+        }
+
         Object.keys(data).forEach(function (x) {
             console.log(x + ' ' + JSON.stringify(data[x]));
         });
+    }, {
+        pinPath: require("../config/config").pinPath
     });
 }

scripts/pinneddata.js

@@ -2,6 +2,7 @@
 const Fs = require('fs');
 const Semaphore = require('saferphore');
 const nThen = require('nthen');
+const Path = require('path');
 
 /*
     takes contents of a pinFile (UTF8 string)
@@ -63,9 +64,14 @@ const pinned = {}; // map of pinned files
 // define a function: 'load' which takes a config
 // and a callback
 module.exports.load = function (config, cb) {
+    var filePath = config.filePath || './datastore';
+    var blobPath = config.blobPath || './blob';
+    var pinPath = config.pinPath || './pins';
+
+
     nThen((waitFor) => {
         // read the subdirectories in the datastore
-        Fs.readdir('../datastore', waitFor((err, list) => {
+        Fs.readdir(filePath, waitFor((err, list) => {
             if (err) { throw err; }
             dirList = list;
         }));
@@ -76,15 +82,15 @@ module.exports.load = function (config, cb) {
             sema.take((returnAfter) => {
                 // get the list of files in every subdirectory
                 // and push them to 'fileList'
-                Fs.readdir('../datastore/' + f, waitFor(returnAfter((err, list2) => {
+                Fs.readdir(Path.join(filePath, f), waitFor(returnAfter((err, list2) => {
                     if (err) { throw err; }
-                    list2.forEach((ff) => { fileList.push('../datastore/' + f + '/' + ff); });
+                    list2.forEach((ff) => { fileList.push(Path.join(filePath, f, ff)); });
                 })));
             });
         });
     }).nThen((waitFor) => {
         // read the subdirectories in 'blob'
-        Fs.readdir('../blob', waitFor((err, list) => {
+        Fs.readdir(blobPath, waitFor((err, list) => {
             if (err) { throw err; }
             // overwrite dirList
             dirList = list;
@@ -96,9 +102,9 @@ module.exports.load = function (config, cb) {
             sema.take((returnAfter) => {
                 // get the list of files in every subdirectory
                 // and push them to 'fileList'
-                Fs.readdir('../blob/' + f, waitFor(returnAfter((err, list2) => {
+                Fs.readdir(Path.join(blobPath, f), waitFor(returnAfter((err, list2) => {
                     if (err) { throw err; }
-                    list2.forEach((ff) => { fileList.push('../blob/' + f + '/' + ff); });
+                    list2.forEach((ff) => { fileList.push(Path.join(blobPath, f, ff)); });
                 })));
             });
         });
@@ -118,7 +124,7 @@ module.exports.load = function (config, cb) {
         });
     }).nThen((waitFor) => {
         // read the subdirectories in the pinstore
-        Fs.readdir('../pins', waitFor((err, list) => {
+        Fs.readdir(pinPath, waitFor((err, list) => {
             if (err) { throw err; }
             dirList = list;
         }));
@@ -131,9 +137,9 @@ module.exports.load = function (config, cb) {
             sema.take((returnAfter) => {
                 // get the list of files in every subdirectory
                 // and push them to 'fileList' (which is empty because we keep reusing it)
-                Fs.readdir('../pins/' + f, waitFor(returnAfter((err, list2) => {
+                Fs.readdir(Path.join(pinPath, f), waitFor(returnAfter((err, list2) => {
                     if (err) { throw err; }
-                    list2.forEach((ff) => { fileList.push('../pins/' + f + '/' + ff); });
+                    list2.forEach((ff) => { fileList.push(Path.join(pinPath, f, ff)); });
                 })));
             });
         });

www/common/cryptpad-common.js

@@ -1295,7 +1295,7 @@ define([
                 if (!noWorker && !noSharedWorker && typeof(SharedWorker) !== "undefined") {
                     worker = new SharedWorker('/common/outer/sharedworker.js?' + urlArgs);
                     worker.onerror = function (e) {
-                        console.error(e.message);
+                        console.error(e.message); // FIXME seeing lots of errors here as of 2.20.0
                     };
                     worker.port.onmessage = function (ev) {
                         if (ev.data === "SW_READY") {